Hello everyone, my question pertains to a specific problem related to shader permutation. I am aware of a lot of other posts that touched on the topic but despite reading quite a lot of them I couldn't find the solution.

So far, I have managed to implement a system that produced 192 different vertex shaders for me. Each permutation is based on the following parameters (a flag means it's taking in, processing and outputting whatever is specified):

enum SHG_VS_SETTINGS : uint32_t
{
	SHG_VS_TEXCOORDS	= (1 << 0),
	SHG_VS_NORMALS		= (1 << 1),
	SHG_VS_COLOUR		= (1 << 2),
	SHG_VS_TANGENT		= (1 << 3 | SHG_VS_NORMALS),
	SHG_VS_BITANGENT	= (1 << 4 | SHG_VS_NORMALS | SHG_VS_TANGENT),
	SHG_VS_SKINIDXWGT	= (1 << 5),
	SHG_VS_INSTANCING	= (1 << 7),
	SHG_VS_WORLD_POS	= (1 << 8)
};

struct ShaderOption
{
	D3D_SHADER_MACRO _macro;
	uint64_t _bitmask;
};

Position is assumed to be there for the purposes of the tool generating shaders, the rest are simple yes/no flags.
Bitwise or-s are used to filter whether a #define is included based on existence of other #define-s. The loop looks like this:

// take a vector of ShaderOption as param to the function etc.
UINT optionCount	= optionSet.size();
std::set<uint64_t> existing;
for (uint64_t i = 0; i < ( 1 << optionCount); ++i)
{
	uint64_t total = 0;

	for (UINT j = 0; j < optionCount; ++j)
	{
		uint64_t requestedOption = optionSet[j]._bitmask;
		uint64_t andResult = requestedOption & i;

		if (andResult == requestedOption)
		{
			matchingPermOptions.push_back(optionSet[j]._macro);
			total += andResult;
		}
	}

	// Eliminate doubles
	if (!_existing.insert(total).second)
	{
		matchingPermOptions.clear();
		continue;
	}

	matchingPermOptions.push_back({ NULL, NULL });	// Required by d3d api
	createShPerm(textBuffer, matchingPermOptions, total);
	matchingPermOptions.clear();
}

Some initialization and clean up code is left out for clarity but basically that's that. It works as expected, tested.
However, this can't handle options containing multiple bits.

So I tried something else. Struct Option is changed, and code is slightly different as well.

struct Option
{
  std::string optName;
  uint32_t min;
  uint32_t max;
  std::vector<const char*> defines;
};

// Same as above
for(uint32_t i = 0; i < (1 << numBits); ++i)
{
    for(uint32_t j = 0; j < numOptions; ++j)
    {
        Option&; o = options[j];
          
        uint32_t shifted = i >> o.min;
        uint32_t bitSpan = o.max - o.min; // Determine bit width of this particular option
        uint32_t bitMask = (~(~0u << bitSpan)); // Set "bitSpan" least sig. bits to 1
        uint32_t result = shifted & bitMask;
         
        std::cout << o.optName << ": " << o.defines[result] << std::endl;
    }
    std::cout << "----------------------------------------" << std::endl;
}

This works. I can “isolate” a value by shifting right so the meaningful part of the key is in the lowest bits, mask against something like 0000 0111 (if 3 bits are needed for the option, for example) and then take whichever value I need. This will populate D3D_SHADER\_MACRO in a real implementation, cout is temporary.

Am I going the right way? Each option has a width and a set of define values that could be attached to a name?
One big issue I see here is using something like PBR as a value of lighting model (it's one of the values I was intending to add) when in fact that's a whole technique implying roughness and metallic textures or at least uniforms etc. meaning it might actually be better to split things like pbr, lambert, phong etc and have different options for each?

Edit TLDR: Should permutations be used at a level of a technique (if we assume a technique is something like a phong based material or a Cook torrance based material) or do we create every shader from a single initial file per shader type?

Well… I reworked the code a lot, did all the stuff. It works, it's actually great how much one can infer just from loaded mesh/material data and make a shader that fits that data automatically. Very content, except…

I also just designed myself into a biggie baddie... now i run into things like textures being packed in a variety of ways, sometimes opacity is in diffuse fourth channel, sometimes it's standalone, specular and shininess maps are also either packed or not... how many options is all this going to take? Feels like the only way to allow maximum flexibility is the node editor :| or the shader key needs to be way bigger than I anticipated. Doubt I can get away with 128 bits if things like this keep popping up.

Initially I wanted to do it this way in order to put a sane limit to how many shaders are used concurrently in a scene so shader switching doesn't destroy perf but seeing how I'm basically testing off free internet assets it's proving to be really hard given that I can't enforce a policy on anything going in, and making more options just seems to be leading to the same problem (and bigger keys, which isn't the end of the world but alas).

J. Rakocevic said:
I also just designed myself into a biggie baddie... now i run into things like textures being packed in a variety of ways, sometimes opacity is in diffuse fourth channel, sometimes it's standalone, specular and shininess maps are also either packed or not... how many options is all this going to take? Feels like the only way to allow maximum flexibility is the node editor :| or the shader key needs to be way bigger than I anticipated. Doubt I can get away with 128 bits if things like this keep popping up.

And this is one of the reasons I've gone with the node editor route in my project. Basically the number and type of possible inputs for materials are endless, so either one sets up a rigid convention, and everything has to fit that, or throws every convention out, and lets the art side of the project use and do whatever they want (within reason). In the end even if you go down the node editor path, the end result will be a limited set of materials that will be reused, so optimizing for shader switching will still be a thing.

But if you don' t want to build a full material editor, you can still do a texture import step, that converts and packs stuff in a way that your shaders expect it. You basically set up a standard set of inputs you expect, and map the image files to those ( or a single channel of those files ).

I think I'll take the second option. Not that I particularly dislike node editors, but the amount of work and time I sunk into asset loading and handling corner cases is huge.
It's mostly caused by working with free files from the net, most of which seem to have at least some error(s) (even in commercial engines and 3D editors it's wrong, not only in my code), and is really burning me out. Hell, I know the windows username of a lot of these authors because all their textures show up in my debug output with absolute paths Q_Q, and that's only the tip of the iceberg.

Building a node editor to get around that on top of everything might just make me quit tbh. I want to get away from the asset hell cheap and fast, and copying a few texture channels around seems trivial.