So, in real life, incoming dot normal at the silhouette is always 0. With smooth shaded meshes, it never is, not naturally, not outside of contrived situations. (Not with flat shaded meshes either, I guess.)
And incoming dot normal is one of the bedrocks of CG. Probably the equal of 4x4 matrix multiplication. Problems with silhouette normals show up in Fresnel, in diffuse lighting, in environment mapping.... everywhere. But I can't really find anybody talking about it. (Maybe I'm not Googling the right terms.)
Obviously, the problem decreases as poly count goes up, eventually reaching a point where it's dwarfed by other silhouette problems (like translucency or micro-occlusion) that CG doesn't handle well either. But, if I'm reasoning correctly, normal maps don't improve the problem-- they're as likely to exacerbate it as improve it, and the exacerbations are, aesthetically speaking, probably worse than the improvements are better.
I've tried playing with crude fixes-- basically, rotating normals toward incoming by a percentage, or of course clamping incoming dot normal (like we all have to do) to prevent it from bending behind the mesh. Nothing I've tried looks good. I suppose the best option might be to rotate normals to perpendicular to incoming at the silhouette and then interpolate to the nearest inflection point of something like screen space depth to preserve curvature, but the math for how to do that is beyond me, and I'm not sure it would look any better. Or maybe, instead, somehow, adjust the drawn silhouette to match the silhouette defined by incoming dot normal? Not even sure how that would work, not if the normal was pointing away from incoming.
I don't know-- is this a solvable problem? Has anyone tried other stuff and given up, pursued anything that was promising but too expensive, anything like that? Are there any papers I'm missing? It's really surprising to me that I can't find anyone else talking about this.
(Apologies if I chose the wrong subforum for this. I considered art forums, but I felt that people frequenting the programming forums would have more to say on the subject.)
