Hi all! It's been a while, but since i have vacation now i decided to go on with my project. When I stopped i got the stencil shadows working. It just doesn't look right. This is the problem:

* = vertex
_ / | \ = face
                      ___
                     *   \___
lightsource >        |       *
                     |       |
                     *_______*

I have 4 verts and four faces in this 2D example... Normally opengl lights the vertexes acording to their normals. So the top and bottom face would be lit at the left vertexes and change to dark at the right ones. All exept the left face are in the shadow volume, so are rendered with only ambient light. So there is a sudden drop in light from the front face to the top and bottom faces which would normally be a gradient from the topleft (lit) vertex to the top right (unlit) vertex. Is there a way to solve this? I should change the normals in some way, but I don't want flat shading. Thanx, Marty

It sounds perfectly normal for shadow volumes, shadow volumes are pretty solid if you do them right, but this is not your problem.
Now if the light behaves strangely it is because it is not lit correctly or that your models are not built for stencil lighting, in your case it's because the normals are "wonky", specificly because none of the normals in the polygons points in the actual normal of the plane, try beveling the edges a little, or subdividing your faces.

Hi,

Beveling the edges sounds like a great idear... My models should look like they have smooth surfaces, so i'll need the normals this way. I guess the only solution is to use more faces on big surfaces and near sharp corners.

Thanks,
Marty

Yes, or you could do what i do.
I have a algorithm that decides if the edge/corner should be sharp or not and then assigns a separate normal to each vertic of each polygon rendered.

IT behaves ok, but it is not perfect.

the best solution is to use normal maps, but that is pretty advanced.