Shadow mapping works! It is rather inefficient at the moment, but it does indeed work.



I think it's about time for another issue of "The Devil's Details", this time regarding shadow mapping.

Pixel shader doesn't know its own depth
You do not have access to the depth of the pixel you are working with in the pixel shader. Because of this, when creating the shadow map you need to calculate the depth in the vertex shader and interpolate that value across the primitive.

No built-in method to go back to the default render target
After changing the current render target to create the shadow map, you want to switch back to the default render target so that you can render the scene from the camera's view. Unfortunately, you cannot do this simply by passing NULL to SetRenderTarget(). To get around this problem, keep a reference to the default render target by calling GetRenderTarget() during initialization (or at least before the first call to SetRenderTarget()) and pass it to SetRenderTarget() when you want to switch back to it.

D3DCOLOR_ARGB accepts inputs in the range 0-255, not 0-1
This can catch you off guard, (especially late at night after working all day trying to get shadow maps to work) because colors in the shaders are 0-1, not 0-255. Just remember this and you won't be confused trying to figure out why the shadow map isn't being cleared to white.

Transform from projection space to texture space
When calculating the location of the pixel in the shadow map for finding out if it is in shadow, you need to perform an additional transformation after the normal light view-projection transformation. After the projection transformation (and homogeneous divide) you are in a coordinate space ranging in x and y from -1 to 1. Texture coordinates have their origin at the top-left and range from 0 to 1, and the y axis points downwards (I am using a left-handed coordinate system where y points up). Because of this, you need to translate the positions by (1, 1) and divide by 2, then negate the y coordinate. You can do this before the homogeneous divide, so these transformations can be rolled into the light's view-projection matrix (just make sure you only do this for the lighting stage, not the shadow map creation stage).

Do not perform the homogeneous divide in the vertex shader!
I'm still not sure why this is the case (probably a problem with interpolation), but when you are doing the lighting stage (after the shadow map has been created), performing the homogeneous divide on the pixel's shadow map space coordinates must be done in the pixel shader. I wasn't doing this originally, and the character's shadow on the ground was extremely thin and angled sharply away from the character.

Render targets and depth buffers are best in pairs
At least when the render targets are of differing size. One of the most annoying problems I encountered while writing the shadow mapping system was that the entire shadow map was not being rendered to. The shadow map I was using was 1024x1024 and the window was 800x600, which means that the default depth buffer was also 800x600. Apparently if a pixel in the render target is outside the bounds of the depth buffer, the z-test automatically fails. To fix this problem, a 1024x1024 depth buffer needs to be created along with the shadow map, and set when the shadow map is set. There doesn't appear to be a built-in method to set the default depth buffer after you have set a new one, so in a similar fashion to a previous devil's detail you will need to keep track of a reference to the default depth buffer yourself.

[size="1"]Reposted from http://invisiblegdev.blogspot.com/