I have been designing a object outlining method for my home-grown game engine. I thought the stencil buffer would do it , but I have having problems with the results. I have been trying to make outlines that are always visible through all surfaces and do not join together. The method I designed is like many I have seen on the web , where you render to the stencil buffer writing 1's in buffer , then scale the object up and only render a outline colour to where the stencil is 0. That approach does not work on anything but a edge case. Maybe a cube with no other objects in the world. I need to select multiple overlapping objects at the same time with differing depths etc. ( real world example. )

I designed a way to do this , and it goes like this.

1. ) I render all the non-outlined stuff first.

2. ) Then I cycle through a linked list of Objects to draw with an outline , and then draw them scaled up in the outline colour Instead of writing a 1 , I use the GL_INCR ( increment ). Here is a picture of what that looks like with three scaled up squares that form intersections. It's a little hard to see , but where they intersect is a sum of all the increments. ( The three's ).

          11111111111111111111111111111
          11111111111111111111111111111
          11111111111111111111111111111
111111111122222222222222211111111111111
111111111122222222222222211111111111111
111111111122222222222222211111111111111
111111111122223333333333322222222111111
111111111122223333333333322222222111111
111111111122223333333333322222222111111
111111111122223333333333222222222111111
111111111122223333333333322222222111111
111111111122223333333333322222222111111
111111111122223333333333222222222111111
111111111111112222222222222222222111111
              1111111111111111111     
              1111111111111111111
              1111111111111111111
              1111111111111111111
              1111111111111111111
              1111111111111111111

Then I cycle again through the same objects , but with normal scale , and I subtract 1 ( GL_DECR ) instead , and what should result is this…

          11111111111111111111111111111
          10000000000000000000000000001
          10000000000000000000000000001
111111111121111111111111100000000000001
100000000010000000000000100000000000001
100000000010000000000000100000000000001
100000000010001111111111211111111000001
100000000010001000000000100000001000001
100000000010001000000000100000001000001
100000000010001000000000100000001000001
100000000010001000000000100000001000001
100000000011112111111111211111111111111
100000000000001000000000100000001
111111111111112111111111211111112
              1000000000000000001     
              1000000000000000001
              1000000000000000001
              1000000000000000001
              1000000000000000001
              1111111111111111111

where all the outlines remain as non zero , I then want to use this mask to render the stencil outline to the screen with glStencilFunc( GL_GREATER , 0 , 0xFF ) ; Unfortunately , what actually happens , is that the renderer , OpenGL in my case , increments the bits , not only once , but for every fragment of the the same pixel location from objects behind it , other front facing polygons , etc. I have not been able to get around this behaviour , and am wondering if anyone knows how to stop all the other fragments from incrementing the same pixel location. If someone knows , then this is a far superior outlining algo than anything I have seen on the web.

Sorry about the second pic , I messed up typing in the 1 2 and 3's. Its a little different but should be good enuf to understand what is really going on there.

This sounds a lot like your depth state is incorrectly set up. If all your fragments pass the depth test, each one will increment your stencil once.

Either, make sure that your fragments can fail the depth test when rendering the stencil.

Or to improve your entire setup. Use the replace stencil instead and skip the second step of your stencil rendering.

Then when you want to render your outline you just do a fullscreen pass, adding colors to the stenciled pixels. Then you render the scene as normal on top with the standard scale. In theory this would leave some stenciled pixels colored "outside" of your geometry.

Thank you for the response Baemz.  I agree it does sound like a problem with the depth somehow ,  I just do not know how to set it properly. Multiple fragments in the same location ARE incrementing the stencil and that IS the problem.  I need to make it increment only once per render.
 
I use a linked list of objects , and scan thru the list looking for objects with a variable ‘use_stencil_active’ set to 1. I render these to the stencil buffer.

The if( i == 1 ) block is for the multiple draw passes where:
The 0th pass is drawing all non-stenciled objects.
The 1th pass is drawing all stenciled objects scaled up.
  Stencil is incremented.
The 2th pass is drawing all stenciled objects normal scale.
  Stencil is decremented.
I only show the 1th to demo the code I am using.

The use_stencil_draw variable is to tell the drawing routine to not separate the rendering of multiple materials in the same object into multiple draw calls If it renders them with multiple draw calls it outlines each different material.

The uniform variables are for telling the vertex shader to scale the stenciled objects in the 1th pass up.

After all the rendering is done , I draw a screen sized quad in ortho mode using the created stencil.

My end goal is to be able to show the outline of all stenciled objects at the same time without any lines being draw on top of.  That is the reason for drawing the quad last in the stencil color.

Can you see what I am doing wrong in this code segment?  I can supply more if needed but I thought for brevity I would only include the stencil part.

if( i == 1 ){ // Draw outlined objects scaled up , writing to the stencil buffer.
 if( current_object_pointer->use_stencil_outline == 0 ){
 current_object_pointer = current_object_pointer->pointer_to_next_object_node ;
 continue ;
} else {
  use_stencil_draw = 1 ;
  glColorMask( GL_FALSE , GL_FALSE , GL_FALSE , GL_FALSE ) ;
  glEnable( GL_DEPTH_TEST ) ; // Enable the depth test.
  glStencilFunc( GL_ALWAYS , 1 , 0xFF ) ; // Stencil Pass always succeeds.
  glStencilOp( GL_KEEP , GL_KEEP , GL_INCR ) ; // Increment stencil
  glStencilMask( 0xFF ) ; // Enable writing to the stencil buffer.
  glUniform1i(
  pointer_to_active_enviornment->pointer_to_start_of_uniform_location_array
  [ VERTEX_STENCIL_ACTIVE ] , 1 ) ;
  glUniform1f(
  pointer_to_active_enviornment->pointer_to_start_of_uniform_location_array
  [ STENCIL_SCALAR_VALUE ] , .02 ) ;
  glUniform1i(
 }
}

If I understand your problem, during your second step of incrementing the stencil buffer values. Some pixels got incremented multiple times, from back faces or hidden part of your mesh.

If that's so i faced a similar problem when working with transparency. You should probably do a pass before to fill your depth buffer with only the fragment that should actually pass the depth test.

1)Fill your depth buffer by rendering your desired objects in which ever order.

2)Do your “incremental” stencil pass with glDepthFunc(GL_EQUAL); By using EQUAl only the desired fragments would affect the stencil buffer.

Hope it helps

Ok , I had a little success with the “Fill your z-buffer” first method. I now can use

glStencilFunc( GL_ALWAYS , 1 , 0xFF ) ;

glStencilOp( GL_KEEP , GL_KEEP , GL_INCR ) ;

to increment the stencil with the model scaled up. I am still getting many extra fragments of garbage from back facings polygons though , but they are 0 or 1 instead of 0 and 1,2,3,4,5,etc , it is no longer incrementing multiple times in the same place. This is a GOOD thing! ( I check this by using a variable in the glStencilFunc( GL_EQUAL , my_check-variable , 0xFF ) ; when drawing my colored quad over top of everything with the stencil.

Then I use the same operation to decrement the stencil , with the model normal scale.

This does not decrement the stencil values at all , and no pixels are removed.

But , If I use the same scale on both the increment AND decrement , the stencil values increment AND decrement. Unfortunately , I cant cut the center out if I use the same scale…..

I have just read the entire GLSL4.6 spec and the OpenGL4.6core spec and have found no help in there as well. ( What a tedious read those were… )

It seems like to me that the set of pixels in the normally scaled render is not the same set of pixels being used in the scaled up render operation.

I tried refilling the z-buffer before both renders to see if that would work , but there was no change.

I have been working on this now for what seems like a month. I am almost thinking that the MESA implementation of the stencil buffer might just be broken and non-working.

I am now looking into the vertex and fragment shader parts to see if I can find anything there , but have not found much to bite on. I really do not know where in the shaders a problem like this may be caused.

Any more help would be very appreciated.