Is there any way to get an effect like the stencil buffer but with intermidiate values? Like be able to draw a grey scale image then when you draw using the buffer it only draws using an alpha of the buffer? anyone get what I mean?

Yes and no, it depends on what you want to do with it.

Basically what I want to do can be accomplished using the AUX buffer (what I'm doing now) but this is not accelerated (unless you have a high end card, like a geforce 6200 or better, or a radeon 9500 or better) and so it is kinda slow, is there a buffer like the AUX buffer that is hardware accelerated? (I've looked into pBuffers, but they won't work because I cannot make one that is the size of the screen)

Use a multipass algorithm, first render the stencil part to the framebuffer, then copy those pixels to one or more textures (it's possible with just one, but if you use more you can save in on texture memory).
clear everything and lastly render it the way you wanted to using those textures.

It's allso possible to do it with a 8bit stencil buffer.
It does requires you to do a pretty advanced stencil hack.
It is fairly hard to build the stencil and the rendering stage is a bit complicated.
But it is supported by hardware. :)

If one method doesn't work, there is allways another way, and another, and another.