What Is a Stencil Buffer?
A stencil buffer is similar to a depth buffer. In fact, it uses part of the depth buffer (for this reason, the depth buffer is often called the depth-stencil buffer). The stencil buffer allows the programmer to set a stencil function that will test the "reference" stencil value - a global value - against the value already in the stencil buffer each time a pixel is rendered.
The outcome of that stencil test determines if the color value of the pixel is written to the render target, and if the depth value of that pixel is written to the depth buffer.
For example, if you render some objects to a scene while the reference stencil is 0, and the stencil buffer has been cleared to 1, the stencil buffer will contain a cut-out pattern of zeros where those objects were rendered. If the reference value was then set to 1, and the StencilFunction was set to CompareFunction.LessEqual, the only pixels that could be rendered to the scene are those that are not in the same location where the stencil value was set to 0. This is a basic way to use the stencil buffer to create an area that is off-limits to the current rendering pass.
The stencil buffer can be used in more sophisticated ways. It is possible to specify StencilOperations that go beyond replace or discard, to increment or decrement the stencil buffer after each successful stencil test. This can be combined with the StencilMask value to ensure that the stencil test only operates on a part of the stencil buffer.
To use the stencil buffer, the DepthFormat must reserve some bits for the stencil buffer. The DepthFormat.Depth24Stencil8 depth format allows 8 bits for a stencil buffer. When you combine this with the RenderState.StencilMask Property, this can provide eight different stencil buffers for the programmer. The Depth24Stencil4 depth format uses 4 bits for the stencil buffer, while the DepthFormat.Depth15Stencil1 format only allows one bit. The stencil buffer can be cleared separately by passing ClearOptions.StencilBuffer to the GraphicsDevice.Clear method.
You can use the DepthStencilBuffer class to create your own depth-stencil buffer. You may need to create your own depth-stencil buffer when you create a custom render target
See Also
Concepts
3D Graphics Overview
What Is a Depth Buffer?
What Is a Render Target?
Tasks
How To: Draw a Shadow
How To: Create a Depth Texture
Reference
DepthStencilBuffer
StencilFunction
StencilMask
StencilOperation
DepthFormat