Lazy Foo' Productions

The Stencil Buffer

Last Updated 10/05/14
Up until now, we've always been rendering to the color buffer. You can also put values in the stencil buffer. We can then use those values to define the regions we want to draw or mask.
Say we have this quad:

And let's say when the polygon rasterizes, it turns into this 3x3 set of pixels.

Pixels are nothing but GLuint RGB values that go into the color buffer. The color buffer isn't the only buffer we can rasterize into. We can also put integer values in the stencil buffer:

We can then use stencil operations to say whether we do or not want to render where certain stencil values are at.
From LUtil.cpp
//OpenGL texture LTexture gTexture; //Polygon attributes GLfloat gPolygonAngle = 0.f; GLfloat gPolygonX = SCREEN_WIDTH / 2.f, gPolygonY = SCREEN_HEIGHT / 2.f; //Stencil operation GLuint gStencilRenderOp = GL_NOTEQUAL;
At the top of LUtil.cpp, we declare the texture we're going to render. In this tutorial, we're also going to have a spinning triangle render to the stencil buffer. "gPolygonAngle", "gPolygonX", and "gPolygonY" are used to define the triangle.

Lastly, we have "gStencilRenderOp" which defines the stencil operation we're going to use when rendering the texture to the color buffer. In this tutorial, we're going to either not render where the triangle is at, only render where the triangle is at, or ignore the triangle entirely and render everything.
From LUtil.cpp
bool initGL() { //Initialize GLEW GLenum glewError = glewInit(); if( glewError != GLEW_OK ) { printf( "Error initializing GLEW! %s\n", glewGetErrorString( glewError ) ); return false; } //Make sure OpenGL 2.1 is supported if( !GLEW_VERSION_2_1 ) { printf( "OpenGL 2.1 not supported!\n" ); return false; } //Set the viewport glViewport( 0.f, 0.f, SCREEN_WIDTH, SCREEN_HEIGHT ); //Initialize Projection Matrix glMatrixMode( GL_PROJECTION ); glLoadIdentity(); glOrtho( 0.0, SCREEN_WIDTH, SCREEN_HEIGHT, 0.0, 1.0, -1.0 ); //Initialize Modelview Matrix glMatrixMode( GL_MODELVIEW ); glLoadIdentity(); //Initialize clear color glClearColor( 0.f, 0.f, 0.f, 1.f ); //Enable texturing glEnable( GL_TEXTURE_2D ); //Set blending glEnable( GL_BLEND ); glDisable( GL_DEPTH_TEST ); glBlendFunc( GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA ); //Initialize stencil clear value glClearStencil( 0 ); //Check for error GLenum error = glGetError(); if( error != GL_NO_ERROR ) { printf( "Error initializing OpenGL! %s\n", gluErrorString( error ) ); return false; } //Initialize DevIL and DevILU ilInit(); iluInit(); ilClearColour( 255, 255, 255, 000 ); //Check for error ILenum ilError = ilGetError(); if( ilError != IL_NO_ERROR ) { printf( "Error initializing DevIL! %s\n", iluErrorString( ilError ) ); return false; } return true; }
The function initGL() now has a call to glClearStencil() which ironically doesn't clear the stencil. This function works like glClearColor(), in that it defines what value the stencil value gets stencil buffer cleared with when you call glClear().
From LUtil.cpp
bool loadMedia() { //Load texture if( !gTexture.loadTextureFromFile32( "26_the_stencil_buffer/opengl.png" ) ) { printf( "Unable to load texture!\n" ); return false; } return true; } void update() { //Rotate triangle gPolygonAngle += 6.f; }
In loadMedia() we load our texture and in update() we update the triangle's rotation angle.
From LUtil.cpp
void render() { //Clear color and stencil buffer glClear( GL_COLOR_BUFFER_BIT | GL_STENCIL_BUFFER_BIT ); glLoadIdentity();
At the top of render we clear both the color buffer and stencil buffer.
From LUtil.cpp
//Disable rendering to the color buffer glColorMask( GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE ); //Start using the stencil glEnable( GL_STENCIL_TEST );
Now we need to render the triangle to the stencil buffer. Using glColorMask(), we'll disable rendering to the R, G, B, and A values in the color buffer. This way the triangle doesn't render to the color buffer and is invisible to the user. Then we enable stencil testing so we can start doing our stencil operations.
From LUtil.cpp
//Place a 1 where rendered glStencilFunc( GL_ALWAYS, 1, 1 ); //Replace where rendered glStencilOp( GL_REPLACE, GL_REPLACE, GL_REPLACE ); //Render stencil triangle glTranslatef( gPolygonX, gPolygonY, 0.f ); glRotatef( gPolygonAngle, 0.f, 0.f, 1.f ); glBegin( GL_TRIANGLES ); glVertex2f( -0.f / 4.f, -SCREEN_HEIGHT / 4.f ); glVertex2f( SCREEN_WIDTH / 4.f, SCREEN_HEIGHT / 4.f ); glVertex2f( -SCREEN_WIDTH / 4.f, SCREEN_HEIGHT / 4.f ); glEnd();
This call to glStencilFunc() will handle how we render to the stencil buffer. "GL_ALWAYS" makes sure that where ever our triangle renders, a "1" (the second argument) will be rendered to the stencil buffer. The last argument is a mask that ANDs the second argument when we render. You'll want to use a different mask if you're using stencil values other than one.

This call to glStencilOp() controls what happens with the stencil values we try to render. We want the values we render (which is the second argument in our call to glStencilFunc()) to the stencil buffer to replace what's in the stencil buffer so the first argument is "GL_REPLACE". The 2nd/3rd arguments have to do with the depth buffer, but since it's disabled it doesn't matter for this demo.

Finally, we render our triangle to the stencil buffer.
From LUtil.cpp
//Reenable color glColorMask( GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE ); //Where a 1 was not rendered glStencilFunc( gStencilRenderOp, 1, 1 ); //Keep the pixel glStencilOp( GL_KEEP, GL_KEEP, GL_KEEP );
With our triangle rendered to the stencil buffer, we turn on rendering to the color buffer again with glColorMask().

This next call controls how the textured quad is going to be rendered using the stencil buffer. The default value we gave to "gStencilRenderOp" was "GL_NOTEQUAL", so only the places where the triangle did not get rendered will the texture render. If "gStencilRenderOp" gets changed to "GL_EQUAL", only in the areas where the triangle was rendered will the texture show up. If it's set to "GL_ALWAYS", it will disregard the stencil test and render the whole texture.

Lastly, we don't want to mess with the existing stencil values we have glStencilOp() keep them all.
From LUtil.cpp
//Render stenciled texture glLoadIdentity(); gTexture.render( ( SCREEN_WIDTH - gTexture.imageWidth() ) / 2.f, ( SCREEN_HEIGHT - gTexture.imageHeight() ) / 2.f ); //Finished using stencil glDisable( GL_STENCIL_TEST ); //Update screen glutSwapBuffers(); }
Here we render the stenciled texture. Then we disable the stencil test since we're done using it.
From LUtil.cpp
void handleKeys( unsigned char key, int x, int y ) { //If the user presses q if( key == 'q' ) { //Cycle through stencil operations if( gStencilRenderOp == GL_NOTEQUAL ) { //Render where stencil polygon was rendered gStencilRenderOp = GL_EQUAL; } else if( gStencilRenderOp == GL_EQUAL ) { //Render everything gStencilRenderOp = GL_ALWAYS; } else if( gStencilRenderOp == GL_ALWAYS ) { //Render where stencil polygon was not rendered gStencilRenderOp = GL_NOTEQUAL; } } }
When the user presses 'q', we cycle through the various ways to stencil the texture.
From LUtil.cpp
void handleMouseMotion( int x, int y ) { //Set polygon position gPolygonX = x; gPolygonY = y; }
When the user moves the mouse, we change the position of the triangle. Note: when you click the mouse the triangle won't move, it's only when you move the mouse when the mouse button is down.
From main.cpp
#include "LUtil.h" int main( int argc, char* args[] ) { //Initialize FreeGLUT glutInit( &argc, args ); //Create OpenGL 2.1 context glutInitContextVersion( 2, 1 ); //Create Double Buffered Window glutInitDisplayMode( GLUT_DOUBLE | GLUT_STENCIL ); glutInitWindowSize( SCREEN_WIDTH, SCREEN_HEIGHT ); glutCreateWindow( "OpenGL" ); //Do post window/context creation initialization if( !initGL() ) { printf( "Unable to initialize graphics library!\n" ); return 1; } //Load media if( !loadMedia() ) { printf( "Unable to load media!\n" ); return 2; } //Set key handler glutKeyboardFunc( handleKeys ); //Set mouse handler glutMotionFunc( handleMouseMotion ); //Set rendering function glutDisplayFunc( render ); //Set main loop glutTimerFunc( 1000 / SCREEN_FPS, runMainLoop, 0 ); //Start GLUT main loop glutMainLoop(); return 0; }
Lastly, in the main() function we have to set the mouse motion handler with glutMotionFunc() and pass in GLUT_STENCIL to glutInitDisplayMode().
Download the media and source code for this tutorial here.
Back to OpenGL Tutorials