The Stencil Buffer

Last Updated: Oct 5th, 2014

Up until now, we've always been rendering to the color buffer. You can also put values in the stencil buffer. We can then use those values to define the regions we want to draw or mask.

Say we have this quad:

And let's say when the polygon rasterizes, it turns into this 3x3 set of pixels.

Pixels are nothing but GLuint RGB values that go into the color buffer. The color buffer isn't the only buffer we can rasterize into. We can also put integer values in the stencil buffer:

We can then use stencil operations to say whether we do or not want to render where certain stencil values are at.

From LUtil.cpp

//OpenGL texture
LTexture gTexture;

//Polygon attributes
GLfloat gPolygonAngle = 0.f;
GLfloat gPolygonX = SCREEN_WIDTH / 2.f, gPolygonY = SCREEN_HEIGHT / 2.f;

//Stencil operation
GLuint gStencilRenderOp = GL_NOTEQUAL;

At the top of LUtil.cpp, we declare the texture we're going to render. In this tutorial, we're also going to have a spinning triangle render to the stencil buffer. "gPolygonAngle", "gPolygonX", and "gPolygonY" are used to define the triangle.

Lastly, we have "gStencilRenderOp" which defines the stencil operation we're going to use when rendering the texture to the color buffer. In this tutorial, we're going to either not render where the triangle is at, only render where the triangle is at, or ignore the triangle entirely and render everything.

From LUtil.cpp

bool initGL()
{
    //Initialize GLEW
    GLenum glewError = glewInit();
    if( glewError != GLEW_OK )
    {
        printf( "Error initializing GLEW! %s\n", glewGetErrorString( glewError ) );
        return false;
    }

    //Make sure OpenGL 2.1 is supported
    if( !GLEW_VERSION_2_1 )
    {
        printf( "OpenGL 2.1 not supported!\n" );
        return false;
    }

    //Set the viewport
    glViewport( 0.f, 0.f, SCREEN_WIDTH, SCREEN_HEIGHT );

    //Initialize Projection Matrix
    glMatrixMode( GL_PROJECTION );
    glLoadIdentity();
    glOrtho( 0.0, SCREEN_WIDTH, SCREEN_HEIGHT, 0.0, 1.0, -1.0 );

    //Initialize Modelview Matrix
    glMatrixMode( GL_MODELVIEW );
    glLoadIdentity();

    //Initialize clear color
    glClearColor( 0.f, 0.f, 0.f, 1.f );

    //Enable texturing
    glEnable( GL_TEXTURE_2D );

    //Set blending
    glEnable( GL_BLEND );
    glDisable( GL_DEPTH_TEST );
    glBlendFunc( GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA );

    //Initialize stencil clear value
    glClearStencil( 0 );

    //Check for error
    GLenum error = glGetError();
    if( error != GL_NO_ERROR )
    {
        printf( "Error initializing OpenGL! %s\n", gluErrorString( error ) );
        return false;
    }

    //Initialize DevIL and DevILU
    ilInit();
    iluInit();
    ilClearColour( 255, 255, 255, 000 );

    //Check for error
    ILenum ilError = ilGetError();
    if( ilError != IL_NO_ERROR )
    {
        printf( "Error initializing DevIL! %s\n", iluErrorString( ilError ) );
        return false;
    }

    return true;
}

The function initGL() now has a call to glClearStencil() which ironically doesn't clear the stencil. This function works like glClearColor(), in that it defines what value the stencil value gets stencil buffer cleared with when you call glClear().

From LUtil.cpp

bool loadMedia()
{
    //Load texture
    if( !gTexture.loadTextureFromFile32( "26_the_stencil_buffer/opengl.png" ) )
    {
        printf( "Unable to load texture!\n" );
        return false;
    }

    return true;
}

void update()
{
    //Rotate triangle
    gPolygonAngle += 6.f;
}

In loadMedia() we load our texture and in update() we update the triangle's rotation angle.

From LUtil.cpp

void render()
{
    //Clear color and stencil buffer
    glClear( GL_COLOR_BUFFER_BIT | GL_STENCIL_BUFFER_BIT );
    glLoadIdentity();

At the top of render we clear both the color buffer and stencil buffer.

From LUtil.cpp

    //Disable rendering to the color buffer
    glColorMask( GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE );

    //Start using the stencil
    glEnable( GL_STENCIL_TEST );

Now we need to render the triangle to the stencil buffer. Using glColorMask(), we'll disable rendering to the R, G, B, and A values in the color buffer. This way the triangle doesn't render to the color buffer and is invisible to the user. Then we enable stencil testing so we can start doing our stencil operations.

From LUtil.cpp

    //Place a 1 where rendered
    glStencilFunc( GL_ALWAYS, 1, 1 );

    //Replace where rendered
    glStencilOp( GL_REPLACE, GL_REPLACE, GL_REPLACE );

    //Render stencil triangle
    glTranslatef( gPolygonX, gPolygonY, 0.f );
    glRotatef( gPolygonAngle, 0.f, 0.f, 1.f );
    glBegin( GL_TRIANGLES );
        glVertex2f(           -0.f / 4.f, -SCREEN_HEIGHT / 4.f );
        glVertex2f(   SCREEN_WIDTH / 4.f,  SCREEN_HEIGHT / 4.f );
        glVertex2f(  -SCREEN_WIDTH / 4.f,  SCREEN_HEIGHT / 4.f );
    glEnd();

This call to glStencilFunc() will handle how we render to the stencil buffer. "GL_ALWAYS" makes sure that where ever our triangle renders, a "1" (the second argument) will be rendered to the stencil buffer. The last argument is a mask that ANDs the second argument when we render. You'll want to use a different mask if you're using stencil values other than one.

This call to glStencilOp() controls what happens with the stencil values we try to render. We want the values we render (which is the second argument in our call to glStencilFunc()) to the stencil buffer to replace what's in the stencil buffer so the first argument is "GL_REPLACE". The 2nd/3rd arguments have to do with the depth buffer, but since it's disabled it doesn't matter for this demo.

Finally, we render our triangle to the stencil buffer.

From LUtil.cpp

    //Reenable color
    glColorMask( GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE );

    //Where a 1 was not rendered
    glStencilFunc( gStencilRenderOp, 1, 1 );

    //Keep the pixel
    glStencilOp( GL_KEEP, GL_KEEP, GL_KEEP );

With our triangle rendered to the stencil buffer, we turn on rendering to the color buffer again with glColorMask().

This next call controls how the textured quad is going to be rendered using the stencil buffer. The default value we gave to "gStencilRenderOp" was "GL_NOTEQUAL", so only the places where the triangle did not get rendered will the texture render. If "gStencilRenderOp" gets changed to "GL_EQUAL", only in the areas where the triangle was rendered will the texture show up. If it's set to "GL_ALWAYS", it will disregard the stencil test and render the whole texture.

Lastly, we don't want to mess with the existing stencil values we have glStencilOp() keep them all.

From LUtil.cpp

    //Render stenciled texture
    glLoadIdentity();
    gTexture.render( ( SCREEN_WIDTH - gTexture.imageWidth() ) / 2.f, ( SCREEN_HEIGHT - gTexture.imageHeight() ) / 2.f );

    //Finished using stencil
    glDisable( GL_STENCIL_TEST );

    //Update screen
    glutSwapBuffers();
}

Here we render the stenciled texture. Then we disable the stencil test since we're done using it.

From LUtil.cpp

void handleKeys( unsigned char key, int x, int y )
{
    //If the user presses q
    if( key == 'q' )
    {
        //Cycle through stencil operations
        if( gStencilRenderOp == GL_NOTEQUAL )
        {
            //Render where stencil polygon was rendered
            gStencilRenderOp = GL_EQUAL;
        }
        else if( gStencilRenderOp == GL_EQUAL )
        {
            //Render everything
            gStencilRenderOp = GL_ALWAYS;
        }
        else if( gStencilRenderOp == GL_ALWAYS )
        {
            //Render where stencil polygon was not rendered
            gStencilRenderOp = GL_NOTEQUAL;
        }
    }
}

When the user presses 'q', we cycle through the various ways to stencil the texture.

From LUtil.cpp

void handleMouseMotion( int x, int y )
{
    //Set polygon position
    gPolygonX = x;
    gPolygonY = y;
}

When the user moves the mouse, we change the position of the triangle. Note: when you click the mouse the triangle won't move, it's only when you move the mouse when the mouse button is down.

From main.cpp

#include "LUtil.h"

int main( int argc, char* args[] )
{
    //Initialize FreeGLUT
    glutInit( &argc, args );

    //Create OpenGL 2.1 context
    glutInitContextVersion( 2, 1 );

    //Create Double Buffered Window
    glutInitDisplayMode( GLUT_DOUBLE | GLUT_STENCIL );
    glutInitWindowSize( SCREEN_WIDTH, SCREEN_HEIGHT );
    glutCreateWindow( "OpenGL" );

    //Do post window/context creation initialization
    if( !initGL() )
    {
        printf( "Unable to initialize graphics library!\n" );
        return 1;
    }

    //Load media
    if( !loadMedia() )
    {
        printf( "Unable to load media!\n" );
        return 2;
    }

    //Set key handler
    glutKeyboardFunc( handleKeys );

    //Set mouse handler
    glutMotionFunc( handleMouseMotion );

    //Set rendering function
    glutDisplayFunc( render );

    //Set main loop
    glutTimerFunc( 1000 / SCREEN_FPS, runMainLoop, 0 );

    //Start GLUT main loop
    glutMainLoop();

    return 0;
}

Lastly, in the main() function we have to set the mouse motion handler with glutMotionFunc() and pass in GLUT_STENCIL to glutInitDisplayMode().

Download the media and source code for this demo here.

Back to Index