Matching calibrated cameras with OpenGL

When working with calibrated cameras it is often useful to be able to display things on screen for debugging purposes.  However the camera model used by OpenGL is quite different from the calibration parameters from, for example, OpenCV.  The linear parameters that OpenCV provides are the following:



where (from http://en.wikipedia.org/wiki/Camera_resectioning) is the skew between the x and y axes, are the image principle point , with f being the focal length and being scale factors relating pixels to distance.  Multiplying a point by this matrix and dividing by resulting z-coordinate then gives the point projected into the image.

The OpenGL parameters are quite different.  Generally the projection is set using the glFrustum command, which takes the left, right, top, bottom, near and far clip plane locations as parameters and maps these into "normalized device coordinates" which range from [-1, 1].  The normalized device coordinates are then transformed by the current viewport, which maps them onto the final image plane.  Because of the differences, obtaining an OpenGL projection matrix which matches a given set of intrinsic parameters is somewhat complicated.

Roughly following this post, (update: a much-improved update from Kyle, the post's author is available here) the following code will produce an OpenGL projection matrix and viewport.  I have tested this code against the OS-X OpenGL implementation (using gluProject) to verify that for randomly generated intrinsic parameters, the corresponding OpenGL frustum and viewport reproduce the x and y coordinates of the projected point.  The code works by multiplying a perspective projection matrix by an orthographic projection to map into normalized device coordinates, and setting the appropriate box for the glViewport command.

/**
 @brief basic function to produce an OpenGL projection matrix and associated viewport parameters
 which match a given set of camera intrinsics. This is currently written for the Eigen linear
 algebra library, however it should be straightforward to port to any 4x4 matrix class.
 @param[out] frustum Eigen::Matrix4d projection matrix.  Eigen stores these matrices in column-major (i.e. OpenGL) order.
 @param[out] viewport 4-component OpenGL viewport values, as might be retrieved by glGetIntegerv( GL_VIEWPORT, &viewport[0] )
 @param[in]  alpha x-axis focal length, from camera intrinsic matrix
 @param[in]  alpha y-axis focal length, from camera intrinsic matrix
 @param[in]  skew  x and y axis skew, from camera intrinsic matrix
 @param[in]  u0 image origin x-coordinate, from camera intrinsic matrix
 @param[in]  v0 image origin y-coordinate, from camera intrinsic matrix
 @param[in]  img_width image width, in pixels
 @param[in]  img_height image height, in pixels
 @param[in]  near_clip near clipping plane z-location, can be set arbitrarily > 0, controls the mapping of z-coordinates for OpenGL
 @param[in]  far_clip  far clipping plane z-location, can be set arbitrarily > near_clip, controls the mapping of z-coordinate for OpenGL
*/
void build_opengl_projection_for_intrinsics( Eigen::Matrix4d &frustum, int *viewport, double alpha, double beta, double skew, double u0, double v0, int img_width, int img_height, double near_clip, double far_clip ){
    
    // These parameters define the final viewport that is rendered into by
    // the camera.
    double L = 0;
    double R = img_width;
    double B = 0;
    double T = img_height;
    
    // near and far clipping planes, these only matter for the mapping from
    // world-space z-coordinate into the depth coordinate for OpenGL
    double N = near_clip;
    double F = far_clip;
    
    // set the viewport parameters
    viewport[0] = L;
    viewport[1] = B;
    viewport[2] = R-L;
    viewport[3] = T-B;
    
    // construct an orthographic matrix which maps from projected
    // coordinates to normalized device coordinates in the range
    // [-1, 1].  OpenGL then maps coordinates in NDC to the current
    // viewport
    Eigen::Matrix4d ortho = Eigen::Matrix4d::Zero();
    ortho(0,0) =  2.0/(R-L); ortho(0,3) = -(R+L)/(R-L);
    ortho(1,1) =  2.0/(T-B); ortho(1,3) = -(T+B)/(T-B);
    ortho(2,2) = -2.0/(F-N); ortho(2,3) = -(F+N)/(F-N);
    ortho(3,3) =  1.0;
    
    // construct a projection matrix, this is identical to the 
    // projection matrix computed for the intrinsicx, except an
    // additional row is inserted to map the z-coordinate to
    // OpenGL. 
    Eigen::Matrix4d tproj = Eigen::Matrix4d::Zero();
    tproj(0,0) = alpha; tproj(0,1) = skew; tproj(0,2) = u0;
                        tproj(1,1) = beta; tproj(1,2) = v0;
                                           tproj(2,2) = -(N+F); tproj(2,3) = -N*F;
                                           tproj(3,2) = 1.0;
    
    // resulting OpenGL frustum is the product of the orthographic
    // mapping to normalized device coordinates and the augmented
    // camera intrinsic matrix
    frustum = ortho*tproj;
}

The code uses the Eigen linear algebra library, which conveniently stored matrices in column-major order, so applying the resulting frustum matrix is as simple as:

glMatrixMode(GL_PROJECTION);
glLoadMatrixd( &frustum(0,0) );

Got my face did!

I recently got to visit a local studio and while I was there they asked if I wanted my face 3D scanned.  Of course I did. Who wouldn't? Well at least one person.  But I did.  And here's the result, after runing it through the quadric-collapse decimator of Meshlab.

It's pretty damn good quality wise, even on challenging areas like my beard.  Now I just have to merge the scans and create a watertight mesh, ready for 3D printing!

Getting things done with qmake

I frequently deal with multiple operating systems on various machines and so make efforts to write portable code and use portable third-party libraries where possible.  On some machines I have administrator access while on others I am a lowly user.  How to consistently configure and build projects across all systems became an issue.

Without much forethought, I began to use qmake from NVidia's Qt toolkit.  I did this originally because I was using Qt to build cross-platform GUIs for code, but continue to do so because it works well and is dead easy. There are other cross-platform build systems out there such as CMake, SCONS, and so on, but I've yet to find one that gives me a compelling reason to switch from qmake.

That said, the process that I went through to finally arrive at this process was not straightforward.  This post is intended to roughly document how I set up my machines to work cleanly and easily with qmake. Ultimately this boils down to:

• Use ONLY portable third-party libraries
• Create consistently-named qmake include files for each library on each system
• Keep ALL those include files in a single directory, ideally with the source-tree of the library
• Define a consistently-named environment variable on all machines that points to the directory

As an example, on OS-X, I have a directory Code/ThirdParty, pointed to by the environment variable $ThirdPartyDir which gives the full path to Code/ThirdParty.  The contents of this directory are:

The *.pri files are the qmake include files, while the directories hold the library source trees. Some libraries, like CGAL, automatically install themselves to a different location on the machine, such as /usr/local.  That doesn't matter, the project include file still goes in the directory pointed to by $ThirdPartyDir.  On Windows, I have another directory ThirdPartyDir, located at a completely different path, pointed to by the environment variable %ThirdPartyDir%, with the same sub-directories and versions of the *.pri files modified to work for Windows.

The *.pri files themselves handle adding linker flags, header search paths and any preprocessor definitions needed to the project.  They look like incomplete qmake project files, as seen by the excerpt of my vtk.pri file:

INCLUDEPATH += "/usr/local/include/vtk-5.8"
LIBS += -L/usr/local/lib/vtk-5.8
LIBS += -lvtkalglib \
        -lvtkCharts \
        -lvtkCommon \
        -lvtkGraphics \
        -lvtkDICOMParser \
        -lvtkexoIIc \
        -lvtkexpat

Now any project that needs vtk can simply include the qmake include file into it's project file.  Provided the include file exists and is correct, all the include paths, linker flags and preprocessor definitions will be set up correctly, as in the following project file:

TEMPLATE = app
QT       += opengl
CONfIG   += console debug_and_release
TARGET   = camera_model

!include( $$(ThirdPartyDir)/eigen.pri )
!include( $$(ThirdPartyDir)/cimg.pri )
!include( $$(ThirdPartyDir)/vtk.pri )

mac {
    CONFIG -= app_bundle
    MOC_DIR = build
}

INCLUDEPATH += include
HEADERS += include/camera.h
SOURCES += src/main.cpp \
           src/camera.cpp 

This project file is now completely portable, able to be built on Windows, OS-X and Linux without change.  The include files themselves eigen.pri, cimg.pri and vtk.pri may differ between systems as necessitated by versions, access permissions and install locations, but the project itself is consistent.

This process is ultimately similar to how CMake is intended to function, however I have wasted hours trying to get CMake find scripts to function, often without success.  By just creating these include files whenever I use third-party code, I've found that much of the frustration of cross-platform development just goes away.

4x4 transformation matching OpenGL

An updated version of this code is available on Github, removing the need for an external linear algebra library to compute matrix inverses, although the remainder of this post is still valid. Please see: http://jamesgregson.blogspot.ca/2013/09/updated-4x4-transformation-matching.html to get the updated code.

Several times now I've wanted to develop/debug graphics applications which involve 3D transformations and visualize the results with OpenGL. I usually end up just using the OpenGL matrix functions like glRotatef(...), glFrustum(...) for everything in order to have consistent transformations between the debugging view and the underlying app.

This works OK, however it means that a valid OpenGL context is available, since many of these functions don't work properly without one. To avoid this, I have duplicated most of the OpenGL functions in a custom 4x4 transformation class, which produces nearly identical results (usually to within 10-6 -> 10-4) but which does not require a context. It also exposes some extra operations, such as transposition and inversion along with the GLU functions gluProject, gluUnProject and gluPerspective. These make it quite easy to duplicate the OpenGL vertex transformation pipelines in custom, non-OpenGL code.

For example:

// example code to pan a view

// declare some variables, grab an OpenGL matrix
double mat[16];
transformation T;
glGetDoublev( GL_MODELVIEW_MATRIX, mat );

// load mat into T
T.from_opengl( mat );

// pan the view 
T = transformation::glTranslate( dx, dy, 0.0 ) * T;

// convert back to OpenGL format and update 
T.to_opengl( mat );
glLoadMatrixd( mat );

Currently the library includes the following OpenGL/GLU functions:

transformation::glIdentity(...)
transformation::glScale(...)
transformation::glTranslate(...)
transformation::glRotate(...)
transformation::glOrtho(...)
transformation::glFrustum(...)
transformation::gluPerspective(...)
transformation::gluProject(...)
transformation::gluUnProject(...)

All of these have been tested against the OpenGL/GLU equivalents, and have matching interfaces except they return a transformation rather than operate on the active matrix stack. There are also a number of additional functions to transpose/invert transformations, access individual elements and get the right/up/forward/position vectors from a transformation.

The code is available to download Github:

https://github.com/jamesgregson/transformation

I hope people find it useful. I will be updating the code with any bug-fixes or improvements over time, please send them my way if you have a suggestion or a problem.