Boost::python + Distutils Python Extension Module Example

In getting my ongoing python CSG library functioning, I needed a way to compile and build python extension modules.  Typically I use qmake from the QT library to build projects, as it's relatively cross-platform (Xcode support is broken as of version 4+) and has relatively little magic sauce making it function (I like transparency in a build system).

However, this isn't so hot for python modules where it's nice to have a simple setup.py script that compiles and installs your module.   Distutils provides just this functionality, while being pretty easy to pick up.  So I decided to do the building with distutils.

Then you have to choose how to actually generate the wrapper code.  Doing this by hand using the python API is manageable but cumbersome and results in a lot more code to maintain.  To improve this, there are a number of other wrapper code generators, including SWIG and boost::python.  I prefer boost::python because it i) is very explicit, you define the exposed interface directly and ii) works with all compilers, without custom build steps or interface file compilation.  It's a bit more typing than SWIG, but I (again) like the transparency.

Of course the downside to boost::python is bjam, the boost build system that appears to summon demons from the ether to build your project through sorcery.  Most tutorials assume you will use bjam but I have yet to get it to work on an even modestly complicated project.  So I set about building python modules wrapped by boost::python using Distutils.  This is actually pretty easy, but I had to dig around through forums and documentation to find out how. Consequently I decided to post a full example from start to finish.

I'm going to assume that we're wrapping two source files functions_to_wrap.cpp and classes_to_wrap.cpp, located in root_dir/src, which have headers functions_to_wrap.h and classes_to_wrap.h located in root_dir/include.  The two headers are:

functions_to_wrap.h

#ifndef FUNCTIONS_TO_WRAP_H
#define FUNCTIONS_TO_WRAP_H

// returns a random string
const char *get_string();

// returns true if values are equal
bool are_values_equal( int a, int b );

// returns the number of supplied arguments to demonstrate 
// boost::python's default argument overloading features
int num_arguments( bool arg0, bool arg1=false, bool arg2=false, bool arg3=false );

#endif

classes_to_wrap.h

#ifndef CLASSES_TO_WRAP_H
#define CLASSES_TO_WRAP_H

class wrapped_class {
private:
 int m_value;
public:
 wrapped_class();
 wrapped_class( const int value );
 
 void set_value( const int value );
 int  get_value( ) const;
};
#endif

There are also the implementation files, but these don't actually matter as far as the build process or wrapper definitions are concerned.  To wrap these, we write an additional c++ file in root_dir/src called boost_python_wrapper.cpp that will define the python interface.  The contents of this file are:


boost_python_wrapper.cpp

#include<boost/python.hpp>

// include the headers for the wrapped functions and classes
#include"functions_to_wrap.h"
#include"classes_to_wrap.h"

using namespace boost::python;

// generate overloads for default arguments, this example is a function
// that takes a minimum of 1 argument and a maximum of four. the 
// overloads are generated in num_arguments_overloads, which is 
// then supplied to boost::python when defining the wrapper
BOOST_PYTHON_FUNCTION_OVERLOADS( num_arguments_overloads, num_arguments, 1, 4 );

// generate wrapper code for both the functions and classes
BOOST_PYTHON_MODULE(ExtensionExample){
 // declare the functions to wrap. note inclusion of the
 // num_arguments_overloads() to include the overloads 
 // that handle default function arguments
 def( "get_string",       get_string );
 def( "are_values_equal", are_values_equal );
 def( "num_arguments",    num_arguments, num_arguments_overloads() );
 
 // declare the classes to wrap. both the default
 // constructor and the constructor taking an 
 // integer argument are exposed
 class_<wrapped_class>("wrapped_class",init<>())
 .def(init<int>())
 .def("get_value", &wrapped_class::get_value )
 .def("set_value", &wrapped_class::set_value )
 ;
}

This file calls a bunch of boost::python macros which instantiate the wrapper code, including handling default arguments for functions and overloaded constructors for classes.  It is also possible to do function overloading, although I've left this out for simplicity.  Note that there's nothing specific about the filenames I chose, as long as the functions and classes to be wrapped are included in the wrapper source code file, you can use any structure/names you want.

When boost_python_wrapper.cpp, functions_to_wrap.cpp and classes_to_wrap.cpp are compiled as a shared library, boost::python will generate wrapper automatically generate wrapper code inside of classes_to_wrap.cpp by macro-expansion, so no additional source files are needed.  Just link the results with the python and boost::python libraries and you get a python module. 

Distutils provides a clean way of doing this and installing the module automatically.  To make a build/install script, we create a script setup.py in the root directory. The contents of this script are little more than compiler and linker flags.

Here's the setup.py script:

from distutils.core import setup, Extension

# define the name of the extension to use
extension_name    = 'ExtensionExample'
extension_version = '1.0'

# define the directories to search for include files
# to get this to work, you may need to include the path
# to your boost installation. Mine was in 
# '/usr/local/include', hence the corresponding entry.
include_dirs = [ '/usr/local/include', 'include' ]

# define the library directories to include any extra
# libraries that may be needed.  The boost::python
# library for me was located in '/usr/local/lib'
library_dirs = [ '/usr/local/lib' ]

# define the libraries to link with the boost python library
libraries = [ 'boost_python-mt' ]

# define the source files for the extension
source_files = [ 'src/boost_python_wrapper.cpp', 'src/functions_to_wrap.cpp', 'src/classes_to_wrap.cpp' ]

# create the extension and add it to the python distribution
setup( name=extension_name, version=extension_version, ext_modules=[Extension( extension_name, source_files, include_dirs=include_dirs, library_dirs=library_dirs, libraries=libraries )] )

Note that the include and library paths are most likely included by default. I have added them simply to demonstrate how you would include (potentially custom) headers and libraries into the build process without overcomplicating the example.

To build and install the module, you can then type:

$ python setup.py install

This should produce a lot of compiler spew.  If there are no errors, you should then be able to use the module.  Here is an example script that does just that:

demo.py

from ExtensionExample import *

print 'calling get_string(), should print out a random string'
print get_string()

print 'calling get_string(), should print out a random string'
print get_string()

print 'calling are_values_equal( 10, 10 ), should print True'
print are_values_equal( 10, 10 )

print 'calling are_values_equal( 10, 1 ), should print False'
print are_values_equal( 10, 1 )

print 'calling num_arguments( True ), should print 1'
print num_arguments( True )

print 'calling num_arguments( True, True ), should print 2'
print num_arguments( True, True )

print 'calling num_arguments( True, True, True ), should print 3'
print num_arguments( True, True, True )

print 'running: a = wrapped_class()'
a = wrapped_class()
print 'running: a.get_value(), should print -1'
print a.get_value()
print 'running: a.set_value( 5 )'
a.set_value( 5 )
print 'running: a.get_value(), should print 5'
print a.get_value()

If you run this script you should see the expected output from the script (evident from the script itself). You can download the entire example as a zip file here which includes all the source/header files as well as the build scripts and a readme.  Hopefully it will be helpful to other people, or just me later on.