Thursday, May 31, 2012

Boost::python + Distutils Python Extension Module Example

In getting my ongoing python CSG library functioning, I needed a way to compile and build python extension modules.  Typically I use qmake from the QT library to build projects, as it's relatively cross-platform (Xcode support is broken as of version 4+) and has relatively little magic sauce making it function (I like transparency in a build system).

However, this isn't so hot for python modules where it's nice to have a simple script that compiles and installs your module.   Distutils provides just this functionality, while being pretty easy to pick up.  So I decided to do the building with distutils.

Then you have to choose how to actually generate the wrapper code.  Doing this by hand using the python API is manageable but cumbersome and results in a lot more code to maintain.  To improve this, there are a number of other wrapper code generators, including SWIG and boost::python.  I prefer boost::python because it i) is very explicit, you define the exposed interface directly and ii) works with all compilers, without custom build steps or interface file compilation.  It's a bit more typing than SWIG, but I (again) like the transparency.

Of course the downside to boost::python is bjam, the boost build system that appears to summon demons from the ether to build your project through sorcery.  Most tutorials assume you will use bjam but I have yet to get it to work on an even modestly complicated project.  So I set about building python modules wrapped by boost::python using Distutils.  This is actually pretty easy, but I had to dig around through forums and documentation to find out how. Consequently I decided to post a full example from start to finish.

I'm going to assume that we're wrapping two source files functions_to_wrap.cpp and classes_to_wrap.cpp, located in root_dir/src, which have headers functions_to_wrap.h and classes_to_wrap.h located in root_dir/include.  The two headers are:


// returns a random string
const char *get_string();

// returns true if values are equal
bool are_values_equal( int a, int b );

// returns the number of supplied arguments to demonstrate 
// boost::python's default argument overloading features
int num_arguments( bool arg0, bool arg1=false, bool arg2=false, bool arg3=false );



class wrapped_class {
 int m_value;
 wrapped_class( const int value );
 void set_value( const int value );
 int  get_value( ) const;

There are also the implementation files, but these don't actually matter as far as the build process or wrapper definitions are concerned.  To wrap these, we write an additional c++ file in root_dir/src called boost_python_wrapper.cpp that will define the python interface.  The contents of this file are:


// include the headers for the wrapped functions and classes

using namespace boost::python;

// generate overloads for default arguments, this example is a function
// that takes a minimum of 1 argument and a maximum of four. the 
// overloads are generated in num_arguments_overloads, which is 
// then supplied to boost::python when defining the wrapper
BOOST_PYTHON_FUNCTION_OVERLOADS( num_arguments_overloads, num_arguments, 1, 4 );

// generate wrapper code for both the functions and classes
 // declare the functions to wrap. note inclusion of the
 // num_arguments_overloads() to include the overloads 
 // that handle default function arguments
 def( "get_string",       get_string );
 def( "are_values_equal", are_values_equal );
 def( "num_arguments",    num_arguments, num_arguments_overloads() );
 // declare the classes to wrap. both the default
 // constructor and the constructor taking an 
 // integer argument are exposed
 .def("get_value", &wrapped_class::get_value )
 .def("set_value", &wrapped_class::set_value )

This file calls a bunch of boost::python macros which instantiate the wrapper code, including handling default arguments for functions and overloaded constructors for classes.  It is also possible to do function overloading, although I've left this out for simplicity.  Note that there's nothing specific about the filenames I chose, as long as the functions and classes to be wrapped are included in the wrapper source code file, you can use any structure/names you want.

When boost_python_wrapper.cpp, functions_to_wrap.cpp and classes_to_wrap.cpp are compiled as a shared library, boost::python will generate wrapper automatically generate wrapper code inside of classes_to_wrap.cpp by macro-expansion, so no additional source files are needed.  Just link the results with the python and boost::python libraries and you get a python module. 

Distutils provides a clean way of doing this and installing the module automatically.  To make a build/install script, we create a script in the root directory. The contents of this script are little more than compiler and linker flags.

Here's the script:
from distutils.core import setup, Extension

# define the name of the extension to use
extension_name    = 'ExtensionExample'
extension_version = '1.0'

# define the directories to search for include files
# to get this to work, you may need to include the path
# to your boost installation. Mine was in 
# '/usr/local/include', hence the corresponding entry.
include_dirs = [ '/usr/local/include', 'include' ]

# define the library directories to include any extra
# libraries that may be needed.  The boost::python
# library for me was located in '/usr/local/lib'
library_dirs = [ '/usr/local/lib' ]

# define the libraries to link with the boost python library
libraries = [ 'boost_python-mt' ]

# define the source files for the extension
source_files = [ 'src/boost_python_wrapper.cpp', 'src/functions_to_wrap.cpp', 'src/classes_to_wrap.cpp' ]

# create the extension and add it to the python distribution
setup( name=extension_name, version=extension_version, ext_modules=[Extension( extension_name, source_files, include_dirs=include_dirs, library_dirs=library_dirs, libraries=libraries )] )

Note that the include and library paths are most likely included by default. I have added them simply to demonstrate how you would include (potentially custom) headers and libraries into the build process without overcomplicating the example.

To build and install the module, you can then type:

$ python install

This should produce a lot of compiler spew.  If there are no errors, you should then be able to use the module.  Here is an example script that does just that:
from ExtensionExample import *

print 'calling get_string(), should print out a random string'
print get_string()

print 'calling get_string(), should print out a random string'
print get_string()

print 'calling are_values_equal( 10, 10 ), should print True'
print are_values_equal( 10, 10 )

print 'calling are_values_equal( 10, 1 ), should print False'
print are_values_equal( 10, 1 )

print 'calling num_arguments( True ), should print 1'
print num_arguments( True )

print 'calling num_arguments( True, True ), should print 2'
print num_arguments( True, True )

print 'calling num_arguments( True, True, True ), should print 3'
print num_arguments( True, True, True )

print 'running: a = wrapped_class()'
a = wrapped_class()
print 'running: a.get_value(), should print -1'
print a.get_value()
print 'running: a.set_value( 5 )'
a.set_value( 5 )
print 'running: a.get_value(), should print 5'
print a.get_value()

If you run this script you should see the expected output from the script (evident from the script itself). You can download the entire example as a zip file here which includes all the source/header files as well as the build scripts and a readme.  Hopefully it will be helpful to other people, or just me later on.


Defcronyke said...

Thank you for this! I was having a hard time deciding on the trade-off between really low-level CPython API (but with a good build system), and a higher-level Boost Python API (but with a terrible bjam build system). Thanks to your post, I don't have to pick one or the other, I can use the high-level API and the good build system! Yay!

James Gregson said...

I'm glad you found it useful!

Naveen said...

This is awesome! Thanks for posting a way of using boost with distutils.

James Gregson said...

My pleasure. It took a while to figure out so I thought I'd try to save others the difficulty.

Miklós Tóth said...

Very useful post!
One question though:
When I want to import the module its missing the boost::python library. Why can it not find it?
Or if I have a 3rd party library in my extension code, how would find the python that library?

James Gregson said...


You'll need to install boost ( with the boost::python option enabled. The build script should work provided that the boost header files are in the search patch.

On my system (OS-X with boost installed via HomeBrew) these are in:


But you will probably have to modify it for your system


Glen Luckjiff said...
This comment has been removed by the author.