Compiling RPython Programs

Inspired by a recent discussion on Reddit about a Python-to-C++ compiler called Shed Skin, I decided to write up my own experiences on compiling (a restricted subset of) Python to a stand-alone executable. My tool of choice is the translation toolchain from the PyPy project – a project, by the way, that every Python programmer should take a look at.

Take this very exciting (EDIT: and needlessly inefficient) python script, which we'll assume is in a file "factors.py":

def factors(n):
    """Calculate all the factors of n."""
    for i in xrange(2,n / 2 + 1):
       if n % i == 0:
           return [i] + factors(n / i)
    return [n]

def main(argv):
    n = int(argv[1])
    print "factors of", n, "are", factors(n)

if __name__ == "__main__":
    import sys
    main(sys.argv)

We can of course run this from the command-line using the python interpreter, but gosh that's boring:

$> python factors.py 987654321
factors of 987654321 are [3, 3, 17, 17, 379721]

Instead, let's compile it into a stand-alone executable! Grab the latest source tarball from the PyPy downloads page and unzip it in your work directory:

$> ls
pypy-1.3  factors.py

To compile this script using the PyPy translator, it will need to be in a restricted subset of Python known as "RPython". Among other things this means no generators, no runtime function definitions, and no changing the type of a variable. Unfortunately there is no complete definition of what RPython is – it's basically whatever subset of Python the PyPy developers have managed to make work. A good reference for getting started is the PyPy coding guide.

Fortunately, our simple script is already valid RPython :-)

Next we need to add a hook telling the PyPy compiler where execution of our script begins. Rather than the classic name == "main" block, PyPy loads and executes a special function named "target" to find the entry point for the script:

def factors(n):
    """Calculate all the factors of n."""
    for i in xrange(2,n / 2):
       if n % i == 0:
           return [i] + factors(n / i)
    return [n]

def main(argv):
    n = int(argv[1])
    print "factors of", n, "are", factors(n)
    return 0

def target(driver,args):
    return main,None

Note that the "target" function returns the desired entry-point for the script as a function object. I've no idea why it has to return a tuple – perhaps one of the PyPy devs will stumble past and enlighten us. Also note that the main routine is required to return an integer status code.

Now, we can simply invoke the PyPy translation toolchain on our script:

$> python pypy-1.3/pypy/translator/goal/translate.py --batch --output factors.exe factors.py
...
...lots and lots of output
...
[Timer] Total:                         --- 17.9 s
$>
$> ls
pypy-1.3  factors.exe  factors.py

Yep, it takes almost 20 seconds to compile on my machine. That should give you some idea of how much work PyPy is doing behind the scenes.

The --output option lets you specify the name of the output file, and the --batch option prevents the compiler from trying to open an interactive debug window. You can of course pass --help to get a list of available options, but most of them are specific to building PyPy's standlone python interpreter (e.g. they select the GC engine to use, whether to include stackless extensions, etc) and are not relevant for a generic RPython program.

The resulting executable does just what you'd expect:

$> ./factors.exe 987654321
factors of 987654321 are [3, 3, 17, 17, 379721]

Neat.

For a real-life example of this toolchain in action, check out the experimental "pypy-compile" branch of esky. The script esky/bootstrap.py is valid RPython, while the function compile_to_bootstrap_exe in esky/bdist_esky/init.py automates the process shown here, calling out to PyPy to compile this bootstrapping script into a stand-alone executable.

As for why you might want to go to all this trouble, here's a simple demonstration:

$> time python factors.py 987654321
factors of 987654321 are [3, 3, 17, 17, 379721]

real	0m0.040s
user	0m0.032s
sys	0m0.004s

$> time ./factors.exe 987654321
factors of 987654321 are [3, 3, 17, 17, 379721]

real	0m0.005s
user	0m0.004s
sys	0m0.000s

That's an order-of-magnitude speedup.