While it was a lot of fun to see a web-based python interpreter beat my system python on a single carefully-tuned benchmark, that result obviously didn't say much about the usefulness of PyPy.js for any real-world applications. I'm keen to find out whether the web can support dynamic language interpreters for general-purpose use in a way that's truly competitive with a native environment.

Inspired by the PyPy speed center and the fine Mozilla tradition of publicly visualising performance metrics, I've been working on a benchmark suite and metrics-tracking site for PyPy.js. The initial version is finally live:

Are we Python yet?

TL;DR:  not really, not yet – but we're tracking slowly towards that goal.

OK OK, I couldn't resist that title but it probably goes a bit far. Let me try for a little more nuance:

PyPy.js: Now faster than CPython, on a single carefully-tuned benchmark, after JIT warmup.

It has been the better part of a year since I first started hacking on PyPy.js, an experiment in bringing a fast and compliant python interpreter to the web. I've been pretty quiet during that time but have certainly been keeping busy. Some of the big changes since my previous update include:

• An asmjs-to-python converter, so that PyPy's comprehensive JIT testsuite can be run on the asmjs backend.
• Some new optimizations in the emscripten compiler, which greatly reduce compiled code size.
• A basic interactive console, so you can try PyPy.js straight from your browser.
• And even uncovering an apparent bug in an LLVM optimization pass.

The result, while still rough in a lot of places, is nonetheless an exciting milestone: the full PyPy interpreter, compiled down to javascript and re-targeted to emit asmjs from its JIT, running the pystone benchmark faster in a browser than the native CPython interpreter runs it on bare metal.

Two weeks ago I hatched a plan to port Python to the Web Platform by using Emscripten to translate PyPy into JavaScript. My hope is to produce something than can run in the browser with performance comparable to a standard Python interpreter, a hope which hinges on two key ingredients:

• PyPy's powerful just-in-time compiler, which can optimize the hot loops of your program into efficient native code.
• The recent work on asm.js, a low-level subset of JavaScript that can act like an efficient virtual machine in the browser.

By translating the PyPy interpreter into asm.js code, and by having its JIT backend emit specialized asm.js code at runtime, it should theoretically be possible to have an in-browser Python implementation whose hot loops perform within a factor of two of native code.

I'm excited to report a small but important milestone on the road to making this a reality.

It's certainly not a full Python interpreter, and it comes with many caveats and question-marks and todos, but I have been able to produce a simple demo interpreter, with JIT, that approaches the theoretical factor-of-two comparison to native code under some circumstances. There's a long way to go, but this seems like a very promising start.

TL;DR? Feel free to jump straight to the important graph.

I've been spending a lot of time in JavaScript land lately. It's not totally unexpected – when I first applied for a job with Mozilla, I was warned only semi-jokingly that "they hire all the best Python programmers and then force them to write JavaScript". I've no deep love or hate for it as a language, but JavaScript is pretty interesting to me as a platform, as a kind of runs-everywhere lowest-common-denominator environment that is slowly being molded and coerced into a pretty decent universal runtime. But if "the Web is the Platform", what's a stranded Pythonista to do?

Port Python to JavaScript, of course!

This has been done before in a variety of ways. Skulpt and Brython are impressive re-implementations of Python on top of JavaScript, including interactive consoles that make for a very compelling demo. Pyjamas lets you translate Python apps into JavaScript so they can be run in the browser. There are many more examples with varying degrees of success and completeness.

I don't want to down-play the phenomenal efforts behind projects like this. But personally, I'm a little wary of the re-implementation treadmill that they risk being stuck on. I'd much prefer to leverage the work that's already been done on making a fantastic Python interpreter, along with the work that's already been done on making a fantastic JavaScript runtime, and re-implement as little as possible while gluing them both together.

I've finally taken my first tentative steps down that path, by combining two amazing projects open-source projects: PyPy and Emscripten.

Mozilla Persona went into beta release last week, and it has been very exciting to see the interest and positive press it has generated. I particularly liked the release of pyramid_persona and the accompanying article Quick authentication on pyramid with persona – it is cleaner, simpler, and more feature-full than my attempt at Pyramid/BrowserID integration from last year. Great stuff!

But there is one weak point to using Persona for authentication: it makes automated access difficult. The login flow depends heavily on javascript and assumes that authentication will involve a real live user and a full-blown web-browser. Scripting access to your site via something like requests would be tricky at best, and downright impossible in the general case.

Most of the webapps we build at Mozilla Services are meant for machines, not people, so we've had to tackle this problem head-on. We have found that a combined authentication approach works very nicely – we provide Persona for live users, and the much more automation-friendly MAC Access Authentication for machines.

In this post I'll show you how easy it can be to combine the two, using pyramid_persona and pyramid_macauth for the heavy lifting, and pyramid_multiauth to tie them both together.

I've got a little bit of downtime at the moment, and I'm determined to spend it keeping my Python skills and projects up to date. First on the list: testing.

Almost all of my open-source projects come with automated tests of some kind. Some are woefully under-tested (m2wsgi, myppy) while others have collected a test case for just about every feature (pyenchant, esky). But they could all stand to benefit from a little more testing discipline.

If you have any of your own python projects, ask yourself: do they work on older versions of python? How about on 2.7? On 3.2? How about on Windows or OSX? Are you sure?

I want to be able to answer a confident yes to all these questions, so I'm embarking on a rather ambitious plan over the coming months: full test suites, with 100% code coverage, tested on Linux, Windows and OSX, across python 2.5, 2.6, 2.7 and 3.2.

If you've seen any of my latest python projects, you know that I spend a lot of time thinking about freezing python programs – taking a python script and packaging it up into a stand-alone application that can be deployed to an end user. My latest quest has been freezing an application that uses PySide for its GUI, and trying to make the resulting distribution bundle as small as possible. The result is a neat little tool called Hatchet.

This post is part motivational, part example. I'll show you a basic "hello world" app in PySide, take you through the process of freezing it into a stand-alone application, then show how to use Hatchet to shrink the distribution down to a manageable size. If that seems a little too academic for you, consider this for real-world motivation: using the techniques shown in this post, I was able to chop over 40MB off of the SaltDrive application bundle for Mac OSX.

The latest release of PyEnchant now contains an experimental binary distribution for OSX, as both an mpkg installer and a python egg. In theory, users on OSX 10.4 or later should be able to just drop pyenchant-1.6.3-py2.6-macosx-10.4-universal.egg somewhere on sys.path and be up and running and spellchecking with ease.

If you're a Mac user, please try it out and let me know if anything doesn't work the way you expect.

Inspired by a recent discussion on Reddit about a Python-to-C++ compiler called Shed Skin, I decided to write up my own experiences on compiling (a restricted subset of) Python to a stand-alone executable. My tool of choice is the translation toolchain from the PyPy project – a project, by the way, that every Python programmer should take a look at.

Take this very exciting (EDIT: and needlessly inefficient) python script, which we'll assume is in a file "factors.py":

def factors(n):
    """Calculate all the factors of n."""
    for i in xrange(2,n / 2 + 1):
       if n % i == 0:
           return [i] + factors(n / i)
    return [n]

def main(argv):
    n = int(argv[1])
    print "factors of", n, "are", factors(n)

if __name__ == "__main__":
    import sys
    main(sys.argv)

We can of course run this from the command-line using the python interpreter, but gosh that's boring:

$> python factors.py 987654321
factors of 987654321 are [3, 3, 17, 17, 379721]

Instead, let's compile it into a stand-alone executable! Grab the latest source tarball from the PyPy downloads page and unzip it in your work directory: read more >>

I've just spent a few days trying to improve the performance of a frozen Python app - specifically, the time it takes to start up and present a login window. Most of the improvements were down to good old-fashioned writing of better code, but I also put together a couple of tricks to help shave off even more milliseconds. They both target one of the major sources of slowness when starting up a Python app: imports.