profile picture

Are we Python yet?

January 28, 2015

While it was a lot of fun to see a web-based python interpreter beat my system python on a single carefully-tuned benchmark, that result obviously didn't say much about the usefulness of PyPy.js for any real-world applications. I'm keen to find out whether the web can support dynamic language interpreters for general-purpose use in a way that's truly competitive with a native environment.

Inspired by the PyPy speed center and the fine Mozilla tradition of publicly visualising performance metrics, I've been working on a benchmark suite and metrics-tracking site for PyPy.js. The initial version is finally live:

Are we Python yet?

TL;DR:  not really, not yet – but we're tracking slowly towards that goal.

There are a lot of different ways to look at the data on this site, but here are a few highlights from my own explorations:

So where are we at, and where to from here?

Given the above, I think it's fair to say that PyPy.js is not yet competitive with a native python environment. The combination of large download size and startup overhead, expensive JIT compilation pauses, and occasional embarrassing performance penalties will be a show-stopper for anything but very specialised applications. And we're not yet even tracking another important metric: the performance of interacting with the host javascript environment and DOM.

But I do see a lot of scope for continuing to improve these metrics.

On the performance front, there will always be some penalty to the approach taken in PyPy.js – while a native PyPy interpreter can just write a chunk of native code into memory and then jump to it, we must build a javascript source string in memory, copy it out of the asmjs heap into a native string object, ask the host environment to compile it for us, then indirectly execute it via a helper function. There is, however, plenty that could be done to reduce this overhead from the current implementation, which naively builds all the code for a loop into a single function and re-compiles it on every change. Since javascript compilation time can increase non-linearly with function size, just breaking things up into smaller functions should go a long way here.

When it comes to download size, I think I can still squeeze out maybe a MB or two through improvements to the code generation process, such as this in-progress patch for reducing the number of writes to PyPy's shadowstack. After that we'll have to reach for more drastic techniques, such as separating the code for builtin modules from that of the interpreter itself so that they can be loaded on demand.

Can these improvements bring us within reach of competing with native? While I'm hopeful, I honestly don't know. But if nothing else, having these graphs to watch is highly motivating! We may not be python yet, but we're also not done yet. Stay tuned.