Summary
Presenting "airspeed velocity", a new tool for benchmarking Python software projects over their lifetime.
Description
As software projects mature and become more robust against bugs, they may also lose some of their runtime performance and memory efficiency. Airspeed velocity (asv) is a new tool to help find those performance degradations before they get out to end users. It automatically runs a benchmark suite over a range of commits in a project's repository, as well as in a matrix of configurations of Python versions and other dependencies. The results, possibly from multiple machines, are then collated and published in a web-based report.
While filling a similar role as projects such as "codespeed" and "vbench", airspeed velocity is designed to be easier to set up and deploy, since it uses only a DVCS repository as its database and the report is deployable to any static web server.
Airspeed velocity provides an easy way to write benchmarks, inspired by "nosetests" and "py.test". It is possible to benchmark runtime, memory usage, or any user-defined metric.
Other features either implemented or in the planning stages include:
- tight integration with existing profiling tools, such as RunSnakeRun
- parameterized benchmarks to investigate how an algorithm scales with data size
- automatic search for degrading commits
The presentation will provide a demo of airspeed velocity, and discuss its early usage for benchmarking the astropy project.