Automated performance testing for Drupal core

The performance gate has greatly improved the number of core patches that are assessed for performance implications. However, the question of "how does this patch affect performance" has in the vast majority of cases consisted of very shallow tests - a single page xhprof benchmark or ab2 before/after runs, normally with tiny amounts of data. This approach is incomplete for catching many types of regressions and completely insufficient for evaluating complex patches.

One major area where there is a lack of coverage is front-end performance tests. We have had regressions in js/css aggregation performance that would have been caught with automated tests. We have also had some serious Javascript performance regressions that took a while to notice. Given the increasing size/complexity of front end codebase and the rise of mobile we simply cannot ignore this.

Our current approach also does not capture the back-end performance pain points most sites experience, which skews our performance testing. For example, our overall performance is quite reliant on caches (not only our own caches, but also database, filesystem etc) - testing a single page not only gives us incorrect information for that page, but is fundamentally unable to give us information about the actual effectiveness of our caching strategies. This kind of problem can be rectified by having a test plan with a mix of anonymous and authenticated users, with realistic data volumes and user navigation patterns, including form submissions.

In this conversation I will:

• Provide a high level overview of the problem space and the main components that I think a solution should provide.
• Propose a high-level architecture and prototype code for discussion. This architecture includes: generating large data sets for testing, generating realistic test plans for user agents, capturing detailed data at all levels of the stack (front-end, php, web-server, database, OS metrics), running benchmarks for both historical commits and pending patches and providing a user interface for time-series drill-down analysis as well as specific comparisons.

The discussion will gather feedback on the problem space and high-level architecture with a goal of gathering a working group to develop this effort further (which may include development, infrastructure/devops, fund raising and governance). The primary goal here is to generate action and momentum to make this happen - even a simple MVP approach is going to enable data-driven decisions and enable us to build Drupal as fast and scalable as possible.