Very Large Development: How to Run Code Review for 1000 Developers

### Overview In this talk, Joe Gordon, an upstream OpenStack developer at HP, discusses the challenges and engineering practices behind managing code review and continuous integration for OpenStack—one of the largest open-source Python projects in existence. Over the preceding 12 months, OpenStack saw over 1,200 developers contributing across millions of lines of code. Gordon details the toolchain, automated gating mechanisms, and architectural policies used to sustain rapid development across hundreds of contributors while preventing trunk from breaking.

### Key Topics - **Scale of Very Large Development:** Scaling an open-source Python project with thousands of contributors, over 1.4 million lines of Python code, and massive enterprise deployments reaching up to 16,000–18,000 nodes (such as Bluehost). - **The Gatekeeping Pipeline:** Moving away from human-driven merges toward fully automated gating systems where no individual developer or reviewer has manual merge privileges. - **Enforcing Code Style and Quality:** Offloading human review burden by enforcing strict automated static checks, PEP 8 standards, and licensing rules via tools like Flake8 and custom plugins. - **Integration Testing and Pipelining:** Running thousands of integration tests on virtual environments and optimistically pipelining merge queues using Zuul and DevStack. - **Diagnosing Transient Failures:** Managing flaky or race-condition failures at scale using Logstash, Elasticsearch, and Elastic Recheck to automatically classify known bugs.

### Technical Details - **Code Review and Merging Workflow:** Developers use `git-review` to submit patches directly to Gerrit. Gerrit and Jenkins/Zuul trigger automated test suites. Patches require two core reviewers to approve, after which the patch enters an automated gate test. If tests pass, the system automatically merges the code. - **Test Environment & Tooling:** Unit and integration testing use `tox` for managing Python environments across versions (Python 2.6, 2.7, with work toward 3.3 and PyPy), and `testrepository` (developed by Robert Collins) to execute unit and integration tests concurrently in parallel threads, keeping test suite execution under an hour. - **Style and Custom Gate Checks:** Flake8, PEP 8, PyFlakes, and the OpenStack `hacking` plugin enforce rules such as an 80-character line limit, no unused imports, defined variable checks, single-line imports in alphabetical order (separated by standard, third-party, and project groups to eliminate merge conflicts), and mandatory Apache 2.0 license headers (enforced via checks H102 and H103). - **Optimistic Pipelining with Zuul:** Because integration tests across multiple services (Nova, Swift, Glance, Keystone, Cinder, Neutron) take substantial time, Zuul pipelines patches speculatively in queue order to allow merging far more than 24 patches per day across donated cloud resources (from HP and Rackspace public clouds). - **Automated Failure Analysis:** With 10–20 MB of logs generated per Tempest test run, OpenStack feeds test logs into Logstash and Elasticsearch. Tools like `recheck` and `elastic-recheck` query logs against known bug patterns (e.g., HTTP 503 errors), reducing manual log inspection and lowering transient gate failure rates from ~10% down to approximately 1%.

### Takeaways To scale a project to thousands of developers, human reviewers must be treated as the most constrained resource. Routine tasks such as style verification, dependency enforcement, and unit/integration verification must be strictly offloaded to automated gating tools where no human has merge rights. Maintaining an egalitarian, trunk-based continuous deployment workflow requires robust log aggregation and automated bug classification to prevent infrastructure and code regressions from blocking developers.