What breaks when your team ships 4 times more code

We made development 4 times faster, and bugs went down 28%. That is the headline, and it is the least interesting thing about the last year.

The interesting part is what broke on the way there, because none of it was the code generation, and almost all of it was predictable from arithmetic we should have done first.

Here is the general shape of the mistake, and I have now watched several teams make it. You did not build a faster team. You pointed a firehose at a pipeline that was sized, tuned, and staffed for the rate at which humans type. Every stage downstream of writing code has a service rate. None of those service rates changed.

The arithmetic nobody runs

Little's Law is the whole story: the amount of work sitting in a system equals the arrival rate multiplied by how long each item spends inside it.

The consequence people forget is what happens when the arrival rate approaches the service rate. Queue length does not rise smoothly. It rises toward infinity. At 80% utilization, a queue is unpleasant. At 95% it is unusable. At 100% it never drains, and the work in progress grows without limit until something outside the system intervenes, usually a human deciding to skip a step.

So when you multiply the arrival rate of changes by 4 and leave every downstream service rate untouched, you have not made your organization faster. You have moved the constraint and made it worse, and the symptom will show up somewhere you were not watching.

They showed up for us in this order.

Review, which is a fixed human resource

A good reviewer does maybe 6 to 8 meaningful reviews a day before quality degrades. That number is a property of human attention, and it has not improved since 1975.

Go from 5 pull requests a day to 20, and you are asking for roughly 3 times the review capacity you have. The queue does not stabilize at "a bit slower". It grows every day, and then it resolves itself the ugly way: people start approving without reading. You will not see this in your metrics, because a rubber-stamp approval and a real one look identical in the API.

Two things fixed it, and only one of them is the obvious one.

The obvious one is splitting review. The engine does the mechanical pass: style, obvious defects, missing error handling, test coverage of the changed paths. A human still approves, always, but they open a pull request where the boring work is done and spend their attention on whether this is the right solution.

The non-obvious one matters more. We changed what an agent is asked to produce. The unit of work is no longer "implement this ticket"; it is "produce a change a human can review in under 15 minutes". Those are different instructions, and they generate different diffs. Agents will happily produce a 2,000-line pull request that is technically correct and functionally unreviewable, and if you accept those, you have not increased throughput; you have converted review from a check into a ritual.

Continuous integration, which you are now paying for 4 times

This one is arithmetic that lands directly on the bill. 4 times the pull requests means 4 times the pipeline runs, and CI cost scales close to linearly with runs. If your monthly CI spend was tolerable, it now is not.

Worse, wall clock time became a throttle. A 20-minute suite is fine when 5 changes a day go through it and intolerable when 40 do, because the runs queue behind each other and the effective feedback loop stretches to hours.

The fix is test impact analysis: work out which tests could possibly be affected by the changed files and run only those on the pull request, keeping the full suite for the merge and for a scheduled run. This is not new technology; it is just that most teams never needed it. At the old arrival rate, the full suite was cheap enough to be lazy about.

Flaky tests, which stop being an annoyance and become a wall

Here is the calculation I wish I had done two years earlier.

Take a suite of 5,000 tests where each individual test is 99.99% reliable per run. That sounds excellent. Almost nobody measures per-test flakiness that carefully, and if they did, they would be pleased with that number.

The probability that the whole suite passes is 0.9999 to the power of 5,000, which is about 0.61. So roughly 39% of your runs fail for reasons that have nothing to do with the change being tested.

At 5 runs a day, that is 2 spurious failures, which your team absorbs as background noise. At 40 runs a day, it is 15, and each one costs a human interruption and a judgment call about whether to re-run or investigate. Human judgment is the fixed resource, so what was a papercut becomes the dominant cost in the system. Then people start reflexively hitting re-run, and at that moment your test suite has stopped being a signal.

We treated flakiness as a defect class with its own budget. A test that flakes twice gets quarantined automatically, and quarantined tests either get fixed within a sprint or deleted. Deleting a flaky test is not vandalism. A test that fails randomly provides no information, and it costs attention every time it runs.

Semantic conflicts, which are the genuinely hard ones

This is the failure mode I had not seen before and the one I would warn people about first.

When several agents work in parallel across a codebase, you get pull requests that touch different files, merge cleanly with no textual conflict, and pass their own tests independently. Then they are merged together, and the system is broken.

One changed the shape of a function's return value and updated its two callers. The other added a third caller against the old shape, in a file the first one never touched. Git sees no conflict, because git compares text. The conflict is semantic, and it only exists in the combined state, which no pipeline run ever evaluated.

Human teams hit this too, but rarely, because humans work slowly enough that one merges before the other starts, and because a person holding the whole change in their head often notices. Neither of those protections survives parallel agents.

The fix is structural. Test the merged result, not the branch. A merge queue that builds each candidate on top of the current head and runs the suite against that combination catches exactly this class of failure, and it is the single highest value piece of infrastructure we added. Beyond that: give parallel tasks non-overlapping ownership where you can, and treat any change to a shared interface as a serialization point rather than something two agents may hold at once.

Test quality, which quietly inverts

Now the one I find most uncomfortable, because it is invisible on every dashboard.

Ask an agent to write tests, and it will write tests that pass. That is the objective it optimizes. You will see coverage go up, and you will feel good about it.

Coverage measures which lines executed during a test run. It does not measure whether anything would have failed had those lines been wrong. Those are entirely different properties, and the gap between them widens when a machine is optimizing for the first one. I have read generated tests that call a function, assert that it returns something not null, and add four percentage points of coverage while verifying nothing at all.

The honest measure here is mutation testing. Deliberately introduce small faults into your code, flip a comparison, change a boundary, remove a call, then check whether any test notices. The proportion caught is your mutation score, and it is a direct measurement of whether your suite can actually detect a defect. Tools exist for most ecosystems, and they have existed for years, largely unused because they are slow and the answer is humbling.

Run it on one important module. If your coverage is 85% and your mutation score is 30%, you do not have a safety net; you have a decoration. That gap is where the 28% bug reduction quietly reverses in an organization that never looks.

Environments and attribution

Two smaller ones worth naming.

Preview environments become a contended resource, because 40 changes a day all want somewhere to run. If they are not ephemeral and created per change, they become a queue in front of a queue.

And regression attribution gets much harder. When 5 things shipped today, finding the culprit is a conversation. When 40 did, it is a search problem, and it needs deploy markers in your observability data, automated bisection, and feature flags so that deploying and releasing are separate events. Without those, the mean time to identify a bad change grows with throughput, which cancels a meaningful part of what you gained.

The principle underneath all of it

The unit of engineering work is no longer the change. It is the verified change.

Generation is now cheap, near-instant and effectively unlimited. Verification is expensive, slow, and bounded by human attention. Any investment in generating more, when verification is your constraint, produces exactly nothing except a longer queue and a more tired team.

So the work is to raise the service rate of everything downstream of writing code. Faster and more selective test execution. Merge queues that evaluate combined states. Flakiness treated as a defect class with a budget. Mutation score rather than coverage. Diffs sized for human attention rather than for tickets. Ephemeral environments. Deploy markers and bisection.

None of that is glamorous, and none of it is AI. It is the same pipeline engineering we have known about for 20 years, suddenly load-bearing, because the arrival rate quadrupled and every one of those stages was tuned for a world that no longer exists.

What to measure instead of velocity

Stop counting merged pull requests. It is now a measure of how fast your generator runs, which you no longer control and which is not your constraint.

Measure the time from pull request opened to merged, which tells you the length of your real queue. Measure the share of pipeline runs that fail for reasons unrelated to the change, which tells you whether your signal still means anything. Measure mutation score on your most important modules, which tells you whether your tests would catch a regression. And measure the ratio of review time to generation time, which tells you where the constraint actually sits today.

When those four are healthy, shipping 4 times more code makes you 4 times more effective. When they are not, it makes you 4 times more exposed, and you will find out which in an incident rather than in a dashboard.

If you want a hand rebuilding the pipeline for the new arrival rate, that is the work I do.