DoorDash Reviews 10,000 PRs a Week With AI. The Model Isn't the Interesting Part.
DoorDash runs an AI reviewer across 10,000 pull requests a week and engineers accept 60% of what it says. A single-pass model on their own benchmark caught 31% of real issues. The jump to production didn't come from a better model — it came from the harness wrapped around it.
DoorDash put out how their AI code reviewer works, and the headline number is the kind that gets screenshotted: 10,000 pull requests a week, across 56 repos, and engineers accept about 60% of what it flags. Real changes to real code, not a demo.
The part worth sitting with isn’t the model. They tested a single-pass reviewer — one strong model, good context, one shot — against a benchmark of their own historical PRs with human-confirmed issues. It caught 31%. Two out of three real problems went unflagged.
Their production reviewer, wrapped in a scout-and-verify harness, reached 53.6% on the same benchmark. And here’s the tell that the harness is doing the work: they can swap the models inside it — they later moved to a cheaper, open-weight scout paired with a frontier reviewer — and it still comes out ahead, at lower cost. So the interesting question isn’t “which model.” It’s what they built around it.
The baseline nobody wants to hear
DoorDash built a benchmark they call DashBench: real PRs replayed with the issues a human reviewer actually caught, so you can measure a reviewer against ground truth instead of vibes.
A single-pass reviewer scored 30.7% weighted recall on it. That’s the honest starting point for “point a good model at a diff and ask it to review.” It finds a third of what matters and misses the rest. If you’ve ever wired up an LLM to comment on PRs and quietly turned it off a month later, that number is why.
The naive read is that the model isn’t good enough yet. The read DoorDash landed on is that a reviewer is not a single call. It’s a pipeline, and the pipeline is where the recall was hiding.
One scout, two reviewers
The structure that moved the number is a split. Instead of one model doing everything, the work is divided by job.
A lead scout scans the diff. This is the fast pass — a cheaper, quicker model whose only job is to flag suspicious areas. It does not verify anything. It’s not trying to be right, it’s trying to not miss. Wide net, low cost, deliberately noisy.
Deep reviewers verify each lead. Every spot the scout flagged gets handed to a stronger model that checks the claim against the actual code, with quoted evidence and exact line anchors. This is where the expensive tokens go, and they go only where there’s already a reason to look.
Only survivors get posted. A flagged issue that fails verification never reaches the engineer. It dies inside the pipeline.
DoorDash says adding the scout was the single change that mattered most. That’s the counterintuitive bit. The gain didn’t come from reaching for a smarter reviewer. It came from putting a cheap, wide pass in front of the deep one so it had somewhere to aim. Their no-scout baseline landed at 30.7% weighted recall; the scout-and-verify setup reached 53.6% on the same cases.
The rule that makes it trustworthy
Recall gets you found issues. Trust comes from what you don’t post.
Before a comment goes out, the agent tries to refute its own finding. It argues the other side. If the objection holds — if the code is actually fine, if CI already catches this, if there’s no evidence in the repo for the rule being cited — the comment is killed. Only findings that survive their own disprove pass get posted.
The logic behind it is worth stating plainly: one wrong comment costs more trust than ten missed ones. An engineer who gets a confidently-wrong review comment learns to ignore the bot. An engineer who occasionally gets a missed issue just treats the bot as a helpful-but-partial second pair of eyes. The asymmetry is huge, and the disprove-it pass is built around it.
So the pipeline is tuned in two directions at once. The scout widens the net to raise recall. The disprove-it pass narrows what escapes to protect precision. Neither is a model choice. Both are harness.
What it costs to run
The numbers that come out the other side:
- 60.2% acceptance on high and critical findings — engineers actually change the code.
- 10,000+ PRs reviewed weekly across 56 repos, spanning Go, iOS, Android, web, and infra.
- ~7 minutes from a PR being opened to a full review posted.
- $3.91 per PR on their 105-case benchmark run.
Four dollars and seven minutes for a review that lands more than half the time is a real tool, not a party trick. And none of it required a frontier model nobody else has. It’s the same API you can call, arranged so the cheap model does the wide work and the expensive model does the narrow work.
Three rules that transfer
You don’t run DoorDash’s infrastructure, but the shape of this moves to almost any agent you’re building.
Split scouting from verifying. Let a fast, cheap model flag suspicious spots. Spend your strong model only on confirming those spots. You get wide coverage and deep checking without paying frontier prices on every token. This is the general move — a cheap wide pass feeding an expensive narrow one — and it’s not specific to code review.
Add a disprove-it pass. Before you trust a finding, make the agent argue against it. Kill anything that doesn’t survive the objection. This is the cheapest precision upgrade you can bolt onto an agent, and it’s the difference between a tool people trust and one they mute.
Write your rules from your post-mortems. DoorDash’s review rules came from real incidents and their own PR history, not a generic lint config. If CI already enforces a rule, drop it — the agent shouldn’t spend attention on things a linter catches for free. The value is in the judgment calls: silent behavior changes, cross-file drift, deleted code that still had a job.
The model is rented. The harness is yours.
Everyone can call the same models. That’s the part that commoditized. What DoorDash actually owns is the arrangement — the scout/verify split, the disprove-it pass, the rules mined from their own incidents. None of that came in the box. They proved the point themselves: swap in cheaper, open-weight models for the easy work and the harness still comes out ahead of an all-frontier setup.
That’s the reframe I keep coming back to. When your agent underperforms, the instinct is to wait for a better model. Usually the model is fine. It’s doing one pass at a job that needed three, with nothing checking its work and no cheap step out front to point it at the right place. The recall was never locked inside the weights. It was sitting in the harness the whole time, waiting for someone to build it.