AI Code Review: Tier the Risk, Cap the Rounds
One pull request went through more than fifty review rounds overnight; a one-line fix elsewhere took eleven. On 15 October 2026 Claude Code Review and GitHub's agentic autofix both reached general availability. Set the risk tier from the diff, count only people, and give every round an end.
One pull request to the Claude Code CLI went through more than fifty rounds of review and fixes overnight before its author stopped it. The loop needs nobody to keep it turning. “An agent author and an agent reviewer can keep each other busy indefinitely,” Anthropic writes: the reviewer posts findings, an agent picks up the fix overnight, the push starts another review, and the remedies Anthropic lists afterwards (diff-only re-review, no repeated findings) tell you what the rounds were spent on. Cockroach Labs met the small version in its own agent pipeline. Issue 1777 was marked urgent and the fix was one line, calling an existing function in one code path as three sibling paths already did. It took eleven rework rounds over two days, spent on comment hygiene, a false claim in the description, a minor deviation from the plan, test quality, two rebases, two failed workflows and a human approval gate.
Both teams answered with a rule about rounds, and both wrote it up within a week of each other: Cockroach Labs' five-month retrospective on 9 October, and Anthropic's post on how the Claude Code team reviews code on 15 October, the day Code Review left research preview for Claude Team and Enterprise organisations. GitHub's agentic autofix and AI Scan for pull requests went generally available the same day; both are code scanning features, and AI Scan's findings are advisory and cannot block a merge. The review side of the pull request now has agents of its own, posting findings faster than people clear them. What is left to design is the gate: who has to approve, when a round ends, and what each round costs.
Review is a pipeline, and the gate is its last stage
The capacity problem has numbers. A Harvard working paper by Fiona Chen and James Stratton, current version dated 4 August 2026 and widely discussed after Ars Technica covered it on 12 October, uses Jellyfish telemetry from 718 firms. After adopting coding agents, developers opened about 23% more pull requests with no statistically significant change in issues resolved. Time from pull request to merge rose 49% from a 7.03-day baseline, and the share sent back for changes nearly doubled. QGIS drew the blunt conclusion. QEP428, merged on 7 October, bars AI-generated contributions, one maintainer writing that the AI stream is “a lot higher than what human review can take seriously”.
Anthropic, which says Claude writes about 90 percent of Claude Code's own code, runs review as a pipeline. Several agents hunt for bugs at the same time, a separate group votes on each finding, and a finding is posted only if enough of them agree it is real. A security review is a separate pass, open findings from either block the merge, required CI blocks on red, and a classifier sets the approval tier from the change: no person, one, or two. Anthropic reports pull requests with substantive review comments up from 16% to 54%, one-round passes from 65% to 78%, pull requests needing five or more rounds down from 8.9% to 1.5%, and mean reviews to pass from 2.08 to 1.32. Those are the vendor's own figures; the loudest pushback in the Hacker News thread of 16 October was on whether merge volume is the right measure. The shape is what transfers: find, verify, rank, then a gate that counts people.
The product that shipped on 15 October stops short of that gate on purpose. The documentation says a review “never approves or blocks the pull request”, and its check run, named Claude Code Review, does not conclude as a failure. Admins choose Quick, Standard or Deep for each repository, and most reviews finish in 6 to 10 minutes against about 20 in the preview. The findings are the product; the gate is yours to build.
The diff picks the tier; the check counts the humans
Anthropic's tiers are set by what the change touches. Low is docs, copy, internal-only features and additive changes behind a flag: no person once automated review and the security checks are green, for authors on the repository's trusted list, with a second classifier able to demand a person anyway. Medium is ordinary product changes plus sensitive areas such as caching, the auto-updater or system prompts: one person. High is the highest-risk code plus a small set of paths that always need two. “An agent's approval never counts as a person's.” “An approval covers one specific commit.” And on an agent-written pull request “the person directing the agent counts as one of the required reviewers”, so a Medium change driven by one engineer merges on that engineer's approval. Decide that knowingly rather than inherit it.
GitHub carries most of this in branch protection and rulesets, under these names.
- Require approvals. One count for every pull request. In rulesets, Required reviewers adds a per-team count, 0 to 10, for files matching a pattern: organisation-owned repositories only, and the approvals must come from team members.
- Require review from Code Owners. The path rule. “Any pull request that affects code with a code owner must be approved by that code owner”, so
CODEOWNERSis where auth, migrations and.github/get named people. - Dismiss stale pull request approvals when new commits are pushed. Drops an approval “when commits are pushed that affect the diff”.
- Require approval of the most recent reviewable push. The last push “must be approved by someone other than the person who pushed it”.
- Require conversation resolution before merging. Every finding ends before the merge button lights.
Two people from the database team for a migration and none for a docs change is a ruleset on an organisation-owned repository, so set that first. What no setting does is refuse an approval because of who gave it, or bind the count to one commit whether or not the diff changed. The workflow below reads the diff, picks the tier, counts the human approvals on the head commit and fails until the count is met; make it a required status check and the pipeline enforces the tier, on a user-owned repository too.
on:
pull_request:
pull_request_review:
jobs:
approval-tier:
runs-on: ubuntu-latest
permissions:
pull-requests: read
env:
GH_TOKEN: ${{github.token}}
GH_REPO: ${{github.repository}}
steps:
- run: |
pr=${GITHUB_REF_NAME%/merge}
# TIER: paths pick 0, 1 or 2
gh pr diff $pr --name-only >f
need=1
grep -qv '^docs/' f || need=0
hi='^(auth|db|\.github)/'
grep -Eq "$hi" f && need=2
# HUMANS: no bot, this commit
p=repos/$GH_REPO/pulls/$pr
sha=$(gh api $p -q .head.sha)
export sha
q='[.[]
| select(.state=="APPROVED")
| select(.user.type=="User")
| select(.commit_id==env.sha)
| .user.login]
| unique | length'
n=$(gh api -X GET -q "$q" \
-F per_page=100 $p/reviews)
# GATE: the required check
test "$n" -ge "$need"
The choices in it carry the argument. The tier defaults to one person and drops to zero only when every file sits under docs/; auth, the database directory and .github/ raise it to two, because a changed workflow file can read any repository secret it names. The user.type filter counts people and nothing else, whatever the platform settings allow. Binding to head.sha is stricter than GitHub's dismiss option, which fires only when a push changes the diff. -X GET is there because gh api switches to POST as soon as a field is passed, and a POST to /reviews creates a review rather than listing them. per_page=100 is there because the default page is thirty and a pull request a dozen rounds in carries more; past a hundred reviews, paginate and count across the pages. Because the job also runs on review events, an approval or a dismissal re-evaluates the gate without a new commit.
A bot approval counts if someone switched it on
“Never counts as a person's” is a policy you set; GitHub does not guarantee it. Since 1 September 2026 a Copilot code review approval “counts toward the repository's required-approvals rule” when enabled: off by default, configurable at enterprise, organisation and repository level, dismissed on a new push like a human's. The organisation setting Allow GitHub Actions to create and approve pull requests decides whether GITHUB_TOKEN can approve; new organisations start with it off. Check both, and keep the user.type filter in the check above.
Three rounds, then a person
A loop converges when its findings run out, and Anthropic's rules are about making them run out. “Every finding has to be answered. You either fix it or resolve the thread with a reason.” Re-review is diff-only after the first few passes, so the reviewer stops re-reading code the round did not touch. After three rounds “the reviewer asks the author to step back and reconsider the approach”. That is a request to a person, not a stop. Cockroach Labs' circuit breaker escalates too, to another agent. After four consecutive rework rounds that each surfaced a new blocking finding, or six substantive rounds of any kind, the reviewing agent hands the pull request off to an Attending, a separate agent instance running a prompt built around finding fault rather than fixing. A change estimated above 1,000 lines including tests is split before it starts. Cockroach Labs reports 1,238 pull requests, well over a million lines, 7 reverts and just over $135,000 in Claude tokens over five months.
The cap this article prescribes is the hard form of both: three rounds, then the check goes amber and a named person owns the next move, usually a different approach or a smaller change. With diff-only re-review from round two, fix or dismiss with a reason as the only ways to close a thread, and conversation resolution required at the branch, by round three a pull request has either converged or reached a human. Nothing runs overnight. Anthropic reports around 2,500 changes merged a day in May and over 30,000 in September, writes that “A proper merge queue, for example, doesn't make much sense at agent scale”, and tests post-merge instead; most teams are nowhere near that volume, and the gate above comes first.
Cost is the other reason for the cap. Anthropic's 9 March preview announcement put the average review at $15 to $25, “scaling with PR size and complexity”. The costs page now says Code Review “has no flat per-review price”: each review is billed on token usage and scales with the review level, the size of the change and how many candidate issues need verification. A repository added from the admin page starts with its trigger set to After every push, “multiplying cost by the number of pushes”, and a monthly spend limit applies per organisation. On that setting fifty rounds is fifty reviews. Set the round cap and the spend limit as one decision, and give review a line in the same budget as the runner minutes in CI at Agent Speed.
Watch the rounds converge or run away
Press Run review below. One Medium-tier pull request goes through review three ways. In the open loop every push triggers a full re-review and threads stay open, so findings never reach zero and the review bill climbs by the round. With the cap on, re-review covers only the new diff from round two and every finding is fixed or dismissed with a reason; the check goes green by round three with one human approval still owed, or goes amber at round three and names a person. The per-round cost is a model figure, the midpoint of Anthropic's preview average.
A backlog of findings has a release date
PostgreSQL 19 shows what a reviewer that raises the bar after the code is written does to a schedule, and its committers disagree about how much of it is the reviewer. Peter Eisentraut wrote on 9 October that about 8 to 10 significant features had been reverted. With LLM-assisted review, he writes, edge-case defects “can get found much faster, and then you're staring at a list of like forty defect reports”, and “there is a fixed overhead to dealing with each of these, and then you run out of time”. He also names senior developers tied up in security fixes and a thinner beta review, “probably a combination of these”. Tomas Vondra, another committer, had gone through the twelve reverts on 1 October and counted four as AI or probably AI, the rest from regular human review: “I don't think that narrative matches reality, and I think it matters to correct that.” On 10 October the release team moved general availability from 29 October to 19 November, with a second release candidate on 12 November “given the volume of fixes that have been added to the back branches”; its message does not mention AI. The undisputed part is narrower. Findings arrived late, and the capacity to triage them set the date.
Three things follow. Run the reviewer in the pull request, where the code is being written, rather than at the release freeze. Eisentraut's own answer is LLM-assisted fuzzing during feature development, and he points to an “experimental” status the community has discussed, which would let a feature mature in the main tree instead of being reverted; he says they have not yet worked out how to do it. Count the overhead per finding (triage, reproduction, a decision, a fix and a review) and schedule it against the date with named capacity; a list of forty is a sprint, whoever found it. And feed the misses back: Anthropic built an eval from past pull requests that caused incidents and runs the reviewer against it daily. The revert count, which both Anthropic (as a share of merges) and Cockroach Labs (as a count) chose to report, is the number that rises when a tier is set too low or a cap is lifted.
Takeaways
- Rulesets vary required reviewers by file pattern (teams, organisation repositories only); a required check that reads the diff, picks the tier and counts the human approvals on the head commit adds the people-only count and the commit binding.
- Filter approvals by user type: since 1 September 2026 a Copilot approval can count towards required approvals once an admin enables it.
- A thread closes two ways, fixed or resolved with a reason; re-review covers the diff after the first pass; the third round names a person instead of starting a fourth.
- Review is billed per run with no flat price, and the After every push trigger multiplies that by the pushes; the round cap and the spend limit are one decision.
- A reviewer that finds defects late creates a backlog with a date on it: triage capacity against the date decides whether the feature ships, waits behind a flag or reverts.
Who approves the code your agents write?
We wire AI review into the merge path: a risk tier read from each change, approvals that only people can satisfy, rounds that end by three, and a review budget that sits next to CI.
Tier your merge gate