Same bugs, four reviewers, four different reviews

A while back I said the next thing I'd do was put CodeRabbit and Greptile on the same pull requests as Bugbot, with the same answer key: the bugs I already knew were sitting in my own git history. This is that experiment. It grew a fourth reviewer and went somewhere I didn't expect.

The setup is what makes it worth doing. Most "which AI reviewer is best" comparisons are vibes. You run a few PRs through each tool, read the output, and decide which one felt smarter. I wanted a scored test instead, and I had the material for one. Three of my own pull requests had real bugs I found and fixed before merge. Encrypted database backups, automatic post series, and Neon preview branching. I know exactly what was wrong with each, because I'm the one who fixed it. That's the answer key.

Four reviewers went up against it. CodeRabbit and Greptile, the two I set out to test. The Claude GitHub Action, which I'd just wired into this repo to auto-review every PR, so it went into the lineup too. And Bugbot itself, the reviewer that had flagged several of these bugs the first time around. Bugbot needed on-demand usage switched on before it would run at all, but once it did, it reviewed the same diffs as the rest. The bugs are the answer key. All four tools are what I'm scoring against it, Bugbot included, with no credit for having seen any of this before.

How I rebuilt the bugs

There's a wrinkle. Those three PRs merged weeks ago, and the bugs are long fixed. A reviewer only looks at open pull requests, and it needs to see the broken code, not the corrected version sitting in main today.

So I reconstructed each one. For every PR I took its first commit, the buggy state before the fixes, and opened it as a fresh pull request against a branch pointing at the exact commit the feature originally forked from. The diff each reviewer sees is byte-for-byte the original. I gave the PRs plain feature titles and descriptions with no hint that they were a test or that anything was wrong, so nobody got coached. Four reviewers, three identical known-buggy diffs, blind.

The answer key

Here's what each PR actually contained.

The backups PR had two serious problems. First, it set cacheControlMaxAge: 0 on the Vercel Blob upload:

await put(key, encrypted, {
  access: 'public',
  cacheControlMaxAge: 0,
})

That passes type-checking and works locally against a mock. It fails only against the real Blob API, which rejects the value. You find out in production or not at all. Second, the export was documented as containing "no credential material," but it shipped every user's sessions array straight into the backup, along with lockUntil and loginAttempts. Live session tokens in a file you're storing off-site. There was also a smaller pagination bug where the Blob list only ever read the first page.

The series PR deleted a post's series siblings from revalidation:

hooks: {
  afterDelete: [revalidateSeriesMembersOnDelete],
}

The hook reverse-queries which series a deleted post belonged to. But afterDelete runs after the row is gone, and the join table has ON DELETE CASCADE, so the membership rows it needs to answer that question have already been deleted. The lookup always returns nothing, siblings never revalidate, and their pages keep a dead link to the deleted post. The fix is to capture the siblings in beforeDelete. There was also a migration down() that wasn't idempotent.

The Neon PR had the one I still find the scariest:

const KEEP = Number(process.env.NEON_PREMIGRATION_KEEP || '5')
// ...
const stale = snapshots.slice(KEEP)

If that environment variable is ever set to something non-numeric, a typo in the Vercel dashboard, Number() returns NaN, and slice(NaN) behaves like slice(0). Every backup gets marked stale and deleted, including the one created moments earlier in the same run, right before the migration that backup exists to protect. The build reports success the whole time.

Four headline bugs, then. Two are self-contained logic errors you can reason out from the diff. Two need something more: knowing that the real Blob API rejects a value your types accept, and knowing what actually lives in Payload's user records.

The result: same diffs, four different reviews

I expected a leaderboard. Reviewer A catches four, reviewer B catches three, you rank them. That's not what happened.

The four reviewers agreed on almost nothing. The only findings all of them shared were the two self-contained logic bugs, the slice(NaN) on the Neon PR and the afterDelete cascade on the series PR. Beyond those the overlap was thin: CodeRabbit and Bugbot doubled up on the secondary infrastructure bugs, and nearly every other finding came from a single tool alone.

The clearest illustration is the backups PR, the one with the two serious bugs that need outside knowledge. Here's what the same diff produced from four reviewers:

• Claude flagged an unjustified type assertion, a non-constant-time secret comparison, and a note about missing tests and key rotation.
• Greptile gave it a confidence score of 5 out of 5, "safe to merge," and posted zero findings.
• CodeRabbit flagged one edge case about a single backup exceeding the size limit.
• Bugbot reviewed it and reported no issues at all.

Four reviewers, one diff, and not one of them caught cacheControlMaxAge: 0 or the leaked session tokens. They couldn't even agree on whether the PR was shippable. This is the PR where, during the original development, those bugs were caught. On the reconstruction, on these reviews, everyone missed them.

The logic bugs tell the opposite story. All four caught the Neon slice bug and the series cascade bug, and the cascade one is genuinely hard. It requires knowing that Payload's afterDelete fires after the row is gone and that the foreign key cascades. They all got it anyway, and all four run on GitHub with no editor involved.

So the pattern isn't that one tool is better. It's that the class of bug decides who catches it. Self-contained logic that lives in the diff is now table stakes. The bug that only fails against a real API, and the security bug that needs domain knowledge about your own data model, were a blind spot shared across all four.

One reviewer disagreeing with itself

That would have been the whole story, except for what happened when I ran Bugbot on the backups PR a second time.

Same commit. Nothing changed. This time it reported reviewing "using high effort," and it found the session leak, with the sharpest explanation of that bug any reviewer gave all week: it worked out that Payload marks sessions as admin.disabled rather than hidden, so the default field stripping never removes it and the tokens ride along into the backup. First run, nothing. Second run, same diff, a clean catch. It still didn't flag cacheControlMaxAge either time.

That's the finding that outgrew the leaderboard. The review isn't a stable function of the diff. The same tool, on the same code, gave me two different answers on two runs. Which means the original development that produced this answer key caught those backup bugs on a good run, and could just as easily have missed them. A single pass over-reports how reliable any of these tools are, mine included.

Four personalities

Run the same PRs past all four and you learn that they aren't better or worse versions of the same product. They're built on different bets.

CodeRabbit is breadth and verification. It writes the most, catches the widest range, and it shared with Bugbot the only catches of the secondary infrastructure bugs the other two missed, the fatal prune on the Neon script and the non-idempotent migration rollback. On the cascade bug it went and read Payload's hook documentation and PostgreSQL's DROP TABLE CASCADE docs and cited them to confirm its own finding. The cost is noise: walkthroughs, sequence diagrams, a docstring-coverage warning, offers to generate tests. You wade through a lot of chrome per real finding.

Greptile is signal and reach. Its output is the cleanest of the four, terse findings with severity badges and nothing extra. It reasons about the whole repository rather than the diff, and it showed: the two genuinely new bugs that weren't even in my answer key both came from Greptile. A race between overlapping production deploys deleting each other's backups, and a design gap where a post in more than one series gets an arbitrary banner. Neither is visible in the diff alone. The weakness is the flip side of that single confidence number. On the backups PR it said 5 out of 5, ship it, and moved on.

Claude is depth and restraint. It gave the deepest single explanation of any bug it caught, and it was the only reviewer that also worked out a negative-value edge on the Neon bug and reasoned about why the build still reports success. It knows the repo's conventions and flagged a type assertion against a rule in the project's own CLAUDE.md. But it's the narrowest of the four. Fewest findings, and it missed the secondary infrastructure bugs.

Bugbot is the highest ceiling and the least stable floor. On the series and Neon PRs it matched CodeRabbit for breadth, and it was the only reviewer to catch the session leak, with the best write-up of that bug anyone produced. But it landed that catch on a second run of a diff its first run had passed clean, and it rated the Neon backup-wipe merely medium where the bug is plainly worse. When Bugbot is good it's the best in the group. You just can't count on getting its good run.

How much of this do I actually need

Here's the question the experiment pushed me toward, and it's not the one I started with. A lot of what these tools do, I already do, for free, in GitHub Actions.

Linting is the obvious case. Why would I pay a language model to run ESLint when a required check already runs it in seconds, deterministically, and blocks the merge? The same goes for secret scanning, dependency vulnerabilities, and running the test suite. My repo already gates all of that. Having a reviewer re-run it launders a deterministic result through a probabilistic model, which is slower, costs tokens, and can miss or invent. Even Greptile's headline trick, catching a changed function signature whose callers you forgot to update, is mostly just the TypeScript compiler in a typed codebase.

That gives a clean rule. If a cheap deterministic check can do it, it belongs in a GitHub Action, not a review comment. What's left for the reviewer is the judgment layer that no rule engine can express: whether the implementation matches the intent, whether a change is safe against an external API's real contract, the semantic cross-file reasoning that types can't see, the repo-specific gotcha that isn't worth a lint rule. That residue is real, and it's smaller than the feature lists suggest.

These tools bundle the linting and the scanning because they're standalone products and can't assume your CI exists. That bundling is their benefit, not mine.

Two honest caveats

First, my results contradict the public benchmarks. The independent numbers I've seen put Greptile as the high-recall, high-noise option and CodeRabbit as the precise, quieter one. On my three PRs I saw close to the opposite surface: Greptile was the quietest and missed the backups PR entirely, while CodeRabbit wrote the most and caught the most. That contradiction is the actual finding. No aggregate benchmark predicts your repository, which is the whole reason to run your own diffs with your own answer key.

Second, I mostly tested each tool close to its defaults. CodeRabbit on its free tier and its "chill" profile, Greptile without its sandbox test-running agent visibly firing, Bugbot on a normal run before that second high-effort pass. So the right claim is "on default settings, they didn't catch it," not "they can't." The high-effort Bugbot run is the proof: the session leak was reachable, just not on the standard pass. Greptile has an agent that writes and runs tests in a sandbox, and CodeRabbit has an autofix agent that verifies findings; either could plausibly close more of the gap if it were turned on. I measured out-of-the-box behavior, not each tool at full stretch.

Where the corner actually sits

That was the question underneath all of this. I pay for Cursor partly because Bugbot catches things, and I wanted to know whether that catch quality is special or just table stakes now.

The self-contained logic bugs are table stakes; all four caught them, and on the retry too. The session leak is catchable but flaky; one reviewer of the four got it, and only on a second run at higher effort. And cacheControlMaxAge: 0, the one that only fails against the real Blob API, was missed by every reviewer on every run I did. That corner isn't Bugbot's, and it isn't anyone's. It's empty.

Which means the reviewer I'd actually want isn't any one of these. It's a small one that assumes my CI already handles the deterministic work, spends its budget on the judgment layer, and knows enough about my stack to catch the bug that only fails against the real thing. None of the four is quite that yet. But now I know what I'm looking for, because I finally have an answer key to check them against, and I know better than to trust a single run of it.

So that's the next experiment. I'm leaving CodeRabbit and Greptile running on real pull requests for a while, not reconstructed ones, to see how each behaves past a single scored run: where the noise settles once it's reviewing my day-to-day work, what they surface that I'd have missed, how often that confidence score is wrong. One run on default settings tells you the shape of a tool. Living with it on real work tells you whether you'd keep it. Then I want to take whatever earns its place and fold it into the Claude action I already have running, so the reviewer I end up with is a small one built around my own stack and my own answer key instead of anyone's feature list.