Code review tooling: OCR delegation mode vs. the production reviewer

Status: 4-PR sample complete (one per bucket, both arms — 8 real Claude Code invocations, ~35 minutes of aggregate wall-clock across them, run mostly in parallel). The full ~10-PR sweep (6 more PRs) has not been run — see "Next step." This sample is directional, not a verdict: n=1 per bucket, one model draw each. Treat every number below as a data point to weigh, not a percentage to trust in isolation.

Why this evaluation exists

.github/workflows/claude-code-review.yml runs anthropics/claude-code-action@v1 with the upstream code-review@claude-code-plugins marketplace plugin on every PR. It has a documented quality problem, recorded in the workflow's own comments: PR #786 ran 20 review rounds over four hours, several of them fixing regressions the previous round introduced, because the plugin re-derives findings from the whole diff on every push and always finds another nit. Commit 41a7de9 (#797) contained the blast radius with MAX_REVIEW_ROUNDS: '3' and a round-2+ prompt telling it to only report merge-blockers — a cap on the symptom, not a fix.

Alibaba's open-code-review (OCR) targets that exact failure mode: it splits review into a deterministic pipeline (which files, which rules, which line) and an LLM (what's wrong), and its own benchmark (50 repos, 200 PRs, 10 languages, 1,505 annotated issues, baselined against Claude Code as a general-purpose reviewer) claims higher precision and F1 at ~1/9 the tokens, with deliberately lower recall — precision over noise. That's the trade this repo wants. The open question is whether it holds up on this codebase: OCR's headline ruleset (NPE / thread-safety / XSS / SQL injection) reads Java-first, with no documented claim that TypeScript/Vue are first-class.

Method

Integration path: delegation mode only. ocr delegate preview + ocr delegate rule resolve OCR's scope and ruleset locally with no OCR-side LLM key — confirmed directly: ocr delegate preview runs cleanly against this repo with no credentials configured, and the CLI's own --help states "no LLM required" for the delegate subcommand. Claude Code (the same CLAUDE_CODE_OAUTH_TOKEN-backed session used everywhere else in this repo) then reasons over that resolved scope/ruleset and writes findings. This isolates OCR's pipeline from a model change — both arms run on the same model — at the cost of not exercising the native alibaba/open-code-review GitHub Action's incremental (IoU dedup against prior bot comments) and route_severity_below (push nits to the summary instead of inline) features, which are plausibly the more direct fix for the #786 ping-pong. That action needs its own ANTHROPIC_API_KEY (metered spend beyond the subscription); if OCR's finding quality wins here but the noise numbers are close, that becomes the real follow-up question.

Replay, not shadow. Both arms are run offline against ~10 already-merged PRs at their original headSha, in a scratch git worktree, entirely read-only against GitHub (neither arm is given a posting tool). This is controlled and repeatable, and ground truth is available for some PRs — a later commit fixed a defect the PR shipped.

Harness: scripts/review-eval/ (run.mjs, score.mjs, score.test.mjs, prs.json) — see scripts/review-eval/README.md for how to run it.

Golden set

10 PRs across four buckets (down from a planned 12 — see prs.json's _meta for why the later-fixed-bug bucket has 1 entry instead of 3: this repo's shallow-clone-obscured history and a repo-wide Prettier reformat made reliable single-PR blame attribution too expensive to chase for the other two candidates within budget):

#786 is deliberately included in risky-backend as the sharpest available noise probe — it's the PR that generated 20 review rounds in production.

Headline metrics (definitions — see score.mjs)

• Position validity — fraction of findings whose file:line lands on a line the diff actually added. Tests OCR's headline claim about line-number accuracy directly.
• Duplicate rate — findings within 2 lines of each other on the same file, within one run.
• Recall — fraction of a PR's hand-annotated knownDefects that some finding matches (same file, line within ±3).
• Severity mix — blocking / should-fix / nit / other, self-reported by the arm.
• Noise rate (the headline metric, not raw finding count) — nits + false positives per PR, especially on the low-risk bucket. The problem being solved is noise-driven review ping-pong; a reviewer that finds more things is not automatically the better one.
• Cost/latency — tokens and durationMs per finding-set, from the Claude Code session's own usage reporting.
• Blind severity/false-positive judge — not yet automated (see scripts/review-eval/README.md); planned as a manual pass over the full sweep's output, shuffled and stripped of arm identity, labeled with the 🔴/🟡/🔵 vocabulary from .claude/commands/code-review.md.

Sample results — 1 PR per bucket, both arms (reports/sample-1/)

Raw findings: scripts/review-eval/reports/sample-1/*.json. Scored summary: scripts/review-eval/reports/sample-1/summary.json (regenerate with node scripts/review-eval/score.mjs --run scripts/review-eval/reports/sample-1 --out scripts/review-eval/reports/sample-1/summary.json — both are gitignored, so pull them from the run that produced this table if you need to re-inspect).

#413 — both arms found real, different issues; neither caught the seeded one

Both arms produced substantive, position-accurate findings (100% of both arms' findings landed on real added diff lines) — this is a 44-file, 3,853-insertion PR, so "nothing to say" was never a live outcome here.

• Control (3 findings, all should-fix): an unescaped-comma bug in a PostgREST .or() filter that silently breaks the new web-intelligence feature for any topic prompt containing a comma; dead first-crawl-detection code (a count query result that's fetched and never used, so the "only on first crawl" comment is false); a dropped "generate image" suggested-action regression from the createPosts.ts → pipeline refactor, confirmed via a repo-wide grep for the removed field. All three read as real, non-speculative, correctly attributed to file:line.
• OCR (7 findings — 4 should-fix, 3 nit): two genuine XSS findings (unescaped customer.name/customer.url interpolated into blog and social preview HTML, while a sibling field on an adjacent line is correctly escaped — a real, specific, plausible security finding, the category OCR's ruleset is built around); two sequential-await-in-loop performance findings where independent per-page work could be Promise.all'd; two unused-import nits; one RLS-completeness nit on a new migration (SELECT-only policy, correctly flagged as "worth confirming intentional" rather than asserted as a bug).
• Recall: 0/1, both arms. Neither caught the seeded defect (Azure rejecting minimum/maximum on the critic's structured-output schema). That is a genuinely obscure, provider-specific bug — not a fair test of general review competence, and consistent with OCR's own claimed precision-over-recall trade-off extending to Claude-driven review in general, not something specific to OCR's ruleset.

This is the strongest evidence point so far, and it favors OCR's findings being no worse, and arguably higher-value (the XSS pair is the kind of finding a security-focused ruleset should surface) — at comparable cost (21K vs 24K output tokens) and speed.

#786 and #785 — both arms found nothing, and that's more informative than it sounds

Zero findings from every arm on both PRs was suspicious enough to stop and verify it wasn't a harness bug — #786 in particular is the PR that generated 20 review rounds in production. A full raw-output diagnostic (bypassing the harness's findings-file redirection) on #786's control arm confirmed this is a genuine, non-error, 56-turn, $2.37 review: it tried multiple ways to run pnpm exec tsc/find/grep across node_modules to verify the PR's claims against the actual LangChain library source, hit permission_denials each time (this harness's --allowedTools only grants Bash(git diff|show|log:*), not generic Bash, so it fell back to manual code reading), and concluded "No bugs or CLAUDE.md violations found" after verifying the PR's claims by hand.

Two real caveats, not excuses:

1. The --allowedTools restriction likely cost something. The plugin visibly wanted to type-check and grep library internals and couldn't. Whether the production workflow's own tool grant (mcp__github_inline_comment__create_inline_comment,Bash(gh pr comment/diff/view:*)) is more or less permissive for this kind of verification is unclear without checking what the plugin's own bundled tool policy adds on top — worth checking before trusting this arm's noise numbers as representative.
2. Single-shot replay of the final diff is a different test than what generated the 20-round problem. The production pain was repeated review of an evolving PR across many pushes — each round re-deriving findings from the whole diff, including the previous round's own fixes. Replaying the final merged diff once, as this harness does, doesn't reproduce that dynamic. Zero findings here doesn't mean "the ping-pong wasn't real"; it means this methodology answers a different, narrower question ("is the final shipped diff clean?") than the one that actually hurt in production ("does re-reviewing an evolving PR find diminishing, repetitive, or self-contradicting nits?"). A true replay of the ping-pong would need to feed the reviewer each of #786's real intermediate commits in sequence — not attempted here.

#799 — the noise-floor probe: OCR is chattier, but not wrong

The one clean bucket-vs-bucket contrast in this sample. #799 is a test-only mutation-coverage PR (no production code touched) — exactly the shape of PR a good reviewer should mostly leave alone.

• Control: 0 findings. Consistent with #786/#785 — nothing flagged.
• OCR: 3 findings (1 should-fix, 2 nit), 100% position-valid, 0% duplicates: the same buildChain Supabase-mock helper copy-pasted verbatim into a second test file instead of shared; the same mock.results.find(...) three-line lookup repeated three times in one file; an as any cast added without the ruleset's required explanatory comment. All three are real — they point at duplicated code that genuinely exists at the cited lines — not hallucinated findings, just about test-code hygiene rather than production risk. One finding (the as any nit) even self-qualifies with "the surrounding file already uses this same idiom elsewhere... which should factor into how strictly this is enforced" — a more calibrated nit than a bare rule violation.

This is the sample's clearest evidence that OCR's rule-driven pipeline produces more findings than the control even on low-stakes PRs — worth weighing against the ping-pong problem this evaluation exists to solve. On this evidence the extra findings are grounded rather than noise in the "made-up" sense, but volume itself has a cost if it's what re-triggers review rounds; a full sweep result and the deferred blind-judge pass (see below) would sharpen whether "grounded but more numerous" nets out as an improvement or a milder version of the same problem.

Next step

The 4-bucket, 8-invocation sample is complete. Six PRs remain (#778, #777, #776, #742, #784, #780) to fill out risky-backend and frontend-vue to 3 PRs each and low-risk to 3. Run them with node scripts/review-eval/run.mjs --run-id <id> — sequentially, not in parallel (see the concurrency note in run.mjs's header comment; racing git worktree add/remove across processes corrupted a worktree's registration during this sample run). Budget ~3-6 minutes and non-trivial token spend per invocation, so ~12 more invocations is a real commitment, not something to run silently.

Once complete, add to this report:

1. The full per-bucket score table and the position-validity/duplicate/ recall numbers for the remaining buckets.
2. A blind severity/false-positive judge pass across the full finding set (shuffle arm identity, label 🔴/🟡/🔵 per .claude/commands/code-review.md's vocabulary) — not yet automated or run even informally beyond the by-hand reads above.
3. Whether the #786/#785 zero-findings pattern and #799's "OCR finds more, control finds nothing" pattern hold across the rest of the sample, or were specific to those three PRs.
4. A recommendation: keep the current plugin, switch to OCR delegation mode, or evaluate the native OCR GitHub Action (its own ANTHROPIC_API_KEY cost, incremental dedup, and route_severity_below routing) as the more direct fix for the ping-pong specifically.

Preliminary read, pending the fuller sample: two consistent signals so far. On the one PR with a substantial, unambiguous diff (#413), OCR's findings were at least as good as the control's — different bugs, one clearly higher-value (the XSS pair) — at similar cost and speed. On the PRs where the control found nothing at all (#786, #785, #799), OCR still found nothing on #786/#785 but produced real, grounded (if minor) findings on #799 — a more thorough reviewer on this evidence, not a noisier one in the hallucination sense, though volume alone could still feed the ping-pong dynamic this evaluation exists to solve. Nothing here yet contradicts the thesis that OCR is a viable, plausibly better-targeted alternative; nothing here yet confirms it either. The #786/#785 zero-findings result specifically argues for testing the iterative re-review scenario — not just a single-shot replay of the final diff — before drawing any conclusion about the ping-pong problem itself.

This task deliberately stops short of switching claude-code-review.yml — that's a follow-up gated on completing the sweep above, not this PR.