Does codeweb work? We pre-registered the answer.
Before collecting any data, we wrote down 33 pass/fail checks across determinism, correctness, detection accuracy, performance, and agent outcomes — then ran them against six real repositories pinned by SHA and independent reference implementations. The raw data is committed in the repo.
Designed to resist reward hacking
Pre-registered
The 33 checks and their pass criteria were fixed in writing before data collection — so the bar couldn't move to fit the result.
Independent oracles
Cycles, impact, and callers are checked against separate implementations (Kosaraju SCC, reverse-BFS, an edit applier) — not against codeweb's own logic.
Frozen truth
The efficiency pilot grades agents against a hand-verified caller set, reconciled with an exhaustive independent grep. Coverage gaps count against codeweb, honestly.
A/B levers
Every bug fix sits behind a reproducible legacy lever, so re-running proves the fix is load-bearing rather than coincidental.
Every claim, with its number, sample, and source
Tagged honestly: Validated stands on pre-registered checks against oracles; Preliminary is a fresh pilot signal; Null is a result that did not move — published anyway.
| Claim | Result | Sample | Evidence | Status |
|---|---|---|---|---|
| Dependents discovery vs grep (compiler-graded) | recall 1.00 · precision 0.94 (grep 0.87) · 1/3 grep's context cost · 30 sampled vite symbols | TS LanguageService oracle, seed 42 | oracle-ab.json | Validated |
| Blast-radius context cost vs recursive grep | ~1KB/call vs ~130KB (126x), grep loop simulated generously (graph-assisted frontier) | 30 vite symbols, mean impact 18 symbols | oracle-ab.json | Validated |
| Byte-deterministic pipeline | 1 distinct digest per repo across 20 runs, all 6 repos | R=20 × 6 repos | determinism.json | Validated |
| Incremental refresh ≡ full rebuild | 0 mismatches | 360 seeded edits | determinism.json | Validated |
| Cycles, impact, callers, callees, context match independent oracles | 0 disagreements | ~490k comparisons | correctness-query.json | Validated |
| Edit-safety pre-flight predicts the real gate verdict | 0 violations | 10,000 ops + 120 CLI trials | edit-safety.json | Validated |
| Exact-clone detection beats name-matching | F1 1.0 (CI 0.9977–1.0) vs 0.67 | synthetic K∈{20,50,100} | detection-accuracy.json | Validated |
| Renamed (Type-2) clone recall | 1.0 structural vs 0.0 lexical | 30 planted pairs | detection-accuracy.json | Validated |
| Reuse ranking (find-similar) | MRR 0.9875 vs 0.107 random | 400 queries | detection-accuracy.json | Validated |
| Dead-code safe-tier precision (after fix) | 1.0 precision & recall (was 0.52) | 120 synthetic graphs | detection-accuracy.json | Validated |
| Sub-quadratic scaling | exponent b=0.33, CI [0.13, 0.53] | 100–3,201 symbols | performance.json | Validated |
| Zero runtime dependencies | full pipeline runs on empty node_modules | non-vacuity control passes | performance.json | Validated |
| Multi-language extraction parity | 11 native languages (JS/TS/Py/Rust/Go receipts; Java/C#/Ruby/PHP/Kotlin/Swift pinned by per-language suites) | per-language fixtures (JS/TS/Py/Rust/Go receipts; the rest pinned by tests/lang-*.test.mjs) | auxiliary.json | Validated |
| MCP tool / CLI parity | 27 / 27 tools | JSON-RPC conformance (suite: tests/mcp.test.mjs) | auxiliary.json | Validated |
| Agent caller-discovery recall on budgeted responses (vs grep) — the current run | +0.310 ± 0.039 recall (all 5 reps positive), precision +0.234 ± 0.080 | 5 engine-frozen reps, v0.9.0, budgeted treatment (MCP-parity) | efficiency-pilot.reps5-v090.json | Preliminary |
| Agent runtime tokens / tool-calls on budgeted responses (vs grep) — the current run | a wash: totalTokens −84k ± 381k, toolCalls +0.95 ± 3.6 — equal cost, reported plainly | 5 reps, 39/39 agents joined from transcripts | efficiency-pilot.usage-v090.json | Null (honest) |
| Agent caller-discovery recall (vs grep) — earlier run, different base model | +0.265 ± 0.045 (all 8 reps positive, ~6× the noise floor) | 8 engine-frozen reps, claude-opus-4-8, unbudgeted responses | efficiency-pilot.reps8.json | Preliminary |
| Agent discovery steps (vs grep) — earlier run; savings did NOT replicate on v0.9.0 | −6.84 ± 3.33 steps (~34% fewer) then; +0.9 ± 3.7 (wash) on the current run | 8 engine-frozen reps, claude-opus-4-8 | efficiency-pilot.reps8.json | Preliminary |
| Agent edit-quality on clean, well-scoped tasks | paired difference = 0 (floor effect, not a power result) | 8 paired tasks (H18) | agent-ab.json | Null (honest) |
| Blind replay of a real historical caller-breakage (ambient loop A/B) | both arms 8/8 caller-file coverage, 0 structural regressions — ceiling on the one task that survived validity guards | 1 mined axios task (allowAbsoluteUrls) x 2 arms x 4 reps, blind protocol, fixed-function grading | replay-ab.json | Null (honest) |
Sources are files under bench/results/ and bench/experiments/ in the repository. Browse the raw data →
Frontier agents find more, at equal cost
On a caller-discovery micro-benchmark against a grep baseline — engine frozen, treatment consuming the budgeted responses agents actually receive over MCP — codeweb improved a frontier agent's recall by +0.310 ± 0.039 (every one of the 5 reps positive) and precision by +0.234 ± 0.080, at equal token cost (v0.9.0).
Runtime tokens and tool-calls were a wash — reported plainly, not rounded up. The mechanism is deterministic: fixing import-alias and member-access resolution took axios AxiosError's recovered callers from 1 to 20 — no model involved. We label the pilot preliminary because grading is strict at the symbol level. One target regressed (flask, a Python import-edge gap) — since bisected to a real extractor regression and fixed (Spec Q, closed): the same pre-flight now finds every truth site.
Why it's believable
The paired design cancels the shared oracle noise that swung the grep baseline run-to-run, the recall gain clears the measurement noise by ~8×, and the null half of the result (no token savings) is published beside the win.
Source: bench/experiments/efficiency-pilot.reps5-v090.json · efficiency-pilot.usage-v090.json
The earlier run — and what didn't replicate
An earlier 8-rep run (different base model, unbudgeted responses) found the same recall advantage (+0.265 ± 0.045, all reps positive) plus large savings: ~34% fewer discovery steps and ~44% fewer tokens. On the current, more frugal base agent those savings did not replicate — the advantage moved from spend to completeness. We lead with the current number and keep this one as labeled history, because that is what the receipts say.
Source: bench/experiments/efficiency-pilot.reps8.json · efficiency-pilot.usage.json
The honest null: agent edit quality (H18)
We also tested whether agents produce better edits with codeweb on 8 paired tasks. The result was a clean zero — a bootstrap CI of [0, 0]. On clean, well-scoped tasks, a capable base model already avoids regressions, so there was no headroom to improve: a floor effect, not a power result. The thesis does not rest on this check, and we report it rather than bury it.
Source: bench/results/agent-ab.json
The honest null, part two: the blind replay A/B (2026)
We mined git history for changes that provably broke callers (the repo itself needed follow-up fixes), then had agents replay them blind — no answer key, no history, coverage graded by a fixed function. On the single task that survived our own validity guards, both arms hit the ceiling: with or without codeweb, agents fixed every caller the original developer missed. The run validates the instrument and bounds the ambient effect near zero on small, well-scoped changes; discriminating between arms needs bigger, messier tasks. Published anyway.
Source: bench/results/replay-ab.json · protocol: docs/specs/replay-run.md
It found bugs its own test suite missed
The study is credible partly because it failed first. It surfaced a determinism leak (file-enumeration order bleeding into domain assignment, plus an overlap stack-overflow on express) and a dead-code precision failure (test-file functions wrongly marked safe-to-delete, 0.52 precision). Both were fixed surgically behind A/B levers; re-runs establish the fixes. The engine's test suite (286 tests at study time) had been green throughout.
What we deliberately don't claim
The fastest way to lose trust is to overstate. Here's where we hold back, and the precise claim we make instead.
- "Finds all clones / 100% recall" → High-precision, body-confirmed overlap on the clones it surfaces (F1 1.0 synthetic, 0.98 on real axios) — it deliberately drops ambiguous edges because a false positive costs more than a miss.
- "Replaces human review" → It produces candidate lists and shows which merges are cycle-safe to apply — a structural pre-flight, not a semantic verdict.
- "Agents provably edit better" → It improves caller-discovery recall (+0.31, all reps positive) at equal token cost on the budgeted responses agents receive (v0.9.0). An earlier run on a different base model also showed ~34% fewer steps; that savings did not replicate, and we say so. The full edit-quality A/B (H18) was a null on clean tasks — a floor effect, reported honestly.
- "100% deterministic" → Byte-deterministic, set-valued output — one digest across 20 runs per repo for nodes, edges, domains, and overlaps.
- "Sub-second every time" → Typical query ~95–120 ms; worst-case p95 264 ms on 3,201 symbols, where Node startup (~63 ms) dominates.
- "Beats commercial tools" → We compare only to stated, reproducible baselines (name-matching, lexical, random, the legacy fallback) — no head-to-head against tools we can't reproduce.