Skip to content
repo-ai

Evidence Lab

This page is rendered from docs/engineering/evidence-matrix.json in the source tree, a condensed, machine-readable ledger of docs/STATUS.md, docs/launch/ACCEPTANCE.json and the study results they cite. When an entry and its source disagree, the source is the record. Results that did not go repo-ai’s way stay here on purpose.

Read every number with its scope. Most agent studies are small, synthetic and written by an AI agent: they are not independent evaluations. How the studies were run is on Evaluation & methodology.

Statuses

  • DEMONSTRATED2Measured on fresh or held-out data under a frozen protocol; the claim holds within the stated scope.
  • EXPERIMENTAL8Observed working live, but on very small samples (often n = 1) or constructed scenarios. A workflow demonstration, not a performance result.
  • NEGATIVE RESULT6A study or qualification found the claim false or its gate missed. Kept visible on purpose.
  • INCONCLUSIVE2A study was run, but its differences are within noise: neither benefit nor harm is shown.
  • NOT YET DEMONSTRATED4No study has shown it. Either none was run, or the studies run could not separate the arms.
  • BLOCKED1Could not be run or qualified because of an external or safety blocker; the blocker is named.
  • UNSUPPORTED2Outside what repo-ai supports by design. Stated so nobody assumes otherwise.

DEMONSTRATEDMeasured on fresh or held-out data under a frozen protocol; the claim holds within the stated scope.

  • 0 / 127

    verify gives no false assurance on fresh held-out data

    0 false assurances in 127 failing states; 0 new failures classified as pre-existing; 0 false alarms; runtime no-miss 96.7%.

    Date
    (Session A qualification on build 903984a, completed before the RC1 freeze of 2026-10-27)
    Sample
    127 failing states across 5 projects
    Method
    Held-out-7: frozen, independently labelled projects (pytest and node --test), run in podman containers on build 903984a.
    Limits
    Verdict CONDITIONAL_WITH_EXPLICIT_LIMITATIONS, for that build and container scope; not re-run on 0.9.0. unittest, TypeScript, Jest/Vitest and host sandboxes are not qualified on held-out data. Not re-run on 0.9.0 or 0.10.0.
    Source for “verify gives no false assurance on fresh held-out data”: docs/release/0.8.0rc1.md

  • 234 / 234

    No working tree was touched in 234 verify runs

    Working tree unchanged in 234 of 234 runs; the integrity check also fails any run that touches it.

    Date
    (same held-out-7 evaluation as above)
    Sample
    234 runs
    Method
    Held-out-7, as above: the source tree is hashed before and after every run.
    Limits
    Containers only on held-out data; host sandboxes (seatbelt, bubblewrap) are covered by the test suite and CI sandbox jobs. Not re-run on 0.9.0 or 0.10.0.
    Source for “No working tree was touched in 234 verify runs”: docs/release/0.8.0rc1.md

EXPERIMENTALObserved working live, but on very small samples (often n = 1) or constructed scenarios. A workflow demonstration, not a performance result.

  • 0.858

    context finds named code from identifiers

    recall@1 0.858 for an exact symbol, 0.883 for a renamed symbol, 0.551 for a symbol named in an issue.

    Date
    (Session A retrieval holdout, October 2026)
    Sample
    18 repositories
    Method
    Session A's retrieval holdout with the shipped ranker (M1).
    Limits
    Label is stable candidate: no fresh preregistered validation of the whole command. Prose queries are weak (next entry). Labelled a stable candidate in STATUS.md: not yet validated on a fresh, preregistered set.
    Source for “context finds named code from identifiers”: docs/STATUS.md

  • n = 1

    Live coding-agent candidates, independently re-verified

    Codex CLI 0.165.0 produced a PASS LIVE_CANDIDATE_VERIFIED change through repo-ai do, with coverage, apply (post-apply tests 4/4) and undo (bytes restored); agy 1.4.9 took part in the tournament and handoff journeys.

    Date
    Sample
    1 workflow per journey per build (builds dfb6880, c505f74 and the freeze 9663ac3)
    Method
    Installed wheel in a clean venv on macOS seatbelt; synthetic projects written for the check; providers only through repo-ai do --run --allow-live-provider.
    Limits
    Synthetic tasks; the same author wrote task, tests and hidden oracle; costs UNKNOWN (subscriptions). IBM Bob produced live re-verified changes earlier (0.9.0.dev0 cycle) and was not re-run on 0.9.0.
    Source for “Live coding-agent candidates, independently re-verified”: docs/launch/evidence/LIVE-JOURNEYS-2026-10-29.md

  • n = 1

    Hand unfinished work to a different agent

    Codex → agy: Codex left validated partial progress, the capsule verified VALID, agy kept Codex's diff byte-identical and reached PASS; independent hidden oracle 714/714. An earlier agy → Codex transfer also reached PASS.

    Date
    (Codex → agy on 2026-10-29 (builds c505f74 and 9663ac3); agy → Codex earlier in October 2026)
    Sample
    1 constructed scenario per build
    Method
    Constructed scenario (stage A scoped to one file to force partial progress); live providers through repo-ai's mission path.
    Limits
    Constructed, n = 1; a handoff beating a restart is not shown. The earlier agy → Codex transfer is recorded in docs/GUIDE.md, not in the cited journey file.
    Source for “Hand unfinished work to a different agent”: docs/launch/evidence/LIVE-JOURNEYS-2026-10-29.md

  • 4 / 4

    Tournaments abstain instead of picking a winner

    In abc-v3 both providers passed with different diffs on every task, so the no-tie-breaker rule abstained 4 of 4; on 2026-10-29 one live comparison abstained (distinct passing diffs) and one byte-identical pair was resolved by the equivalence rule.

    Date
    (abc-v3 preregistered in the 0.9.0.dev2 cycle; launch journeys on 2026-10-29)
    Sample
    4 tasks (abc-v3) + 2 live comparisons
    Method
    Codex + agy tournaments through repo-ai do --tournament; preregistered abc-v3 manifest.
    Limits
    A behaviourally different live tie was not exercised; that a tournament beats a single agent is not shown.
    Source for “Tournaments abstain instead of picking a winner”: benchmarks/abc-v3/RESULTS.md

  • PASS, oracle 17/17

    Claude Code as an isolated execution provider

    On the published 0.10.0 wheel, Claude Code 2.1.314 produced a model-generated fix inside repo-ai's isolation; repo-ai re-ran the frozen tests (PASS LIVE_CANDIDATE_VERIFIED), an author-written hidden oracle passed 17/17, apply and undo were byte-exact; a hostile repository's injected instructions were not followed.

    Date
    Sample
    n = 1 per journey (2 on 0.10.0)
    Method
    pip install repo-ai-cli==0.10.0 into a clean venv; repo-ai do --agent claude --run --allow-live-provider; Claude's own login (dedicated repo-ai login dir), file tools only, seatbelt, per-run isolation probe.
    Limits
    n = 1 per journey, synthetic tasks, single author. The model refused boundary probes, so the OS boundary rests on direct sandbox probes and two independent reviews; the first review found the read rules could expose HOME (fixed before 0.10.0; earlier journeys ran on development builds). Token mode and Linux are not live-verified.
    Source for “Claude Code as an isolated execution provider”: docs/engineering/evidence/JOURNEYS-0.10.0-PYPI.md

  • 1 of 4 PASS

    An account-free local model as a repo-ai provider

    OpenCode 2.2.1 with qwen3:8b (Ollama, loopback only, $0): on the published 0.10.0 wheel one attempt was refused (broken edit) and one passed the visible tests while failing a hidden case (it dropped a docstring requirement no test pinned; repo-ai labelled the claim ASSURANCE_PARTIAL); on development builds 1 of 4 passed (hidden oracle 9/9). repo-ai refused every broken candidate.

    Date
    Sample
    7 attempts across builds
    Method
    Installed wheel; repo-ai do --agent opencode; deny-by-default tools, loopback-only network.
    Limits
    One small model, synthetic tasks, single author. Shows the account-free path works and that broken candidates are refused, not that local models are good at this; a mission PASS is not correctness.
    Source for “An account-free local model as a repo-ai provider”: docs/engineering/evidence/JOURNEYS-0.10.0-PYPI.md

  • 2 hosts

    A repo-ai-installed skill loads on two real hosts

    A skill vetted and installed by repo-ai was listed and invoked by Claude Code (paired run: the convention it states appeared in 2 of 2 tests with the skill, 0 of 2 without) and loaded by OpenCode from .agents/skills on request; uninstall restored every file byte-for-byte.

    Date
    Sample
    n = 1 per arm
    Method
    repo-ai skills discover --from-path / install --approve / disable; host-mode runs.
    Limits
    Load and invocation only: whether a skill improves outcomes is not shown (no blinded trial). The 2/2 vs 0/2 result is one haiku pair; a sonnet pair stopped at the cost cap before writing. Host mode exposed the account's synced MCP connectors (a shell tool among them) in both arms. OpenCode's model never chose the skill unprompted.
    Source for “A repo-ai-installed skill loads on two real hosts”: docs/engineering/evidence/SKILL-HOSTS-2026-10-30.md

  • PASS_WITH_PRE_EXISTING_FAILURES

    A fix verified on a real repository with unrelated failures

    On un33k/python-slugify (a deliberate regression committed as a bad commit), a live Codex candidate was verified through a scoped mission (--expect-fix with the three tests the regression broke): the declared tests went from failing to passing, nothing that passed at the base failed, 20 unrelated failures were listed; the candidate is byte-identical to the upstream line. An unscoped mission on an equivalent candidate (Claude's recorded fix, replayed) could not be verified because of the 20 unrelated failures.

    Date
    Sample
    1 real repository; 1 successful live run (after 3 failed attempts on 0.11.0)
    Method
    repo-ai do --agent codex --expect-fix … on a development build of the init-then-do fix (0.11.0 itself refused the run because of that first-run bug).
    Limits
    One repository, one task, regression injected by the author; the claim stays ASSURANCE_PARTIAL. The passing run used a development build of the init-then-do fix (PR #11, released in 0.11.1): 0.11.0 itself refused the run. The Claude provider attempt was blocked by a revoked login. The post-apply run in the real checkout reported 0 failures where the sandbox saw 20 (unexplained).
    Source for “A fix verified on a real repository with unrelated failures”: docs/engineering/evidence/REAL-PROJECT-2026-10-30.md

NEGATIVE RESULTA study or qualification found the claim false or its gate missed. Kept visible on purpose.

  • 0.320

    context on plain-prose and error-message queries

    recall@5 0.320 on natural-language and error-message queries: prose queries are weak.

    Date
    (same retrieval holdout)
    Sample
    18 repositories
    Method
    Session A's retrieval holdout, natural-language and error-message query classes.
    Limits
    BM25 hybrid (PR #57 in the development history) was tried and not adopted: it broke identity-first ranking.
    Source for “context on plain-prose and error-message queries”: docs/STATUS.md

  • 1 of 4

    repo-ai reported PASS for a false green

    On node-slug, repo-ai reported PASS / LIVE_CANDIDATE_VERIFIED for a candidate that failed a hidden requirement the visible tests never stated.

    Date
    (abc-v1, 0.9.0.dev0 cycle)
    Sample
    4 tasks, IBM Bob 2.0.9, one run per cell
    Method
    Preregistered abc-v1 with a hidden grader run in a separate copy.
    Limits
    This is the documented meaning of PASS (the visible checks passed), and this run is its counterexample.
    Source for “repo-ai reported PASS for a false green”: benchmarks/abc-v1/RESULTS.md

  • 1.44–2.31×

    verify is faster than running the native suite

    On held-out-7, verify --run was slower than the native suite on 3 of 5 projects (1.44×, 1.70× and 2.31× the native time) and faster only on the two larger ones.

    Date
    (held-out-7, before the RC1 freeze of 2026-10-27)
    Sample
    5 projects
    Method
    Wall-clock comparison inside the held-out-7 evaluation (podman containers).
    Limits
    Earlier, held-out-5 showed slower runs on 4 of 7 short suites (1.22–2.52×). On small suites the value is the classification, not time.
    Source for “verify is faster than running the native suite”: docs/release/0.8.0rc1.md

  • 0.385 vs 0.80

    Capability recommendation meets its recall gate

    FAIL: top-3 recall 0.385 against a gate of 0.80; shortlist recall@6 0.736; top-3 precision 0.712; 5 constraint-forbidden recommendations reached the top 3. Still above the lexical (0.088) and repository-only (0.286) baselines.

    Date
    Sample
    120-task fresh corpus (Corpus C)
    Method
    Preregistered protocol frozen beforehand, measured once; tuning of the deterministic matcher stopped after three unsuccessful passes.
    Limits
    The raw results file sits in the private development history; the public record is docs/CAPABILITIES.md and docs/STATUS.md.
    Source for “Capability recommendation meets its recall gate”: docs/CAPABILITIES.md

  • 12 of 42

    Plugin inspection passes its fresh validation

    FAILED: 12 of 42 benign cases were blocked; executable-resolution coverage 0.25 (0.43 after a post-hoc corrective pass, still a fail). Nothing critical or unresolvable was reported eligible.

    Date
    Sample
    44 cases (42 benign), analyser frozen
    Method
    Preregistered fresh validation of skills vet / inventory on plugin components.
    Limits
    Attack detection was not freshly validated (the synthetic attacks could not be produced). Since then a plugin is never reported eligible.
    Source for “Plugin inspection passes its fresh validation”: docs/SKILL_NETWORK.md

  • 3 classes

    ProofLab (prove) qualification

    NOT QUALIFIED: the release qualification found false proofs inside the narrowed scope (swallowed-assertion, importorskip-zero-entry-points, always-equal-class). The sealed certification is blocked.

    Date
    (Session A release qualification, RC1 cycle)
    Sample
    3 open false-proof classes, kept as strict expected failures
    Method
    Session A release qualification of the narrowed ProofLab scope.
    Limits
    prove stays experimental, opt-in and excluded from the supported capabilities; no hook or gate calls it.
    Source for “ProofLab (prove) qualification”: docs/STATUS.md

INCONCLUSIVEA study was run, but its differences are within noise: neither benefit nor harm is shown.

  • 92 / 90 / 88 %

    Claude Code succeeds more often with repo-ai context

    Plain Claude 92%, repo-ai Core 90%, repo-ai Full 88%; Holm-adjusted p = 1.0 for every comparison; equivalence not shown either. Full cost about 10% more.

    Date
    Sample
    150 preregistered runs (145 valid), 25 real-repository tasks
    Method
    Option A: preregistered, run blind; 25 tasks × 3 arms × 2 runs, Claude Sonnet.
    Limits
    Intervals about ±15 points: a gain or loss of 10–15 points is not ruled out. One clear harm with a known cause (guidance steered agents to lint instead of testing on one task).
    Source for “Claude Code succeeds more often with repo-ai context”: benchmarks/results/coding-option-a-v0.7.5.md

  • 92 / 84 %

    Core vs plain Claude on sealed-v2

    Plain solved 23 of 25 (92%), Core 21 of 25 (84%); −8 points, 95% CI −22 … +7, p = 0.5.

    Date
    Sample
    25 tasks, one run per arm
    Method
    Exploratory, as registered; sealed-v2 is no longer untouched.
    Limits
    Only 2 discordant tasks: the comparison carried almost no information.
    Source for “Core vs plain Claude on sealed-v2”: benchmarks/results/coding-sealed-v2-pairs-v0.7.7.md

NOT YET DEMONSTRATEDNo study has shown it. Either none was run, or the studies run could not separate the arms.

  • 14 tasks

    Lift over a standalone coding agent

    In abc-v2 to v4 the bare provider already solved every task, so nothing could separate the arms; in abc-v1 the one-task difference of four is consistent with noise.

    Date
    (abc-v1 to abc-v4 ran in the 0.9.0.dev0 to dev4 cycles)
    Sample
    4 studies, 14 tasks (4 + 4 + 4 + 2), one run per cell
    Method
    Preregistered manifests frozen before any live call; hidden graders outside the agent workspace; free controls (untouched FAIL, reference CORRECT, decoy FALSE_GREEN) per task.
    Limits
    Tasks, hidden tests and harness written by an AI agent; small synthetic tasks. Measuring lift needs tasks the bare provider fails at a non-trivial rate, at least 20 paired tasks and repeats.
    Source for “Lift over a standalone coding agent”: docs/launch/EVALUATION.md

  • 0 of 4

    repo-ai catches false greens better than “tests pass”

    Calibration only: repo-ai (arm B) accepted 5 of 9 semantic decoys and 0 of 10 test-gaming controls, against 9 and 10 for “tests pass”; task-level discrimination 0 of 4. The confirmatory phase was not run.

    Date
    Sample
    4 synthetic tasks, 29 candidates, no model calls
    Method
    Preregistered abc-v5 with a sealed grader, immutable manifests and exact power analysis; calibration run only.
    Limits
    One author wrote tasks, decoys and oracle. The thesis was narrowed to “names the requirements your tests do not pin, and resists test gaming” (DECISIONS L-007). Also: B declined 2 of 4 correct candidates; its semantic catches were abstentions it also gave the correct reference; a careful-developer baseline accepted only 2 of 10 test-gaming controls.
    Source for “repo-ai catches false greens better than “tests pass””: docs/launch/EVALUATION.md

  • The opt-in verify completion gate changes agent outcomes

    Its own tests pass; its effect on agent outcomes has never been measured.

    Date
    Sample
    none
    Method
    No study run.
    Limits
    Earlier recorded runs showed agents rarely call optional tools (0 skill invocations in 212 runs), which is why the gate exists.
    Source for “The opt-in verify completion gate changes agent outcomes”: docs/STATUS.md

  • Windows, on fresh data

    Unit tests pass in CI (Python 3.10 and 3.13); no held-out or fresh-data run on Windows.

    Date
    Sample
    none on fresh data
    Method
    CI unit tests only.
    Limits
    Windows has no host sandbox: verify --run needs --sandbox podman or docker.
    Source for “Windows, on fresh data”: docs/STATUS.md

BLOCKEDCould not be run or qualified because of an external or safety blocker; the blocker is named.

  • exit 41

    Gemini CLI as an isolated execution provider

    A bounded retry through repo-ai do failed authentication (Gemini CLI exit 41) before any edit, for the tested account. From 0.10.0 repo-ai no longer copies ~/.gemini into the sandbox; live Gemini needs GEMINI_API_KEY or GOOGLE_API_KEY.

    Date
    Sample
    1 bounded retry
    Method
    Installed wheel; repo-ai do --agent gemini --run --allow-live-provider (Gemini CLI 0.65.0, macOS).
    Limits
    Blocked by the tested account's authentication, not by a repo-ai verdict; not tried with an API key.
    Source for “Gemini CLI as an isolated execution provider”: docs/engineering/INTEGRATIONS.md

UNSUPPORTEDOutside what repo-ai supports by design. Stated so nobody assumes otherwise.