Local-first CLI · macOS, Linux, Windows · Python 3.10–3.14

Context before the change.
Evidence after it.

repo-ai indexes your repository offline, runs the tests your change can break in a sandboxed throwaway copy, and compares them with the base commit. Coding agents are optional. When you use one, its work goes through the same checks and reaches your checkout only when you say so.

A pass means the tests that ran observed a pass. It never means the task is correct.

pipx install repo-ai-cli

On PyPI as repo-ai-cli 0.11.3; the command is repo-ai. No daemon, no telemetry.

What it does

Three parts. Each one works without the next.

Most tools in this space start from the agent. repo-ai starts from the repository and the evidence, and treats the agent as an optional input that has to pass the same checks as you.

  1. Context

    no agent needed

    Where to look first, from an offline index.

    repo-ai indexes imports, test links, routes, entry points and git co-change history on your machine. It returns a ranked shortlist with the reason for every file, and returns nothing when it has no evidence.

    • repo-ai context "…"
    • repo-ai explain "…"
    • repo-ai impact --changed
  2. Evidence

    no agent needed

    What your change can break, checked skeptically.

    It picks the tests linked to your change, runs them sandboxed in a disposable copy, and compares every failure with the base commit, so new failures are kept apart from old ones. repo-ai coverage checks the requirement-to-test map you declare against what the tests actually showed.

    • repo-ai verify
    • repo-ai verify --run
    • repo-ai coverage ID --requirements reqs.json
  3. Agents

    experimental

    Optional. Under frozen tests, with your consent.

    repo-ai do can hand a task to a coding agent CLI you already use. The agent works in an isolated worktree, the acceptance tests are frozen, and nothing reaches your checkout until you run repo-ai apply. A tournament between providers abstains when more than one candidate passes.

    • repo-ai do "…" --dry-run
    • repo-ai apply ID · undo ID
    • repo-ai shortlist ID · handoff ID

A real session

Six commands on a small Python and TypeScript project

Captured on macOS on 2026-10-29 with the 0.9.0 release candidate (it reports itself as 0.9.0.dev4), in a throwaway copy of the test fixture that ships in the repository. Lines are removed where marked, paths shortened; nothing is added.

Step 1 Index the repository offline. No agent wiring with --agent none.

repo-ai init --agent none
[  0.01s] repo-ai 0.9.0.dev4 — initialising …/demo
[  0.02s] scanning …/demo
[  0.03s] scan complete: 25 indexable files (30 seen, 6 skipped, 0 unchanged)
[  0.03s] parsing 25 files (stdlib ast + regex fallback)
[  0.03s] found 5 manifests, 10 declared dependencies
[  0.44s] git: main@72990b0, 1 commits, 1 contributors
[  0.45s] analysis complete in 0.4s
 
  repo-ai 0.9.0.dev4 — demo: 25 files, 212 LOC, 2 component(s) [Python, TypeScript]
  agents: none (docs only)   learning: suggest   skills: curated   layout: .repo-ai/context/
  changed 13 file(s)
    create  .gitignore
    create  .repo-ai/config.toml
    (+ 11 generated docs under .repo-ai/context/)
  rules: 0 active, 3 proposed (`repo-ai learn` to review)
  ! tree-sitter not installed: non-Python symbol extraction uses the regex fallback. Install with `pip install 'repo-ai-cli[ast]'` for higher fidelity.

Step 2 A ranked shortlist for a task, with the reason for each file.

repo-ai context "fix the login validation bug"
<repo-ai-context>
Retrieved by repo-ai for this request. Confidence: high. Task class: auth, bug. Start with these files; ignore the rest of the repository unless they prove insufficient.
 
Verify with: `make test`
 
| # | File | Why | Look at |
|---|------|-----|---------|
| 1 | `app/auth/login.py` — Login flow: credential validation and token issuance. | keyword match, named like the task, auth path, imported by test_login.py | L1-24 (class LoginError, function validate_credentials) |
| 2 | `tests/test_login.py` | keyword match, named like the task, auth path, imports login.py | whole file (8 lines) |
| 3 | `web/src/components/LoginForm.tsx` | keyword match, named like the task, auth path, imported by LoginForm.test.tsx | L1-10 (function LoginForm) |
… rows 4–9 and the @-mentions line trimmed
</repo-ai-context>

Step 3 Read-only plan: what the change touched and which tests can show a break.

repo-ai verify
repo-ai verify: 1 changed file against HEAD
 
changed
  app/auth/login.py  [source, modified, units: validate_credentials]
      evidence: 1 linked test file(s); runtime evidence none
 
selected 1 of 3 test files
  tests/test_login.py  [static] imports app/auth/login.py
 
confidence: MEDIUM: some of the change rests on static evidence only (imports or file names), not on runtime evidence
… uncertainty notes trimmed
run
  targeted:   python -m pytest -q -p no:cacheprovider tests/test_login.py
  full suite: python -m pytest -q tests   (before merging)
  sandboxed:  repo-ai verify --run   (--full runs the full suite instead)
  sharper:    repo-ai verify --record   (once: records which code each test runs)

Step 4 Run the selection sandboxed, in a disposable copy; the tree is checked before and after.

repo-ai verify --run
verifying 1 selected test file(s) in the seatbelt sandbox, in a disposable copy of the working tree…
  command: make test  [declared in the project's manifests, CI or docs]
  isolated copy: 31 files (clone, 0.0s)
… strategy line trimmed
  pass    tests/test_login.py (0.4s)
… 3 summary lines trimmed
 
PASSED: every selected test passed (the full suite was not run)
  1 passed
  source tree unchanged
  ! no runtime evidence: selection rests on static imports only (`repo-ai verify --record` records which code each test executes)
… one more caveat trimmed
evidence saved; `repo-ai evidence` shows whether it still covers the working tree.
This covers only the tests that ran; it is not proof the change is correct.

Step 5 Plan a mission offline. No provider, no tests, no writes.

repo-ai do "Reject emails with surrounding whitespace in login validation" --dry-run
ASK_APPROVAL: External execution requires verified provider policy, explicit disclosure and budget authorization; planning only; trivial change classification refused unnecessary specialist fan-out
Planning only; no provider or tests invoked.
  app/auth/login.py: keyword match; named like the task; auth path
  web/src/components/LoginForm.tsx: keyword match; named like the task; auth path
  app/api/users.py: keyword match; auth path; imports login.py
… 3 more files trimmed
Unknowns: static import graph only: dynamic imports, reflection, dependency injection, plugin registries and string-based routing are not visible
Orchestration (trivial): merge_owner=controller

Step 6 Which agent CLIs are on PATH. Static check only; nothing is started.

repo-ai agents list
codex: PLANNING_ONLY | version not probed | auth UNKNOWN
claude: PLANNING_ONLY | version not probed | auth UNKNOWN
gemini: PLANNING_ONLY | version not probed | auth UNKNOWN
bob: PLANNING_ONLY | version not probed | auth UNKNOWN
agy: PLANNING_ONLY | version not probed | auth UNKNOWN
No model was invoked. Installed executable does not establish authenticated runnable availability.

How it works

Index once. Verify in a copy. Decide yourself.

How repo-ai fits togetherThree lanes. Context: the repository and its git history are indexed locally, which feeds ranked context and change impact. Evidence: your change is planned read-only against the base, linked tests run in a disposable sandboxed copy, failures are compared with the base, and an evidence record is written. Agents, optional: with explicit consent a task runs in an isolated worktree with frozen tests, one or more candidates are produced, the same checks run, and the result is PASS or ABSTAIN; you decide whether to apply or undo. The index supplies test links to the plan, and missions reuse the same sandbox.1 · Contextoffline, no agentRepositoryworking tree+ git historyLocal indeximports, tests, routeschurn, co-changeRanked contextcontext · explainonly with evidenceChange impactimpact · packetreview2 · Evidenceruns code only with --runYour changediff against baseHEAD or --basePlan (read-only)changed units →tests + confidenceDisposable copysandboxed, no networkintegrity-checkedBase comparisonnew vs pre-existingper testEvidence recordwhat ran, what passed+ coverage map3 · Agentsoptional, explicit consentTask + consent--run--allow-live-providerIsolated worktreefrozen testsallowed write pathsCandidate(s)one provider, or a2–3 way tournamentSame checksbaseline · candidate· regression runPASS or ABSTAINyou decide:apply · undotest linkssame sandbox and comparisonHow repo-ai fits togetherContext: repository, local index, ranked context, change impact. Evidence: your change, read-only plan, disposable sandboxed copy, base comparison, evidence record. Agents, optional: task with consent, isolated worktree, candidates, the same checks, PASS or ABSTAIN, then you apply or undo.1 · Contextoffline, no agentRepositoryworking tree · + git historyLocal indeximports, tests, routes · churn, co-changeRanked contextcontext · explain · only with evidenceChange impactimpact · packet · review2 · Evidenceruns code only with --runYour changediff against base · HEAD or --basePlan (read-only)changed units → tests + confidenceDisposable copysandboxed, no network · integrity-checkedBase comparisonnew vs pre-existing · per testEvidence recordwhat ran, what passed · + coverage map3 · Agentsoptional, explicit consentTask + consent--run · --allow-live-providerIsolated worktreefrozen tests · allowed write pathsCandidate(s)one provider, or a · 2–3 way tournamentSame checksbaseline · candidate · regression runPASS or ABSTAINyou decide: · apply · undo
One engine, three entry points. The index is built once and kept fresh by repo-ai update. The verifier never runs in your working tree. Missions add an agent in front of the same checks — they do not replace them.
Index
SQLite with full-text search under .repo-ai/state/, gitignored. No embeddings, no server.
Sandbox
seatbelt on macOS, bubblewrap on Linux, podman or docker anywhere. No network, a clean environment, writes only to the copy.
Comparison
Failures are re-checked per test at the base revision. A dependency change makes them “not comparable”, never pre-existing.
Apply
Re-verifies, writes receipt-backed bytes, refuses on later edits, symlinks or scope mismatch. undo restores them.

Use cases

What it is for, and what backs each claim

  • Start a task in an unfamiliar repository

    repo-ai context "fix the login validation bug"

    A shortlist of files with line ranges and the evidence for each, plus the test command to run.

    Backed by Captured for this page on the bundled fixture. Identifier lookups are a stable candidate; plain-prose queries are weak (recall@5 0.320 on the retrieval holdout).

  • Check a change before you push it

    repo-ai verify --run

    The linked tests run sandboxed in a throwaway copy of the working tree; the source tree is snapshotted before and after, and any difference invalidates the run.

    Backed by Held-out-7: 0 false assurances in 127 failing states; working tree unchanged in 234 of 234 runs.

  • Tell a new failure from one that was already there

    repo-ai verify --run

    Every failing test is re-checked at the base revision and labelled a regression or already failing.

    Backed by Held-out-7: no new failure classified as pre-existing, no false alarms (pytest and node --test, in containers).

  • Pin the requirements that matter to real tests

    repo-ai coverage ID --requirements reqs.json

    Each declared requirement is matched to the tests that should show it, and checked against observed before/after outcomes.

    Backed by Reproduced on the node-slug false green: a decoy and the reference look identical until the missing requirement is mapped.

  • Let an agent attempt a fix without touching your checkout

    repo-ai do "…" --agent codex --allow-live-provider --run --allow-write src/x.py

    The candidate is produced in an isolated worktree, checked by the same engine, and applied only by repo-ai apply.

    Backed by Live, independently re-verified changes from Codex CLI and Antigravity on the 0.9.0 freeze build (2026-10-29, n = 1 per journey), and from IBM Bob in an earlier pre-release. Small tasks, small n.

  • Hand unfinished work to a different agent

    repo-ai handoff ID --to bob --json > capsule.json

    An integrity-checked capsule; validated partial progress is re-derived from the local store and re-verified from the frozen base.

    Backed by Observed live on constructed scenarios, n = 1 each (experimental): Codex to agy on 2026-10-29 (hidden oracle 714/714), and agy to Codex earlier.

Support, as of 2026-10-30

Providers and platforms, stated plainly

No provider is needed for context or verification. The agent rows describe repo-ai’s isolated execution path only: what we have observed, on small tasks, with the accounts we tested.

Coding agents and repo-ai, as of
ProviderLabelIsolated execution (repo-ai do)Host (repo-ai init)Cost reporting
OpenAI Codex CLI--agent codexEXECUTIONLive, independently re-verified change through repo-ai do on the published 0.10.0 wheel (2026-10-30, hidden oracle 17/17, undo byte-exact), with the 0.10.0 sandbox that hides the rest of HOME. Runs inside repo-ai's outer sandbox with Codex's own sandbox bypassed, only once the outer one is confirmed. Disclosed: the login token copied into its sandbox is readable by a model with shell access; use a separate repo-ai login (repo-ai agents setup codex). Opt-in from 0.11.3 (REPO_AI_CODEX_BROKER=1, macOS): a credential broker keeps the token outside the sandbox and limits Codex to one loopback port. It was live-verified on 2026-10-31 with a ChatGPT login on a development build (two runs; an API-key login is not yet verified). · version 0.165.0 · n = 1 per journey · macOS (seatbelt) verified live; Linux bubblewrap path covered by tests, no live mission this cycle · evidence for OpenAI Codex CLINot wired as a host by repo-ai init.UNKNOWN under subscription; a positive --max-cost-usd is refused (no hard USD flag)
Google Antigravity (agy)--agent agyEXECUTIONLive, independently re-verified change through repo-ai do on the published 0.10.0 wheel (2026-10-30, hidden oracle 17/17, undo byte-exact); earlier tournament and handoff journeys on 0.9.0. Must read the encrypted login keychain files to authenticate; refused on other platforms. · version 1.4.9 · n = 1 per journey · macOS only · evidence for Google Antigravity (agy)Not wired as a host by repo-ai init.UNKNOWN under subscription
IBM Bob--agent bobEXECUTIONLive, model-generated changes independently re-verified through repo-ai do on an earlier pre-release (including abc-v1). Not re-run on 0.9.0 or 0.10.0; account entitlement not re-checked. · version 2.0.9 · small tasks, small n (abc-v1: 4 tasks, one run per cell) · macOS · evidence for IBM Bobrepo-ai init --agent bob (also the default when --agent is omitted) writes .bob/ skills, rules, commands and hooks and an AGENTS.md block.Provider-reported USD; hard cap through Bob's own flag
Claude Code--agent claudeEXECUTIONLive, independently re-verified changes through repo-ai do on the published 0.10.0 wheel (2026-10-30): mission PASS, hidden oracle 17/17, apply and undo byte-exact, and a hostile-repository run whose injected instructions were not followed. Uses Claude Code's own login, nothing copied: a separate repo-ai login (live-verified) or CLAUDE_CODE_OAUTH_TOKEN from claude setup-token (not yet live-verified). The model gets file tools only, and every live run first checks, at no cost, that the session would load nothing else. · version 2.1.314 · n = 1 per journey (2 journeys on 0.10.0) · macOS (seatbelt) verified live; Linux bubblewrap profile covered by tests, not live-verified · evidence for Claude CodeFully supported as a host: repo-ai init --agent claude writes a marked CLAUDE.md block, path-scoped rules, curated skills and hooks that attach ranked context to prompts.Claude's own USD cap via --max-cost-usd, checked between turns (one turn of overshoot observed); on a subscription the reported cost is notional
OpenCode (local model)--agent opencodeEXECUTIONOpenCode against a local, loopback-only model endpoint (Ollama), account-free. On the published 0.10.0 wheel (2026-10-30) with qwen3:8b: one attempt refused (broken edit), one PASS of the visible tests that a hidden case failed (the model dropped a docstring requirement no test pinned); on development builds 1 of 4 passed. repo-ai refused every broken candidate. Deny-by-default tools, network to loopback only. · version 2.2.1 with qwen3:8b via Ollama 0.40 · 7 attempts across builds · macOS only (no tested restriction elsewhere: refused) · evidence for OpenCode (local model)Reads repo-ai-installed skills from .agents/skills (observed 2026-10-30).$0 (local model); cloud-routed models are refused
OpenCode over ACP (local model)--agent opencode-acpEXECUTIONOpenCode driven over the Agent Client Protocol (`opencode acp`) under exactly the opencode isolation: seatbelt, scratch HOME, deny-by-default inline config, loopback-only network. The ACP client stays outside the sandbox and serves file reads and writes only within the run's paths. Any protocol problem, refusal, timeout or stop reason other than end_turn fails the run, and OpenCode's own edit is still checked. Live $0 on a development build (2026-10-31, qwen3:8b): PASS on a scratch fixture in 60 s. · version 2.2.1 with qwen3:8b via Ollama · n = 1 · macOS only (as opencode) · evidence for OpenCode over ACP (local model)Same binary as opencode; a live tournament refuses the pair (not independent).$0 (local model, loopback only); a positive --max-cost-usd is refused
Gemini CLI--agent geminiBLOCKEDRejected for the tested account (exit 41, authentication) on 2026-10-30. From 0.10.0 repo-ai no longer copies ~/.gemini into the sandbox; live Gemini needs GEMINI_API_KEY or GOOGLE_API_KEY. · version 0.65.0 · 1 bounded retry · macOS · evidence for Gemini CLINot wired as a host by repo-ai init.—
Platforms
CapabilitymacOSLinuxWindows
Index, context, impact, reviewyesyesyes
Host sandbox for verify --runseatbeltbubblewrapnone: use --sandbox podman or docker
Missions (repo-ai do execution)yes (seatbelt; live-verified)bubblewrap path covered by tests, no live mission recorded; agy and OpenCode refusednot supported
Measured on fresh held-out dataheld-out-7, in podman (Linux) containers onlynever

Python 3.10–3.14. Runtime dependencies: pathspec, plus tomli on 3.10. Test runners split per file:pytest, unittest, node --test. Jest, Vitest and Mocha run only as an opaque full suite. Provider labels and setup notes · Full status matrix

Local-first

Your code stays on your machine unless you send it

Nothing leaves the machine on any default path. When you authorise a live provider, the task text and the files that agent opens go to that provider under its terms. That is the whole list of exceptions, plus the opt-in network commands named in the docs.

Security & privacy details
  • No daemon, no telemetry, no MCP server. A CLI you run, and hooks that fail open if you wire an agent.
  • Tests never run in your working tree. A disposable copy, sandboxed, with no network; the source tree is hashed before and after every run.
  • No unsandboxed fallback. No supported sandbox means nothing executes.
  • Secrets are quarantined before they are read. Credential files never reach the index; findings show a short prefix and a length, never the value.
  • Repository text is data, never instructions. repo-ai never adds permission allow rules or bypassPermissions.
  • You can see every store. repo-ai privacy audit lists what is kept, where, and whether it is committed; repo-ai uninstall --dry-run shows exactly what removal deletes.

Evidence Lab

What we have and haven’t shown

A tool that checks claims should be strict about its own. Every claim below carries its status, date, sample size, method and source, negative results included. The full ledger has 25 entries, with their limits.

  • DEMONSTRATED

    0 / 127

    verify gives no false assurance on fresh held-out data

    0 false assurances in 127 failing states; 0 new failures classified as pre-existing; 0 false alarms; runtime no-miss 96.7%.

    Date
    Sample
    127 failing states across 5 projects
    Method
    Held-out-7: frozen, independently labelled projects (pytest and node --test), run in podman containers on build 903984a.
    Source for “verify gives no false assurance on fresh held-out data”: docs/release/0.8.0rc1.md
  • DEMONSTRATED

    234 / 234

    No working tree was touched in 234 verify runs

    Working tree unchanged in 234 of 234 runs; the integrity check also fails any run that touches it.

    Date
    Sample
    234 runs
    Method
    Held-out-7, as above: the source tree is hashed before and after every run.
    Source for “No working tree was touched in 234 verify runs”: docs/release/0.8.0rc1.md
  • EXPERIMENTAL

    n = 1

    Live coding-agent candidates, independently re-verified

    Codex CLI 0.165.0 produced a PASS LIVE_CANDIDATE_VERIFIED change through repo-ai do, with coverage, apply (post-apply tests 4/4) and undo (bytes restored); agy 1.4.9 took part in the tournament and handoff journeys.

    Date
    Sample
    1 workflow per journey per build (builds dfb6880, c505f74 and the freeze 9663ac3)
    Method
    Installed wheel in a clean venv on macOS seatbelt; synthetic projects written for the check; providers only through repo-ai do --run --allow-live-provider.
    Source for “Live coding-agent candidates, independently re-verified”: docs/launch/evidence/LIVE-JOURNEYS-2026-10-29.md
  • EXPERIMENTAL

    PASS, oracle 17/17

    Claude Code as an isolated execution provider

    On the published 0.10.0 wheel, Claude Code 2.1.314 produced a model-generated fix inside repo-ai's isolation; repo-ai re-ran the frozen tests (PASS LIVE_CANDIDATE_VERIFIED), an author-written hidden oracle passed 17/17, apply and undo were byte-exact; a hostile repository's injected instructions were not followed.

    Date
    Sample
    n = 1 per journey (2 on 0.10.0)
    Method
    pip install repo-ai-cli==0.10.0 into a clean venv; repo-ai do --agent claude --run --allow-live-provider; Claude's own login (dedicated repo-ai login dir), file tools only, seatbelt, per-run isolation probe.
    Source for “Claude Code as an isolated execution provider”: docs/engineering/evidence/JOURNEYS-0.10.0-PYPI.md
  • EXPERIMENTAL

    1 of 4 PASS

    An account-free local model as a repo-ai provider

    OpenCode 2.2.1 with qwen3:8b (Ollama, loopback only, $0): on the published 0.10.0 wheel one attempt was refused (broken edit) and one passed the visible tests while failing a hidden case (it dropped a docstring requirement no test pinned; repo-ai labelled the claim ASSURANCE_PARTIAL); on development builds 1 of 4 passed (hidden oracle 9/9). repo-ai refused every broken candidate.

    Date
    Sample
    7 attempts across builds
    Method
    Installed wheel; repo-ai do --agent opencode; deny-by-default tools, loopback-only network.
    Source for “An account-free local model as a repo-ai provider”: docs/engineering/evidence/JOURNEYS-0.10.0-PYPI.md
  • EXPERIMENTAL

    2 hosts

    A repo-ai-installed skill loads on two real hosts

    A skill vetted and installed by repo-ai was listed and invoked by Claude Code (paired run: the convention it states appeared in 2 of 2 tests with the skill, 0 of 2 without) and loaded by OpenCode from .agents/skills on request; uninstall restored every file byte-for-byte.

    Date
    Sample
    n = 1 per arm
    Method
    repo-ai skills discover --from-path / install --approve / disable; host-mode runs.
    Source for “A repo-ai-installed skill loads on two real hosts”: docs/engineering/evidence/SKILL-HOSTS-2026-10-30.md
  • EXPERIMENTAL

    PASS_WITH_PRE_EXISTING_FAILURES

    A fix verified on a real repository with unrelated failures

    On un33k/python-slugify (a deliberate regression committed as a bad commit), a live Codex candidate was verified through a scoped mission (--expect-fix with the three tests the regression broke): the declared tests went from failing to passing, nothing that passed at the base failed, 20 unrelated failures were listed; the candidate is byte-identical to the upstream line. An unscoped mission on an equivalent candidate (Claude's recorded fix, replayed) could not be verified because of the 20 unrelated failures.

    Date
    Sample
    1 real repository; 1 successful live run (after 3 failed attempts on 0.11.0)
    Method
    repo-ai do --agent codex --expect-fix … on a development build of the init-then-do fix (0.11.0 itself refused the run because of that first-run bug).
    Source for “A fix verified on a real repository with unrelated failures”: docs/engineering/evidence/REAL-PROJECT-2026-10-30.md
  • NEGATIVE RESULT

    1 of 4

    repo-ai reported PASS for a false green

    On node-slug, repo-ai reported PASS / LIVE_CANDIDATE_VERIFIED for a candidate that failed a hidden requirement the visible tests never stated.

    Date
    Sample
    4 tasks, IBM Bob 2.0.9, one run per cell
    Method
    Preregistered abc-v1 with a hidden grader run in a separate copy.
    Source for “repo-ai reported PASS for a false green”: benchmarks/abc-v1/RESULTS.md
  • NEGATIVE RESULT

    1.44–2.31×

    verify is faster than running the native suite

    On held-out-7, verify --run was slower than the native suite on 3 of 5 projects (1.44×, 1.70× and 2.31× the native time) and faster only on the two larger ones.

    Date
    Sample
    5 projects
    Method
    Wall-clock comparison inside the held-out-7 evaluation (podman containers).
    Source for “verify is faster than running the native suite”: docs/release/0.8.0rc1.md
  • NEGATIVE RESULT

    0.385 vs 0.80

    Capability recommendation meets its recall gate

    FAIL: top-3 recall 0.385 against a gate of 0.80; shortlist recall@6 0.736; top-3 precision 0.712; 5 constraint-forbidden recommendations reached the top 3. Still above the lexical (0.088) and repository-only (0.286) baselines.

    Date
    Sample
    120-task fresh corpus (Corpus C)
    Method
    Preregistered protocol frozen beforehand, measured once; tuning of the deterministic matcher stopped after three unsuccessful passes.
    Source for “Capability recommendation meets its recall gate”: docs/CAPABILITIES.md
  • NEGATIVE RESULT

    12 of 42

    Plugin inspection passes its fresh validation

    FAILED: 12 of 42 benign cases were blocked; executable-resolution coverage 0.25 (0.43 after a post-hoc corrective pass, still a fail). Nothing critical or unresolvable was reported eligible.

    Date
    Sample
    44 cases (42 benign), analyser frozen
    Method
    Preregistered fresh validation of skills vet / inventory on plugin components.
    Source for “Plugin inspection passes its fresh validation”: docs/SKILL_NETWORK.md
  • INCONCLUSIVE

    92 / 90 / 88 %

    Claude Code succeeds more often with repo-ai context

    Plain Claude 92%, repo-ai Core 90%, repo-ai Full 88%; Holm-adjusted p = 1.0 for every comparison; equivalence not shown either. Full cost about 10% more.

    Date
    Sample
    150 preregistered runs (145 valid), 25 real-repository tasks
    Method
    Option A: preregistered, run blind; 25 tasks × 3 arms × 2 runs, Claude Sonnet.
    Source for “Claude Code succeeds more often with repo-ai context”: benchmarks/results/coding-option-a-v0.7.5.md
  • NOT YET DEMONSTRATED

    14 tasks

    Lift over a standalone coding agent

    In abc-v2 to v4 the bare provider already solved every task, so nothing could separate the arms; in abc-v1 the one-task difference of four is consistent with noise.

    Date
    Sample
    4 studies, 14 tasks (4 + 4 + 4 + 2), one run per cell
    Method
    Preregistered manifests frozen before any live call; hidden graders outside the agent workspace; free controls (untouched FAIL, reference CORRECT, decoy FALSE_GREEN) per task.
    Source for “Lift over a standalone coding agent”: docs/launch/EVALUATION.md
  • NOT YET DEMONSTRATED

    Windows, on fresh data

    Unit tests pass in CI (Python 3.10 and 3.13); no held-out or fresh-data run on Windows.

    Date
    Sample
    none on fresh data
    Method
    CI unit tests only.
    Source for “Windows, on fresh data”: docs/STATUS.md
  • UNSUPPORTED

    Missions (repo-ai do execution) on Windows

    Not supported: apply needs POSIX no-follow file operations.

    Date
    Sample
    n/a (design limit)
    Method
    Design limit, stated in the platform matrix.
    Source for “Missions (repo-ai do execution) on Windows”: docs/launch/LAUNCH_REPORT.md

How these were measured. Each card states its own method, sample and limits. The abc studies froze manifests before any live call and used free controls (untouched FAIL, reference CORRECT, decoy FALSE_GREEN); the provider and skill journeys are n = 1 demonstrations without such controls. Most tasks, hidden tests and harnesses were written by an AI agent, and the samples are small: these are not independent evaluations. All 25 entries · Full methodology

Start with the parts that need nothing but your repository.

Index, ask for context, verify a change. Add an agent later, or never.