Local-first CLI · macOS, Linux, Windows · Python 3.10–3.14
Context before the change.
Evidence after it.
repo-ai indexes your repository offline, runs the tests your change can break in a sandboxed throwaway copy, and compares them with the base commit. Coding agents are optional. When you use one, its work goes through the same checks and reaches your checkout only when you say so.
A pass means the tests that ran observed a pass. It never means the task is correct.
pipx install repo-ai-cliWhat it does
Three parts. Each one works without the next.
Most tools in this space start from the agent. repo-ai starts from the repository and the evidence, and treats the agent as an optional input that has to pass the same checks as you.
Context
no agent neededWhere to look first, from an offline index.
repo-ai indexes imports, test links, routes, entry points and git co-change history on your machine. It returns a ranked shortlist with the reason for every file, and returns nothing when it has no evidence.
repo-ai context "…"repo-ai explain "…"repo-ai impact --changed
Evidence
no agent neededWhat your change can break, checked skeptically.
It picks the tests linked to your change, runs them sandboxed in a disposable copy, and compares every failure with the base commit, so new failures are kept apart from old ones.
repo-ai coveragechecks the requirement-to-test map you declare against what the tests actually showed.repo-ai verifyrepo-ai verify --runrepo-ai coverage ID --requirements reqs.json
Agents
experimentalOptional. Under frozen tests, with your consent.
repo-ai docan hand a task to a coding agent CLI you already use. The agent works in an isolated worktree, the acceptance tests are frozen, and nothing reaches your checkout until you runrepo-ai apply. A tournament between providers abstains when more than one candidate passes.repo-ai do "…" --dry-runrepo-ai apply ID · undo IDrepo-ai shortlist ID · handoff ID
A real session
Six commands on a small Python and TypeScript project
Captured on macOS on 2026-10-29 with the 0.9.0 release candidate (it reports itself as 0.9.0.dev4), in a throwaway copy of the test fixture that ships in the repository. Lines are removed where marked, paths shortened; nothing is added.
Step 1 Index the repository offline. No agent wiring with --agent none.
repo-ai init --agent none
[ 0.01s] repo-ai 0.9.0.dev4 — initialising …/demo
[ 0.02s] scanning …/demo
[ 0.03s] scan complete: 25 indexable files (30 seen, 6 skipped, 0 unchanged)
[ 0.03s] parsing 25 files (stdlib ast + regex fallback)
[ 0.03s] found 5 manifests, 10 declared dependencies
[ 0.44s] git: main@72990b0, 1 commits, 1 contributors
[ 0.45s] analysis complete in 0.4s
repo-ai 0.9.0.dev4 — demo: 25 files, 212 LOC, 2 component(s) [Python, TypeScript]
agents: none (docs only) learning: suggest skills: curated layout: .repo-ai/context/
changed 13 file(s)
create .gitignore
create .repo-ai/config.toml
(+ 11 generated docs under .repo-ai/context/)
rules: 0 active, 3 proposed (`repo-ai learn` to review)
! tree-sitter not installed: non-Python symbol extraction uses the regex fallback. Install with `pip install 'repo-ai-cli[ast]'` for higher fidelity.
Step 2 A ranked shortlist for a task, with the reason for each file.
repo-ai context "fix the login validation bug"
<repo-ai-context>
Retrieved by repo-ai for this request. Confidence: high. Task class: auth, bug. Start with these files; ignore the rest of the repository unless they prove insufficient.
Verify with: `make test`
| # | File | Why | Look at |
|---|------|-----|---------|
| 1 | `app/auth/login.py` — Login flow: credential validation and token issuance. | keyword match, named like the task, auth path, imported by test_login.py | L1-24 (class LoginError, function validate_credentials) |
| 2 | `tests/test_login.py` | keyword match, named like the task, auth path, imports login.py | whole file (8 lines) |
| 3 | `web/src/components/LoginForm.tsx` | keyword match, named like the task, auth path, imported by LoginForm.test.tsx | L1-10 (function LoginForm) |
… rows 4–9 and the @-mentions line trimmed
</repo-ai-context>
Step 3 Read-only plan: what the change touched and which tests can show a break.
repo-ai verify
repo-ai verify: 1 changed file against HEAD
changed
app/auth/login.py [source, modified, units: validate_credentials]
evidence: 1 linked test file(s); runtime evidence none
selected 1 of 3 test files
tests/test_login.py [static] imports app/auth/login.py
confidence: MEDIUM: some of the change rests on static evidence only (imports or file names), not on runtime evidence
… uncertainty notes trimmed
run
targeted: python -m pytest -q -p no:cacheprovider tests/test_login.py
full suite: python -m pytest -q tests (before merging)
sandboxed: repo-ai verify --run (--full runs the full suite instead)
sharper: repo-ai verify --record (once: records which code each test runs)
Step 4 Run the selection sandboxed, in a disposable copy; the tree is checked before and after.
repo-ai verify --run
verifying 1 selected test file(s) in the seatbelt sandbox, in a disposable copy of the working tree…
command: make test [declared in the project's manifests, CI or docs]
isolated copy: 31 files (clone, 0.0s)
… strategy line trimmed
pass tests/test_login.py (0.4s)
… 3 summary lines trimmed
PASSED: every selected test passed (the full suite was not run)
1 passed
source tree unchanged
! no runtime evidence: selection rests on static imports only (`repo-ai verify --record` records which code each test executes)
… one more caveat trimmed
evidence saved; `repo-ai evidence` shows whether it still covers the working tree.
This covers only the tests that ran; it is not proof the change is correct.
Step 5 Plan a mission offline. No provider, no tests, no writes.
repo-ai do "Reject emails with surrounding whitespace in login validation" --dry-run
ASK_APPROVAL: External execution requires verified provider policy, explicit disclosure and budget authorization; planning only; trivial change classification refused unnecessary specialist fan-out
Planning only; no provider or tests invoked.
app/auth/login.py: keyword match; named like the task; auth path
web/src/components/LoginForm.tsx: keyword match; named like the task; auth path
app/api/users.py: keyword match; auth path; imports login.py
… 3 more files trimmed
Unknowns: static import graph only: dynamic imports, reflection, dependency injection, plugin registries and string-based routing are not visible
Orchestration (trivial): merge_owner=controller
Step 6 Which agent CLIs are on PATH. Static check only; nothing is started.
repo-ai agents list
codex: PLANNING_ONLY | version not probed | auth UNKNOWN
claude: PLANNING_ONLY | version not probed | auth UNKNOWN
gemini: PLANNING_ONLY | version not probed | auth UNKNOWN
bob: PLANNING_ONLY | version not probed | auth UNKNOWN
agy: PLANNING_ONLY | version not probed | auth UNKNOWN
No model was invoked. Installed executable does not establish authenticated runnable availability.
How it works
Index once. Verify in a copy. Decide yourself.
repo-ai update. The verifier never runs in your working tree. Missions add an agent in front of the same checks — they do not replace them.- Index
- SQLite with full-text search under
.repo-ai/state/, gitignored. No embeddings, no server. - Sandbox
- seatbelt on macOS, bubblewrap on Linux, podman or docker anywhere. No network, a clean environment, writes only to the copy.
- Comparison
- Failures are re-checked per test at the base revision. A dependency change makes them “not comparable”, never pre-existing.
- Apply
- Re-verifies, writes receipt-backed bytes, refuses on later edits, symlinks or scope mismatch.
undorestores them.
Use cases
What it is for, and what backs each claim
Start a task in an unfamiliar repository
repo-ai context "fix the login validation bug"A shortlist of files with line ranges and the evidence for each, plus the test command to run.
Backed by Captured for this page on the bundled fixture. Identifier lookups are a stable candidate; plain-prose queries are weak (recall@5 0.320 on the retrieval holdout).
Check a change before you push it
repo-ai verify --runThe linked tests run sandboxed in a throwaway copy of the working tree; the source tree is snapshotted before and after, and any difference invalidates the run.
Backed by Held-out-7: 0 false assurances in 127 failing states; working tree unchanged in 234 of 234 runs.
Tell a new failure from one that was already there
repo-ai verify --runEvery failing test is re-checked at the base revision and labelled a regression or already failing.
Backed by Held-out-7: no new failure classified as pre-existing, no false alarms (pytest and node --test, in containers).
Pin the requirements that matter to real tests
repo-ai coverage ID --requirements reqs.jsonEach declared requirement is matched to the tests that should show it, and checked against observed before/after outcomes.
Backed by Reproduced on the node-slug false green: a decoy and the reference look identical until the missing requirement is mapped.
Let an agent attempt a fix without touching your checkout
repo-ai do "…" --agent codex --allow-live-provider --run --allow-write src/x.pyThe candidate is produced in an isolated worktree, checked by the same engine, and applied only by
repo-ai apply.Backed by Live, independently re-verified changes from Codex CLI and Antigravity on the 0.9.0 freeze build (2026-10-29, n = 1 per journey), and from IBM Bob in an earlier pre-release. Small tasks, small n.
Hand unfinished work to a different agent
repo-ai handoff ID --to bob --json > capsule.jsonAn integrity-checked capsule; validated partial progress is re-derived from the local store and re-verified from the frozen base.
Backed by Observed live on constructed scenarios, n = 1 each (experimental): Codex to agy on 2026-10-29 (hidden oracle 714/714), and agy to Codex earlier.
Support, as of 2026-10-30
Providers and platforms, stated plainly
No provider is needed for context or verification. The agent rows describe repo-ai’s isolated execution path only: what we have observed, on small tasks, with the accounts we tested.
| Provider | Label | Isolated execution (repo-ai do) | Host (repo-ai init) | Cost reporting |
|---|---|---|---|---|
OpenAI Codex CLI--agent codex | EXECUTION | Live, independently re-verified change through repo-ai do on the published 0.10.0 wheel (2026-10-30, hidden oracle 17/17, undo byte-exact), with the 0.10.0 sandbox that hides the rest of HOME. Runs inside repo-ai's outer sandbox with Codex's own sandbox bypassed, only once the outer one is confirmed. Disclosed: the login token copied into its sandbox is readable by a model with shell access; use a separate repo-ai login (repo-ai agents setup codex). Opt-in from 0.11.3 (REPO_AI_CODEX_BROKER=1, macOS): a credential broker keeps the token outside the sandbox and limits Codex to one loopback port. It was live-verified on 2026-10-31 with a ChatGPT login on a development build (two runs; an API-key login is not yet verified). · version 0.165.0 · n = 1 per journey · macOS (seatbelt) verified live; Linux bubblewrap path covered by tests, no live mission this cycle · evidence for OpenAI Codex CLI | Not wired as a host by repo-ai init. | UNKNOWN under subscription; a positive --max-cost-usd is refused (no hard USD flag) |
Google Antigravity (agy)--agent agy | EXECUTION | Live, independently re-verified change through repo-ai do on the published 0.10.0 wheel (2026-10-30, hidden oracle 17/17, undo byte-exact); earlier tournament and handoff journeys on 0.9.0. Must read the encrypted login keychain files to authenticate; refused on other platforms. · version 1.4.9 · n = 1 per journey · macOS only · evidence for Google Antigravity (agy) | Not wired as a host by repo-ai init. | UNKNOWN under subscription |
IBM Bob--agent bob | EXECUTION | Live, model-generated changes independently re-verified through repo-ai do on an earlier pre-release (including abc-v1). Not re-run on 0.9.0 or 0.10.0; account entitlement not re-checked. · version 2.0.9 · small tasks, small n (abc-v1: 4 tasks, one run per cell) · macOS · evidence for IBM Bob | repo-ai init --agent bob (also the default when --agent is omitted) writes .bob/ skills, rules, commands and hooks and an AGENTS.md block. | Provider-reported USD; hard cap through Bob's own flag |
Claude Code--agent claude | EXECUTION | Live, independently re-verified changes through repo-ai do on the published 0.10.0 wheel (2026-10-30): mission PASS, hidden oracle 17/17, apply and undo byte-exact, and a hostile-repository run whose injected instructions were not followed. Uses Claude Code's own login, nothing copied: a separate repo-ai login (live-verified) or CLAUDE_CODE_OAUTH_TOKEN from claude setup-token (not yet live-verified). The model gets file tools only, and every live run first checks, at no cost, that the session would load nothing else. · version 2.1.314 · n = 1 per journey (2 journeys on 0.10.0) · macOS (seatbelt) verified live; Linux bubblewrap profile covered by tests, not live-verified · evidence for Claude Code | Fully supported as a host: repo-ai init --agent claude writes a marked CLAUDE.md block, path-scoped rules, curated skills and hooks that attach ranked context to prompts. | Claude's own USD cap via --max-cost-usd, checked between turns (one turn of overshoot observed); on a subscription the reported cost is notional |
OpenCode (local model)--agent opencode | EXECUTION | OpenCode against a local, loopback-only model endpoint (Ollama), account-free. On the published 0.10.0 wheel (2026-10-30) with qwen3:8b: one attempt refused (broken edit), one PASS of the visible tests that a hidden case failed (the model dropped a docstring requirement no test pinned); on development builds 1 of 4 passed. repo-ai refused every broken candidate. Deny-by-default tools, network to loopback only. · version 2.2.1 with qwen3:8b via Ollama 0.40 · 7 attempts across builds · macOS only (no tested restriction elsewhere: refused) · evidence for OpenCode (local model) | Reads repo-ai-installed skills from .agents/skills (observed 2026-10-30). | $0 (local model); cloud-routed models are refused |
OpenCode over ACP (local model)--agent opencode-acp | EXECUTION | OpenCode driven over the Agent Client Protocol (`opencode acp`) under exactly the opencode isolation: seatbelt, scratch HOME, deny-by-default inline config, loopback-only network. The ACP client stays outside the sandbox and serves file reads and writes only within the run's paths. Any protocol problem, refusal, timeout or stop reason other than end_turn fails the run, and OpenCode's own edit is still checked. Live $0 on a development build (2026-10-31, qwen3:8b): PASS on a scratch fixture in 60 s. · version 2.2.1 with qwen3:8b via Ollama · n = 1 · macOS only (as opencode) · evidence for OpenCode over ACP (local model) | Same binary as opencode; a live tournament refuses the pair (not independent). | $0 (local model, loopback only); a positive --max-cost-usd is refused |
Gemini CLI--agent gemini | BLOCKED | Rejected for the tested account (exit 41, authentication) on 2026-10-30. From 0.10.0 repo-ai no longer copies ~/.gemini into the sandbox; live Gemini needs GEMINI_API_KEY or GOOGLE_API_KEY. · version 0.65.0 · 1 bounded retry · macOS · evidence for Gemini CLI | Not wired as a host by repo-ai init. | — |
| Capability | macOS | Linux | Windows |
|---|---|---|---|
| Index, context, impact, review | yes | yes | yes |
Host sandbox for verify --run | seatbelt | bubblewrap | none: use --sandbox podman or docker |
Missions (repo-ai do execution) | yes (seatbelt; live-verified) | bubblewrap path covered by tests, no live mission recorded; agy and OpenCode refused | not supported |
| Measured on fresh held-out data | held-out-7, in podman (Linux) containers only | never | |
Python 3.10–3.14. Runtime dependencies: pathspec, plus tomli on 3.10. Test runners split per file:pytest, unittest, node --test. Jest, Vitest and Mocha run only as an opaque full suite. Provider labels and setup notes · Full status matrix
Local-first
Your code stays on your machine unless you send it
Nothing leaves the machine on any default path. When you authorise a live provider, the task text and the files that agent opens go to that provider under its terms. That is the whole list of exceptions, plus the opt-in network commands named in the docs.
Security & privacy details- No daemon, no telemetry, no MCP server. A CLI you run, and hooks that fail open if you wire an agent.
- Tests never run in your working tree. A disposable copy, sandboxed, with no network; the source tree is hashed before and after every run.
- No unsandboxed fallback. No supported sandbox means nothing executes.
- Secrets are quarantined before they are read. Credential files never reach the index; findings show a short prefix and a length, never the value.
- Repository text is data, never instructions. repo-ai never adds permission allow rules or
bypassPermissions. - You can see every store.
repo-ai privacy auditlists what is kept, where, and whether it is committed;repo-ai uninstall --dry-runshows exactly what removal deletes.
Evidence Lab
What we have and haven’t shown
A tool that checks claims should be strict about its own. Every claim below carries its status, date, sample size, method and source, negative results included. The full ledger has 25 entries, with their limits.
Statuses across all 25 entries
- DEMONSTRATED
0 / 127
verify gives no false assurance on fresh held-out data
0 false assurances in 127 failing states; 0 new failures classified as pre-existing; 0 false alarms; runtime no-miss 96.7%.
Source for “verify gives no false assurance on fresh held-out data”:docs/release/0.8.0rc1.md - DEMONSTRATED
234 / 234
No working tree was touched in 234 verify runs
Working tree unchanged in 234 of 234 runs; the integrity check also fails any run that touches it.
Source for “No working tree was touched in 234 verify runs”:docs/release/0.8.0rc1.md - EXPERIMENTAL
n = 1
Live coding-agent candidates, independently re-verified
Codex CLI 0.165.0 produced a PASS LIVE_CANDIDATE_VERIFIED change through repo-ai do, with coverage, apply (post-apply tests 4/4) and undo (bytes restored); agy 1.4.9 took part in the tournament and handoff journeys.
Source for “Live coding-agent candidates, independently re-verified”:docs/launch/evidence/LIVE-JOURNEYS-2026-10-29.md - EXPERIMENTAL
PASS, oracle 17/17
Claude Code as an isolated execution provider
On the published 0.10.0 wheel, Claude Code 2.1.314 produced a model-generated fix inside repo-ai's isolation; repo-ai re-ran the frozen tests (PASS LIVE_CANDIDATE_VERIFIED), an author-written hidden oracle passed 17/17, apply and undo were byte-exact; a hostile repository's injected instructions were not followed.
Source for “Claude Code as an isolated execution provider”:docs/engineering/evidence/JOURNEYS-0.10.0-PYPI.md - EXPERIMENTAL
1 of 4 PASS
An account-free local model as a repo-ai provider
OpenCode 2.2.1 with qwen3:8b (Ollama, loopback only, $0): on the published 0.10.0 wheel one attempt was refused (broken edit) and one passed the visible tests while failing a hidden case (it dropped a docstring requirement no test pinned; repo-ai labelled the claim ASSURANCE_PARTIAL); on development builds 1 of 4 passed (hidden oracle 9/9). repo-ai refused every broken candidate.
Source for “An account-free local model as a repo-ai provider”:docs/engineering/evidence/JOURNEYS-0.10.0-PYPI.md - EXPERIMENTAL
2 hosts
A repo-ai-installed skill loads on two real hosts
A skill vetted and installed by repo-ai was listed and invoked by Claude Code (paired run: the convention it states appeared in 2 of 2 tests with the skill, 0 of 2 without) and loaded by OpenCode from .agents/skills on request; uninstall restored every file byte-for-byte.
Source for “A repo-ai-installed skill loads on two real hosts”:docs/engineering/evidence/SKILL-HOSTS-2026-10-30.md - EXPERIMENTAL
PASS_WITH_PRE_EXISTING_FAILURES
A fix verified on a real repository with unrelated failures
On un33k/python-slugify (a deliberate regression committed as a bad commit), a live Codex candidate was verified through a scoped mission (--expect-fix with the three tests the regression broke): the declared tests went from failing to passing, nothing that passed at the base failed, 20 unrelated failures were listed; the candidate is byte-identical to the upstream line. An unscoped mission on an equivalent candidate (Claude's recorded fix, replayed) could not be verified because of the 20 unrelated failures.
Source for “A fix verified on a real repository with unrelated failures”:docs/engineering/evidence/REAL-PROJECT-2026-10-30.md - NEGATIVE RESULT
1 of 4
repo-ai reported PASS for a false green
On node-slug, repo-ai reported PASS / LIVE_CANDIDATE_VERIFIED for a candidate that failed a hidden requirement the visible tests never stated.
Source for “repo-ai reported PASS for a false green”:benchmarks/abc-v1/RESULTS.md - NEGATIVE RESULT
1.44–2.31×
verify is faster than running the native suite
On held-out-7, verify --run was slower than the native suite on 3 of 5 projects (1.44×, 1.70× and 2.31× the native time) and faster only on the two larger ones.
Source for “verify is faster than running the native suite”:docs/release/0.8.0rc1.md - NEGATIVE RESULT
0.385 vs 0.80
Capability recommendation meets its recall gate
FAIL: top-3 recall 0.385 against a gate of 0.80; shortlist recall@6 0.736; top-3 precision 0.712; 5 constraint-forbidden recommendations reached the top 3. Still above the lexical (0.088) and repository-only (0.286) baselines.
Source for “Capability recommendation meets its recall gate”:docs/CAPABILITIES.md - NEGATIVE RESULT
12 of 42
Plugin inspection passes its fresh validation
FAILED: 12 of 42 benign cases were blocked; executable-resolution coverage 0.25 (0.43 after a post-hoc corrective pass, still a fail). Nothing critical or unresolvable was reported eligible.
Source for “Plugin inspection passes its fresh validation”:docs/SKILL_NETWORK.md - INCONCLUSIVE
92 / 90 / 88 %
Claude Code succeeds more often with repo-ai context
Plain Claude 92%, repo-ai Core 90%, repo-ai Full 88%; Holm-adjusted p = 1.0 for every comparison; equivalence not shown either. Full cost about 10% more.
Source for “Claude Code succeeds more often with repo-ai context”:benchmarks/results/coding-option-a-v0.7.5.md - NOT YET DEMONSTRATED
14 tasks
Lift over a standalone coding agent
In abc-v2 to v4 the bare provider already solved every task, so nothing could separate the arms; in abc-v1 the one-task difference of four is consistent with noise.
Source for “Lift over a standalone coding agent”:docs/launch/EVALUATION.md - NOT YET DEMONSTRATED
Windows, on fresh data
Unit tests pass in CI (Python 3.10 and 3.13); no held-out or fresh-data run on Windows.
Source for “Windows, on fresh data”:docs/STATUS.md - UNSUPPORTED
Missions (repo-ai do execution) on Windows
Not supported: apply needs POSIX no-follow file operations.
Source for “Missions (repo-ai do execution) on Windows”:docs/launch/LAUNCH_REPORT.md
How these were measured. Each card states its own method, sample and limits. The abc studies froze manifests before any live call and used free controls (untouched FAIL, reference CORRECT, decoy FALSE_GREEN); the provider and skill journeys are n = 1 demonstrations without such controls. Most tasks, hidden tests and harnesses were written by an AI agent, and the samples are small: these are not independent evaluations. All 25 entries · Full methodology
Start with the parts that need nothing but your repository.
Index, ask for context, verify a change. Add an agent later, or never.