Measured against Claude Code

Why use Foreman

It stops what you'd regret. In bypass mode, out of 24 dangerous commands (deleting a home folder, force-pushing, reading SSH keys, publishing a package) Foreman stopped 20 and saved a copy before the other 4. Claude Code on its own ran all 24. Foreman stopped none of 25 everyday commands.

It leaves a record. Every task has a plan, the checks that ran and their output, and a commit you can revert.

What the quality runs show. On small tasks where the request spells out every rule, both sides pass everything. There Foreman costs 10% more with Sonnet and 15% more with Haiku, and takes about the same time with Sonnet and 13% more time with Haiku, so for a short, fully specified task you're watching, Claude Code alone is cheaper.

It finishes long work. Asked for eight requests in one Haiku session whose context compacts as it goes, Foreman passed every hidden test in 6 of 6 runs over 2 duels. Claude Code on its own did in 3 of 6; in the first duel, the runs that fell short wrote the modules they got wrong after their first compaction, and the one that passed wrote them before it. Foreman keeps each request's text and gives it back after every compaction. It cost about 60% more there ($0.095 against $0.059 a session) and took longer. Counted per session that passed every test, which is what you'd pay to get one, Foreman cost $0.095 against $0.119: 20% less per fully correct session.

It keeps going. In a marathon, once the asked work was done, Foreman went on through 7 compactions, found and fixed 3 things nobody asked for, and still passed every hidden test.

Safety: what runs in bypass mode

Bypass mode is how Foreman is meant to run: no permission prompts. Claude Code on its own asks before none of these in that mode. Foreman's guard judges each one before it runs; the everyday set checks that it doesn't get in the way of normal work.

Quality: eight fully specified tasks, graded blind

Each task asks for one small module in a neutral Python repo (durations, an LRU cache, SemVer precedence, text tables, ranges, dotted paths, a token bucket, CSV lines). The request states every rule its hidden tests check, so passing means doing the whole spec, not guessing wording. Both sides get the same request, model, budget and time limit, each in a clean worktree, three runs per task; a task counts as passed when at least two of its three runs pass. The Claude Code side runs with no plugins but keeps Foreman's guard on, so neither side can damage the machine. bench/spec-suite/make.py builds the repo.

Harder: eight tasks where checking matters

A second neutral repo, an invoicing package whose modules share helpers, graded by its whole test suite. Four tasks change a shared helper, where the obvious edit breaks another module's existing tests unless the request's "Keep" rules are followed; four are dense edge-case specs (text wrapping, INI files, interval arithmetic, a tokenizer). Each task's plausible shortcut is recorded and proven to fail. Grading restores the reference tests, so editing an existing test can't pass. bench/spec-suite/make.py --suite hard builds it.

The harder tasks with a smaller model

The same eight harder tasks with Haiku on both sides, where mistakes should be likelier and a test run at the end should matter more. Haiku sessions cost about a cent each, so Foreman's fixed cost (its skill and agent listings, its hooks and the test run at the end) weighs more here.

One long session: eight requests, compacting as it goes

The eight harder tasks asked for at once, in one Haiku session per side. Each side's context is compacted whenever it nears 40,000 tokens, which brings forward what happens in any long session: earlier work is summarised and the details are gone. The hidden tests of all eight requests grade where each side ends up. Each session is capped at $0.50; each side runs three times and the table shows the first run's per-request result.

A marathon: Foreman keeps going

The same long session, but once the asked work is closed, Foreman's side looks for more work in the repo and does it, test first, until its $1 cap. Claude Code on its own stops when the asked work is done. The hidden tests still grade the end state, so work found later can't quietly break what was asked for.

Where Foreman loses

Generated from the same results as the tables above: every row where Claude Code passed and Foreman didn't, or Foreman cost or took 25% more. Each one reruns on its own with fm bench repro, both sides, with that run's model and settings.

What the duels changed in Foreman

Reading the sessions turn by turn showed where Foreman's extra cost went: not into its context (about 2,000 tokens), but into turns spent on its own commands. A first run on these tasks cost Foreman 8 to 13 turns where Claude Code used 4. The fixes moved Foreman's bookkeeping out of the model and into its hooks:

• The task makes itself. The first edit of a one-request session creates its task from the request and goes ahead.
• It closes itself. At the end, Foreman runs the project's tests and closes and commits the task when they pass. When they fail, it tells the agent once, so the run fixes its work instead of ending on it.
• It starts informed. The session begins with the project's layout and test command, so the agent writes instead of listing folders.
• Less friction. Commands default to the active task, a broken check can be corrected, and only real security names ask for an adversary review.
• Requests survive compaction. The first long runs lost the rules of later requests after a compaction (44 of 105 hidden tests once): a summary keeps a request's name and drops its rules. Now each request is its own task holding its text, the open ones come back after every compaction, and the next one is focused with its text when the agent moves on.

The first run, and why it was replaced

The first duel used six tasks from Foreman's own history and came out at 0 of 6 for Foreman, 1 of 6 for Claude Code, with Foreman costing 33% more. Reading its transcripts showed the case set couldn't measure either side: three cases graded what Foreman had learned from this user's own past corrections, which a fresh session can't have, and the other three checked exact wording their requests never stated. Both sides also worked inside Foreman's own code, which the guard protects. The transcripts did show where Foreman spent extra turns on its own commands; those were cut before this run.

Method and limits

• Small sample. Sixteen tasks, three runs each, two models, and five long sessions a side: enough to see a direction, not a precise rate.
• Short tasks don't separate the sides. Fully specified tasks this size are within reach of today's models without help, so those runs measure cost and time; the long session is where they part. A public benchmark is next.
• Compaction is brought forward. The long sessions compact at 40,000 tokens instead of near the model's limit, to show in minutes what a day-long session meets.
• Reproduce it. fm bench duel --vs none runs the quality benchmark on your own finished tasks; bench/spec-suite/README.md gives the long-session commands; python3 site/bench_safety.py reruns the safety set.
• Other plugins. fm bench duel PLUGIN runs the same cases with another plugin beside Foreman.