Skip to content

test: publish the two-agent guardrail benchmark - #325

Merged
arminazimi merged 1 commit into
mainfrom
test/116-guard-eval
Nov 18, 2026
Merged

arminazimi merged 1 commit into
mainfrom
test/116-guard-eval

Conversation

@arminazimi

Copy link
Copy Markdown
Contributor

Closes #273
Closes #280
Closes #284
Closes #116

What changed

  • Repeated the 39 Codex pairs interrupted by quota with the same CLI and model. The runner preserved the original attempts in previousAttempts. The complete Cursor run from test: benchmark installed Offing guardrails across agent hosts #315 is reused.
  • Updated the benchmark and launch note using the 400 scored live attempts across both hosts. The report includes the per-host outcomes, the denominator for the headline, and commands to rerun and rescore the benchmark.

With hooks on, Offing denied 153 of 160 requested risky edits (95.6%) and 0 of 40 clean controls. The requested risky edit landed in 154 of 160 attempts with hooks off and 4 of 160 with hooks on.

Quality bar

The diff covers the three remaining tasks in the epic. All 100 task pairs for each host were checked against the task manifest. The committed live JSON came from the benchmark runner.

Checks CI cannot make

  • No application behavior changed. Benchmark results and documentation do not require a changelog fragment.

Verification

  • npm run check
  • Full offline replay on both host formats, 400 attempts
  • Full Codex live retry with --retry-failed and rescoring both live JSON files with --report

Notes for reviewers

The headline describes single-file writes with the two recorded host and model combinations on the fixed task set. The benchmark used the packaged hooks and recorded hook decisions and final fixture files. It did not run fixture code or install dependencies from the fixtures.

@arminazimi
arminazimi merged commit df99608 into main Nov 18, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant