obedience-benchmark: The benchmark for when agents know the answer but ignore the instructions
a5c-ai/obedience-benchmark measures process fidelity, not just task completion, with Docker isolation, structured logs, and a judge that scores how closely an agent followed the script.
- obedience-benchmark turns agent evaluation into a process-fidelity test, so a clever shortcut counts as a failure even when the final answer looks plausible.
- Its architecture is a clean pipeline, Catalog to Prepare to Run to Judge to Report, with structured logs acting as the evidence trail.
- The runner is the real safety boundary, because the benchmark is built to contain an agent that can execute shell commands without inheriting the whole host environment.
- The project exposes a gap that capability benchmarks miss, which is that a model can be useful and still be unreliable when the workflow itself matters.
Most evals ask whether an agent got the right answer. obedience-benchmark asks a harder question: did it follow the instructions in the order they were given, with the constraints intact? That distinction sounds subtle until you are debugging an agent that keeps reaching the destination by taking the wrong road.
The metric most benchmarks skip
The repo is built around a simple but sharp idea: capability and obedience are different things. A model can be smart enough to solve the task and still be a poor fit for a workflow where sequence, dependency, and error handling matter. In software systems, that gap is not academic. It is the difference between a helpful agent and one that quietly rewrites the operating rules.
| Benchmark | What it measures | Why it matters |
|---|---|---|
| SWE-Bench and variants | Whether the agent can solve real GitHub issues and land a correct patch. | It is a strong proxy for coding ability, but it says little about whether the agent respected a required process. |
| Terminal-Bench and Harbor | Whether agents can operate reliably inside terminal or containerized environments. | It is useful for reproducible execution, but the main question is still task success. |
| obedience-benchmark | Whether the agent followed the requested steps, constraints, ordering, and fallback logic. | It measures process fidelity, which is what you need when the workflow itself is the product. |
How the machine is wired
The implementation reads like a lab instrument rather than a demo app. The central orchestrator, `benchmarker.ts`, moves through a fixed pipeline: Catalog, Prepare, Run, Judge, Report. Dynamic imports for the judge and reporter let the system keep moving even if one module is missing or still under construction, which is a practical choice for a benchmark that expects partial implementations.
That evidence trail is the point of `log-collector.ts`. It records events like `tool_call_start`, `step_start`, and `message_received`, then packages them into a structured log that the judge can replay against the task definition. In other words, the benchmark does not just score outcomes. It captures behavior.
What the judge actually scores
The judge compares the `ExecutionTrace` against the `task.yaml` and scores obedience across seven dimensions. That makes the rubric richer than a simple pass or fail, because two runs can both finish and still differ in whether they respected the path that was asked for.
- Completeness: did the agent do every required step?
- Ordering: did it do those steps in the right sequence?
- Conditionality: did it branch only when the instructions allowed it?
- Parallelism: did it parallelize only where parallelism was permitted?
- Granularity: did it keep the right level of detail instead of compressing steps away?
- Aggregation: did it preserve the requested grouping of results?
- Error handling: did it react to failures in the way the task required?
This is the kind of rubric that exposes shortcut behavior. An agent can land on a plausible output while skipping a dependency, merging steps that should stay separate, or handling an error in a way that violates the instructions. The benchmark makes those failures legible.
Why the runner matters more than the headline
The most consequential code lives in `runner.ts` and `docker-runner.ts`. The `Runner` interface keeps the orchestrator agnostic about where the agent executes, but the Docker path is where the security story gets serious. `resolveHarnessCredentials` explicitly selects only the credentials the run needs, which keeps the benchmark from handing an untrusted agent the entire host environment.
The Docker setup also separates read-only inputs from writable outputs and manages shutdown with a grace period before force-killing a container. That matters because the benchmark is testing behavior that may be inconvenient, evasive, or simply messy. A serious evaluator has to assume the candidate can be both clever and uncooperative.
What this project is really saying
The deeper argument here is not about obedience as a moral ideal. It is about operational discipline. If an agent is going to run in a workflow where step order, conditional behavior, and failure handling all matter, then a final answer is not enough. The process is part of the product.
That is why the repo feels more like research infrastructure than a polished consumer tool. It is built as a Claude Code plugin, it leans on `CLAUDE.md` for prompt injection, it persists `run-state.json` at transition points, and it uses ULIDs for run IDs. Those are the fingerprints of a system designed for reproducibility, not flash.
