code-fix-harness: AutoFixer: The Model Proposes, the Test Suite Decides
A local 7B model writes the patch. A fail-to-pass rule decides if it ships. Inside a Python harness that treats AI code generation as a reliability problem, not a capability race.
- There is no AutoFixer without the fail-to-pass rule: a 7B local model writes every patch, but the test suite decides whether any patch ships.
- The reliability lives in the harness, which catches a small model's predictable failures with a five-strategy parsing cascade, a 12KB context budget, and a verifier that writes diagnostics back into the prompt.
- When pytest collects no tests, the harness tells the model it assumes it is trying to cheat, which is a cleaner statement of the threat model than any policy document.
- The pattern is orthogonal to model quality, and the repo's two drifting entry points show how much polish a good pattern still needs.
In Fix Mode, the Test Has to Fail First
Before the model touches a file, AutoFixer runs the failing test and confirms it fails. After the patch, it runs the same test again and confirms it passes. Fail-to-pass, or the patch does not ship.
A second, quieter rule runs alongside it: pass-to-pass. The existing tests cannot break. Two sentences turn an unreliable code generator into a reviewable artifact.
The physical product of the whole machine is a single file called remediation.patch. Nothing reaches your source tree until that file exists, and nothing reaches GitHub until you say so.
Why Build This Around a 7B Model?
AutoFixer defaults to qwen2.5-coder:7b and runs entirely on Ollama. No cloud keys, no API spend, no data leaving the machine.
The reason is not cost. It is that the harness, not the model, is where the reliability lives. A frontier model with a loose harness is still a coin flip. A small model with a strict harness is a repeatable process.
The defaults make the bet concrete: a 7B coder model, a max of 3 attempts, and a cap of 12 iterations. If the model cannot produce a passing patch inside that budget, the run ends and the diagnostic is yours to read.
Five Strategies for a Model That Won't Follow Instructions
Small local models do not reliably emit FILE: path headers or valid JSON. So _extract_code_files does not ask them to. It tries five extraction strategies in order and stops at the first one that yields code.
FILE:or###headers followed by a fenced block.- Paths embedded in fence language tags, such as
```python:path/to/file.py. - JSON objects or arrays with
pathandcontentkeys, fenced or bare. - Any fenced block, assigned to a known default filename, with the largest block winning.
- Raw text that looks like Python, with no fences at all.
The cascade favors recall over precision. Heuristics like "largest block wins" can silently pick the wrong block, and the raw-text strategy can accept prose containing the word import as code. The verifier is what keeps that from becoming a correctness problem.
# _extract_code_files: five strategies, first hit wins
# 1. FILE: path / ### path headers followed by a fenced block
# 2. Paths embedded in fence language tags (```python:path/to/file.py)
# 3. JSON objects or arrays with "path" and "content" keys
# 4. Any fenced block assigned to a default filename (largest wins)
# 5. Raw Python-looking text with no fences
for strategy in (headers, fence_paths, json_objects, any_fence, raw_python):
files = strategy(text, default_filename)
if files:
return files # stop at the first strategy that yields code
When pytest Says 'No Tests Collected,' the Harness Writes a Lecture
The verifier does more than return pass or fail. When pytest exits with code 5, meaning no tests were collected, format_diagnostic() appends a message to the model's next prompt.
if result.returncode == 5: # pytest: no tests collected
diagnostic += (
"\nNo tests were collected. Create a test file that imports "
"the real repository modules. Trivial tests like `assert True` "
"without source changes will be rejected."
)
The message instructs the model to create a test file that imports real repository modules and warns that trivial tests like assert True without source changes will be rejected. That is the threat model in one sentence: the harness assumes the model will try to game it.
Diagnostics as prompt engineering is a pattern worth stealing. The verifier writes guidance back into the model's context at the point of failure, which is exactly when the model is most likely to change course.
New Google paper shows that agents score higher when each task gets a specialized harness from a shared family, not 1 global harness. An agent harness is the prompts, tools and control code around a model. Most tuning methods evolve 1 global harness, though a fix for 1 task htt
Eight Files, Twelve Kilobytes, and a Cap of 28 Messages
_collect_workspace_source_files walks the repo, filters by extension, sorts shallow files first, and returns at most 8 files of 12KB or less. The conversation history is capped at 28 messages.
Every input the model sees is deliberately bounded. This is the opposite of stuffing the whole repo into the context window. A budget forces choices, and choices are easier to reason about than a dump.
The trade-off is real. A fix that depends on a file outside the budget will never be attempted, and the harness will report a failure that looks like the model's fault. But a bounded failure is still a failure you can inspect.
Cloud Agents, Static-Analysis Autofixers, and Everything in Between
AutoFixer is not Cursor, not Claude Code, not CodeQL autofix, not Semgrep. It is a local, sandboxed, test-gated patch generator with no tool-calling loop and no autonomous commit.
The table's payoff is the "How success is decided" column. Only AutoFixer and the linters are deterministic. Everything else relies on the model or the user to declare victory.
| Where the model runs | How success is decided | What it outputs | Human gate | Failure mode | |
|---|---|---|---|---|---|
| AutoFixer (code-fix-harness) | Local (Ollama, qwen2.5-coder:7b) | A fail-to-pass test run in a sandboxed venv | A remediation.patch file | Two prompts: apply to source, publish to GitHub | Patch rejected, diagnostic fed back to the model (up to 3 attempts) |
| Cloud coding agents (Cursor, Claude Code) | Vendor cloud | Model self-reports success or user reviews | Live edits in the working tree | Usually none | Flaky edits the user has to catch |
| Static-analysis autofix (CodeQL, DeepSource) | Cloud or local | Static re-scan of the alert | Suggested patch in the scanner UI | Review before apply | Fixes that pass the scan but break tests |
| Rule-based linters (Semgrep, ESLint) | Local | Deterministic rule match | Warnings and occasional auto-fixes | None | False positives and missed semantic bugs |
The point is not that AutoFixer wins. It is that the harness pattern is orthogonal to the model. Line the trade-offs up and the choice becomes legible: you are picking where the model runs, who declares success, and how much damage a wrong answer can do.
Building an AI agent? Don't start with a 20-step workflow full of if/else branches. Start with a harness that already works. The order LangChain's Harrison Chase recommends: 1. Start with a harness. Claude Code, Codex or his own deepagents. The agent loop and tool calling are
What You'd Fix Next
The root-level autofix.py has an argv-parsing bug. Passing -i as the third argument is interpreted as the mode string and raises ValueError.
if len(sys.argv) >= 4:
mode_str = sys.argv[3] # "-i" lands here, RunnerMode("-i") raises ValueError
That is the smallest of the seams. The rest are documented drift between two entry points that were supposed to do the same thing.
- The argparse CLI in
src/main.pycallsclassify_intent()withoutuse_llm=True, so "auto" mode silently uses keyword heuristics only, contradicting the README. _clean_patch_diffdrops binary changes without warning, and its substring matching can drop a source file that merely mentions.venvin a comment.- The two entry points have drifted apart.
main.pysupports--modeland--keep-workspace;autofix.pydoes not. Onlyautofix.pyhas the GitHub publish flow. - There is one contributor, no license, and no description in the repository metadata.
None of these are fatal. All of them are the kind of thing that only surfaces when someone other than the author reads the code, which is the point of publishing it. The harness pattern is the interesting artifact. The polish is the work left on the bench.





