code-fix-harness: AutoFixer: The Model Proposes, the Test Suite Decides

A local 7B model writes the patch. A fail-to-pass rule decides if it ships. Inside a Python harness that treats AI code generation as a reliability problem, not a capability race.

11 min read • View on GitHub • More from sameerlokhande

A close-up editorial illustration of two hands above a workbench. On the left, a small ink-stained hand holds up a single sheet of dense, crooked handwriting labeled remediation.patch at the top edge. On the right, a steady ringed hand holds a heavy stamp poised over the paper, casting a hard rectangular shadow. Behind them, a workshop wall with a single locked door and a stopwatch on a nail. The image illustrates how the model proposes a patch and the verifier decides whether it ships.
The model offers a patch. The verifier holds the stamp. Nothing reaches the source tree until the test agrees.
Key Takeaways

In Fix Mode, the Test Has to Fail First

Before the model touches a file, AutoFixer runs the failing test and confirms it fails. After the patch, it runs the same test again and confirms it passes. Fail-to-pass, or the patch does not ship.

A second, quieter rule runs alongside it: pass-to-pass. The existing tests cannot break. Two sentences turn an unreliable code generator into a reviewable artifact.

The physical product of the whole machine is a single file called remediation.patch. Nothing reaches your source tree until that file exists, and nothing reaches GitHub until you say so.

The loop is the point. A failed verification is not an error, it is the next input.

Why Build This Around a 7B Model?

AutoFixer defaults to qwen2.5-coder:7b and runs entirely on Ollama. No cloud keys, no API spend, no data leaving the machine.

The reason is not cost. It is that the harness, not the model, is where the reliability lives. A frontier model with a loose harness is still a coin flip. A small model with a strict harness is a repeatable process.

The defaults make the bet concrete: a 7B coder model, a max of 3 attempts, and a cap of 12 iterations. If the model cannot produce a passing patch inside that budget, the run ends and the diagnostic is yours to read.

Five Strategies for a Model That Won't Follow Instructions

Small local models do not reliably emit FILE: path headers or valid JSON. So _extract_code_files does not ask them to. It tries five extraction strategies in order and stops at the first one that yields code.

  1. FILE: or ### headers followed by a fenced block.
  2. Paths embedded in fence language tags, such as ```python:path/to/file.py.
  3. JSON objects or arrays with path and content keys, fenced or bare.
  4. Any fenced block, assigned to a known default filename, with the largest block winning.
  5. Raw text that looks like Python, with no fences at all.

The cascade favors recall over precision. Heuristics like "largest block wins" can silently pick the wrong block, and the raw-text strategy can accept prose containing the word import as code. The verifier is what keeps that from becoming a correctness problem.

A medium-wide editorial illustration of a sloped wooden trough. A chaotic river of text, braces, quotes, and half-formed function signatures pours in from the top. Five progressively finer wire sieves sit along the trough at angles, catching larger chunks first and smaller JSON-shaped pebbles later, until a single clean sheet of paper drops into a wooden bin labeled parsed files. A small apron-wearing figure watches with a clipboard. The image explains the five-strategy parsing cascade that extracts structured edits from an unpredictable model.
Five sieves, ordered coarse to fine. The parser would rather catch a wrong block than miss a real one, because the verifier runs next.
# _extract_code_files: five strategies, first hit wins
# 1. FILE: path / ### path headers followed by a fenced block
# 2. Paths embedded in fence language tags (```python:path/to/file.py)
# 3. JSON objects or arrays with "path" and "content" keys
# 4. Any fenced block assigned to a default filename (largest wins)
# 5. Raw Python-looking text with no fences

for strategy in (headers, fence_paths, json_objects, any_fence, raw_python):
    files = strategy(text, default_filename)
    if files:
        return files  # stop at the first strategy that yields code

When pytest Says 'No Tests Collected,' the Harness Writes a Lecture

The verifier does more than return pass or fail. When pytest exits with code 5, meaning no tests were collected, format_diagnostic() appends a message to the model's next prompt.

if result.returncode == 5:  # pytest: no tests collected
    diagnostic += (
        "\nNo tests were collected. Create a test file that imports "
        "the real repository modules. Trivial tests like `assert True` "
        "without source changes will be rejected."
    )

The message instructs the model to create a test file that imports real repository modules and warns that trivial tests like assert True without source changes will be rejected. That is the threat model in one sentence: the harness assumes the model will try to game it.

Diagnostics as prompt engineering is a pattern worth stealing. The verifier writes guidance back into the model's context at the point of failure, which is exactly when the model is most likely to change course.

New Google paper shows that agents score higher when each task gets a specialized harness from a shared family, not 1 global harness. An agent harness is the prompts, tools and control code around a model. Most tuning methods evolve 1 global harness, though a fix for 1 task htt

Rohan Paul, rohanpaul_ai · @rohanpaul_ai on X

Eight Files, Twelve Kilobytes, and a Cap of 28 Messages

_collect_workspace_source_files walks the repo, filters by extension, sorts shallow files first, and returns at most 8 files of 12KB or less. The conversation history is capped at 28 messages.

Every input the model sees is deliberately bounded. This is the opposite of stuffing the whole repo into the context window. A budget forces choices, and choices are easier to reason about than a dump.

The trade-off is real. A fix that depends on a file outside the budget will never be attempted, and the harness will report a failure that looks like the model's fault. But a bounded failure is still a failure you can inspect.

Cloud Agents, Static-Analysis Autofixers, and Everything in Between

AutoFixer is not Cursor, not Claude Code, not CodeQL autofix, not Semgrep. It is a local, sandboxed, test-gated patch generator with no tool-calling loop and no autonomous commit.

The table's payoff is the "How success is decided" column. Only AutoFixer and the linters are deterministic. Everything else relies on the model or the user to declare victory.

Where the model runsHow success is decidedWhat it outputsHuman gateFailure mode
AutoFixer (code-fix-harness)Local (Ollama, qwen2.5-coder:7b)A fail-to-pass test run in a sandboxed venvA remediation.patch fileTwo prompts: apply to source, publish to GitHubPatch rejected, diagnostic fed back to the model (up to 3 attempts)
Cloud coding agents (Cursor, Claude Code)Vendor cloudModel self-reports success or user reviewsLive edits in the working treeUsually noneFlaky edits the user has to catch
Static-analysis autofix (CodeQL, DeepSource)Cloud or localStatic re-scan of the alertSuggested patch in the scanner UIReview before applyFixes that pass the scan but break tests
Rule-based linters (Semgrep, ESLint)LocalDeterministic rule matchWarnings and occasional auto-fixesNoneFalse positives and missed semantic bugs

The point is not that AutoFixer wins. It is that the harness pattern is orthogonal to the model. Line the trade-offs up and the choice becomes legible: you are picking where the model runs, who declares success, and how much damage a wrong answer can do.

Building an AI agent? Don't start with a 20-step workflow full of if/else branches. Start with a harness that already works. The order LangChain's Harrison Chase recommends: 1. Start with a harness. Claude Code, Codex or his own deepagents. The agent loop and tool calling are

aiartgallerie, aiartgallerie · @aiartgallerie on X

What You'd Fix Next

The root-level autofix.py has an argv-parsing bug. Passing -i as the third argument is interpreted as the mode string and raises ValueError.

if len(sys.argv) >= 4:
    mode_str = sys.argv[3]  # "-i" lands here, RunnerMode("-i") raises ValueError

That is the smallest of the seams. The rest are documented drift between two entry points that were supposed to do the same thing.

None of these are fatal. All of them are the kind of thing that only surfaces when someone other than the author reads the code, which is the point of publishing it. The harness pattern is the interesting artifact. The polish is the work left on the bench.