autoresearch-ai-plugin: When Claude Code Becomes an Autonomous Experimenter
A Claude Code plugin that edits, measures, reverts, and remembers. It uses Git, JSONL, and confidence scoring to turn optimization into a closed-loop research system.
- This plugin treats software optimization as a research loop, not a chat session.
- Git and JSONL give the agent memory, rollback, and continuity across resets.
- Confidence scoring keeps the system from mistaking noise for improvement.
- The ML skill shows the same loop can stretch from app code to GPU training.
The loop that keeps going
The big idea here is not autocomplete. It is a machine that can run experiments end to end: edit code, commit the change, run a benchmark, read the metric, decide whether the result is real, then keep or revert the patch. That turns Claude Code from a helper into an optimizer with state.
The repo says the quiet part plainly: “Edit code → commit → run benchmark → measure metric → keep improvement or revert → repeat forever.” That is the entire product in one sentence. Everything else is scaffolding around that loop.
Autonomous Experiment Loops for Claude Code — Let AI optimize while you sleep
Why memory is the breakthrough
The memory file, autoresearch.jsonl, is the part that changes the feel of the whole system. It is not just persistence. It is continuity across context resets, so the agent does not start every session from amnesia.
That matters because optimization produces a lot of dead ends. If the system only remembered the final winner, it would repeat the same failed ideas every time the context window refreshed. By keeping structured notes on what it tried and why it lost, the plugin builds a working history instead of a pile of discarded guesses.
The README frames the ambition in one more line: “Works for any optimization target: LLM training loss, test speed, bundle size, build time, Lighthouse scores, and more.” The point is not the metric list. It is that the memory model is generic enough to survive across domains.
Git as the memory and the undo button
This is the repo’s sharpest move. Instead of inventing a custom state layer, it uses Git as the experiment journal and the rollback mechanism. Each commit becomes a snapshot of the agent’s hypothesis, and each revert becomes a clean failure state.
| Concern | Reactive assistant | autoresearch-ai-plugin |
|---|---|---|
| State | Mostly conversational context | Git commits plus JSONL history |
| Failure handling | User manually fixes the mess | Automated revert to the previous experiment |
| Memory across sessions | Fragile unless re-prompted | Persistent and append-only |
| User role | Reviewer and prompt writer | Operator of an optimization campaign |
| Output | A code change | A ranked sequence of measured experiments |
How the agent decides what counts as a win
Measurement is where the system stops being clever and starts being disciplined. The plugin expects benchmark output in a strict format, like METRIC name=value, so the loop can parse results without guessing what the code meant.
That constraint sounds small, but it is the boundary that makes the whole project work. If the target cannot be measured repeatably, the agent can still edit code, but it cannot optimize anything. The repository’s confidence scoring exists to keep the loop honest when measurements wobble.
METRIC validation_loss=1.842
METRIC eval_ms=412.7
METRIC bundle_size_kb=183.4
The point of the MAD-based noise handling is simple: a tiny gain may be fake. If the system treats every faster run as a victory, it will learn to chase randomness. By comparing against the noise floor, it only promotes changes that clear a meaningful threshold.
| Signal | Looks like | Actually means |
|---|---|---|
| Small timing improvement | Maybe better | Could be ordinary noise |
| Large repeatable improvement | Likely real | Worth keeping |
| Reverted change | Failure | Still valuable if the reason is stored |
| Simpler code at equal performance | Neutral to humans | A win for maintainability |
The ML skill shows the ceiling
The autoresearch-ml skill is the most revealing extension in the repo. It takes the same loop and points it at GPU-bound model training, where wall-clock time, attention kernels, and training recipes become the things under optimization.
That makes the project feel broader than a Claude Code plugin. It becomes a template for autonomous performance work, whether the target is a web app, a test suite, or an LLM training run that only gets a few minutes to prove itself.
| Dimension | General autoresearch | autoresearch-ml |
|---|---|---|
| Target | Any measurable developer metric | LLM training and GPU optimization |
| Constraint | Benchmark repeatability | Fixed time budgets and wall-clock fairness |
| Tools | Git, JSONL, shell scripts | Plus PyTorch, Flash Attention 3, training templates |
| Audience | Software teams | ML practitioners and GPU tinkerers |
What makes this different from a normal coding assistant
Claude Code, Copilot, and Cursor are reactive tools. They help you write or revise code when you ask. This plugin is closer to an experiment manager. It does not wait for a new prompt each time. It keeps running until the metric says stop.
That distinction matters. A coding assistant helps you act faster. An experiment loop helps you discover better answers with less supervision. One is a productivity tool. The other is a search process.
| Tool type | Primary mode | Persistence | Best for |
|---|---|---|---|
| Reactive assistants | Respond to prompts | Weak or session-bound | Drafting and local edits |
| Standalone research loops | Automate experiment search | Strong within their own runtime | Model tuning and benchmarks |
| autoresearch-ai-plugin | Run optimization inside Claude Code | Git and JSONL backed | Ongoing software and ML optimization |
The safety rail is the existing developer toolchain
The safety story is refreshingly unspecial. Instead of inventing a proprietary guardrail stack, the repo leans on things developers already understand: append-only logs, Git history, and file protection scripts. That keeps the agent inside a system that is inspectable by default.
There is real elegance in that choice. The plugin does not make autonomy safe by hiding it behind a new abstraction. It makes autonomy safer by forcing it to leave evidence in tools the team can audit.
The result is a system that can explore aggressively without becoming opaque. If an experiment looks suspicious, the trail is there. If a result needs to be replayed, the commit is there. If the agent has forgotten why it rejected a path, the JSONL record is there.
Why this matters now
This repo points at a broader shift in AI tooling. The category is moving from copilots that respond to prompts toward systems that can run campaigns, compare outcomes, and preserve lessons. That is a very different product shape.
`autoresearch-ai-plugin` is still small, but the idea is crisp. It shows how far you can get by combining a few familiar primitives and refusing to let the agent forget what it learned. That makes it less like a novelty plugin and more like a prototype for a new operating mode.





