autoresearch-ai-plugin: When Claude Code Becomes an Autonomous Experimenter

A Claude Code plugin that edits, measures, reverts, and remembers. It uses Git, JSONL, and confidence scoring to turn optimization into a closed-loop research system.

8 min read • View on GitHub • More from proyecto26

A wide editorial scene of a developer toolchain turned into a laboratory. A looping mechanism built from a Git commit graph, a JSONL log, and a benchmark gauge sits on a white workbench while a hand labeled Claude Code places a code patch into the loop. The image explains that the plugin treats optimization as an experiment cycle, not a one-shot code edit.
The plugin turns ordinary developer tools into an autonomous research loop: patch, measure, decide, and remember.
Key Takeaways

The loop that keeps going

The big idea here is not autocomplete. It is a machine that can run experiments end to end: edit code, commit the change, run a benchmark, read the metric, decide whether the result is real, then keep or revert the patch. That turns Claude Code from a helper into an optimizer with state.

The repo says the quiet part plainly: “Edit code → commit → run benchmark → measure metric → keep improvement or revert → repeat forever.” That is the entire product in one sentence. Everything else is scaffolding around that loop.

The loop is simple, but the memory and confidence layer make it durable. A reverted experiment still changes future behavior because the reason it failed is saved.

Autonomous Experiment Loops for Claude Code — Let AI optimize while you sleep

proyecto26/autoresearch-ai-plugin README, Project Documentation · proyecto26/autoresearch-ai-plugin

Why memory is the breakthrough

The memory file, autoresearch.jsonl, is the part that changes the feel of the whole system. It is not just persistence. It is continuity across context resets, so the agent does not start every session from amnesia.

That matters because optimization produces a lot of dead ends. If the system only remembered the final winner, it would repeat the same failed ideas every time the context window refreshed. By keeping structured notes on what it tried and why it lost, the plugin builds a working history instead of a pile of discarded guesses.

The README frames the ambition in one more line: “Works for any optimization target: LLM training loss, test speed, bundle size, build time, Lighthouse scores, and more.” The point is not the metric list. It is that the memory model is generic enough to survive across domains.

Git as the memory and the undo button

This is the repo’s sharpest move. Instead of inventing a custom state layer, it uses Git as the experiment journal and the rollback mechanism. Each commit becomes a snapshot of the agent’s hypothesis, and each revert becomes a clean failure state.

ConcernReactive assistantautoresearch-ai-plugin
StateMostly conversational contextGit commits plus JSONL history
Failure handlingUser manually fixes the messAutomated revert to the previous experiment
Memory across sessionsFragile unless re-promptedPersistent and append-only
User roleReviewer and prompt writerOperator of an optimization campaign
OutputA code changeA ranked sequence of measured experiments
A close-up editorial scene of a mechanical filing cabinet built from Git history. One drawer holds a green commit card, another holds a red reverted card, and a thin ledger strip labeled JSONL runs through both drawers. The image explains how the repo uses Git for snapshots and rollback while preserving the reasons behind discarded experiments.
Git is not just version control here. It is the experiment log, the undo button, and the trust anchor.

How the agent decides what counts as a win

Measurement is where the system stops being clever and starts being disciplined. The plugin expects benchmark output in a strict format, like METRIC name=value, so the loop can parse results without guessing what the code meant.

That constraint sounds small, but it is the boundary that makes the whole project work. If the target cannot be measured repeatably, the agent can still edit code, but it cannot optimize anything. The repository’s confidence scoring exists to keep the loop honest when measurements wobble.

METRIC validation_loss=1.842
METRIC eval_ms=412.7
METRIC bundle_size_kb=183.4

The point of the MAD-based noise handling is simple: a tiny gain may be fake. If the system treats every faster run as a victory, it will learn to chase randomness. By comparing against the noise floor, it only promotes changes that clear a meaningful threshold.

SignalLooks likeActually means
Small timing improvementMaybe betterCould be ordinary noise
Large repeatable improvementLikely realWorth keeping
Reverted changeFailureStill valuable if the reason is stored
Simpler code at equal performanceNeutral to humansA win for maintainability

The ML skill shows the ceiling

The autoresearch-ml skill is the most revealing extension in the repo. It takes the same loop and points it at GPU-bound model training, where wall-clock time, attention kernels, and training recipes become the things under optimization.

That makes the project feel broader than a Claude Code plugin. It becomes a template for autonomous performance work, whether the target is a web app, a test suite, or an LLM training run that only gets a few minutes to prove itself.

DimensionGeneral autoresearchautoresearch-ml
TargetAny measurable developer metricLLM training and GPU optimization
ConstraintBenchmark repeatabilityFixed time budgets and wall-clock fairness
ToolsGit, JSONL, shell scriptsPlus PyTorch, Flash Attention 3, training templates
AudienceSoftware teamsML practitioners and GPU tinkerers

What makes this different from a normal coding assistant

Claude Code, Copilot, and Cursor are reactive tools. They help you write or revise code when you ask. This plugin is closer to an experiment manager. It does not wait for a new prompt each time. It keeps running until the metric says stop.

That distinction matters. A coding assistant helps you act faster. An experiment loop helps you discover better answers with less supervision. One is a productivity tool. The other is a search process.

Tool typePrimary modePersistenceBest for
Reactive assistantsRespond to promptsWeak or session-boundDrafting and local edits
Standalone research loopsAutomate experiment searchStrong within their own runtimeModel tuning and benchmarks
autoresearch-ai-pluginRun optimization inside Claude CodeGit and JSONL backedOngoing software and ML optimization

The safety rail is the existing developer toolchain

The safety story is refreshingly unspecial. Instead of inventing a proprietary guardrail stack, the repo leans on things developers already understand: append-only logs, Git history, and file protection scripts. That keeps the agent inside a system that is inspectable by default.

There is real elegance in that choice. The plugin does not make autonomy safe by hiding it behind a new abstraction. It makes autonomy safer by forcing it to leave evidence in tools the team can audit.

The result is a system that can explore aggressively without becoming opaque. If an experiment looks suspicious, the trail is there. If a result needs to be replayed, the commit is there. If the agent has forgotten why it rejected a path, the JSONL record is there.

Why this matters now

This repo points at a broader shift in AI tooling. The category is moving from copilots that respond to prompts toward systems that can run campaigns, compare outcomes, and preserve lessons. That is a very different product shape.

`autoresearch-ai-plugin` is still small, but the idea is crisp. It shows how far you can get by combining a few familiar primitives and refusing to let the agent forget what it learned. That makes it less like a novelty plugin and more like a prototype for a new operating mode.