OpenSRE: The Incident-Response Agent That Refuses to Guess

A deep dive into the LangGraph loop, reroute logic, and evidence validation system that turns outage triage into a proof-driven workflow.

10 min read • View on GitHub • More from Tracer-Cloud

A wide editorial scene shows an incident commander surrounded by logs, metrics, and cloud evidence, with a mechanical decision wheel cycling through planning, investigation, diagnosis, and a loop back to planning. The image explains that OpenSRE is built to keep searching until the evidence supports a conclusion.
OpenSRE treats incident response as a loop, not a one-way script.
Key Takeaways

OpenSRE starts with a simple refusal: it will not treat a guess like an answer. That makes it more interesting than another AI wrapper around alerts, because the repo is really about how an agent decides it does not know enough yet.

The Agent That Can Say It Needs More Evidence

The project’s core move is a loop. An alert or user prompt enters the system, evidence is gathered, a diagnosis is attempted, and if the claim is still weak, the agent routes back to planning instead of pretending confidence is a substitute for proof.

Build your own AI SRE agents. The open source toolkit for the AI era ✨

N/A, GitHub Repository Description · Tracer-Cloud/opensre

That distinction matters because incident response is not a tidy workflow. Logs, traces, metrics, Slack messages, and cloud audit trails rarely agree on timing, and the useful clue is often the one that changes the frame entirely.

Problem shapeLinear automationOpenSRE
Missing evidenceKeeps moving anywayLoops back to plan again
Tool choiceFixed playbookReroutes based on clues
Final answerWhatever the script producedOnly what can be validated
EvaluationAd hoc demosSynthetic incident tests

Inside the Investigation Graph

OpenSRE’s architecture is built around LangGraph in `app/pipeline/graph.py`, with separate paths for chat mode and investigation mode. The interesting part is the conditional edge out of diagnosis, which lets the system decide whether to publish a conclusion or go back and search for missing proof.

The key idea is the conditional loop: diagnosis can publish, or it can force another planning pass when evidence is thin.

def route_investigation_loop(state):
    if state.is_healthy:
        return "publish"
    if not state.evidence_validated:
        return "plan_actions"
    return "publish"

That loop is not just a control-flow trick. It is the project’s philosophy made executable: the system should know when to stop, when to ask for more evidence, and when to change tools because the current path is not producing the right signals.

A close-up desk scene shows two trays of investigation tools. One tray contains the obvious sources, like logs, metrics, and traces. The other tray is revealed behind a pivot sign and contains a cloud audit key, a Lambda log sheet, and a Grafana service map, showing how the agent changes its toolset when a clue points elsewhere.
The reroute logic is really a change of frame, not just another tool call.

Planning Is Where the Agent Learns to Reroute

The planning node is where OpenSRE stops being a generic agent framework and starts acting like an SRE. The code looks for clues such as S3 audit evidence or a Grafana service name, then pivots to the next best evidence source instead of exhausting the same obvious path.

That reroute behavior is supported by a tool budget, which matters more than it sounds. The budget makes the system finite, discourages endless loops, and forces the planner to spend each step like a scarce resource.

tool_budget = 10

if detect_reroute_trigger(evidence):
    tools = select_new_toolset(evidence)
else:
    tools = default_tools

Parallel Execution Without Pretending the Cloud Is Reliable

Once the plan is set, OpenSRE fans out work in parallel. That choice fits the reality of production APIs, where one service may respond, another may throttle, and a third may throw a transient error for no interesting reason at all.

The implementation leans on retries and backoff because resilience is part of the product, not just the plumbing. A serious incident system has to expect noisy infrastructure, not a sanitized demo environment.

with ThreadPoolExecutor() as executor:
    futures = [executor.submit(run_action, action) for action in actions]
    results = [future.result() for future in futures]

The Final Gate: Prove the Root Cause or Loop Again

The final diagnosis node is where OpenSRE earns its credibility. It can short-circuit if the system is clearly healthy, and it can validate claims against gathered evidence before publishing anything that sounds definitive.

That is the strongest trust signal in the repo. A lot of agents can produce a root-cause-shaped paragraph. Far fewer can check whether the paragraph is actually grounded in the telemetry.

We do that because SWE-bench<sup>1</sup> gave coding agents scalable training data and clear feedback. Production incident response still lacks an equivalent.

That line explains the benchmark mindset behind the repo. OpenSRE is not only trying to solve incidents, it is trying to create the evaluation surface that makes incident-solving agents measurable.

OpenSRE vs. the Usual Incident Stack

LayerTraditional stackOpenSRE
AlertingPage the right peopleStart the investigation loop
ObservabilityShow telemetryReason across telemetry sources
AI agent frameworksGeneral purpose orchestrationVerticalized SRE investigation
Confidence modelHuman judgment after the factEvidence validation before publication

The project does not replace observability tools or incident managers. It sits above them as a reasoning layer, which is a more precise and more ambitious claim than trying to own the whole workflow.

Why the Synthetic Tests Matter

The synthetic incident suite is the part that suggests this repo wants to be more than a clever prototype. By creating ground truth scenarios, OpenSRE is trying to benchmark agent behavior in the same way coding agents were benchmarked against SWE-bench.

That choice is quietly important. In incident response, the hard problem is not generating prose. It is determining whether the system can actually navigate uncertainty, identify the missing evidence, and stop only when the conclusion is earned.