cli: Circuit Breaker Labs Built a Red-Team Tool With No Attack Logic
The open-source cbl CLI probes chatbots for mental-health risk across multi-turn conversations. The adversarial brain lives on the server, and the client is a careful, privacy-conscious relay to your model.
- The cbl CLI ships zero attack logic: every adversarial probe, escalation decision, and scoring rule lives on Circuit Breaker Labs' server, and the open-source client is a relay.
- Mental-health risk in a chatbot is not a keyword match; it is coded language and accumulated context across turns, which is why the evaluation loop is multi-turn and adaptive.
- One WebSocket loop serves both single-turn and multi-turn evaluations through an EvaluationMode trait, so the transport logic is testable without a live socket.
- The LiveKit voice path lets the same hosted service probe agents that never produce a text transcript at all.
The Client That Holds No Weapons
Install cbl, point it at a model, and run an evaluation. The binary opens a WebSocket to Circuit Breaker Labs' backend, receives a stream of completion requests, forwards each one to your model, and ships the answers back. That is the whole job.
There is no probe library in the repository. No attack templates. No scoring function. No list of crisis phrases. The adversarial intelligence, the part you would expect a red-team tool to guard jealously, never leaves the server. The client is a pipe.
That inversion is deliberate, and it buys something concrete. Your model endpoint, your API tokens, and your customers' conversation data stay on your machine. The server sees a target label and the model's replies, not your credentials or your infrastructure. For a tool that companies point at their own production chatbots, that is the argument.
// The shape of the relay loop in evaluations/engine.rs, simplified.
// No attack logic here. Classify the frame, forward it, send the answer back.
loop {
match classify_transport_message(frame) {
TransportEvent::Text(text) => {
let request = mode.parse_text(&text)?;
completion_tasks.spawn(async move {
let response = provider.complete(request).await?;
Ok(OutboundEvent::Protocol(mode.serialize_response(&response)?))
});
}
TransportEvent::Ping => { /* reply with a Pong */ }
TransportEvent::Close => break,
TransportEvent::Ignore => {}
TransportEvent::ReadError(e) => return Err(e),
}
}
The comment on that loop is the article in miniature. The client classifies what arrived, hands the payload to whichever evaluation mode is active, and lets a provider turn it into a model call. Everything that decides what to ask lives on the other end of the wire.
Why Keyword Scanning Fails
The domain is why the design has to work this way. A chatbot that fails a vulnerable user rarely fails on an obvious phrase. It fails on slang, on typos, on a coded half-sentence that only reads as distress in the context of the four turns before it. A model can respond to each message in isolation and still miss the pattern entirely.
That is the case for adaptive, multi-turn testing, and it is the reason a static probe list would be the wrong tool. The evaluation has to read the model's answer and decide what to say next based on it. Which means the intelligence cannot be a fixed file shipped in a binary. It has to live somewhere that can generate and revise on the fly.
For some reason, in this industry, safety is just taken at face value from the developers. And I think a lot of these big companies are taking efforts to make sure that their products are safe, but it’s really hard to see your own blind spots,
Who Built It and Why
Circuit Breaker Labs launched publicly in September 2025. The co-founders, Shirali Nigam and Arul Nigam, pointed the first product at a single, uncomfortable category: suicide-ideation scenarios. Not a broad sweep of model-quality checks. One domain, tested hard.
Treat that as a choice rather than a limit. General-purpose scanners spread their coverage across prompt injection, data leakage, jailbreaks, and toxicity. Circuit Breaker Labs went the other way. The bet is that depth in one high-stakes domain beats shallow coverage everywhere, especially when the failure mode is a person in crisis being met with the wrong response.
The autonomy is the second half of the pitch. Nigam framed the API as something you run at volume rather than once.
Because it’s an API that’s autonomous, you can run like hundreds of thousands of test cases with it,
One Engine, Two Conversations
Open the engine and the cleverness is almost entirely structural. Single-turn evaluations and multi-turn evaluations share one WebSocket loop. They differ only in the message types they send and the progress they report, and that difference is captured by a trait called EvaluationMode.
The trait names four associated types (request, final response, optional message, progress message) and a handful of methods for serializing requests, parsing incoming text, and emitting progress. SingleTurnMode implements it. Multi-turn implements it too. The loop never knows which one it has.
Two details make the file worth reading. First, incoming frames are turned into a TransportEvent enum by a pure function. Sockets are messy; a pure classifier over raw frames is not. The unit under test is the classification and serialization logic, and it needs no network to run. Second, outbound work is described by an OutboundEvent enum that gets converted into a tungstenite::Message at the edge. Same discipline: keep the decision separate from the transport.
Concurrency runs through a JoinSet. When a completion request arrives, the engine spawns a task rather than blocking the read loop, so several requests can be in flight against the model at once. The socket keeps draining while the provider works.
The Escalation Loop
Here is the part the client refuses to do, and the part that makes the product interesting. The server generates a probe, the model answers, and the server reads that answer to choose a subtler probe for the next turn. The blunt opening questions come first. What follows depends entirely on what the model just said.
The repository does not ship this logic. It ships the flags that shape it: a --threshold, a --maximum-iteration-layers, and a --variations setting. Those names describe a loop that iterates, escalates, and varies its approach. The design points to an adaptive generator on the server. The code excerpts do not contain the generator itself, so read it as the reported methodology rather than a line you can open.
Testing a Voice With No Transcript
Most red-team tools assume text. They send a prompt and score a string. A voice agent produces neither. It produces audio, in real time, with no transcript sitting around to inspect.
The CLI takes a different route for this. It connects to a customer-managed voice agent over LiveKit, plays caller audio into the agent, and streams the agent's audio back to the API for transcription and scoring. The API receives a target label and the audio. The connection configuration and the customer's credentials stay in the CLI, on the customer's side.
That is a real gap closed. A team testing a phone-based crisis line now has a path that matches how the product actually behaves, instead of a text proxy that quietly changes the thing being measured. Keep the claims to what the docs state: the CLI connects, plays caller audio, streams the agent's audio for transcription and scoring, and holds the credentials itself.
Where It Fits
The open-source red-team field is crowded and good. garak, PyRIT, promptfoo, Inspect AI, and DeepEval all matter. The honest comparison is not about which is best. It is about where the adversarial intelligence lives and what it is aimed at.
| Project | Approach | Domain focus | Where the intelligence lives | Hosted or self-hosted |
|---|---|---|---|---|
| Circuit Breaker Labs cbl | Autonomous, multi-turn, agentic | Mental-health risk scenarios | Server-side; client is a relay | Hosted API, open-source client |
| garak (NVIDIA) | Probe and detector library | Broad failure classes | In the open-source tool | Self-hosted |
| PyRIT (Azure) | Orchestrated adversarial attacks | General-purpose red teaming | In the open-source framework | Self-hosted |
| promptfoo | Config-driven test suites | Model and app testing, CI | In the open-source tool | Self-hosted (or hosted option) |
| Inspect AI (UK AISI) | Tasks, solvers, scorers | Flexible evaluation substrate | In the open-source framework | Self-hosted |
| mpathic | Human-led red teaming | Behavioral-health AI safety | With licensed clinician reviewers | Commercial service |
Read the table down the middle. Circuit Breaker Labs is the only row where the adversarial logic sits on a server you do not control and the domain is mental health. mpathic is the closest neighbor by domain, but its red team is people, not an autonomous loop. The rest put the attack logic in the open-source tool and spread it across broad categories.
This is an early project. The repository carried five stars and nine open issues in the snapshot reviewed here. State that plainly: small, young, and not a signal either way about the quality of the evaluations it runs.
What It Won't Do
The evaluation service is hosted, not open source. The client is MIT-licensed and you can read every line, but the generator, the scorer, and the scenario packs are on the far side of the WebSocket. If you need to audit the attack logic itself, this is not the tool for that.
Access runs through an API key you request by email. The value of the whole thing depends on how relevant and how valid the domain scenarios are, and on your willingness to send model replies to a hosted service. The CLI's privacy design keeps your credentials and endpoints local, but the conversation still travels.
None of that is a flaw. It is the trade the project made on purpose: a thin, auditable, fast client that refuses to hold the weapons, pointed at a domain where the wrong answer has real cost.





