security-audit-skill: Cloudflare Built a Security Auditor That Tries to Disprove Itself
security-audit-skill is not a scanner. It is a set of prompts, two validators, and a rule that every finding must survive a fresh adversary before it counts.
- Most of security-audit-skill is Markdown written to be executed by a model; the only deterministic code is two Node.js validators that decide whether an agent's output counts.
- Every candidate finding is handed to a fresh verifier that did not hunt it and never sees another verifier's conclusion, so independence is enforced by process topology rather than a polite request to be objective.
- The model is never trusted to produce valid output: a finding that fails schema validation is not a finding, no matter how persuasive the prose behind it.
- Uncertainty is a first-class verdict, and a finding keeps a stable fingerprint as it moves between confirmed, needs_validation, and rejected states.
It Ships Prompts, Not a Scanner
You would not guess it from the language bar. GitHub reports cloudflare/security-audit-skill as 100 percent JavaScript, and that number is technically true and substantively misleading. The JavaScript is two .cjs validators and their tests. Everything else is Markdown.
SKILL.md, RECONNAISSANCE.md, HUNTING.md, VALIDATION-AND-REPORTING.md, ATTACK-CLASSES.md, and ten domain files covering memory safety, AI and LLM handling, web auth, client-side code, supply chain, cloud deployment, RPC, resource exhaustion, data isolation, and desktop and mobile IPC. That is the product. Prose written to be executed by a model.
Install is one line:
npx skills add https://github.com/cloudflare/security-audit-skill --skill security-audit
Loading the skill does not start an audit. Guidance mode is the default, and full audit mode requires an explicit request for a codebase audit, a pen test, or a report artifact. If the request is ambiguous, the skill asks one focused question before it creates any files. That single rule sidesteps the failure mode of agentic tools that quietly write artifacts into your repository.
The architecture has three layers, and the split is the whole idea. The methodology lives in Markdown phase files. The contract lives in report-schema.json. The enforcement lives in the validators. The model supplies judgment. The code supplies the veto.
With the recent wave of vulnerability disclosures, it’s a good time to take a closer look at your own code. Turn your coding agent into a team of security auditors with the open-source skill that seeded Cloudflare’s vulnerability discovery harness. The 6 phase approach: 1. Map the codebase 2. Hunt for vulnerabilities 3. Challenge each candidate 4. Record the findings 5. Verify the findings 6. Report what’s confirmed, rejected, or still needs validation Start your hunt here: github.com/cloudflare/securi…
A Fresh Agent Tries to Disprove Every Finding
The most interesting mechanic is not the hunting. It is what happens to a candidate after it is found.
Every unique candidate goes to a fresh verifier. That verifier did not hunt it. It must not receive another verifier's conclusion. The instruction is blunt: "You did not write this candidate. Try to refute it from repository source and bounded local evidence."
Independence here is not a personality trait requested of the model. It is a property of the process graph. The hunter and the verifier are separate agents with separate context, and the only thing that crosses between them is the candidate itself. The hunter's reasoning is withheld on purpose, because a conclusion is the easiest thing to anchor on.
The verifier returns one of three verdicts. confirmed requires a complete source trace and a bounded observed result. needs_validation carries an exact unresolved fact and no severity. rejected records a disproved candidate. That third state matters as much as the first: a finding that has been actively refuted is a recorded fact, not a silent deletion.
One more rule holds the whole thing together. Verifiers must preserve the fingerprint for the same source-derived root cause across every state. A finding that starts as needs_validation and later earns confirmed keeps its identity, so a later run can recognize it as the same root cause instead of rediscovering it from scratch.
Only confirm established boundary failures.
Six Phases, One Writer
The pipeline runs in six phases: reconnaissance, coverage-led hunting, adversarial candidate validation, structured findings, independent verification of final records, and target-neutral reporting. The flow is conventional. The coordination rules are not.
Reconnaissance fans out to four read-only research agents. One maps the product, stack, and build and test commands, with an explicit instruction not to run them. One maps principals, authority, and controls. One maps entry surfaces, copies, and sinks. One checks local execution and deployment visibility, including whether the host can actually enforce a sandbox. Every recon agent returns repository-relative file:line references and writes nothing.
That last part is a rule repeated everywhere: only the parent writes shared state. Agents write into their own scratch/ directories and nowhere else. Even their artifacts/ output is parent-owned and promoted through a race-safe copy. In a field where agents routinely share a common scratch space, this is an unusually strict isolation model.
Each hunter is built from a fixed prompt contract, and the hunting method itself is a numbered procedure that turns a hunch into a falsifiable claim:
- Name the lower-trust principal and starting capability.
- Name the accepted value, action, state transition, or resource selector.
- Locate the control that should reject, bind, isolate, limit, or revoke it.
- Trace the exact source path after that decision.
- Stop at the smallest affected dummy record, wrong return value, process-integrity effect, or locally observable shared-resource effect.
- State a source-level change and regression case that enforce the invariant.
Principal, capability, control, path, bounded effect, fix. The shape is designed to be refuted. It also builds in a depth bound: stop a line of investigation as soon as the invariant is settled either way.
Coverage is tracked in a ledger with 19 required fields per unit and statuses that include planned, in_progress, covered, candidate, blocked, and deferred. A unit the budget cannot reach must be explicitly deferred with a stated reason. When hunters are capped, prioritization is deterministic: unauthenticated entry surfaces first, then boundaries protecting credentials or cross-tenant data, then prior-run gaps, with ties broken lexicographically by coverage ID. A stochastic system gets a deterministic scheduler.
The Audit Tool That Audits Itself
The paranoia is where the project earns trust. The validators treat their own inputs as hostile, because the content they check is attacker-influenced by design.
- The validator escapes its own error messages. A malicious
findings.jsoncannot inject terminal escape sequences or spoof console output, because the diagnostic text is sanitized before it is printed. - A
visibleContentregex rejects strings made only of whitespace, control characters, format characters, or default-ignorable code points. A title made of zero-width spaces fails validation. - A Unicode-scalar check rejects lone surrogates, closing another route to a plausible-looking but meaningless finding.
- Agent IDs must match
^[a-z0-9][a-z0-9_-]{0,63}$and cannot be Windows device names likecon,nul,com1, orlpt9. Lowercase is required to prevent case-fold collisions. - Hard limits cap the input at 5 MB, nesting at 64 levels, and error output at 100 messages, with strict UTF-8 decoding that fails on malformed bytes.
- The promotion procedure uses no-follow opens, re-checks file identity with
fstat, creates the destination exclusively, and never recursively copies, globs, or extracts archives. - Unsupported schema keywords are rejected rather than silently ignored, so a schema change cannot quietly weaken validation.
The promotion procedure is the sharpest example. The audited code runs inside a sandbox and could plant a symlink or a FIFO in a scratch directory to trick the parent into copying or blocking on something it should not touch. The defense is a directory-relative, re-verified copy that never reopens a file by path. It is a time-of-check-to-time-of-use defense, applied to the audit tool's own output.
Where It Sits
This is not a leaderboard. The agent-security harness landscape is a set of different answers to one question: what counts as proof? Cloudflare's answer is a structured record that survives adversarial verification and schema validation. Others answer with a crash, a revalidated match, or an expert checklist.
| Project | Approach | What counts as proof |
|---|---|---|
| cloudflare/security-audit-skill | Six-phase agent workflow with coverage ledger | A structured finding that survives adversarial verification and schema validation |
| Anthropic defending-code-reference-harness | Exploit generation for C/C++ memory bugs | A reproducible crash |
| Vercel deepsec | SAST prefilter plus LLM analysis and revalidation | A revalidated static match |
| Trail of Bits skills | Expert-oriented skill toolbox | Expert review against a specialist checklist |
| Visa visa-vulnerability-agentic-harness | 11-stage SAST-plus-LLM pipeline | An adversarially verified static finding |
| Capital One vulnhunter | Composable find, fix, verify skills | A verified find-fix-verify cycle |
| Google mantis | About 15 security-focused skills | Flexible, no fixed pipeline |
| Cisco ai-deep-sast | SAST plus LLM | A triaged static match |
These approaches are complementary, not interchangeable. SAST-plus-LLM systems offer more deterministic scanner inputs. Exploit-generation systems offer concrete runtime evidence for supported targets. Cloudflare foregrounds orchestration and evidence records rather than any particular scanner.
The Honest Limits
The project does not oversell itself, and neither should this article. Several costs are structural, not incidental.
- Cost and latency. A fresh verifier per candidate, plus a four-agent reconnaissance fan-out, trades tokens and wall-clock time for independence and auditability.
- Sandbox dependence. The skill cannot execute target code unless the host provides an OS-enforced sandbox with no external network, an allowlisted environment, and scratch-only writes. Many agent platforms cannot enforce that. The fallback is to report a blocker and stop.
- Hand-rolled schema validation. Writing a JSON Schema interpreter avoids dependencies, which matters for supply-chain-sensitive tooling, but the authors must maintain it.
- Coverage over confidence. The README reports that in the project's own test runs, a single run found roughly half the vulnerabilities found across repeated runs. That is a README-reported result, not an independent benchmark, and it is the clearest reason the design treats repeat runs as additive rather than suppressing.
There is also a practical concern raised in public discussion: security-framed audit tasks can trigger refusals from some frontier models, which guard against misuse. A commenter on Hacker News suggested a workaround of running bug-class skills without the security framing and combining results afterward. That is a claim from an individual, not an established finding, but it points at how a model's safety behavior can interact with a security workflow.
Tip for security professionals using LLMs to audit code: audit skills that explicitly frame the task as security research sometimes trigger refusals from the top OpenAI and Anthropic models because they guard against misuse. What works for me: separate skills for bug classes (and bugs in general) without the security framing, plus another skill that combines their findings to spot security bugs.
Except their validation doesn't seem to work. I've gone through both (both personally and for my company), and.. no response for weeks.
Taken together, the honest picture is a tool that is careful about what it claims and explicit about what it cannot prove. That is the point. The interesting thing about security-audit-skill is not that it finds bugs. It is that it refuses to trust the agent that found them, including in the small places where trusting would have been easier.





