Adversarial Reviewers: A System for AI Auditing AI

A single review pass is already too late. When AI writes code, tests, documentation, and configuration, another AI can review it—but only if the reviewer is designed to challenge the assumptions of the author.

This article describes a three-persona adversarial review system and the lessons learned while applying it to 131 AI-generated skills.

---

Why AI-generated work needs adversarial review

Self-review has a structural weakness: the model that produced an answer tends to preserve the same assumptions when it reviews that answer. A clean format and confident explanation can therefore hide unsafe commands, incomplete prerequisites, weak validation, or invented behavior.

The solution is not a single, more elaborate reviewer prompt. It is a small review team whose members approach the same artifact from deliberately different failure models.

The three reviewer personas

Saboteur

The Saboteur tries to break the target. It supplies malformed input, stresses concurrent operations, interrupts state transitions, and exhausts resources. Its question is not “does this look correct?” but “what sequence of actions makes it fail?”

Typical probes include:

• empty values, NUL bytes, oversized strings, and shell payloads;
• concurrent access to shared files or state;
• process termination during critical writes;
• memory, file-descriptor, and disk exhaustion.

New Hire

The New Hire has no project context and follows the instructions literally. This persona exposes undocumented dependencies, contradictory paths, ambiguous terms, and steps that only work when the reader silently supplies tribal knowledge.

The key constraint is important: ambiguity must be reported as ambiguity. The reviewer must not invent the missing step and then judge the invention.

Security Auditor

The Security Auditor maps external input to sensitive operations and checks the result against common vulnerability classes. It examines path traversal, command injection, secret leakage, unsafe permissions, and missing authorization.

Keyword matching is insufficient. Before reporting a vulnerability, the auditor must trace the value through validation and sanitization calls.

The result-processing pipeline

Independent findings enter a shared pipeline with three controls.

Forced discovery

Any CRITICAL finding enters deep review even when the other personas disagree. To prevent unsupported alarmism, it must include complete reproduction steps.

Cross-escalation

When two personas independently reproduce related LOW or MEDIUM findings, the combined issue is promoted. Two individually minor weaknesses can form a complete attack path.

Three-layer false-positive filtering

Every finding must pass:

1. Reproducibility — can it be triggered in the real environment?
2. Input validity — can a legitimate caller provide the required input, or is it already guarded?
3. Actual impact — does it cause data loss, a security breach, outage, recoverable degradation, or no user-visible effect?

This filter matters because adversarial reviewers are intentionally biased toward reporting danger. Their raw output should be treated as evidence to verify, not as a final verdict.

Evolving the attack pool

A static prompt eventually becomes predictable. We maintain an attack pool and score each vector by the number of new, reproducible failures it discovers. High-fitness vectors are retained and mutated; low-fitness vectors are removed.

Mutations vary parameters, encodings, ordering, and context. This produced repository-specific combinations that were not present in the original hand-written pool, including multi-stage encoding bypasses.

Detecting phantom work

An AI reviewer can produce a persuasive report for a bug that does not exist. We call this phantom work. Three signals help detect it:

In the 131-skill audit, 17 findings showed clear phantom-work characteristics and were removed before the final report.

Results from 131 skills

The audit extracted 159 claims and produced 417 raw findings. After filtering, 138 findings remained—a 66.9% false-positive elimination rate. Of three initial CRITICAL reports, only one survived end-to-end verification.

That result is not evidence that adversarial review failed. It shows why generation and verification must be separate stages: broad discovery improves recall, while reproducibility and impact analysis restore precision.

Design principles

1. Diversity beats repetition. Three distinct failure models cover more ground than three copies of the same expert.
2. False positives are an expected property. Build a verification pipeline instead of pretending they can be prompted away.
3. Audit the audit. A reviewer finding is another AI-generated artifact and requires evidence.
4. Evolution beats a frozen checklist. Let successful attacks adapt to the codebase and retire ineffective ones.

The practical goal is not to make one model infallible. It is to create a system in which authoring, attack, verification, and learning continuously correct one another.