Adversarial Review + Property-Based Testing: Let AI Find Logic Bugs in Your Code

Integrating Anthropic's agentic-pbt research into our adversarial code review system: using Hypothesis to auto-derive invariants and search for counterexamples, catching pure logic bugs that security, design, and runtime reviews miss.

1. 209 Tests Passed, Configuration Wiped

Last month, I was reviewing headroom's opencode registrar. 209 pytest tests all green. 30 E2E tests all passing. Looked ready to merge.

Then I tested with adversarial input — crafted a config file with JSONC comments and hit the registrar API.

The entire configuration was wiped clean.

Here's what happened: _read_json used json.loads(). When the JSONC file contained // comment lines, parsing failed. The failure didn't throw an exception — it silently returned {}. Then _write_json took that empty dict and happily overwrote the entire config file.

Provider configs, model configs, MCP configs — all gone.

All three bugs followed the same pattern: normal test cases only test normal inputs. Adversarial inputs are what expose real vulnerabilities. This later became pitfall #28 in our code review skill: 100% test pass rate ≠ no bugs.

Since then, we've built an adversarial review workflow — don't read code, don't look at diffs. Just attack the code with concurrent stress, malicious inputs, and crash recovery. It works: from silent config corruption to flock+os.replace lock losses, every high-value bug has been caught this way.

But there's a blind spot.

2. What Adversarial Review Can't Catch

Our adversarial review looks roughly like this:

Phase 1: Baseline (confirm existing tests pass)
Phase 2: Attack —
  • Concurrent chaos (8 threads × 20 read/writes)
  • Malicious input (null, NUL bytes, path traversal, shell injection)
  • Crash recovery (kill -9 mid-write)
  • Config mutation (JSONC comment variants, truncation, empty files)
  • E2E live run (fork+exec, no mocking)
Phase 3: Fix + verify
Phase 4: Dead code cleanup + pattern consistency check

This combo catches four categories of bugs: race conditions, crash recovery failures, path traversal/injection from malicious input, and design flaws (like config propagation inconsistencies). Every recent high-value bug — silent config corruption, flock lock loss, IndexError from empty commands — was found this way.

But there's a structural blind spot: pure logic bugs.

I don't mean crashes, injections, or data corruption — I mean code that is logically wrong. Like:

• numpy.random.wald sometimes returns negative numbers, but the Wald distribution's domain is positive
• slice_dictionary repeatedly returns the first chunk because the iterator never advances
• item_hash returns the same value for all list inputs because .sort() returns None, not the sorted list
• calculate_label_colors produces invalid HSL color codes due to a missing closing bracket

These bugs share a common trait: they don't crash, don't corrupt data, don't trigger security vulnerabilities — they're just wrong. Concurrent testing can't find them (deterministic errors). Malicious input can't find them (they fail on normal input). Crash recovery can't find them (not I/O operations).

They only surface under one condition: someone explicitly knows "what properties this function should satisfy" and feeds data to verify those properties.

That's what property-based testing does.

3. Anthropic's agentic-pbt

In January this year, Anthropic published a paper: using a Claude-driven agent to automatically write property-based tests for Python packages, then using the Hypothesis framework to search for counterexamples. They ran 984 targets across 100+ PyPI packages — 56% of the reports were genuine bugs, 32% worth filing issues for.

This aligns perfectly with our adversarial review philosophy — don't find bugs by reading code; find them by running tests. But the approach differs: we design attack scenarios manually; they let AI derive mathematical properties.

Their agent workflow:

1. Read target code (introspection + trace import chains)
2. Derive "claimed properties" from the code
   — Extract evidence from docstrings, type annotations, function names, caller usage
   — Only test properties the code explicitly claims; no assumptions
3. Write property-based tests with Hypothesis
4. Run pytest (500 random examples)
5. Three-stage triage:
   a) Is it reproducible?
   b) Is the input legitimate? (check if callers perform precondition checks)
   c) Does it affect real users?
6. Self-reflection loop: if all tests pass, check if properties are too weak
   (One agent discovered a passing test wrapped in try-catch)
7. Output standardized bug report (with reproduction script + git diff fix)

A few design decisions in this workflow that I really appreciate.

First, only test "claimed" properties. The agent doesn't guess what mathematical relationships a function should satisfy — it extracts contracts the code has committed to, from docstrings and type annotations. "This function should return a sorted list" — if the docstring doesn't say that, the agent won't test for it. This dramatically reduces false positives — the paper's 44% false positive rate isn't low, but if you let AI freely assume "this function should satisfy…" it could double.

Second, the self-reflection loop. This is what separates a PBT agent from a regular fuzzer. Traditional fuzzers only care about crashes; a PBT agent also asks "are my tests too weak?" The paper gives an example — an agent wrote a test that passed, but during self-reflection noticed the entire test was wrapped in try-catch. Removing the try-catch made the test fail, revealing a real bug.

Third, caller validation. After finding a case where input is valid but output is wrong, the agent checks whether callers perform precondition validation on the input. If all callers guarantee the input stays within a certain range, this "bug" won't trigger in practice — it's not a false positive, but a "theoretical bug, protected by callers in practice."

They found some representative bugs:

All of these are the kind of bug where "the code looks fine, tests are green, but it's just wrong." Exactly the blind spot in our adversarial review.

4. How to Integrate

After discovering agentic-pbt, my first thought was: can we just use it directly?

Good news: it's open source under MIT. The core is a Claude Code command (hypo.md, a 200-line prompt) + a parallel runner (run.py). Bad news: it's written for Claude Code, and we use Hermes — different ecosystem.

But the core logic — "read code → derive properties → write Hypothesis tests → triage" — is a pure AI agent workflow, tool-agnostic. Just adapt the prompt to Hermes's delegate_task system.

Then came the "where to put it" question. Three options:

A: Embed as a new adversarial review phase. Run a round of PBT after baseline testing, before concurrent testing. Problem: PBT is expensive — each run uses Claude to read code and write tests, burning maybe $0.5-2 per module. Embedding it in the standard flow means running it on every review — terrible ROI. Plus, PBT works best on library code with clear invariants (math functions, serialization, data structures), not every PR has that kind of code.

B: Add as the fourth lens in parallel review. We already have "go all out" mode, dispatching three parallel sub-agents for security, design, and runtime review. PBT is a natural fourth — the Logic Lens. Only run it during deep reviews — no token waste, and parallel execution means no added time.

C: Make it a standalone skill. Run PBT on third-party dependencies independently, without needing a full adversarial review.

I chose B + C.

B is the primary path — when you tell me "go all out on this," all four lenses fire simultaneously:

├── Security  → path traversal, injection, env var attacks
├── Design    → backward compat, config propagation, caller sync
├── Runtime   → import chains, real binary verification, fallback correctness
└── Logic     → [NEW] PBT invariant testing

Results come back from all four sub-agents, get deduplicated, and if the security lens finds something related to a Logic Lens invariant violation, it gets auto-escalated in priority.

C is for flexibility — want to review a third-party library function on the fly? Just say "PBT review on requests.get" and dispatch a sub-agent.

Implementation-wise, the Logic Lens prompt is adapted from agentic-pbt's hypo.md, stripped of Claude Code-specific instructions, keeping the core 6-step flow. The standalone skill is similar, with added target format specs and advanced options (adjust test intensity, batch review).

The whole thing touched four files, created two new ones, and took an afternoon.

5. Is This Worth It?

Rationally: PBT has solid data behind it — 56% genuine bug rate, 32% worth filing. For PRs that deserve deep review, adding this layer will significantly reduce logic bug escape.

But emotionally, I think the bigger value isn't in the numbers.

In our year-plus of adversarial reviewing, we've built a body of experience — test value matrices (empty string, ., .., /, NUL bytes, etc.), concurrent stress models (8 threads × 20 iterations, register+unregister+force chaos), crash recovery verification (kill -9 + os.replace atomicity checks). This knowledge was hard-won, one pitfall at a time, out of 30 accumulated lessons.

But ultimately, these are all human-imagined attack surfaces. You can only attack in directions you can think of.

PBT is different. AI reads code, discovers properties the code claims for itself, then uses random search to find counterexamples — this isn't "humans attacking code," it's "code attacking itself." The AI doesn't need to understand concurrency, doesn't need to understand path traversal — it only needs to understand "you said you'd do X, but with input Y, you didn't."

That's the real upgrade. Not making review faster — making review see things humans can't see.

---

Our adversarial review now looks like this:

Mode 5 (Standard):   Baseline → Attack → Fix → E2E → Cleanup
Mode 5 (Deep):       4-lens parallel → Security + Design + Runtime + Logic
PBT (Standalone):    Run Hypothesis directly on module/function

Three paths, choose by need. Lightweight PRs run Mode 5 standard. Core changes run 4-lens. Suspicious functions get standalone PBT.

The code lives in Hermes skills — Logic Lens in the code-review skill + standalone pbt-review skill. Anthropic's original project is at mmaaz-git/agentic-pbt (MIT), paper on arXiv 2510.09907.

I'll keep pushing on this. Plenty of automation potential ahead — auto-running PBT on functions touched in CI diffs, or verifying that agent-generated functions satisfy their docstring contracts. But for now, it's enough.