Persistent AI Code Teams: Maker/Checker Architecture in Production

209 Tests All Green, Then Configuration Vanished

Here's what happened.

A while back, an open-source project had a perfect CI run — all 209 test cases passed. Code coverage looked great. Lint was clean. Type checks were silent. Couldn't have been more stable.

But when it shipped, users found the configuration feature completely broken. Not some edge case — the entire configuration module was gone. No runtime error, because when the refactor happened, the config file was never put under test coverage in the first place. 209 green lights, none told you the config was missing.

That's what got me thinking seriously about "what are we actually reviewing in code review?"

Humans reviewing code get tired, miss things, and tend to trust code that "looks fine." But the bigger problem: the person reviewing the code and the person writing it are the same person (or the same model), with completely overlapping blind spots. You know what you won't test, so you don't check that part during review either.

This is why the Maker/Checker architecture genuinely excited me after MetaGPT. Not because it adds another reviewer, but because it attacks the problem at the team structure level.

Why "Persistent" Teams

Most AI coding workflows today look like this: you ask Claude or ChatGPT to write some code, skim it, decide it looks okay, and merge. Next time you come back, it doesn't remember what it wrote last time, what traps it stepped in, or what conventions your project uses.

One-shot review works fine within a single session. But across projects, across weeks, across months, what you really need is team memory.

We spent two months stumbling through this. The current solution has three persistent roles:

• Maker — writes code only (executed via Pi), no tests, no review
• Checker — reviews only (independent session running validation), never touches a line of code
• System — orchestrates SOPs, extracts learnings, maintains the skill library

Each role runs in its own session, its own model instance. Core principle: eliminate same-source bias.

Did I coin "same-source bias"? No. Anthropic raised this when discussing Generator/Evaluator separation — ask the same model to review its own output, and it tends to agree with itself. Humans do this too: fixing bugs in your own bug report is never as thorough as having someone else do it. LLMs suffer even more because their "confidence" lacks the buffer of self-doubt that humans have.

I tried having the same session both write and review code. The result was telling — in the Checker phase it would say "code looks fine, but let me double-check" and then find absolutely nothing wrong. Not because it was slacking off, but because it had a statistical "path dependency" on its own output: since it just spent 2000 tokens generating this code, it's probably correct.

After splitting Maker and Checker into separate model instances, the probability of Checker finding issues doubled. This data comes from our internal benchmark, not conjecture.

Architecture: Three-Way Division of Labor

Maker Role

The Maker's job is brutally simple: take a task description, write code, place it at the specified path. No tests. No validation. No self-review.

We use Pi as the Maker engine — a dedicated code generation agent that outputs code files directly via --print mode. Each invocation creates an independent session to prevent cross-task context pollution.

Pi --print "task description" --provider opencode --model deepseek-v4-flash-free

Notice: Pi's sessions are fire-and-forget. It keeps no memory. Memory is managed by System, not by Maker.

Checker Role

The Checker works differently than most people imagine. **It doesn't "read code" — it "runs code." **

We use delegate_task to dispatch an independent session that validates the Maker's output. The validation process:

1. Run tests (pytest / npm test) — must all pass
2. Type checking (mypy / tsc --noEmit)
3. Construct edge case inputs — null values, special characters, out-of-bounds data
4. Verify module importability — python3 -c "import module"
5. Verify each criterion in the Sprint Contract

The Checker outputs structured scores, each on a 1-5 scale:

scores:
  functionality: 4
  correctness: 3
  code_quality: 4
  bug_detected: true
verdict: REQUEST_CHANGES

Threshold gate: functionality >= 4 AND correctness >= 3 AND no blocking bugs — only then PASS.

System Role

The System is the easiest role to overlook. But honestly, it might be the most important.

System owns:

• Extracting learnings after each cycle
• Updating the Maker and Checker skill libraries
• Maintaining manifest.json statistical state
• Tracking Checker judgment accuracy

Without System, the team has no memory. The Maker falls into the same trap twice. The Checker repeats the same judgment errors.

Sprint Contract: "Ceasefire Agreement" Before Writing Code

Maker and Checker have a natural conflict of interest. The Maker wants to "get it done quickly." The Checker wants to "find more issues." Without a mutually agreed-upon "done" standard, this tension never resolves.

Our solution is the Sprint Contract — before any code is written, the Maker and Checker must sit down (coordinated by System) and agree on one question: What counts as "done"?

A Contract looks like this:

sprint: "email_validator input validation"
criteria:
  - id: "whitespace-check"
    description: "Whitespace-only strings should be treated as empty input"
    test: "validate_email('  ') → InvalidReason.EMPTY"
    threshold: "PASS"
  - id: "length-limit"
    description: "Compliant with RFC 5321 total length ≤ 254"
    test: "validate_email('a@' + 'x'*250 + '.c') → InvalidReason.TOO_LONG"
    threshold: "PASS"
max_iterations: 3

A Contract is not a requirements document. It doesn't specify product specs, UI design, or non-functional requirements. It answers only three questions: what are we doing this sprint, how do we verify it, and what's the pass threshold?

This is exactly where most people slip — they write a Contract that reads like a PRD, then the Maker is handcuffed, and the Checker finds the Maker missed a bunch of things "not in the requirements doc." An over-long Contract kills Maker flexibility.

The Checker Calibration Loop (The Discovery That Surprised Me Most)

Anthropic said something in their harness design article that I still think about:

"An out-of-the-box LLM is a terrible QA agent."

It finds problems, then convinces itself "it's not a big deal," and lets them go. Not maliciously — it's the model's training-driven tendency to "help the user solve problems" at work. It prefers consensus over conflict.

This is devastating. The Checker's job is to create conflict. If it defaults to letting things slide, the entire architecture's artificial tension mechanism breaks down.

Our calibration process:

Checker logs
    ├─ Missed a problem a human caught?   → too lenient → tighten prompt
    ├─ Flagged a non-existent bug?        → too sensitive → add constraints to filter noise
    └─ Score drift?                       → add few-shot examples to anchor

Specifically:

• Update the Checker prompt (not the defect pattern library — that's separate)
• Same bias pattern appears 2 times → permanently baked into the system message
• Run calibration analysis after every full cycle

Honestly: You cannot fully trust the first few rounds of Checker output. A human must review the Checker's early results and calibrate its judgment baseline. This is no different from training a junior QA engineer — you wouldn't expect a new hire to do effective code review on day one. Checker onboarding cost is unavoidable.

Validation Data: Two Full Cycles

We ran two complete cycles on the baby-harness project. Real data, not hypothetical.

Cycle 1: email_validator

The Maker (Pi) wrote an email validation function. The Checker independently reviewed it.

The Checker found three critical bugs:

The third issue hit me the hardest. The Maker wrote doctests but submitted without running them. The Checker's runtime validation caught this "forgot one step before submitting" mistake. It's not a code quality issue — it's an engineering discipline problem. The Checker acts as a mechanical lock on engineering discipline.

This also drove home a deeper truth: when people (or agents) are in a hurry to deliver, they're most likely to skip the steps they know they should do but find tedious. Same for AI agents. As a Maker session winds down and the token window fills up, it defaults to "output code that's roughly okay" rather than "output code that's been validated." A Checker running as a separate session doesn't carry that "rush to finish" pressure. That's precisely the value of two decoupled sessions.

Cycle 2: Defect Pattern Accumulation

A 100% rejection rate looks extreme. It reflects early calibration — the first contract drafts are too rough, and the Checker's standards haven't stabilized. This number should drop significantly by Cycle 3. I won't predict it'll hit 50%, but the trend is clear: the larger the defect pattern library, the lower the probability of the Maker repeating the same mistakes.

Pitfall Log (Paid For In Blood)

1. Pi Times Out on Large File Output

--print mode outputs the entire file at once. For files over 400 lines, a free-tier model can time out in 60-120 seconds.

Mitigation: Split tasks. Have Pi generate only the new class or function. The main agent assembles them with patch.

2. Checker's "Let It Slide" Tendency

Already covered. Early rounds require human review. This isn't a technical problem — it's a model behavior characteristic.

3. Contract Scope Sensitivity

Too-large Contract = useless. Too-small Contract = no binding force.

Heuristic: task description ≤ 40 characters (e.g., "rename variable to camelCase") → skip Contract. Description ≥ 100 characters (e.g., "add a rectangle fill tool with undo support") → Contract required.

4. manifest.json accumulated_skills Gets Out of Sync

When adding or removing files in the skills/ directory with skill_manage write_file, the manifest's accumulated_skills array doesn't auto-update.

Every time the skills/ directory changes, add a sync verification step in UpdateSkills. I got burned on this once — the manifest claimed 8 skills, the directory had 6 files. The extra 2 were left over from a previous cleanup that never got synced.

5. No Colons in File Paths

Linux rejects filenames containing :. Not a Maker/Checker problem per se, but code file paths coming from Pi should be sanitized in advance.

Comparison with MetaGPT

MetaGPT is groundbreaking work. But in production, it has some practical limitations:

I'm not throwing shade at MetaGPT. It's pioneering in this direction. But when we brought it to production, we found cross-session skill accumulation is the most painful gap. If a lesson learned during a review can't be applied to the next review, that review was wasted.

Code Review Is a Knowledge Management Problem

Most teams treat Code Review as a quality control process. Checklists. Gates. Reviewer rotations. But at its core, Code Review is a knowledge management problem.

You review a piece of code. What did you learn? If nothing, that review delivered zero value. If the lesson stayed in the reviewer's head only, the value is still essentially zero.

What I'm most satisfied with in the Maker/Checker architecture isn't the number of bugs it found (two cycles of data isn't enough to draw conclusions there). It's that knowledge沉淀变成了不需要刻意做、但一定会做的事情 — knowledge crystallizes without deliberate effort, because the System role's sole job is extraction and persistence. You don't even have to think "should I write this lesson down." The process does it for you.

"Team memory" sounds abstract. In the context of AI agents, it means: three months from now, when a new model instance starts up, it knows what the email_validator contract from six months ago specified. That's the real value of a persistent team.

Think of it this way. In a traditional software team, a core member leaves and knowledge goes with them. A new hire needs three months to catch up. A Maker/Checker team has no "resignation" — the skill library is there, the defect pattern library is there, the review history is there. Swap in a new model instance (a new "person"), and the skills don't degrade. This isn't about replacing humans — people are irreplaceable — but it means the energy you spend on "context handover" drops dramatically.

Of course, that's the ideal. We've only run two cycles with eight defect patterns. Not enough data to claim "we solved AI knowledge management." But the direction is right: turn code review from a one-shot quality gate into a continuous knowledge collector. Every review makes the team a little stronger.

Related projects:
— persistent-code-team skill
— Baby Harness — Maker/Checker validation example
— Anthropic — Harness design for long-running application development