Trace → Fix → PR: Building an Agent Self-Healing Pipeline in Three Phases

A few weeks ago I took a close look at PostHog and was struck by their "Self-driving mode" — AI scouts that automatically scan product signals, diagnose problems, and generate pull requests. This is exactly what our self-evolution system was missing.

1. The Gap

Our existing self-evolution pipeline had been running for months:

Hermes session traces → parallel miner → hierarchical distiller → skill updates

It can turn chat history into skills. But there's a blind spot: it evolves "knowledge" (skills), not "code."

When an agent consistently times out, hits import errors, or fails to connect to an API — a skill can tell you "you should add retry logic," but it won't open a PR with the fix. You still have to read logs, analyze, patch, and PR manually.

PostHog's self-driving mode gave us a clear target: automate the chain from error detection in traces to pull request submission.

2. Three-Phase Architecture

Phase A: Trace Healer — Error Pattern Detection

The foundation. Scans bh-traces.db (our span-based SQLite trace store) and matches against 7 built-in error patterns:

Each pattern is a dataclass with a match_fn predicate and a fix_prompt_template. Adding a new pattern is a single dataclass entry.

Design choices:

• Zero external dependencies — stdlib + sqlite3 only
• First-match-wins — prevents double-counting
• CLI + API dual use — python -m baby_harness.trace_healer scan or from trace_healer import scan

from baby_harness.trace_healer import scan, query_spans
 
spans = query_spans(only_errors=True)
report = scan(spans)
for match in report.matches:
    print(f"{match.pattern.severity} {match.pattern.name}: {match.span.name}")

Phase B: AutoFixHook — Reactive Auto-Fix on Failure

Phase A is passive — you run scan manually. Phase B makes it event-driven:

Coordinator executes task → fails → on_task_failed
  ├── 1. Build synthetic SpanRecord from Task + ExecutionOutput (no DB needed)
  ├── 2. trace_healer.scan() matches error patterns
  ├── 3. Generate fix_prompt (zero cost)
  ├── 4. [Optional] PiExecutor executes the fix (tokens)
  └── 5. Persist FixRecord to ~/.hermes/data/auto-fixes/

Key design decisions:

Synthetic SpanRecord: The hook doesn't query the trace DB — it constructs a span from in-memory Task + ExecutionOutput. Simpler, faster, and works without the DB.

auto_run guard: By default, only the prompt is generated (zero cost). You must explicitly pass an executor to enable execution. AutoFixHook(auto_run=True) without an executor raises ValueError.

max_fixes rate limit: 5 fixes per session by default, prevents runaway token burn.

from baby_harness.auto_fix_hook import AutoFixHook
 
# Detect only (no token cost)
detect = AutoFixHook()
 
# Detect + auto-repair (via PiExecutor)
repair = AutoFixHook(executor=pi_executor, auto_run=True, max_fixes=5)

Each persisted FixRecord captures the full context: original error, matched pattern, generated fix prompt, execution output (if any), and timestamps. AutoFixHook.report() produces a readable summary.

Phase C: AutoPRBot — GitHub PR Submission

The last mile. Reads FixRecords and automates the full GitHub PR workflow:

FixRecord → clone repo → checkout -b auto-fix/<pattern>-<task>
  → write to .auto-fixes/ → commit → push → gh pr create --draft
  → update FixRecord with pr_url + submitted_at

Safety measures:

• Draft PR — never auto-merges
• Writes to .auto-fixes/ directory — never modifies source code
• dry_run mode for previewing without git/gh side effects
• submitted_at + pr_url prevents duplicate submissions

from baby_harness.auto_pr_bot import AutoPRBot
 
# Preview
bot = AutoPRBot(repo="lennney/baby-harness", dry_run=True)
bot.process_all()
 
# Submission
bot = AutoPRBot(repo="lennney/baby-harness", executor=pi_executor)
results = bot.process_all(limit=5)

Each PR includes the error pattern, severity, original error, fix prompt, and fix output — so the reviewer sees the full chain of context.

3. Architecture Overview

                              ┌──────────────────┐
                              │  Coordinator      │
                              └────────┬─────────┘
                                       │ failure
                                       ▼
                              ┌──────────────────┐
                              │  AutoFixHook      │
                              │  Phase B          │
                              └────────┬─────────┘
                                       │
                    ┌──────────────────┼──────────────────┐
                    ▼                  ▼                  ▼
           ┌────────────────┐ ┌────────────────┐ ┌────────────────┐
           │ Trace Healer   │ │ Fix Prompt     │ │ PiExecutor     │
           │ Phase A        │ │ (zero cost)    │ │ (opt-in)       │
           └────────────────┘ └────────────────┘ └────────────────┘
                                       │
                                       ▼
                              ┌──────────────────┐
                              │  FixRecord        │
                              │  auto-fixes/      │
                              └────────┬─────────┘
                                       │ (cron / CLI)
                                       ▼
                              ┌──────────────────┐
                              │  AutoPRBot        │
                              │  Phase C          │
                              └────────┬─────────┘
                                       ▼
                              ┌──────────────────┐
                              │  GitHub Draft PR  │
                              │  (awaits review)  │
                              └──────────────────┘

4. PostHog Comparison

PostHog is bolder — their AI scouts write code directly. We're more conservative: generate fix prompt → human reviews → decides whether to merge. This is intentional: in the autonomous agent space, trust needs to be earned gradually.

5. Lessons & Trade-offs

1. Synthetic SpanRecord > DB query

I initially designed AutoFixHook to query the trace DB. Unnecessary — all the information is in on_task_failed(task, output). Synthesis is simpler, faster, and has zero dependencies.

2. Templates > LLM for pattern prompts

Phase A's fix_prompt_template is a hand-written f-string. Phase B's auto_run uses the LLM. This decouples pattern matching (low-cost, high-precision) from fix execution (high-cost, creative).

3. Filename collision is a trap

FixRecords initially used task_id as filename. Two records for the same task would overwrite. Switched to UUID — simple, correct, done.

4. gh CLI beats GitHub API

Phase C initially planned to use PyGithub. But gh pr create --draft is one command. The API is 20 lines + token management. And gh auth persists across sessions.

6. Next Steps

• Phase D: Parse diff blocks from fix output and apply them to the target repo automatically
• Eval: Track PR merge rate — how many auto-generated PRs get merged? How much debug time saved?
• Feedback loop: Feed merged/closed PR outcomes back to the pattern library for continuous improvement