Feeding 275 Documents to an AI Agent: Knowledge Base Structure Design in Practice

A sequel: Format conversion done — now what? 275 documents, 6 bundles, 16 types — it all starts with the brutal realization that "agents don't do full-text search."

---

It All Started With One Prompt

Here's how it went down.

Last month I converted all 270 knowledge documents to the OKF format and wrote the blog post "Switching Your Agent's Knowledge Base to a Standard Format: OKF Conversion in Practice". I felt good — format compliant, frontmatter complete, INDEX auto-generated. I was convinced I'd prepared a feast for my agent.

Then I asked the compound-system to read through the knowledge base and tell me what it learned.

What came back was a set of fuzzy matches — low relevance scores, and a majority of the 275 documents simply never surfaced.

I sat there stunned.

Wait — I'd organized everything. The content was there. Why was the agent performing like someone flipping through a library with their eyes closed?

Because agents don't do full-text search.

At least, not the kind you imagine — "dump everything into a vector database, query matches whatever I ask." When a large language model runs embedding similarity across 275 documents, the results look like fishing with a drinking straw in the ocean. You might catch one, but you never know what you're missing.

This wasn't an embedding technology problem. It was how I organized knowledge.

I had been organizing knowledge with a "human library" mindset. Categories, taxonomies, tags — as long as the table of contents looked clear enough, I figured that was sufficient. But an agent reads entirely differently from a human.

A human scans a TOC: glance at titles, jump to what's interesting. An agent reads a TOC: it literally reads every single line — title and type — character by character.

A human searches: type keywords, skim titles and summaries, pick one to open. An agent searches: feed a query, embedding finds nearest neighbors, top-K cutoff, then pray that the snippet you wanted happened to match.

These are two completely different navigation systems.

I spent two weeks redesigning the knowledge base structure. The results confirmed my hunch: when the knowledge base structure is right, the agent's comprehension accuracy jumps by a full level.

This article is the full record of that restructuring.

Bundle Design Philosophy: Who Consumes It, Who Owns It

Let me show you what the final output looks like.

275 documents, carved into 6 bundles:

Looking at this breakdown, you might think: isn't this just categorizing by "source"?

I thought the same thing at first. Until I realized — I had the premise backwards.

The first version was organized by source. AI Agent Wiki came from a wiki project, Compound System from the self-evolution pipeline, Skills from the skill library… seemed reasonable, right?

But when actually using it, something felt off. Because when an agent faces a task, it doesn't think "oh, this question should be answered in the wiki bundle." It thinks "I'm writing code right now — I need to find relevant reference implementations."

Bundles shouldn't be organized by "where documents come from," but by "who will consume them."

That was the turning point in my thinking.

Here's an example. The Compound System Knowledge Base has 45 Knowledge documents covering architecture decisions, debugging stories, and theoretical derivations from the self-evolution pipeline — all hard lessons from engineering practice. If organized by source, these should sit with the AI Agent Wiki (both related to compound-system). But the consumer of the AI Agent Wiki is "someone who wants to understand what an agent is," whereas the consumer of the Compound System Knowledge Base is "the running self-evolution pipeline agent."

Two completely different consumption scenarios.

The result? The former has only 12 documents (concise overviews), the latter has 62 (granular engineering records). The same Pattern document, placed in the Compound System bundle, gets automatically referenced by the agent when debugging the pipeline. Placed in the Wiki, the agent never connects it to the current task.

Bundle equals consumption context. The final rule I settled on is simple: state who the "primary user" of this bundle is. If you can't articulate that clearly, your partitioning is wrong.

Index First, Content Second

Here's a discovery that genuinely surprised me.

How do you think an agent actually uses a knowledge base of 275 documents? I initially assumed it would embed everything, store vectors, and retrieve by query.

Wrong.

I traced the actual invocation patterns of the compound-system. The overwhelming majority of the time, the very first thing the agent does is: read INDEX.md.

Yes — that auto-generated table of contents file. It reads the index line by line, understands the overall structure, and decides where to dig based on the index titles.

This is actually quite similar to what humans do — but with one critical difference.

A human reads an index to "find the good parts" — stop when something catches their interest. An agent reads an index to "build a navigation map" — it loads the entire knowledge base structure into its context, then drills down layer by layer.

The result?

The information density of the index determines the scope of knowledge the agent can perceive.

If INDEX.md only lists file names, the agent finishes reading and knows only "which files exist." It doesn't know which files it needs right now, nor which bundles relate to the current task.

My current INDEX generation logic includes three layers of information:

1. Structure layer: bundle breakdown, document count per bundle
2. Semantic layer: topic summary for each bundle (auto-extracted from document frontmatter)
3. Navigation layer: each document's title + type + one-sentence summary

When an agent reads:

## Compound System Knowledge Base (62 docs)
- Bug: compression session split fix
- Knowledge: Executor-Curator separation architecture
- Pattern: Trajectory Mining Pipeline template

It can immediately determine "this bundle is relevant to the current task," then read only this bundle's detailed index before deciding which specific document to open.

Front-truncated, back-full. The INDEX only gives a table of contents; full content lives in the individual documents. The agent starts from the index and only goes deeper when needed.

This pattern exceeded my expectations. Before the redesign, the agent would open 8-10 documents on average to find the information it needed. After: down to 2-3.

Type Navigation: Build a Navigation Language for Your Agent

16 types. This was another design decision I agonized over for a long time.

I started with just 5 types: Knowledge, Bug, Pattern, Documentation, Reference. Was that sufficient? Yes. But with 275 documents, the agent needed a finer-grained filtering dimension.

The final type list:

— Plus 6 more sub-types, used internally within specific bundles.

The motivation for this many is simple: type labels are the agent's navigation language.

When an agent is debugging a self-evolution pipeline issue, it needs to quickly filter down to Bug-type documents rather than spending time reading Patterns or Knowledge docs. When it needs to reference an existing skill implementation, it searches directly within the HermesSkill type.

The principle for type design is: the agent's filtering intent determines granularity.

If the agent frequently needs to "find all bug records related to X" — make Bug a first-class type. If the agent never distinguishes between Reference and Documentation — merge them.

Sounds simple. But in practice, you'll find some documents are inherently cross-type. A doc titled "Fixing session splits caused by compression" is both a Bug (documenting a fix) and a Knowledge (explaining the relationship between compression and sessions).

How do you handle this?

I tried two approaches. Approach A: allow a document to have multiple types. Approach B: enforce a single primary type per document, with cross-type information supplemented by tags.

I chose Approach B.

Why B and not A? Because I traced the agent's actual behavior. When filtering, the agent uses exact matches — it asks "give me all documents with type=Bug." If a document has 3 types, it appears in all 3 filter results, creating both duplication and diluting the meaning of the filter.

Consistency matters — but you don't need to chase perfection. Agents can handle some degree of inconsistency. A document tagged as Knowledge that also contains bug information won't cause problems when the agent reads it. It just won't surface when the agent filters for bugs. But tags can fill that gap.

So my rule is: type determines primary classification, tags cover cross-type information.

The Cost of Dual-Form Storage

This decision gave me the most hesitation.

Within the same knowledge base, 46 documents exist in two forms simultaneously:

• As a Hermes Skill (stored under ~/.hermes/skills/)
• As a Source Pattern (stored in the Pi agent's config directory)

Same content, different frontmatter. Hermes Skills require type, description, tags; Pi agent Patterns need trigger, context, workflow.

Why go through this trouble? Couldn't both systems just read the same file?

I tried. It doesn't work.

Hermes and Pi are two completely different agent systems. Hermes loads skills by reading the entire SKILL.md and parsing YAML frontmatter. Pi consumes patterns by matching trigger conditions and injecting context. Their frontmatter schema requirements are fundamentally different — forcing a shared file would mean either sacrificing one system's needs or building an absurdly complex compatibility layer.

Dual-form storage solves a very real problem: the same piece of knowledge can be consumed by two agent systems, each in its own way.

The cost?

Sync overhead. Every time you update the knowledge content, you need to make the same change in two locations. I wrote an auto-sync script (really just a curator agent task), but there are still occasional cases where one copy gets updated and the other doesn't.

But I believe the cost is worth it.

Because without dual-form storage, the biggest loser isn't the agent — it's me. Every time I write a good skill in Hermes and want to reuse it in Pi, I'd have to rewrite it from scratch. A "rewrite everything twice" workflow won't survive a month.

Dual-form storage is essentially: create once, distribute across channels. The cost is sync overhead. The benefit is sustained usability.

Is it worth it? At my current volume (46 dual-form docs), I say yes. Beyond 100 docs, you'd need better automation.

War Stories from the Trenches

Let me share some genuine pitfalls. Not "we encountered a challenge and solved it gracefully" — more like "damn it, broken again."

Auto-Classification Is Inaccurate

I naively wrote an auto-classification script that assigned bundles based on document frontmatter, filename, and keywords.

The result? Roughly 20% of documents landed in the wrong bundle.

The most egregious case: a technical note about "session compression algorithm optimization" was classified into the Skills References bundle. Tracing the logic, the classification script saw the word "compression," noticed the document mentioned a skill usage, and concluded it belonged with Skills.

The root cause of inaccurate classification is that cross-bundle documents are the norm, not the exception.

I no longer do fully automatic classification. I use a "pre-classify + manual confirm" workflow: the auto-script suggests a bundle, and I spend 15 minutes each week reviewing and confirming.

INDEX Bloat

Auto-generated INDEX has a natural problem: it gets more detailed with every iteration.

Initially, the INDEX only contained filenames and types. Then I added descriptions. Then tags. Then one-sentence summaries. Then — the INDEX file became longer than any single content document.

A 285-line INDEX file.

The agent consumes a significant number of context tokens just to read it. And when the content gets too detailed, the agent loses its "navigation" function entirely — it reads everything from the INDEX and stops drilling into the actual documents.

The fix is manual control over INDEX granularity. Only title + type + one-sentence summary. Detail stays in the individual documents. Any summary exceeding three sentences gets automatically truncated.

Missing Frontmatter from Migration

1% of legacy documents had no frontmatter at all.

Yes — the ones written earliest, before any format standards were established. No type, no tags, no timestamp.

This caused the INDEX generation script to crash because the parser couldn't find required fields.

I didn't batch-fix them. For some documents, I no longer know the accurate type — and assigning a wrong type is worse than assigning none.

I added a fallback logic: if frontmatter is missing, auto-assign type: Unknown and status: needs-review. The INDEX lists these "unclassified" docs separately. Fill them in slowly, over time.

Design Your Knowledge Base as a Query Language

By this point, one feeling grew increasingly strong:

Preparing a knowledge base for an AI agent is, at its core, designing a query language.

Think about it. Traditional knowledge base design asks "how do I organize this information well" — categories, indexes, tags, full-text search. That's a human librarian's mindset.

But an agent doesn't need "well-organized" information. It needs information that can be efficiently located and consumed.

What's the difference?

The human knowledge base query path is: browse → find → read. The agent knowledge base query path is: index-locate → type-filter → targeted-read.

So your knowledge base structure is your agent's query language:

• Bundle is the namespace
• Type is the type system
• Tags are the query tags
• INDEX is the query plan

Sound familiar? It resembles a programming language's type system.

Yes. The knowledge base structure is the agent's DSL.

This realization unlocked a much bigger question for me: why do so many RAG systems underperform in production? Not because embeddings aren't good enough, not because chunk size isn't tuned right — but because the knowledge base structure itself was never designed to be queryable.

You have a 10,000-document knowledge base, but no bundles, no type system, no index navigation — then all you have is a file list. Using it feels as painful as working with a database that has no schema.

If you design your knowledge base as the agent's query language from day one, you'll make different decisions at every level:

• When adding a new document: What type should this be? Who will consume it, and in what context?
• When defining a bundle: What is the query intent of this bundle? What task would navigate the agent here?
• When writing a summary: If an agent reads this summary, can it decide whether to go deeper?

That's how I build knowledge bases now.

275 documents. 6 bundles. 16 types.

Not the largest, not the most impressive. But it's the first knowledge base I've seen that an agent actually uses well.

---

This is part one of the knowledge base structure design series. Part two will cover: how to enable "horizontal transfer" in an agent — taking what it learns from one bundle and applying it in another. If you'd like to be notified when it publishes, follow the blog.

Related code and config: the knowledge base structure lives in the compound-system repo; the OKF conversion toolchain lives in a standalone CLI.