Cover image for “Before You Let Agents Run Loose: AI Context Architecture”
· 3 min read

Before You Let Agents Run Loose: AI Context Architecture

Also available in 中文

Agent Engineering LLM

You wired up RAG, but the agent still hallucinates, rifles through the wrong Slack channels, and treats brainstorm threads as shipped facts. The bottleneck is often context architecture — not the model. Stack Overflow’s No Dumb Questions episode lays it out cleanly.

Three terms, three jobs

Agent context guardrails

TermWhat it covers
Context infrastructureStore, index, retrieve, deliver context (vector DBs, rules in Markdown, etc.)
Context architectureWhy and how boundaries are designed — MCP choices, Scopes, trust models, HITL
Context engineeringImplementation — language, rerankers, indexing tactics

Their car analogy: car vs boat vs bike is architecture; make and model is engineering; parts sourcing is infrastructure. MCP skews architectural; RAG touches all three.

Where context architecture should be bought

What architecture actually fixes

1. Retrieval boundaries
Ask for “tires” and the library returns plane, bike, and wheelbarrow manuals — all valid, none useful. Prompts saying “sports car only” are unreliable. Curate what the agent can touch.

2. Agentic memory
Once you refine the task to “sports car,” that state should survive sessions. At 10 or 100 parallel agents, memory becomes infrastructure.

3. Guardrails

  • Trust tiers: high / medium / low — don’t act blindly on shaky knowledge
  • Human-in-the-loop: route to SMEs when data is incomplete
  • Goal: predictable behavior you can delegate

4. Two-layer permissions

  • Inherit source permissions (no channel access → no agent access)
  • Scopes narrow further: “I can see the company, but you only care about my product area”
  • Control write-back too: does output land in the team pool or stay with me?

The technical flow: cast a wide net, then filter

Build vs buy crossroads

Stack Internal chains: pick retrieval strategy per question → context mapping / tabular grabs → trust scores → rerank. Doug’s metaphor: trawl the ocean, toss oysters, keep the right fish, then size-filter. Good architecture also fills gaps you didn’t know existed (e.g., correct tire PSI for your car model) and stays model-agnostic.

Ash adds consistency and predictability — same question, same answer, across people. That’s the bar for “coworker,” not “toy.” Narrower context also saves tokens — an underrated line item in build-vs-buy math.

Build vs buy: the hard part isn’t code

Teams can ship RAG. Ash argues the expensive layer is product philosophy:

  • Conflicts, gaps, and bad data across Slack, Drive, Confluence, Jira…
  • Trust: experts judge “does this match what I expect?”; vibe coders often can’t
  • Systematic ranking, filtering, and trust needs design conversations — not just embeddings

Buying (they pitch Stack Internal) claims ~20 categories of edge cases already earned the hard way.

Takeaways for builders

  1. Harness design matters as much as model pick — see the Saul Agent 24h startup experiment for the failure mode
  2. Scopes, trust, and HITL belong in v1, not as post-launch patches
  3. If you build: how do you resolve conflicts, own dirty data, and earn trust from inexperienced users?
  4. Tighter context is a quality and cost lever

Source: What is AI context architecture? Why not just build your own?

Comments