This case study shares the workflow and outcomes without exposing internal source code, customer material, or proprietary implementation details.
Guru Agents: Building a Governed Knowledge Lifecycle
Design a governed lifecycle in which frontline usage reveals knowledge gaps, AI proposes structured guidance, and qualified humans decide what becomes authoritative.
The working model
From a bounded answer to a governed feedback loop.
01
The challenge
The initial problem looked like a chatbot migration, but the harder issue lived beneath the interface. The existing knowledge base was written primarily for humans, who can infer authority, reconcile qualifications, and separate internal detail from customer-safe guidance. An AI system must reconstruct those distinctions from retrieval. The project therefore asks both how trusted knowledge should be structured and who is allowed to decide what becomes trusted in the first place.
02
The approach
- Constrain the frontline Answer Engine to a curated Guru collection so it retrieves approved guidance without silently expanding its own authority.
- Treat low-confidence or incomplete answers as potential knowledge signals rather than letting them disappear into Slack history.
- Route validated gaps through a Queue and a broader Knowledge Architect workflow that can investigate sources, reconcile contradictions, and prepare structured drafts.
- Keep qualified SME or Sales Engineer review as the authority boundary before proposed guidance enters the trusted retrieval layer.
03
The current architecture
01 · Retrieval
Answer Engine
The frontline layer answers Account Manager questions from a bounded collection of curated knowledge. Its job is narrow: understand the question, preserve qualifications, answer concisely, and avoid unsupported detail.
02 · Signal
Gap detection
Low-confidence, incomplete, or unsupported interactions become potential knowledge signals. Not every failure requires a new card; the goal is to identify reusable gaps rather than collect noise.
03 · Qualification
Reviewer
A reviewer checks whether the interaction represents a real knowledge failure, whether authoritative guidance already exists, and whether specialist input is required before drafting.
04 · Backlog
The Queue
Validated gaps become explicit knowledge work: a visible backlog of questions that need research, correction, structure, or review instead of another transient Slack thread.
05 · Production
Knowledge Architect
A broader-context workflow investigates the full knowledge base and prepares focused, retrieval-ready drafts—generally around 500 words or less—with answers, requirements, boundaries, customer-safe language, sources, and review state.
04
Human review is the trust boundary
AI can propose knowledge. Humans decide what becomes authoritative.
- Source authority and currency
- Technical accuracy and missing qualifications
- Product and operational boundaries
- Internal versus customer-facing information
- Language that could create an unsupported commitment
The workflow can gather evidence, normalize terminology, expose conflicts, and prepare drafts. A qualified human still decides whether the organization is prepared to trust the resulting guidance.
05
How I am evaluating it
Retrieval + groundedness
- Did the system find the authoritative source?
- Did irrelevant material contaminate the answer?
- Can material claims be traced to approved knowledge?
Completeness + abstention
- Were material requirements and qualifications preserved?
- Did the system recognize when approved knowledge was insufficient?
- Did it escalate instead of improvising?
Review efficiency
- Can an SME quickly identify what the AI got right or wrong?
- Does the workflow reduce synthesis work rather than merely moving it?
- Does approved knowledge improve the next retrieval?
06
Where it stands
The project has progressed beyond the original research-and-drafting prototype into an active architecture with distinct answering, gap identification, knowledge-production, and human-review responsibilities. Several components are being exercised in real workflows, while the closed-loop automation and evaluation model continue to mature. I am intentionally not treating the system as finished.
07
What I am learning
Agent architecture begins before the prompt.
Answer quality is shaped by source access, structure, and authority. Prompt design cannot permanently compensate for an ambiguous knowledge system.
Different jobs deserve different permissions.
A fast answering system should not necessarily have the same context or authority as a system investigating organizational knowledge.
Human review is not temporary scaffolding.
For consequential knowledge, review can remain a permanent authority boundary rather than a limitation waiting for a better model.
Failure can become useful telemetry.
Captured carefully, a failed answer reveals where the knowledge architecture is incomplete and where knowledge work should happen next.
08
Related field notes
09