Skip to content
ZJ

Case study /04 · Delivered

Knowledge Copilot

AI · RAG · Enterprise Knowledge

A retrieval-augmented assistant that answers from an organization's own documents — with citations, not confidence tricks.

Knowledge Copilot is retrieval-augmented generation done the unglamorous, correct way: an assistant that answers questions from an organization's internal documents, shows exactly which passages support every claim, and says 'I don't know' when the corpus doesn't contain the answer.

  • TypeScript
  • Python
  • LLM APIs
  • Embeddings
  • pgvector
  • Hybrid search
  • Reranking
  • PostgreSQL
  • Evals
  • Streaming
Knowledge Copilot grounded answer flow: document sources are parsed, chunked, embedded and indexed, then a user question runs through hybrid retrieval with ACL filters applied before grounded generation returns a cited answer or says it does not know.

Context

Every company has the same problem: policy docs, runbooks, contracts and wikis that contain the answer to almost any question — if anyone could find it. Generic chatbots hallucinate confidently; search returns forty links. The useful middle is grounded answering with provenance.

Problem

RAG fails quietly in three places: retrieval that misses the right passage, generation that drifts beyond what was retrieved, and permissions — the assistant must never synthesize an answer from documents the asking user cannot open. All three had to be engineered and measured, not assumed.

Constraints

  • Answers had to carry passage-level citations a user can click and verify.
  • Document permissions (ACLs) had to filter retrieval before generation, not after.
  • Source documents change daily; the index could never drift stale silently.
  • Quality had to be provable to stakeholders — an eval suite, not a demo.

Responsibilities

  • Ingestion pipeline: parsing, structure-aware chunking, metadata and embedding generation.
  • Hybrid retrieval combining vector similarity and keyword search, with a reranking stage.
  • Permission-aware retrieval filtering against the organization's access model.
  • Grounded generation with inline citations and refusal behavior.
  • Evaluation harness with answerable and unanswerable question sets.

Architecture

  • Documents flow through a staged ingestion pipeline — parse, chunk along structural boundaries, embed, index — with per-source sync watermarks so updates re-index incrementally.
  • Retrieval is hybrid: dense vectors (pgvector) and keyword search run in parallel, results merge and pass through a reranker before the top passages reach the model.
  • ACL filters apply inside the retrieval query itself, so unauthorized passages never enter the context window.
  • Generation is constrained to retrieved context with citation markers per claim; a groundedness check flags answers that cite nothing.
  • Every answer stores its full retrieval trace for debugging and evals.

Key decisions

  • Hybrid retrieval over pure vector search: exact terms — invoice numbers, product names, legal phrases — are where pure embeddings embarrass themselves.
  • Chunking follows document structure (headings, clauses, tables) instead of fixed token windows; retrieval quality moved more from chunking than from any model swap.
  • Refusal is a feature: below a retrieval-confidence threshold the copilot says it doesn't know and shows the closest sources instead of improvising.
  • Evals run on every prompt or pipeline change — a fixed suite of real questions with graded answers, including trap questions the corpus cannot answer.

Implementation

  • Answer streaming starts as soon as generation begins; citations resolve progressively as claims complete.
  • The reranking stage cut context size substantially — fewer, better passages beat a stuffed window on both cost and accuracy.
  • Unanswerable-question sets keep refusal behavior calibrated release over release.
  • Retrieval traces power an internal debugging view: what was fetched, what was ranked out, what the model actually cited.

Performance

  • Incremental re-indexing keeps corpus freshness within minutes of source changes.
  • Parallel dense + keyword retrieval keeps end-to-end answer latency interactive.
  • Context discipline (reranked top-k) holds token costs flat as the corpus grows.

Outcomes

  • A copilot whose answers are verifiable — every claim traces to a passage the user can open.
  • Permission-aware retrieval held up under audit: no cross-ACL leakage paths.
  • The eval harness turned model and prompt upgrades from gut calls into measured decisions.
  • [PLACEHOLDER — add corpus size, usage and accuracy figures when publishable.]

Lessons

  • Retrieval quality is the product; the LLM only narrates what retrieval found.
  • Unanswerable questions are the most important part of any eval set.
  • Citations change user behavior — people trust, then verify, then rely.

Software Architecture

Seven-stage flow from enterprise documents to cited, permission-aware answers. 1. Document sources: policies, runbooks, contracts, wikis. 2. Parse and chunk: structure-aware sections and metadata. 3. Embed and index: embeddings plus pgvector plus a keyword index. 4. User question: ask a natural-language question. 5. Hybrid retrieval: dense plus keyword search, then rerank the top passages, with ACL filters applied before generation. 6. Grounded generation: answer only from retrieved context, where refusal is a feature. 7. Cited answer: inline citations, or 'I don't know'. A retrieval trace and debugging store feeds off the retrieval and generation stages, keeping fetched passages, rankings and citations, and a return path shows incremental re-indexing keeping content fresh. Four properties are called out: passage-level citations, permission-aware, hybrid retrieval and grounded answers.
Architecture diagram in three layers showing how retrieval quality, ACL safety and measurable accuracy work together. Ingestion layer: source connectors for docs, wiki, drive and contracts, a parser extracting text, structure and tables, chunking by headings, clauses and sections, then embeddings and metadata holding vectors, source IDs and ACL tags. Retrieval layer: a user query fans out to keyword search and vector search, which merge and rerank — fewer, better passages beat stuffed context — producing top-k context. Answer and trust layer: an ACL filter admitting only documents the user can access, with permissions enforced inside the retrieval query, then LLM answering constrained to retrieved context, citations giving passage-level support per claim, and a refusal check that says 'I don't know' when evidence is weak. Both the retrieval and answer layers report into an eval harness covering answerable questions, unanswerable questions, groundedness, retrieval quality and regression checks. Four properties are called out: hybrid search, ACL-safe, reranking and measured quality.