RAG for Enterprise Documents: What It Takes Beyond the Demo

Retrieval-augmented generation (RAG) is the default architecture for "ask questions of our documents": retrieve the passages most relevant to a question, put them in the model's context, and have the model answer from them with citations. A prototype takes a day. A system that a legal, HR or engineering team will actually rely on takes considerably longer, and almost none of the extra work is in the model call.

This post walks through the parts that decide whether an enterprise RAG system is trustworthy: ingestion, retrieval, permissions, answer generation and evaluation. It follows the same principle as our post on moving LLMs into production: the demo is judged on its best answers, the deployed system on its worst.

RAG or fine-tuning?

Teams often ask whether they should fine-tune a model on their documents instead. For question answering over a changing document set, retrieval is nearly always the better starting point. Fine-tuning changes how a model behaves (tone, format, domain vocabulary); it is a poor way to make it reliably recall specific facts, it cannot cite a source, and it has to be repeated every time the documents change. Retrieval keeps the knowledge outside the model, where you can update it, restrict it per user and point to where an answer came from.

Ingestion is where most quality is won or lost

If a table is flattened into a stream of numbers, or a policy's section heading is separated from its body, no amount of prompt engineering downstream will recover the meaning. Ingestion deserves the same engineering attention as the query path.

Parsing

• Use format-aware parsers. PDFs, slide decks, spreadsheets, wiki pages and scanned documents each need different handling; a single "extract text" step loses headings, tables and reading order.
• Keep structure as metadata: document title, section path (e.g. Leave Policy > Parental Leave > Eligibility), page number, last-modified date, owner and source system.
• Treat scanned documents and complex tables as a separate pipeline with its own quality checks. They are where silent extraction errors hide.

Chunking

Chunks are the unit of retrieval. Too small and the model gets fragments without context; too large and retrieval becomes imprecise and you waste context window. Structure-aware chunking, splitting on headings and paragraphs and then capping size, beats fixed-size windows for most business documents. Prepending the document title and section path to each chunk's text is a cheap change that noticeably improves retrieval of otherwise ambiguous passages.

{
  "chunk_id": "hr-policy-2026-04#s3.2-c1",
  "doc_id": "hr-policy-2026-04",
  "source": "sharepoint",
  "title": "Leave Policy",
  "section_path": [
    "Parental Leave",
    "Eligibility"
  ],
  "text": "Leave Policy > Parental Leave > Eligibility\nEmployees with at least six months of continuous service...",
  "updated_at": "2026-04-02T09:15:00Z",
  "acl": {
    "groups": [
      "all-employees-in"
    ],
    "classification": "internal"
  },
  "region": "IN"
}

Note the acl and region fields. Every chunk should carry enough metadata to decide, at query time, whether this user is allowed to see it and whether it applies to them.

Retrieval: hybrid search, filters and reranking

Pure vector search is good at paraphrase and weak at exact identifiers such as product codes, clause numbers, error codes and people's names. Keyword search is the reverse. Most production systems run both and merge the results (hybrid search), then pass a few dozen candidates through a reranker that scores each passage against the question more carefully than the initial retrieval can.

def retrieve(question: str, user: User, k: int = 8) -> list[Chunk]:
    filters = {
        "acl_groups_any": user.groups,          # permission filter, applied in the index
        "region_in": [user.region, "GLOBAL"],
    }
    dense = vector_index.search(embed(question), top_k=40, filters=filters)
    sparse = keyword_index.search(question, top_k=40, filters=filters)
    candidates = reciprocal_rank_fusion(dense, sparse)[:40]
    ranked = reranker.score(question, candidates)
    return [c for c in ranked[:k] if c.score >= MIN_RELEVANCE]

Two details matter here. First, the relevance threshold: if nothing clears it, the right behaviour is to say so rather than hand the model weak context and hope. Second, the permission filter is applied inside the index query, not afterwards.

Generation: grounded answers and honest refusals

The generation step should be narrow. Instruct the model to answer only from the supplied passages, cite the chunk or document for each claim, and say plainly when the passages do not contain the answer. Then check it: a post-generation step can verify that every citation refers to a passage that was actually retrieved, and flag answers with no citations at all.

• Show citations as links back to the source document and section, so users can verify in one click. This is the single biggest driver of user trust.
• Prefer "I couldn't find this in the documents you have access to" over a plausible answer. Measure how often it happens; a rising rate usually points to an ingestion or coverage gap.
• Keep conflicting versions out of the index, or rank by updated_at. Superseded policies are a common source of confidently wrong answers.

Common failure modes

Evaluate retrieval and answers separately

When a RAG answer is wrong, you need to know whether retrieval failed (the right passage was never found) or generation failed (it was found and misused). Build a test set of real questions with the documents that should answer them, and measure both: did the expected source appear in the top results, and is the final answer correct and faithful to the passages? Our post on evaluating LLM applications covers how to build that harness and run it on every change to chunking, embeddings, prompts or models.

Operating it

• Freshness: incremental sync from source systems, including deletions and permission changes. A document removed from SharePoint should leave the index within a defined window.
• Observability: log the question, retrieved chunk IDs, scores and the answer, under the same data controls as the source content. See AI data privacy and governance.
• Cost: retrieval quality usually lets you send fewer, better passages, which is also the main lever on per-query cost. More in LLM cost control.
• Feedback: a thumbs-down button is only useful if someone reviews the results weekly and turns them into test cases.

RAG is also increasingly one tool among several that an assistant can call, rather than the whole application. If you are heading that way, see AI agents in the enterprise and the Model Context Protocol.

Key takeaways

• Most RAG quality problems are ingestion problems: parsing, structure and metadata.
• Use hybrid retrieval with reranking and a relevance threshold; refusing is a feature.
• Permissions belong in the index query, synced from source systems.
• Evaluate retrieval and generation separately, with a test set built from real questions.

HashTech designs and builds retrieval systems as part of its AI enablement work, starting from the documents and workflows rather than the model.