message.comDevelopers
Developers/Concepts/LLM-grounded RAG

What is LLM-grounded RAG?

LLM-grounded RAG (Retrieval-Augmented Generation) combines a vector search step with a language model so the model answers only from content you have approved. Without grounding, an LLM hallucinates: it makes up plausible-sounding facts. With grounding, the model is constrained to summarize retrieved passages from your knowledge base.

Definition

A grounded RAG pipeline has three stages:

  1. Retrieve. Embed the user's question into a vector. Find the top K most similar chunks from a pre-indexed corpus.
  2. Augment. Inject those chunks into the model's prompt context.
  3. Generate. Ask the model to answer using only the provided context, with citations.

The corpus is the source of truth. The model is the writer. Separating those two roles is what makes the system trustworthy in a customer-support context.

Why not fine-tune

Fine-tuning bakes information into model weights. It is expensive, slow to update, and impossible to audit. RAG keeps the information in a database where you can edit, delete, version, and trace. When a policy changes, you update one knowledge-base entry and the model immediately answers with the new policy. No retraining run, no eval cycle, no risk of forgetting unrelated facts.

How message.com implements it

The stack:

  • Crawler. Customer enters a URL during onboarding. We crawl 50 to 200 pages, two levels deep. See knowledge base.
  • Chunker. Pages are split into 500-token chunks with 50-token overlap to preserve context across boundaries.
  • Embedder. Each chunk goes through an open-weight embedding model running on our infrastructure. See vector embedding.
  • Store. Vectors are stored in our own database alongside conversation state. No separate vector DB to operate.
  • Retriever. Question gets embedded the same way. We rank by cosine similarity (see cosine similarity) and pick the top 5.
  • Generator. An open-weight LLM generates the answer with the 5 retrieved chunks in context.

Grounding discipline

Grounding is only useful if the model actually obeys it. We enforce this with:

  • A system prompt that explicitly forbids drawing on parametric knowledge for anything factual.
  • Citation requirements. Every claim must reference a chunk ID.
  • Confidence thresholds. If the top retrieval similarity is below a floor, the model falls back to "I do not have that information, an agent will follow up" rather than hallucinating.

When RAG fails

RAG is not a silver bullet. It fails when:

  • The right answer is in the corpus but is split across multiple chunks the retriever did not co-rank.
  • The question is multi-hop and requires reasoning that pure retrieval cannot reach.
  • The corpus contradicts itself. The model picks one passage and confidently quotes it.

For those cases, message.com gives the agent a one-tap "take over" affordance so the human can step in without losing context.