message.comDevelopers
Developers/AI/Knowledge base

AI knowledge base Planned

The knowledge base is the source of grounded answers. When you enter your URL during onboarding, we crawl your public pages, chunk them, embed each chunk with our self-hosted embeddings model, and store the vectors in our vector store. Every AI reply retrieves from this index before responding.

This is the retrieval half of retrieval-augmented generation. See the RAG concept page for why grounding matters and how it differs from fine-tuning.

The crawl

During onboarding you supply a URL. We crawl up to 200 pages by default, two levels deep, respecting robots.txt and noindex meta tags. The crawler identifies as message-com-bot and uses a polite rate limit. You can change crawl scope per source after the initial install.

  • Up to 200 pages by default. Configurable per source.
  • Two levels of link depth from the seed URL.
  • HTML only; PDFs and uploaded files use a separate ingestion path.
  • Re-crawl on a configurable cadence (default monthly).

Chunking

Each page is converted to clean markdown, then split into chunks of roughly 800 tokens with 100 tokens of overlap. We try to break on semantic boundaries (headings, paragraph breaks). Each chunk preserves its source URL, page title, and breadcrumb path so retrieval results stay grounded.

Embedding

Chunks are embedded with our self-hosted English-tuned embeddings model. Multilingual variants are on the roadmap once we measure demand. Embeddings run on hardware we own, so chunk text never leaves our infrastructure.

Storage

Vectors go into Postgres with vector search enabled. Per-workspace isolation is enforced via row-level security on every query. There is no separate vector database to operate.

Manage sources via API

Most teams add and edit sources via the dashboard, but every operation is also exposed as REST. Useful for syncing your help-centre CMS into the KB on a schedule.

Add a source

cURL
curl -X POST 'https://app.message.com/api/v1/kb/sources' \
  -H 'Authorization: Bearer YOUR_WORKSPACE_JWT' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://acme.com",
    "name": "Acme docs",
    "maxPages": 200,
    "maxDepth": 2
  }'

The same semantic search the AI uses is exposed for your own apps. Useful for building a help-centre search widget or a chatbot frontend without leaving the platform.

cURL
curl -X POST 'https://app.message.com/api/v1/kb/search' \
  -H 'Authorization: Bearer YOUR_WORKSPACE_JWT' \
  -H 'Content-Type: application/json' \
  -d '{
    "query": "How do I cancel my subscription?",
    "topK": 5
  }'
200 OK
{
  "results": [
    {
      "chunkId": "uuid",
      "sourceId": "uuid",
      "url": "https://acme.com/help/cancel",
      "title": "Cancelling your subscription",
      "score": 0.87,
      "text": "To cancel, open Settings → Billing → Cancel...",
      "lastCrawledAt": "2026-05-12T10:00:00Z"
    }
  ]
}

How AI uses retrieval

  1. A visitor sends a message. The AI workflow runs.
  2. We embed the visitor message with the same self-hosted embeddings model.
  3. We pull the top-5 chunks from the vector store, filtered to the current workspace.
  4. The chunks plus conversation history are stuffed into the prompt as context.
  5. The model drafts a reply, citing the sources it used.

If retrieval comes back empty (no relevant chunk above the similarity threshold), the AI says it does not know and escalates to a human. We do not let it hallucinate.

Freshness

The default recrawl cadence is monthly. For sites that change often (help-centre teams shipping daily, ecommerce stores with weekly inventory shifts), set a weekly or daily recrawl per source. The POST /api/v1/kb/sources/:id/recrawl endpoint also lets you trigger a refresh on demand, useful in CI when you publish docs.

CMS integrations

If you run WordPress, the WordPress plugin auto-indexes wp_posts on save, so the KB is current within sixty seconds of a publish. Shopify product pages and policies are indexed automatically by the Shopify app.

Common pitfalls

  • JavaScript-rendered content. The crawler renders pages with a headless browser, but client-side routes that rely on auth cannot be indexed. Expose a public HTML version for SEO and for us.
  • Stale answers. If your KB hasn't been recrawled since a pricing change, the AI may quote old prices. Trigger a recrawl on launch.
  • Empty chunks. Boilerplate-only pages (nav, footer, no body content) are filtered out automatically. Check the source dashboard for filtered counts.