message.comDevelopers
Developers/AI/Knowledge base

AI knowledge base Live

The knowledge base is the source of grounded answers. When you enter your URL during onboarding, we crawl your public pages, chunk them, embed each chunk with our self-hosted embeddings model, and store the vectors in our vector store. Every AI reply retrieves from this index before responding.

This is the retrieval half of retrieval-augmented generation. See the RAG concept page for why grounding matters and how it differs from fine-tuning.

The crawl

During onboarding you supply a URL. We crawl up to 200 pages by default, two levels deep, respecting robots.txt and noindex meta tags. The crawler identifies as message-com-bot and uses a polite rate limit. You can change crawl scope per source after the initial install.

  • Up to 200 pages by default. Configurable per source.
  • Two levels of link depth from the seed URL.
  • HTML only; PDFs and uploaded files use a separate ingestion path.
  • Re-crawl on a configurable cadence (default monthly).

Chunking

Each page is converted to clean markdown, then split into chunks of roughly 800 tokens with 100 tokens of overlap. We try to break on semantic boundaries (headings, paragraph breaks). Each chunk preserves its source URL, page title, and breadcrumb path so retrieval results stay grounded.

Embedding

Chunks are embedded with our self-hosted English-tuned embeddings model. Multilingual variants are on the roadmap once we measure demand. Embeddings run on hardware we own, so chunk text never leaves our infrastructure.

Storage

Vectors go into Postgres with vector search enabled. Per-workspace isolation is enforced via row-level security on every query. There is no separate vector database to operate.

Managing the KB via API

Most teams add and edit content via the dashboard, but the crawl and retrieval steps are also exposed as REST. Useful for syncing your help-centre CMS into the KB on a schedule, or for building your own search UI on top of it.

Crawl a URL

Queues an async ingestion job: crawl, chunk, and embed. There is no separate "source" resource to create first; posting a URL both registers it and starts the crawl.

cURL
curl -X POST 'https://app.message.com/api/v1/kb/ingest/url' \
  -H 'Authorization: Bearer YOUR_WORKSPACE_JWT' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://acme.com",
    "maxPages": 200,
    "maxDepth": 2
  }'

// 200 OK. crawl runs async; poll GET /api/v1/kb/jobs/:id for status
{ "jobId": "uuid", "status": "queued" }

The same semantic search the AI uses is exposed for your own apps. Useful for building a help-centre search widget or a chatbot frontend without leaving the platform.

cURL
curl -X POST 'https://app.message.com/api/v1/kb/retrieve' \
  -H 'Authorization: Bearer YOUR_WORKSPACE_JWT' \
  -H 'Content-Type: application/json' \
  -d '{
    "query": "How do I cancel my subscription?",
    "topK": 5
  }'
200 OK
{
  "chunks": [
    {
      "chunkId": "uuid",
      "documentId": "uuid",
      "chunkIndex": 3,
      "text": "To cancel, open Settings -> Billing -> Cancel...",
      "score": 0.87,
      "title": "Cancelling your subscription",
      "sourceUrl": "https://acme.com/help/cancel",
      "sourceType": "url",
      "articleId": null,
      "provenance": null
    }
  ],
  "queryEmbedTimeMs": 42,
  "searchTimeMs": 11
}

How AI uses retrieval

  1. A visitor sends a message. The AI workflow runs.
  2. We embed the visitor message with the same self-hosted embeddings model.
  3. We pull the top-5 chunks from the vector store, filtered to the current workspace.
  4. The chunks plus conversation history are stuffed into the prompt as context.
  5. The model drafts a reply, citing the sources it used.

If retrieval comes back empty (no relevant chunk above the similarity threshold), the AI says it does not know and escalates to a human. We do not let it hallucinate.

Freshness

The default recrawl cadence is monthly. There is no dedicated recrawl endpoint: re-posting the same URL to POST /api/v1/kb/ingest/url queues a fresh crawl job and supersedes the prior document, which you can trigger on demand from CI when you publish docs.

CMS integrations

If you run WordPress, the WordPress plugin auto-indexes wp_posts on save, so the KB is current within sixty seconds of a publish. Shopify product pages and policies are indexed automatically by the Shopify app.

Common pitfalls

  • JavaScript-rendered content. The crawler renders pages with a headless browser, but client-side routes that rely on auth cannot be indexed. Expose a public HTML version for SEO and for us.
  • Stale answers. If your KB hasn't been recrawled since a pricing change, the AI may quote old prices. Trigger a recrawl on launch.
  • Empty chunks. Boilerplate-only pages (nav, footer, no body content) are filtered out automatically. Check the source dashboard for filtered counts.