Implementing RAG in your SaaS product.

Why RAG, and why in your product

Retrieval-augmented generation grounds a language model in specific content at answer time: the model reads retrieved passages from your data and answers from them, rather than from its training. For a SaaS product this is the difference between a generic chatbot bolted to your interface and a feature only you can ship, because only you hold the customer's data the answers come from. Support deflection, natural-language search over the customer's own records, document summarisation inside your workflow, onboarding assistants that know the account's actual configuration: all of these are retrieval problems before they are model problems. The model is rented and replaceable. The retrieval layer over your data is the product work, and it is where this guide spends its time.

The architecture that fits an existing product

A production RAG feature adds three components to your stack. An indexing pipeline extracts content from your existing stores, splits it into chunks, embeds them and writes them to a vector index, and, critically, re-runs when the source changes rather than on a nightly schedule, because a stale index answers yesterday's questions confidently. A retrieval service takes the user's query, runs semantic and keyword search over the index, and assembles the context. And a generation layer sends context and query to a model and returns the answer, ideally with citations your UI can render, because an answer that links to its sources earns trust an unsourced paragraph never will.

On the vector store: teams already running PostgreSQL should start with pgvector and move to a dedicated vector database only when scale demands it, which is later than most architecture reviews assume. Where retrieval needs live application state rather than documents, current account data, configuration, recent activity, extend retrieval into the application itself through internal APIs or MCP rather than trying to embed everything. The best answer often combines a retrieved document with a live lookup.

Multi-tenancy is the part you cannot retrofit

In a single-company deployment, a permissions mistake is embarrassing. In a multi-tenant SaaS product, it is one customer reading another customer's data, which is the kind of incident that ends contracts. Tenant isolation has to be structural in the index, not filtered at query time as an afterthought: tenant identifiers on every chunk at minimum, separate namespaces or indexes per tenant where the sensitivity warrants it, and retrieval that runs inside the tenant boundary the same way every other query in your product does.

Within a tenant, the same discipline applies at the user level: retrieval runs as the requesting user, and a record their role cannot open is a record the retriever never returns. Instructing the model not to reveal restricted content does not work, because content in the context window is available to the answer regardless of instructions. We have written up the design in more depth in role-aware RAG; the one-line version is that permissions are a retrieval problem, and solving them there is what lets you connect the data that makes the feature worth shipping.

Grounding, chunking and the quality details

Most quality problems trace to unglamorous causes. Chunking that splits a table or a procedure mid-thought produces retrieval that returns fragments no model can answer from; chunk along your content's natural structure, not a fixed token count. Corpora that grew by accumulation contain three versions of the same answer; curate what gets indexed, because the index is a product surface now. Measure retrieval quality separately from answer quality, since a wrong answer over correct passages and a correct-sounding answer over wrong passages are different bugs with different fixes. And prefer structured output where the feature feeds your product rather than a chat window: a JSON verdict with a reason can drive UI, filters and audit; prose cannot.

Evaluation before launch, observability after

Before the feature ships, build the evaluation harness: a few hundred real questions from support tickets, search logs and customer interviews, paired with correct answers and with the questions that should be refused, scored automatically on correctness and groundedness, run on every change. It is the asset that lets you swap models in an afternoon and catch the prompt change that quietly degraded quality, and we have argued the case in full in a companion post. After launch, log what was asked, what was retrieved, what was answered and what it cost, per tenant. That observability is how you find the tenants whose data needs curation, the queries the index cannot serve, and the cost outliers before the invoice does.

Cost, latency and model choice

Per-query economics decide whether the feature scales. Route by task: small fast models for classification and extraction, larger models where reasoning earns its cost, with fallbacks so no single provider outage takes the feature down. Cache aggressively at the retrieval layer, where repeated questions are common. Set token budgets per tenant tier, because an unmetered AI feature under a flat subscription is a margin problem waiting for a heavy user. And keep model choice a configuration decision, not an architecture decision; the harness makes each new model release an afternoon's experiment rather than a migration.

Where to start

One workflow, chosen because its data is in reasonable condition and its value is obvious to customers. A two-to-four week assessment covering the data, the tenancy model and the constraints. Then a first feature built on production architecture from day one, permissions, evaluation and observability included, because the pilot that skips them gets rewritten before it ships. Our AI practice has run exactly this arc for SaaS products, including the production system in the Grapple case study: document automation, a role-aware assistant and risk scoring, live on AWS Australia.