A practical checklist for your first RAG knowledge assistant
What to prepare, what to test and what to watch before your team relies on it.
Retrieval-augmented generation (RAG) lets an assistant answer from your own documents instead of guessing. Getting a demo working is easy. Getting answers your team will trust takes preparation.
Before you build
- Pick one audience and one job, such as “support agents answering policy questions”
- Collect the 50 questions people actually ask, with the correct answers
- Find the source of truth for each answer and who owns it
While you build
Chunk documents by meaning, not by page. Keep titles and section headings with every chunk. Store permissions alongside content so people only get answers from documents they are allowed to see.
Before you launch
Run your 50 questions and score each answer for correctness and citation. Add “I don’t know” as an acceptable answer — it is far better than a confident wrong one.
After launch
Log every question, let users flag bad answers, and review them weekly. Most improvements come from fixing content, not from changing the model.