A practical checklist for your first RAG knowledge assistant

What to prepare, what to test and what to watch before your team relies on it.

XAXylisys AI Team··1 min read

Retrieval-augmented generation (RAG) lets an assistant answer from your own documents instead of guessing. Getting a demo working is easy. Getting answers your team will trust takes preparation.

Before you build

  • Pick one audience and one job, such as “support agents answering policy questions”
  • Collect the 50 questions people actually ask, with the correct answers
  • Find the source of truth for each answer and who owns it

While you build

Chunk documents by meaning, not by page. Keep titles and section headings with every chunk. Store permissions alongside content so people only get answers from documents they are allowed to see.

Before you launch

Run your 50 questions and score each answer for correctness and citation. Add “I don’t know” as an acceptable answer — it is far better than a confident wrong one.

A good first target: 90% of answers correct with a valid source, and zero answers that cite the wrong document.

After launch

Log every question, let users flag bad answers, and review them weekly. Most improvements come from fixing content, not from changing the model.

Wondering where AI fits in your business?Take the 2-minute AI readiness check.
Start the check
XA
Xylisys AI TeamAI engineering & consulting

The engineers and consultants who design, build and run AI agents, assistants and automations for clients worldwide.