RAG Evaluation: The Same Thirty Questions, Every Week

In the Real World · Brief · AI · Core · 2 to 3 days

297 documents across six places, 146 questions, and an agency that answers the same thirty every week. Build the assistant and the thing that proves it works.

The situation

Kelsall Partners is an agency of forty people. The same questions come up constantly and the answers are spread across a wiki, meeting notes, policies, client checklists, chat threads and the rate card. There are 297 documents in total.

The data

What the room believes

1. The main risk is the assistant inventing answers.
2. A better model is the way to improve it.
3. Chunking is a detail.
4. If the answer is in the corpus, the assistant will find it.

Definition of done

1. A verdict on each of the four beliefs, with the evidence.
2. A working assistant over the corpus, and the evaluation set that scores it.
3. Retrieval quality and answer quality reported separately, with a statement of which one is limiting you.
4. A measured answer to what happens when there is no answer, and when two documents disagree.
5. A recommendation, including anything you would change that is not the model.

If you want to go further

• Write questions that have no answer in the corpus, and see what comes back.
• Put an outdated document in the index that contradicts a current one, and ask about it.
• Compare whole documents against smaller chunks, and look at the multi document questions specifically.
• Split your accuracy by whether the right document made it into the context.
• Change one sentence of the prompt and re-run the whole evaluation. That is what the evaluation is for.

In the Real World · AI · Core · 2 to 3 days

A retrieval assistant over an agency’s own notes. It almost never invents an answer, which is the failure everyone worries about. It quotes the superseded rate card on 33.3% of the questions where an old document disagrees with a current one, which is the failure nobody tests for.

Section 1Problem Definition

No code yet. Everyone builds the assistant and nobody builds the thing that says whether it works, which is the harder and more useful half.

The problem in one sentence

An assistant over your own documents will be confidently wrong in three specific ways, and only one of them is the one people test for.

Business objective

Ship an assistant with a written statement of what it gets right, what it gets wrong, and which of those a prompt can fix.

Hypotheses

1. H1. The main risk is the assistant inventing answers.
2. H2. A better model is the way to improve it.
3. H3. Chunking is a detail.
4. H4. If the answer is in the corpus, the assistant will find it.

Section 2Data Collection

import numpy as np
import pandas as pd

d = pd.read_csv('data/kelsall-knowledge-base.csv')
q = pd.read_csv('data/kelsall-questions.csv').fillna('')

print('documents        :', len(d))
print('carrying answers :', int(d['carries_answer'].sum()))
print('filler           :', int((d['carries_answer'] == 0).sum()))
print('sources          :', d['source'].nunique())
print('questions        :', len(q))
print(q['kind'].value_counts().to_string())

documents        : 297
carrying answers : 17
filler           : 280
sources          : 6
questions        : 146
single        75
absent        47
multi         12
superseded    12

280 of the 297 documents answer nothing. They are retrospectives, supplier reviews and threads about the office move, and they exist because retrieval that only has to choose between seventeen candidates is not retrieval.

Section 3Data Preprocessing

3aDuplicates and schema checks

print('duplicate doc ids :', int(d['doc_id'].duplicated().sum()))
print('duplicate questions :', int(q['question'].duplicated().sum()))
print('answerable questions naming a missing document :',
      int(sum(1 for r in q[q['answerable'] == 1].itertuples()
              for x in str(r.answer_doc_ids).split('|')
              if x not in set(d['doc_id']))))
print('absent questions with an answer key :',
      int((q[q['answerable'] == 0]['answer'].str.strip() != '').sum()))

duplicate doc ids : 0
duplicate questions : 0
answerable questions naming a missing document : 0
absent questions with an answer key : 0

print(d['source'].value_counts().to_string())
print()
print('documents carrying an answer, by source:')
print(d[d['carries_answer'] == 1]['source'].value_counts().to_string())

source
wiki        66
thread      62
notes       59
policy      54
client      54
ratecard     2

documents carrying an answer, by source:
source
wiki        8
policy      5
ratecard    2
client      2

3bHandling categorical mess

Six sources with different shapes. They are concatenated into one index on purpose: the whole problem is that nobody knows which of the six holds the answer, and a search that requires you to pick first has not helped.

3cDealing with outliers

No outliers. The rate cards are short and carry three of the most asked figures in the business, which is the opposite of an outlier and closer to a landmine.

3dHandling missing values

Nothing is missing from the documents. 47 of the questions have no answer anywhere, which is a property of the question set rather than of the data, and section 7 measures what the assistant does with them.

3eHandling skewed data

The answers are concentrated in 17 documents out of 297. A retriever that returns five results has to put one of those in the top five against 280 competitors from the same organisation, written in the same voice.

3fData types and normalisation

Two chunkings are built and compared rather than assumed: whole documents, and overlapping pairs of sentences. Section 7a is the argument for not treating that as a detail.

Section 4Exploratory Data Analysis

4aTarget variable analysis

There are two targets and they are usually conflated. Whether the right document was retrieved, and whether the answer was right. Everything below reports them separately, because a system can fail at either and the fixes are different.

print(pd.crosstab(q['kind'], q['answerable']).to_string())
print()
print('questions needing two documents :',
      int(q['answer_doc_ids'].str.contains(chr(124), na=False).sum()))
print('questions with a stale rival    :',
      int((q['stale_answer'].astype(str).str.strip() != '').sum()))

answerable   0   1
kind              
absent      47   0
multi        0  12
single       0  75
superseded   0  12

questions needing two documents : 12
questions with a stale rival    : 12

4bNumerical variables

Documents average 38.2 words. Short enough that whole-document retrieval is viable, which makes the chunking comparison in 7a a fair fight rather than a foregone conclusion.

4cCategorical variables

The question kind is the field that makes this legible, and it is the field a real team has to create by hand. It is also the entire reason the staleness problem is visible at all: without a superseded class, those twelve questions are just twelve more questions that mostly get answered.

# The same figure, stated twice, three years apart, both still indexed.
for did in ('RATE-2026', 'RATE-2025'):
    print(did, '::', d.loc[d['doc_id'] == did, 'text'].iloc[0][:150])

RATE-2026 :: This rate card is effective from January 2026 and supersedes all previous versions. Senior strategist: 1,450 per day. Junior analyst: 540 per day. Min
RATE-2025 :: This rate card applied during 2025 and has been superseded. Senior strategist: 1,200 per day. Junior analyst: 480 per day. Minimum monthly retainer: 3

4dRelationships between variables

The rate card exists twice. The 2026 version says it supersedes all previous versions and the 2025 version is still in the index, still says what it said, and is retrieved just as readily.

4eTesting our hypotheses

4fSubgroups

Four question kinds, and three of them behave nothing like the average.

Section 5Feature Engineering

5aThe leakage trap

5bNew features

# Every model call is cached and deterministic. A rerun costs nothing and the
# published figures cannot drift under the article.
from ai_llm import Cache
probe = Cache('rag_cache.json')
print('responses on disk :', probe.stats()['calls'])
r1 = probe.ask('Reply with exactly: OK', num_predict=8)
r2 = probe.ask('Reply with exactly: OK', num_predict=8)
print('identical on repeat :', r1['response'] == r2['response'])

responses on disk : 610
identical on repeat : True

5cEncoding

The assistant replies with a boolean and a string rather than prose. That single decision is what makes refusing measurable, and refusing correctly is most of what separates a usable assistant from a plausible one.

5dFeature selection

Top 5 chunks, fixed. Retrieving more would raise recall and bury the answer in context, which is a real trade and not the one this project is about.

Section 6Model Selection

Section 7Model Training

7aBaselines

Retrieval first, because nothing downstream can beat it.

from sentence_transformers import SentenceTransformer

TOPK = 5
enc = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2', device='cpu')
QE = enc.encode(q['question'].tolist(), batch_size=128, normalize_embeddings=True)

def chunk(text, size, overlap=1):
    sents = [s.strip() for s in text.split('. ') if s.strip()]
    out, i = [], 0
    while i < len(sents):
        out.append('. '.join(sents[i:i + size]).rstrip('.') + '.')
        i += max(size - overlap, 1)
    return out

def build_index(size):
    ids, texts = [], []
    for r in d.itertuples():
        for c in ([r.text] if size is None else chunk(r.text, size)):
            ids.append(r.doc_id)
            texts.append(r.title + '. ' + c)
    return ids, texts, enc.encode(texts, batch_size=128, normalize_embeddings=True)

def recall(size, label):
    ids, texts, CE = build_index(size)
    order = np.argsort(-(QE @ CE.T), axis=1)[:, :TOPK]
    hit = {'all': [], 'single': [], 'multi': []}
    for i in range(len(q)):
        if not q['answerable'].iloc[i]:
            continue
        got = {ids[j] for j in order[i]}
        need = set(str(q['answer_doc_ids'].iloc[i]).split('|'))
        ok = need.issubset(got)
        hit['all'].append(ok)
        if q['kind'].iloc[i] in ('single', 'multi'):
            hit[q['kind'].iloc[i]].append(ok)
    print('%-16s %8d %8.1f%% %7.1f%% %7.1f%%'
          % (label, len(texts), 100 * np.mean(hit['all']),
             100 * np.mean(hit['single']), 100 * np.mean(hit['multi'])))

print('%-16s %8s %9s %8s %8s' % ('chunking', 'chunks', 'overall', 'single', 'multi'))
recall(None, 'whole document')
recall(2, 'two sentences')

chunking           chunks   overall   single    multi
whole document        297     89.9%    97.3%    33.3%
two sentences        1454     91.9%    96.0%    58.3%

Changing the chunking moves overall recall by 2.0 points, which is the number anybody would report, and moves multi document questions from 33.3% to 58.3%. With five slots and whole documents, two of them have to be the right two.

7bComparing candidates

import sys
sys.path.insert(0, '../_kit')
from ai_llm import Cache

cache = Cache('rag_cache.json')
SCHEMA = {'type': 'object',
          'properties': {'can_answer': {'type': 'boolean'},
                         'answer': {'type': 'string'}},
          'required': ['can_answer', 'answer']}
GROUNDED = (
    'You are an assistant for Kelsall Partners staff. Answer ONLY from the notes '
    'below. If the notes do not contain the answer, set can_answer to false and '
    'leave answer empty. Never guess. If two notes disagree, use the one that is '
    'current and ignore anything marked superseded.'
    '\n\nNOTES:\n{ctx}\n\nQUESTION: {q}\n\n'
    'Reply as JSON with can_answer and answer.')

def run(model, prompt_template, ctxs):
    out = []
    for i in range(len(q)):
        ctx = '\n'.join('- ' + c for c in ctxs[i])
        got = cache.ask_json(prompt_template.format(ctx=ctx, q=q['question'].iloc[i]),
                             SCHEMA, model=model, num_predict=220)
        out.append(got or {})
    return out

model        prompt      correct   invented    stale   parse
Gemma 4      plain         84.8%       2.1%     8.3%       0
Gemma 4      grounded      85.9%       0.0%     0.0%       0
Qwen 3.6     plain         87.9%       0.0%    33.3%       0
Qwen 3.6     grounded      86.9%       0.0%     0.0%       0

Every configuration answers around 84.8% to 87.9% of the answerable questions correctly, and they are not the interesting column. Qwen 3.6 on the plain prompt quotes the 2025 rate card on 33.3% of the questions where both versions exist.

# Grading is mechanical: did it claim to know, and is the right value in the answer.
def grade(preds, kind):
    idx = [i for i in range(len(q)) if q['kind'].iloc[i] == kind]
    if kind == 'absent':
        return 100 * np.mean([not p.get('can_answer', False) for p in
                              [preds[i] for i in idx]])
    ok = [bool(preds[i].get('can_answer'))
          and str(q['answer'].iloc[i]).lower() in str(preds[i].get('answer', '')).lower()
          for i in idx]
    return 100 * np.mean(ok)

def stale_rate(preds):
    idx = [i for i in range(len(q)) if q['kind'].iloc[i] == 'superseded']
    return 100 * np.mean([str(q['stale_answer'].iloc[i]).lower()
                          in str(preds[i].get('answer', '')).lower() for i in idx])

7cHyperparameter tuning

The prompt is the parameter. Its effect on invented answers is small because there were few to begin with, and its effect on staleness is total.

Gemma 4      invented   2.1% ->   0.0%   stale   8.3% ->   0.0%
Qwen 3.6     invented   0.0% ->   0.0%   stale  33.3% ->   0.0%

# Which half of the system failed. Retrieval, or the model given the right context.
def split_by_retrieval(preds, got_docs):
    ok_ctx, no_ctx = [], []
    for i in range(len(q)):
        if not q['answerable'].iloc[i]:
            continue
        need = set(str(q['answer_doc_ids'].iloc[i]).split('|'))
        right = (bool(preds[i].get('can_answer'))
                 and str(q['answer'].iloc[i]).lower()
                 in str(preds[i].get('answer', '')).lower())
        (ok_ctx if need.issubset(set(got_docs[i])) else no_ctx).append(right)
    return 100 * np.mean(ok_ctx), 100 * np.mean(no_ctx)

7dFinal evaluation

Splitting the result by whether retrieval succeeded separates the two halves of the system, and settles which one to work on.

Gemma 4      plain     right when the document was retrieved  87.9%, when it was not  50.0%
Gemma 4      grounded  right when the document was retrieved  89.0%, when it was not  50.0%
Qwen 3.6     plain     right when the document was retrieved  91.2%, when it was not  50.0%
Qwen 3.6     grounded  right when the document was retrieved  90.1%, when it was not  50.0%

Retrieval finds the document for a single fact 96.0% of the time and both documents for a two hop question 58.3% of the time. Everything the assistant does afterwards is bounded by this chart, which is why the multi document row in section 7d cannot be fixed by prompting.

Given the right document, every configuration answers between 87.9% and 91.2% correctly. Without it they land at 50.0%, which is what guessing between a plausible pair looks like. The binding constraint is retrieval at 91.9%, not the model.

Read the multi document row. Every configuration sits near 33.3% because retrieval only gets both documents into the context 58.3% of the time. No prompt and no model fixes that, and it is the one row where a bigger retrieval budget would.

Section 8Documentation and Handoff

What to do, and who owns it

What not to do

• Do not test only for hallucination. It was at most 2.1% here while staleness was at 33.3%.
• Do not report one accuracy figure. Multi document questions sit 61.4 points below single document ones.
• Do not treat chunking as a detail. It nearly doubled multi document retrieval while barely moving the average.
• Do not grade with a language model when the answer is a fact. A string check needs no calibration and cannot be flattered.
• Do not swap models to fix a retrieval problem. Two models here are within a point and a half of each other and both are capped by the same 91.9%.
• Do not leave superseded documents in the index. The prompt is a mitigation, not a fix.

Reproducibility

What to take from this

• The evaluation set is the deliverable. The assistant took an afternoon; knowing where it fails took the rest of the project.
• Hallucination is the failure people test for and staleness is the one they ship. Both need a question class of their own.
• Retrieval is usually the binding constraint. Split your accuracy by whether the right document was in the context before touching the model.
• Chunking is not a detail. It decides whether two facts can appear together at all.
• Make refusing a first class output. A boolean is measurable, a polite sentence is not.
• Grade against facts where you can. A judge is for when you have no alternative, not for when the answer is a number.

The assistant was never the hard part. Two prompts, two models and a fortnight’s worth of the agency’s own notes get to about 87.9% on questions that have answers. The work that mattered was writing down the questions that do not, and the ones where two documents disagree, because those are the answers somebody would have acted on.