LLM as a Judge

In the Real World · AI · Core · 2 days

The plan was to let a language model grade support answers, because nobody wants to read four thousand of them a month. Two local models were given one hundred answers whose grades were already known. They agreed with those grades 10.0% and 17.0% of the time, and with each other 54.0% of the time. A rubric of five lines of Python agreed 91.0%. The rubric is also the only grader here that cannot notice a lie.

Section 1Problem Definition

The request arrives as a tooling question: which model should we use to score support answers? That question cannot be answered, because it assumes the thing worth checking. Before choosing a judge you have to know what a good grade is, and then you have to catch the judge being wrong about it.

So the project is built backwards. Write the answers first, with the grade each one deserves already decided. Then ask every candidate grader to recover the grades you already know. A grader that cannot recover a grade you constructed on purpose will not recover one you did not.

The five things an answer can be

The padded grade is the control that makes the rest of the project honest. A judge that rewards length will score padded above correct, and because padded contains correct word for word, nothing else can explain the gap.

Section 2Data Collection

Twenty member questions, each with a required fact and a required second point. Five answers per question, one of each grade, giving 100 answers with known grades.

import pandas as pd

ans = pd.read_csv("data/ferrow-answers.csv")
rub = pd.read_csv("data/ferrow-rubric.csv")

print(ans.shape, rub.shape)
print(ans.true_grade.value_counts().to_dict())
print(ans[["true_grade", "answer"]].head(3)
        .to_string(index=False, max_colwidth=58))

(100, 6) (20, 4)
{'correct': 20, 'incomplete': 20, 'wrong': 20, 'evasive': 20, 'padded': 20}
true_grade                                                     answer
   correct A reset link is emailed and expires after 30 minutes, a...
incomplete      A reset link is emailed and expires after 30 minutes.
     wrong A reset link is emailed and expires after 24 hours, and...

Read the correct and the wrong answer for question one next to each other. They differ by two words. That is the whole difficulty of this project.

How a wrong answer is made

# Each wrong answer is the correct one with a single true fact
# replaced by a false one. Everything else is untouched: same
# structure, same tone, same second point.
correct = "A reset link is emailed and expires after 30 minutes, and you must be signed out to request one."
wrong   = correct.replace("30 minutes", "24 hours")

print(wrong)
print("words changed:", sum(a != b for a, b in
                            zip(correct.split(), wrong.split())))

A reset link is emailed and expires after 24 hours, and you must be signed out to request one.
words changed: 2

Section 3Data Preprocessing

3aDuplicates and schema checks

assert len(ans) == len(ans.answer_id.unique())
assert set(ans.true_grade) == {"correct", "padded", "incomplete", "evasive", "wrong"}
assert (ans.groupby("question_id").size() == 5).all()
assert set(ans.question_id) == set(rub.question_id)

print("rows", len(ans), "| questions", ans.question_id.nunique(),
      "| answers per question", ans.groupby("question_id").size().unique())

rows 100 | questions 20 | answers per question [5]

3bHandling categorical mess

The grade column is the only categorical here and it was written by the generator, so it is clean. The check that matters is not spelling but ordering: the grades have to map to numbers before anything can be compared.

DESERVES = {"correct": 5, "padded": 5, "incomplete": 3, "evasive": 1, "wrong": 1}
ans["deserves"] = ans.true_grade.map(DESERVES)

print(ans.groupby("true_grade").deserves.first().sort_values().to_dict())

{'evasive': 1, 'wrong': 1, 'incomplete': 3, 'correct': 5, 'padded': 5}

Two grades share a score deliberately. Correct and padded both deserve 5 because padding is not a defect, and evasive and wrong both deserve 1 because an answer that tells you nothing and an answer that tells you something false are equally unusable.

3cDealing with outliers

There are no numeric outliers to remove. There is one structural check worth making, because the entire length control depends on it.

pairs = ans.pivot(index="question_id", columns="true_grade", values="answer")
longer   = (pairs.padded.str.split().str.len() > pairs.correct.str.split().str.len())
contains = [c in p for c, p in zip(pairs.correct, pairs.padded)]

print("padded longer than correct in all pairs :", bool(longer.all()))
print("padded contains correct in all pairs    :", all(contains))

padded longer than correct in all pairs : True
padded contains correct in all pairs    : True

3dHandling missing values

print("empty answers:", int((ans.answer.fillna("").str.strip() == "").sum()))
print("nulls per column:", ans.isna().sum().to_dict())

empty answers: 0
nulls per column: {'answer_id': 0, 'question_id': 0, 'question': 0, 'true_grade': 0, 'answer': 0, 'answer_words': 0}

Worth stating that a missing value appears later in this project and not here. When a model is asked to write an answer rather than grade one, it can return nothing at all, and that is dealt with in section 7.

3eHandling skewed data

Answer length is skewed by construction, because two of the five grades are long on purpose.

print(ans.groupby("true_grade").answer_words.mean().round(1).to_dict())

{'correct': 16.0, 'evasive': 31.5, 'incomplete': 7.4, 'padded': 57.6, 'wrong': 15.8}

Padded runs to 57.6 words against 16.0 for the correct answer it contains, and incomplete is the shortest at 7.4. A grader that simply preferred longer text would rank padded first and incomplete last, which is a pattern worth being able to recognise later.

Length is confounded with grade on purpose, so that a length-sensitive grader gives itself away.

3fData types and normalisation

ans["answer"] = ans.answer.astype(str).str.strip()
ans["norm"] = ans.answer.str.lower()
rub["required_fact"] = rub.required_fact.str.lower()
rub["required_second"] = rub.required_second.str.lower()

print(ans.dtypes.to_dict())

{'answer_id': dtype('O'), 'question_id': dtype('O'), 'question': dtype('O'),
 'true_grade': dtype('O'), 'answer': dtype('O'), 'answer_words': dtype('int64'),
 'deserves': dtype('int64'), 'norm': dtype('O')}

Section 4Exploratory Data Analysis

4aTarget variable analysis

The target is the grade a human would give, and it is balanced by construction: twenty of each. That balance is not realism, it is measurement. A real queue is mostly correct answers, and a judge that scored everything 5 would look excellent on it.

4bNumerical variables

The only numeric column before grading is word count, covered above. After grading there are three score columns, and the useful view of each is not its average but its spread. A grader that uses one point of a five point scale has not graded.

4cCategorical variables

Twenty questions across policy, claims, billing and account access. No grader in this project is given the topic, so nothing here can be exploited by one and not another.

4dRelationships between variables

The relationship the project turns on is between length and grade, and it is planted. The relationship it hopes to find is between a grader’s score and the grade deserved.

4eTesting our hypotheses

4fSubgroups

The subgroup that decides this project is the twenty wrong answers. Overall agreement hides them, because they are one fifth of the set and every grader here gets most of the other four fifths right or wrong for reasons that do not matter commercially.

Section 5Feature Engineering

5aThe leakage trap

5bNew features

The rubric needs two features per answer, and both are membership tests rather than anything learned.

def features(answer, required_fact, required_second):
    text = answer.lower()
    return {
        "has_fact":   required_fact[:28] in text,
        "has_second": required_second[:24] in text,
    }

print(features("A reset link is emailed and expires after 30 minutes, and you "
               "must be signed out to request one.",
               "a reset link is emailed and expires after 30 minutes",
               "you must be signed out to request one"))

{'has_fact': True, 'has_second': True}

The prefix lengths are a deliberate compromise. Matching the whole required fact would fail on any harmless rewording. Matching a short prefix would pass almost anything. Both numbers were chosen from the question sheet before grading, and the cost of that choice is measured in section 7.

5cEncoding

Two booleans become one score on the same one to five scale as everything else, so that the rubric and the model judges can be compared without a translation step.

def rubric_score(answer, required_fact, required_second):
    f = features(answer, required_fact, required_second)
    if f["has_fact"] and f["has_second"]:
        return 5
    if f["has_fact"]:
        return 3
    return 1

That is the whole grader. Five lines, no model, no network call, and it runs on the full set in under a millisecond.

5dFeature selection

There is nothing to select from. The rubric is deliberately the smallest thing that could work, because its value in this comparison is as a floor. If the floor wins, the finding is about the ceiling.

Section 6Model Selection

Three graders: the rubric above, and two local models asked to score the same answers. Both models run on the machine, at temperature zero, with a fixed seed, and every response is cached on disk.

JUDGE_PROMPT = '''You are grading a customer support answer.

QUESTION: {q}
ANSWER: {ans}

Score the answer from 1 to 5, where 5 is complete and correct and 1 is
useless or misleading. Reply with JSON only.'''

SCHEMA = {"type": "object",
          "properties": {"score": {"type": "integer"},
                         "reason": {"type": "string"}},
          "required": ["score", "reason"]}

Determinism, and why it is not optional

cache = Cache("judge_cache.json")

got = cache.ask_json(JUDGE_PROMPT.format(q=row.question, ans=row.answer),
                     SCHEMA, model="gemma4:latest", num_predict=220)

# Every response is keyed by model, prompt, schema, token budget and
# seed. Re-running the analysis re-reads the file and asks nothing.

cache: {'calls': 320, 'hits': 320, 'misses': 0, 'empty_responses': 0, 'unparsed_responses': 0}

Caching here is not a speed optimisation, though it is that too. It is what makes the quality assurance in this project possible: the gate recomputes every published figure from the cached responses by a second route, and never asks a model anything.

Section 7Model Training

7aBaselines

Two baselines, both of which a grader has to beat before it is worth discussing.

Both model judges come in below the constant baseline. That is not a figure of speech: scoring every answer 5 without reading it agrees with the known grades more often than either model does.

7bComparing candidates

rows = []
for grader in ["rubric", "judge_gemma4", "judge_qwen3.6"]:
    exact = (ans[grader] == ans.deserves).mean()
    within = (abs(ans[grader] - ans.deserves) <= 1).mean()
    by_grade = ans.groupby("true_grade")[grader].mean()
    rows.append({"grader": grader,
                 "exact": round(100 * exact, 1),
                 "within_one": round(100 * within, 1),
                 "length_bias": round(by_grade["padded"] - by_grade["correct"], 2),
                 "catches_wrong": round(by_grade["correct"] - by_grade["wrong"], 2)})

print(pd.DataFrame(rows).to_string(index=False))

       grader  exact  within_one  length_bias  catches_wrong
       rubric   91.0        91.0         +0.00           +2.30
 judge_gemma4   10.0        65.0        -0.15           +0.50
judge_qwen3.6   17.0        81.0        +0.05           +1.05

The cheapest grader in the comparison is the only one that beats a constant.

The within-one column is the charitable reading, and it is where the model judges look least bad: 81.0% for Qwen 3.6. It is also the column that matters least. A grader that is reliably one point out on a five point scale cannot be used to decide anything, because the decisions the team wants to make are at the ends.

Where each grader puts each grade

print(ans.groupby("true_grade")[["rubric", "judge_gemma4", "judge_qwen3.6"]]
        .mean().round(2).to_string())

            rubric  judge_gemma4  judge_qwen3.6
true_grade
correct       5.00          2.60           3.55
evasive       1.00          1.90           1.65
incomplete    3.00          2.35           2.75
padded        5.00          2.45           3.60
wrong         2.70          2.10           2.50

Neither judge separates an answer that is right from one that is false by more than about a point, and one of them barely moves at all.

Gemma 4 puts every grade between 1.90 and 2.60. That is not a grader with poor judgement, it is a grader that is not discriminating: the whole five point scale has collapsed into two thirds of one point. Qwen 3.6 has more range and gets the ordering right, and still puts wrong answers at 2.50, above incomplete answers that are merely missing half the content.

The length hypothesis does not survive

Both judges were expected to reward padding. Neither does: -0.15 and +0.05 points for an answer that is 3.6 times longer and contains the shorter one word for word. This is a real negative result and it is worth reporting as one, because length bias in language model judges is widely assumed and it did not appear here.

7cHyperparameter tuning

There is one setting in this project and getting it wrong silently destroyed a result, so it is worth the space.

Both of these models reason before they answer. Under a response schema that costs nothing. Asked for free text at a small token budget, they spend the entire budget thinking and return an empty string with a completion reason of length. An empty answer is not an error the caller can see. It is a blank string that gets graded, and it grades badly.

for budget, think in [(160, None), (160, False), (600, None)]:
    r = call("gemma4:latest", question, num_predict=budget, think=think)
    print("budget %-4d think %-5s tokens %-4d reason %-7s %r"
          % (budget, think, r["eval_count"], r["done_reason"],
             r["response"][:40]))

budget 160  think None  tokens 160  reason length  ''
budget 160  think False  tokens 55   reason stop    'To reset your password, please visit the'
budget 600  think None  tokens 294  reason stop    'To reset your password, please navigate'

The harness now refuses to cache an empty response that hit the token ceiling. It raises instead, and names the fix.

7dFinal evaluation

The rubric’s blind spot

The headline agreement figure for the rubric, 91.0%, is an average across two very different jobs.

wrong = ans[ans.true_grade == "wrong"]
rest  = ans[ans.true_grade != "wrong"]

print("rubric exact on wrong answers   :",
      round(100 * (wrong.rubric == wrong.deserves).mean(), 1))
print("rubric exact on everything else :",
      round(100 * (rest.rubric == rest.deserves).mean(), 1))

rubric exact on wrong answers   : 55.0
rubric exact on everything else : 100.0

A rubric checks whether the required words are present, which is not the same question as whether the answer is true.

The rubric is perfect on form and close to a coin toss on truth, and the reason is mechanical rather than mysterious. It tests whether the first characters of the required fact appear in the answer. When the swapped fact falls inside that span the test fails and the answer is caught. When it falls outside, the span the rubric reads is intact word for word and the answer passes.

wrong["prefix_intact"] = [fact[:28] in a.lower()
                          for fact, a in zip(wrong.required_fact, wrong.answer)]
wrong["caught"] = wrong.rubric == 1

print(pd.crosstab(wrong.prefix_intact, wrong.caught))
print("every miss explained by the prefix:",
      bool((wrong.prefix_intact != wrong.caught).all()))

caught         False  True
prefix_intact
False              0    11
True               9    0
every miss explained by the prefix: True

All 20 are explained, with no exceptions. Compare the two cases directly.

The judges agree with each other more than with the truth

both = ans[["judge_gemma4", "judge_qwen3.6", "deserves"]]

print("judges with each other :", round(100 * (both.judge_gemma4 ==
                                               both["judge_qwen3.6"]).mean(), 1))
print("gemma with the truth   :", round(100 * (both.judge_gemma4 ==
                                               both.deserves).mean(), 1))
print("qwen with the truth    :", round(100 * (both["judge_qwen3.6"] ==
                                               both.deserves).mean(), 1))

judges with each other : 54.0
gemma with the truth   : 10.0
qwen with the truth    : 17.0

This is the most quietly dangerous number in the project. Two judges agreeing is the usual reassurance that a grading setup works, and here they agree with each other 54.0% of the time while agreeing with the truth 10.0% and 17.0%. They are not converging on the right answer. They are making the same kind of mistake, which is what you would expect from two models trained on overlapping data to be agreeable.

Does a model prefer its own writing

The obvious test is to have each model answer the same twenty questions, then have both judges score everything. The obvious comparison, a judge’s score for its own work against its score for the other model’s, does not work.

cross = pairs.pivot_table(index="author", columns="judge", values="score")
print(cross.round(2).to_string())

judge          Gemma 4  Qwen 3.6
author
Gemma 4        4.10      4.10
Qwen 3.6       4.20      4.90

Both judges rank the same author above the other, so a raw own-against-other gap would confound self-preference with one model simply writing better answers. The measure that separates them compares like with like: how much higher a judge scores a set of answers than the other judge scores those same answers.

for judge in ["gemma", "qwen"]:
    mine = pairs[pairs.author == judge]
    own   = mine[mine.judge == judge].score.mean()
    other = mine[mine.judge != judge].score.mean()
    print("%-6s scores its own work %.2f, the other judge gives it %.2f, gap %+.2f"
          % (judge, own, other, own - other))

gemma  scores its own work 4.10, the other judge gives it 4.10, gap +0.00
qwen   scores its own work 4.90, the other judge gives it 4.20, gap +0.70

One of the two marks its own homework up. The other does not, which is why the controlled version of this measure was worth building.

Qwen 3.6 scores its own answers +0.70 points above what the other judge gives those same answers. Gemma 4 shows +0.00. So self-preference is real but not universal, and it is not something to assume in either direction without measuring it on the models actually in use.

Note also the level. In this experiment both judges score real model-written answers between 4.1 and 4.9 on a five point scale. Handed answers that are genuinely poor, in the graded set, they behaved quite differently. A judge that gives everything a four is not measuring anything.

Section 8Documentation and Handoff

What ships is the harness, not a score. The team gets a script that regrades every answer from cached responses, a rubric they can extend question by question, and a written statement of what each grader cannot see.

The checks that run before any of this is believed

• Every published figure is recomputed from the cached responses by a second route, and no model is called during the gate.
• Every code block in this article is extracted and executed in document order.
• The padded answers are verified to contain the correct answers word for word, because the length result depends on it.
• The prefix explanation for the rubric’s misses is asserted, not asserted-ish: if it does not account for every miss, the build fails rather than printing the claim.
• An empty model response that hit the token ceiling raises instead of being cached.

That last one is in the list because it was learned the hard way, twice, in this series. A failed call that gets recorded as data does not announce itself. It produces a plausible number, and the number is wrong.