GuidesAI reliability

LLM as a judge: grading an AI’s answers automatically

By the Banzai Labs teamUpdated on 2 min read

Answer to gradeJudge modelRubricFaithful to sourcesAnswers the questionRight toneScore: 2 / 3One model grades another’s answers, criterion by criterion
A judge model grades an answer against a rubric, criterion by criterion.

A rubric rather than an overall score

Criterion Question asked to the judge
Faithfulness Is every claim backed by the sources provided?
Relevance Does the answer address the question asked?
Completeness Is any expected information missing?
Tone Does it follow the style guidelines?

A structured grade

const res = await client.messages.create({
  model: process.env.CLAUDE_MODEL!,
  max_tokens: 1024,
  tools: [{
    name: 'grade',
    description: 'Records the grade of the answer, criterion by criterion',
    input_schema: {
      type: 'object',
      properties: {
        faithful: { type: 'boolean' }, relevant: { type: 'boolean' }, complete: { type: 'boolean' },
        justification: { type: 'string' },
      },
      required: ['faithful', 'relevant', 'complete', 'justification'],
    },
  }],
  tool_choice: { type: 'tool', name: 'grade' },
  messages: [{ role: 'user', content: rubric(question, sources, answer) }],
});
const grade = res.content.find((b) => b.type === 'tool_use')?.input;

Calibrating the judge

  1. 1Human sample50 answers graded by hand.
  2. 2Same sample, judgeCompare the two grades.
  3. 3Adjust the rubricUntil agreement is good.

Yes-or-no criteria give more stable grades than a 1 to 10 scale.

Use cases: evaluating a RAG system and test an AI chatbot. The technique: reliable JSON from an LLM.

Frequently asked questions

Can the judge be the same model as the one being evaluated?
It can, but a model tends to prefer its own answers. A different model, or a very factual rubric, limits that bias.
Which biases should I know about?
A preference for long answers, for the first answer shown in a comparison, and for its own style. Shuffle the order and score length separately.
How do I know the judge is reliable?
Have the same sample graded by humans and by the judge, and measure how much they agree. Adjust the rubric while agreement is low.

Read next

All guides