DataFramer

Guide for: AI/ML engineers, product leads, eval, and platform teams

For AI/ML engineers, product leads, evaluation, and platform teams using LLM judges but still checking the important calls by hand.

Problem

LLM Judges Are
Inconsistent, Biased, and Opaque

Solution Calibrate Judges Against Expert Judgment

We spoke to an AI team at a public company that had built its own internal eval platform, including multiple judges running against reviewed data with measured alignment. They planned to use those judges across a lot more production outputs, but they couldn't trust them.

We hear this a lot. Teams use an LLM judge because reviewing every output by hand doesn't scale. Then they realize the judge is another model with its own prompt, model version, biases, etc.

A few noteworthy problems that come up often:

  1. Lack of reliable ground truth

    Judges need human-verified answers to calibrate against, and even experts don't always agree.

    Expand to learn about the ground truth wall and see two doctors disagree on a case

    LLM judges need ground truth backed by human annotations or a golden dataset you trust for the use case.

    This is what the community calls the "ground truth wall." A stakeholder might ask what the judge is being calibrated against. The answer should come back to manual review.

    But in some cases, even human annotations can be difficult to trust. For tasks that require deductive reasoning, two experts may reach different determinations. For ex., for one of our healthcare customers, two doctors often disagree on the same case.

    "You can make a case for either one, but we need to maintain at least uniformity in what kind of logic we employ to make that answer."

    Those disagreements show where the rubric might be vague, or context is missing, or even a case that is genuinely ambiguous. Teams need to either resolve them before treating the labels as ground truth or calibrate a judge to review those cases as ambiguous.

  2. Variability, inconsistency, and reward hacking

    The same query can score differently, and judges can learn to game the rubric instead of judging it.

    Expand to see the specific biases that make judges unreliable, like reward hacking

    The exact same query can get a different score even on consistently reliable ground truth. Judges can be sensitive to prompt wording, model version changes, and certain answer styles. Some of the ways this can manifest are verbosity, self-enhancement, and position bias. Reward hacking, where a judge finds an easy way to satisfy the rubric without judging the output correctly, is also very common. We call these judges "yes-judges".

    These are repeatable failure modes, meaning changing the judge prompt may improve agreement in one area while making another worse.

  3. Poor judge-human alignment and insufficient diversity

    Judges are only checked against the cases that already have human reviews, and those sets can go stale.

    Expand to learn why a diverse dataset beats a bigger one, and how datasets go stale

    An alignment score between how humans and LLMs judge tells us how often the LLM matched the reviewed/annotated examples. But it only covers the datasets that have human reviews. Typically, this is the hardest part to scale for companies.

    Dataset diversity matters more than dataset size. A few hundred diverse examples might be substantially better than a few hundred homogeneous examples.

    A static golden dataset also may go stale over time. Teams tune against the same cases, the score improves, and the judge starts overfitting to the benchmark set. That is why new production failures should be continuously incorporated into these datasets.

    If procuring diverse enough production samples is hard for you, given your industry regulations for instance, you can explore synthetic data generation tools that keep your data true to the required structure and desired scenarios.

SOLUTION:
Calibrate Judges Against Expert Judgment.
Build clear rubrics. Turn expert review into ground truth. Use it to calibrate judges, then automate the checks you trust.

1Build Rubrics

What to do

Our recommendation to AI/ML teams is to start with defining clear rubrics and to not rush beyond this crucial step until they are confidently ready. Rubrics define how experts will review the work (AI interactions) and how LLM judges will grade it later.

This is the first step in operationalizing and standardizing your expert reviews. You need to make sure that reviewers get a clear explanation of what needs to be done, and how they need to evaluate and analyze AI's work.

Here are some helpful tips:

  1. Provide clear instructions. Say what the reviewer should look at, what evidence matters, and how to handle missing context.
  2. Construct clear dimensions. You should think thoroughly about how you want to use the result of the reviewing process and pick the right dimensions. For ex., a scale of 1-5 might fit best for your task or for some tasks, a binary scale might work better.
  3. Then add examples. Include good, bad, and ambiguous ones. Examples matter because a rule that looks clear on paper often breaks on the first borderline case. They also give reviewers a shared point of reference, which reduces variation between their reviews.

Treat rubrics as a working standard. When a review exposes a vague instruction or a missing case, improve it, but keep the old version so earlier decisions still make sense.

2Build Ground Truth

What to do

The finalized rubrics can be used to review real cases with experts who understand the work. And their decisions become the ground truth used to test the LLM judges.

Getting experts out of their day and into a routine of evals is very difficult and their time is expensive, so we need to make the review easy for them. A doctor, lawyer, analyst, or support lead should not need to understand AI traces, their syntax, prompt engineering, or eval tooling before giving useful feedback.

Prioritizing which cases go to experts, and giving them the production context beside the trace, is covered in Scale Expert Review and Build Reliable Ground Truth.

It is a good practice to have more than one expert review the same cases. If they disagree, reading their reasoning might prove insightful. You might need to fix the rubric when the instruction is vague, or add an example when a case is new.

Ideally, a useful review should help with the next similar case, not disappear into a spreadsheet or a one-off conversation. This way, over time, the review processes get faster and more efficient.

The feedback can also reveal a broader failure. Once one reviewer explains what went wrong, it might be prudent to look for other production cases with the same pattern. Some of those failures may have gone unnoticed before. That search is the subject of Discover, Diagnose & Fix AI Failures at Scale.

3Calibrate

What to do

To calibrate, you should run the LLM judge on the same cases the experts reviewed. We use the same rubric, with instructions and examples. This gives you a direct comparison between the judge and human ground truth.

You could test different judge prompts on different models and compare their accuracy and judge-human alignment. Then dive deeper into the disagreements to look for behaviour such as false passes, false fails, and reward hacking.

The dataset you run through the judges is critical. You should include normal cases, known failures, unusual inputs, ambiguous cases, and mistakes that would be expensive to miss in production.

Once the judge is aligned, you can be confident it will preserve the expert knowledge behind it. You should be ready to scale automated judgement in production and monitor how it is doing.

4Automate

What to do

Once an LLM judge has been calibrated against expert review, it can be used for automated evaluations.

Calibration results can help determine what is ready to be automated. Cases with strong agreement between the judge and human reviewers may be suitable for automated evaluation. Clear rules may be handled with deterministic guardrails. Ambiguous or high-risk decisions should continue to involve human review.

This should be established as an ongoing process and automated decisions should be continuously sampled and reviewed. Comparing judge decisions with expert judgments helps identify new problems and areas where the evaluation criteria need to be updated.

When experts review a new or disputed case, that decision can be added to the ground-truth dataset and included in future evaluations.

A judge score on its own does not show whether users were better off. Tying these checks back to the outcomes they are meant to protect is covered in Connect AI Quality to User & Business Outcomes.

FAQ

Frequently asked questions.

Ground truth What is ground truth?
Verified data used as the gold standard to check if an AI's answers are correct.
Judges & Calibration What does it mean to calibrate an LLM judge?
Run the judge on cases people have already reviewed. Then compare its verdict with the human verdict. The cases where they disagree tell you what needs work.
Reviewer agreement What if the human reviewers disagree?
Read those cases before using the labels as ground truth. The rubric may be vague, the evidence may be incomplete, or the task may allow more than one valid answer. DataFramer shows reviewer agreement by rubric dimension so you can see where this is happening.
How many reviewed examples do we need?
Start with enough cases to cover normal behaviour, known failures, difficult edge cases, and mistakes that would be expensive. If the first 200 examples all look alike, adding another 800 like them won't tell you much about the cases the judge misses.
Judge-Human Alignment What is a good Judge-Human Alignment score?
There isn't a useful universal cutoff. A team checking marketing copy can tolerate a different miss rate than one checking a regulated recommendation. Decide from the false passes, false failures, and the worst disagreement cases, not the average score alone.
Judge API Can we use the judge in production?
A calibrated DataFramer judge can grade one interaction through the API and return the verdict inline. The call and verdict are recorded for review. Keep sampling those decisions and send uncertain or high-risk cases back to experts.

Let's see where your judge gets it wrong.

Start free with a $5 credit, add a set of reviewed traces, and compare your judge with the people who understand the work.