LLM-as-Judge: Why It's Hard to Get Right

LLM-as-judge has become the default approach for evaluating AI systems at scale. Running humans through every output is too slow and too expensive, so teams use another LLM to score the first one. It works well enough to get started. The trouble is that most teams don’t realize how much calibration these judges need before you can trust them, and they find out only after they’ve already built a lot on top of them.

What is LLM-as-judge?

It’s a technique where a language model evaluates the outputs of another model. You give the judge some criteria, a set of examples, and the output you want scored, and it returns a score with some explanation.

The typical setup: your base LLM handles user queries. For each output, you send the response along with the original query and any retrieved context to a separate judge LLM. The judge evaluates whether the response met your criteria, hallucinated any facts, followed the instructions, and so on. Teams use judges both inline (real-time evaluation) and in batch (offline evaluation over historical traces).

Where judges actually work

Judges are good at getting you off the ground quickly. If you have no evaluation process at all, adding an LLM judge gives you coverage fast. They’re useful for:

• Catching obvious failures like off-topic responses, formatting violations, or complete non-answers
• Comparing two versions of a prompt or model across a test set
• Running broad quality checks across thousands of traces that would take weeks to review manually
• Flagging outputs for human review rather than making final determinations

That last point matters: judges work best when they’re part of a larger process, not the final word.

At production scale, this often becomes a triage problem: find the traces most likely to contain an accuracy failure, then send the uncertain or high-impact cases to reviewers.

Here’s how to set one up for a simple use case like evaluating student writing:

1. Pick your judge model. A frontier model for complex criteria, a smaller one if you’re evaluating at high volume and cost matters.
2. Define evaluation criteria. Factual accuracy, topic adherence, style requirements, whatever matters for your use case.
3. Write the evaluation prompt. Be specific about the rubric you want applied.

4. Automate at scale. Submit outputs inline or in batches and collect scores automatically.
5. Review results on a cadence. Don’t just trust the scores. Someone needs to audit whether the judge is getting things right.

Why getting a reliable judge is hard

Here’s the core problem: LLMs are trained to generate text, not to score it consistently. Scoring requires applying a fixed rubric the same way every time. LLMs don’t do that.

That image shows GPT-4o-mini giving different scores for the exact same query on separate runs. This isn’t a quirk. Research has documented the problem clearly:

LLMs fail to respect scoring scales given to them

• Large Language Models are Inconsistent and Biased Evaluators (link)

We find limited evidence that 11 state-of-the-art LLMs are ready to replace expert or non-expert human judges, and caution against using LLMs for this purpose.

• LLMs instead of Human Judges? A Large Scale Empirical Study (link)

The important word in that second finding is replace. An uncalibrated judge dropped in to substitute for human review is unreliable. That’s a different thing from a judge that has been calibrated against human-labeled examples and used to extend the reach of expert review rather than replace it. The rest of this article is about getting to that second case, because a well-calibrated judge is the only practical way to evaluate AI quality at production scale.

Beyond inconsistency, judges run into several other reliability problems:

Shared biases with the model they’re judging. If your primary LLM and your judge are both GPT-4o, the judge will likely miss errors that GPT-4o systematically makes. They share blind spots. Using a judge from a different model family helps.

Prompt sensitivity. Small changes to the judge prompt shift scores. Writing good evaluation criteria is harder than it looks, and fragile prompts produce fragile scores.

Cost and latency. Using a strong model as a judge means paying for inference twice: once for the output, once for the evaluation. For real-time applications this often isn’t practical.

Score drift when you switch judges. If you change your judge model (from GPT-4 to Claude, say), your historical scores may no longer be comparable. Teams tracking quality trends suddenly have a discontinuity.

The judge still needs human oversight. The stated goal was to reduce how much humans review. In practice, someone still needs to audit judge outputs regularly to verify it’s scoring correctly. Blindly trusting a judge produces false confidence.

Best practices for building judges that work

Calibrate against human feedback. The most reliable judges have been tuned to match what your human reviewers say is good or bad. Collect examples that domain experts have labeled, test whether the judge agrees, identify where it diverges, and refine the prompt or fine-tune accordingly. Research shows even few-shot examples improve consistency significantly:

Few-shot in-context learning does lead to more consistent LLM-based evaluators

• Assessment and Mitigation of Inconsistencies in LLM-based Evaluations (link)

Don’t use the same model to generate and judge. A model that produced a response is a biased evaluator of that response. Use a different model family as your judge. This paper documents the cognitive biases this introduces in detail.

Provide labeled examples of good and bad outputs. Judges given concrete examples outperform those given only abstract criteria. If your judge has seen twenty examples of what counts as a hallucination in your domain, it catches them more reliably than if you just tell it to “check for factual accuracy.”

The examples also need to represent the range of failures you expect to encounter; coverage matters more than raw dataset size.

Use chain-of-thought prompting. Asking the judge to reason step-by-step before scoring leads to more accurate and consistent evaluations. The reasoning trace also makes it easier to catch when the judge went wrong.

Build a regression suite from your human-reviewed examples. Once you’ve had humans verify a set of scores, use those as tests. When you update your judge prompt or switch models, run the regression suite to detect drift before it affects your production evaluations.

Specific failure modes to watch for

Beyond general inconsistency, judges have documented systematic biases that are worth knowing before you deploy one.

Position bias. Judges tend to favor responses that appear earlier when comparing two outputs side by side. Research from Stanford found that simply swapping the order of two responses being compared could flip a judge’s preference (Zheng et al., 2023, “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”). If you’re using a judge to compare prompt variations, run the comparison both ways and average the results.

Verbosity bias. Judges tend to rate longer, more detailed responses higher regardless of accuracy. A response that confidently elaborates on wrong information can score better than a shorter, correct one. This matters in RAG applications where the LLM might fill gaps with plausible-sounding elaboration.

Self-preference. When a model judges its own outputs, it tends to rate them higher than outputs from other models. Using the same model as both generator and judge compounds this. Panickssery et al. (2024) documented self-preference systematically across multiple model families.

Knowing these biases helps you design around them: run comparisons in both orderings, test whether verbose answers are genuinely better or just superficially preferred, and use a different model family as your judge.

Multi-dimensional judging vs. single scores

A single “good or bad” score hides too much. A response can be factually accurate but fail on instruction following. It can follow instructions but be irrelevant to the actual query. A judge that collapses everything into one score makes it hard to know what’s actually broken.

Breaking evaluation into separate dimensions (factual accuracy, relevance, instruction adherence, completeness) gives you specific signal you can act on. It also makes judge calibration more tractable: it’s easier to show a judge twenty examples of instruction-following violations and teach it to catch those specifically, than to teach it a single holistic quality judgment.

This is especially true for agentic systems, where different steps in a trajectory might fail on different dimensions. A scoring rubric that covers each quality dimension separately gives reviewers a structured framework and makes it possible to track which dimension is degrading over time.

Ask, don’t judge

One useful alternative is to ask the evaluator narrower questions. This approach is described in Ask, Don’t Judge (Cho et al., ICML 2026 workshop), which introduces a framework called BINEVAL.

Consider the difference between these two prompts:

Broad score

”Rate this response from 1 to 5.”

Checkable question

”Does this response include a dollar figure that does not appear in the retrieved context? If so, quote it.”

The first prompt depends on the model applying an abstract scale consistently. The second produces an answer that a reviewer can check against the source.

BINEVAL applies that idea systematically: it breaks an evaluation criterion into atomic yes-or-no questions, answers each one independently, and combines the answers into scores for each dimension. In the paper’s experiments, this training-free approach matched or outperformed G-Eval and UniEval, with particularly strong results on factual consistency.

You do not need to adopt the full framework to use the underlying idea. Start by rewriting broad criteria as questions with observable answers:

• Not “score instruction following” but “the prompt asked for three bullets. How many are there?”
• Not “rate factual grounding” but “list every claim the context doesn’t support.”
• Not “score relevance” but “does this answer the question asked? Quote the part that does.”

You can still produce an overall score, but the score is now assembled using rules you define. More importantly, a reviewer can trace it back to the answers and evidence that produced it.

Narrow questions can also reduce the opportunity for verbosity bias. “Which response is better?” leaves plenty of room for style and length to influence the judge. “Which claims are unsupported by the context?” gives it a more specific job.

The calibration loop matters more than the judge itself

The teams that build reliable judges are the ones that treat calibration as an ongoing process, not a one-time setup. They collect human-reviewed examples, test the judge against them, identify systematic errors, improve the rubric or prompt, and repeat.

That loop requires somewhere to store the reviewed examples, track reviewer agreement, and surface where the judge diverges from humans. Without that infrastructure, calibration tends to happen once and then get forgotten, and the judge drifts silently over time.

Human review is still essential for high-stakes decisions. LLM judges are useful at scale but don’t understand your compliance requirements, your domain nuances, or what “correct” actually means for your specific business context. The best use of a judge is to extend the reach of human review, not to eliminate it.

LLM-as-judge vs. human evaluation

LLM judges and human reviewers have different strengths, which is why production evaluation usually needs both.

	LLM judge	Human review
Cost per output	Usually low, but depends on the model and prompt size	Higher, especially when domain experts are required
Throughput	Can evaluate large volumes automatically	Usually limited to a sample
Consistency	Can vary across runs even with the same rubric	Can vary between reviewers and over long review sessions
Coverage	Can be applied to every trace	Best used where human judgment adds the most value
Domain and compliance judgment	Limited to the context and instructions it receives	Can apply experience and understand what is at stake
Role in calibration	Produces scalable quality signals	Provides the reference labels used to test those signals

Human labels are not automatically perfect; reviewers can disagree or misunderstand a case. But they are still the reference point for deciding whether a judge reflects your standards. The practical goal is to use reviewers to establish and audit those standards, then use the judge to apply them across a much larger volume.

Common questions

Is LLM-as-judge reliable?

An LLM judge can be reliable enough to serve as a production quality signal, but it should not be trusted without testing. Judges can score the same input differently across runs and are susceptible to position, verbosity, and self-preference biases. Before relying on one, compare it with labeled examples from your domain, turn those examples into a regression suite, and continue having people audit a sample of its decisions.

What are the limitations of LLM-as-judge?

The main limitations are inconsistent scoring, sensitivity to prompt wording, shared blind spots with the model being evaluated, and added inference cost. Changing the judge model can also make new scores difficult to compare with historical ones. Domain expertise is another constraint: a judge cannot apply a requirement it was never given or reliably infer the business consequences of getting a decision wrong.

These limitations do not make LLM judges unusable. They do mean that a score is only meaningful in the context of a tested rubric, a known judge model, and a calibration set.

How reliable is multilingual LLM-as-a-judge?

Multilingual LLM judges are generally less consistent than English-language judges, and reliability varies substantially by language. A study of five models across 25 languages and five tasks reported average agreement of roughly 0.3 Fleiss’ Kappa, with lower consistency in many low-resource languages (Fu and Liu, 2025). The study also found that larger models and multilingual training did not, by themselves, solve the problem.

In practice, treat each language as its own evaluation problem. Calibrate against labels from fluent or native-speaking reviewers rather than assuming that a judge tested in English will behave the same way elsewhere.

In model evaluation, what is a documented con of using the LLM-as-judge technique?

One well-documented drawback is inconsistent scoring: the same judge can assign different scores to the same input across separate runs. Research has also documented position bias, verbosity bias, and a preference for outputs produced by the judge’s own model family (Stureborg et al., 2024).

Do LLM benchmarks and leaderboards use LLM judges?

Yes, some prominent benchmarks use LLM judges. MT-Bench uses a strong model to score responses, while Chatbot Arena is based on human preference votes. In the paper that introduced them, GPT-4 agreed with human preferences more than 80% of the time on the authors’ open-ended chat tests (Zheng et al., 2023).

That result helped establish LLM judging as a credible evaluation method, but it should not be treated as a guarantee for every domain. A judge used for legal research, customer support, or an internal agent still needs to be tested on examples from that application. Leaderboard results also inherit the biases of the judge that produced them.

Can an LLM judge evaluate AI agents?

Yes, but evaluating only the final answer misses much of what can go wrong. An agent chooses tools, supplies arguments, interprets results, and decides whether to continue. It may arrive at the right answer despite a poor tool call, or execute the workflow correctly and then produce a misleading summary.

A useful agent evaluation therefore looks at both the outcome and the trajectory. Did the agent select the right tool? Were its arguments valid? Did the final answer reflect what the tool returned? These narrower questions make failures easier to locate and give engineers something specific to fix. An outcome score is still useful as a top-line measure, but trajectory-level checks explain why that score changed.

How DataFramer Helps

DataFramer is the infrastructure that calibration loop needs. It stores your human-reviewed examples, tracks where reviewers agree, and calibrates a judge against them, then shows you exactly where the judge diverges from people so you can fix the rubric or the prompt. The loop keeps running as your data and models change, so the judge stays aligned instead of drifting quietly.