Guide for: AI Teams, Human-in-the-loop Program Leaders, AI Compliance & RAI Leads
For teams that need domain experts like doctors, lawyers, analysts, and underwriters to review production AI.
Scale Human-in-the-Loop Review and Build
Auditable Ground Truth
PROBLEM:
AI Teams struggle to put humans in the loop and turn their judgment
into auditable, reusable ground truth.
An AI leader at a Financial Services company told us he'd tried lists, spreadsheets, and "like 50 other ways" to get people to do evals. Legal and underwriting had the expertise, but getting them to stop their regular work and review AI traces was a different matter. As he put it:
"Getting people out of their day and into a routine of evals is very difficult."
The AI team knows it needs SMEs involved because engineers can't decide whether every medical, legal, financial, or compliance answer is actually correct. At the same time, sending a large random sample to experts is expensive. Some traces contain a normal engineering failure, some repeat an issue the team already knows about, and others need a simple check that never required a person in the first place.
A team building AI for their Security platform triaged this by asking what kind of failure it was.
"If it is an engineering failure, it goes back to engineering. If it completed but the response is suboptimal, then it goes to human review."
The wording there is important. A completed trace can still be wrong. The agent may have followed every technical step and produced an answer that looks right, while a security analyst looks at it and immediately sees that the recommendation doesn't make sense. These are the cases where expert time is useful.
For regulated teams, the review also needs to leave a reliable record. They need to know who reviewed the case, which evidence and rubric were used, why the decision was made, and what changed afterward. This history is difficult to preserve when reviews happen across spreadsheets, tickets, and Slack conversations.
Here is what makes it difficult:
-
Limited expert time and cost
Production AI systems generate more traces than human experts can review. This is true even for teams with dedicated evaluation staff, and it becomes a much bigger issue when reviewers have another full-time job.
A doctor's primary responsibility is treating patients. A lawyer has legal work to do. Underwriters, financial analysts, and security teams also have their own queues. Reviewing AI output is yet another task being added to the day.
One person who leads a large dataset annotation program described the process as "very operationally intense." Every data point eventually gets touched by multiple people.
There exists virtually no way to prioritize human reviewer assignments by the impact the AI has on the business, the opportunity loss from AI problems, or how many of them are happening.
-
Human feedback is difficult to capture and operationalize
End users may say an answer was wrong or didn’t help, which tells the team where to look but leaves someone to work out manually what failed and what should have happened.
Reviewers have to answer questions like:
- What evidence supports the decision?
- Which part of the answer failed?
- What would have been acceptable?
- Does the same logic apply to a slightly different case?
For AI agents and RAG systems, this requires the final answer, retrieved documents, tool calls, trajectory, applicable policies, and business context.
In many cases, an expert can tell that the answer is wrong, but an engineer might need to diagnose and answer: "why?".
We were surprised how many teams used spreadsheets for reviews. And spreadsheets tend to lose this complex context very easily.
The final answer gets copied into a row, a few columns are added for scores, and the rest of the interaction lives elsewhere. The reviewer makes a quick judgment or spends half the review tracking down documents and reconstructing what happened. Neither is a good use of the limited expert time.
-
Expert judgment is difficult to standardize between reviewers
Human reviewers may interpret vague terms differently, weigh evidence in different ways, or bring assumptions that were never written down.
Reviewer agreement gives AI teams a very useful signal. If two reviewers regularly disagree on one dimension, it's worth checking if instructions are vague, or a key document is missing. Maybe one reviewer is grading the final answer and the other is grading the complete agentic workflow.
"Two doctors may disagree."
While working with a leading healthcare data and analytics co., we learnt that two different doctors could make a case for their own determination fairly often.
This becomes especially important when the reviews are used to calibrate an LLM judge. A judge can't be aligned with a human standard that the humans themselves haven't agreed on. Tuning the prompts, switching models, and rerunning evals won't help in these situations.
-
Expert knowledge is difficult to reuse and audit
We see teams maintaining expert feedback in spreadsheets, tickets, or even Slack conversations. It doesn't become a rubric example, evaluation case, regression test, or sometimes even ground truth for calibrating LLM judges. When the same problem appears again, another person has to investigate it.
This also creates an auditability and compliance problem. When an internal governance team, customer, or auditor asks why an AI decision was approved, the team has to reconstruct the answer across several systems. They may not know which policy or rubric version the reviewer used, or whether someone changed the review later.
SOLUTION:
Prioritize cases that need expert judgment. Define clear rubrics and
queues, assign them with the complete context, and turn approved
decisions into reusable, auditable ground truth.
1Prioritize Cases
What to do
The first goal is to find the cases worth taking an expert away from their day for.
Start with real user signals and look for signs of where there may be a production AI problem like user complaints, corrections, or other types of feedback, repeated questions, human escalations, abandoned workflows, or any answers users ignore.
The strongest signal might be a business outcome moving in the wrong direction, for ex., falling completion, rising cycle time, etc. This gives teams a way to rank the review work by impact.
Then look for repeated patterns across the traces behind those signals. One bad answer might be an incident. The same failure across hundreds of cases is a much bigger product problem.
Quick tip: before an expert sees the case, remove obvious engineering errors and duplicates. If a tool timed out or retrieval returned an empty result, engineering has enough information to start investigating. Cases where the system completed the task but the quality of the result is still in question should go to the domain expert.
Capturing user signals and measuring them across the complete journey is covered in Connect AI Quality to User & Business Outcomes. For the process of finding repeated failures across production traces, see Discover, Diagnose & Fix AI Failures at Scale.
2Define Rubrics & Queues
What to do
Our recommendation to AI teams is to start with defining clear rubrics and to not rush beyond this crucial step until they are confidently ready. Rubrics define how experts will review the work and how LLM judges may grade it later.
This is the first step in operationalizing and standardizing expert reviews. Reviewers need a clear explanation of what needs to be done and how they should evaluate and analyze the AI's work.
Here are some helpful tips:
- Provide clear instructions. Say what the reviewer should look at, what evidence matters, and how to handle missing context.
- Construct clear dimensions. Think through how the result of the review will be used and pick the right scoring type. For ex., a scale of 1–5 might fit one task, while a binary score may work better for a required policy check.
- Then add examples. Include good, bad, and ambiguous ones. A rule that looks clear on paper often breaks on the first borderline case. Examples also give reviewers a shared point of reference, which reduces variation between their reviews.
The domain experts need a say in this process. Treat rubrics as a working standard. When a review exposes a vague instruction or a missing case, improve it, but keep the old version so earlier decisions still make sense.
Once the rubric is ready, a queue can help your team route the cases to the right reviewers. A queue should have a clear purpose, the right rubric attached, and a group of reviewers who understand that type of work.
For regulated work, the queue and rubric should also be a part of the review record. The team should be able to tell which standard applied to the case and who was responsible for reviewing it.
3Assign & Review
What to do
Getting experts out of their day and into a routine of evals is very difficult and their time is expensive, so the review needs to be easy for them. A doctor, lawyer, analyst, underwriter, or support lead should not need to understand AI trace syntax, prompt engineering, or eval tooling before giving useful feedback.
Assign the case with the original interaction and the context required to make the decision. Depending on the use case, this may include:
- The user's request and the AI's answer
- Retrieved documents and citations
- Prompts, tool calls, attachments, and policies
- The user actions that followed
Try not to drop a raw page of telemetry in front of the reviewer. Most SMEs don't need every technical field. They need to see the evidence that matters and be able to inspect more of the trace when they have a question.
Ask for enough information to use the review afterward. Usually, this means a grade for the relevant rubric dimensions, a short rationale, and a correction showing what the AI should have done. For an agent, the correction might apply to a tool call, handoff, or another intermediate step rather than the final answer.
Capture the expert's reasoning without turning every case into a long annotation project. A score, a short explanation, and a correction are generally more useful than a large form filled with fields no one uses later.
It is a good practice to have more than one expert review some of the same cases. This doesn't need to happen for every trace. A shared sample lets the team measure reviewer agreement and identify dimensions where the standard isn't being applied consistently.
If reviewers disagree, reading their reasoning might prove insightful. You might need to fix the rubric when an instruction is vague, provide a missing document, or add an example when a case is new. Some cases are genuinely ambiguous and should stay marked that way.
The review should also leave a reliable record. Keep the reviewer, evidence, rubric version, grades, rationale, and correction connected to the original case. This becomes important when an internal governance team, customer, or auditor needs to understand why a decision was approved.
4Use & Reuse
What to do
A useful correction might update the rubric as an example, enter an evaluation dataset, become a regression case, or be used to test an LLM judge.
Ideally, a useful review should help with the next similar case and not disappear into a spreadsheet or a one-off conversation. This way, over time, the review process gets faster and more efficient.
The feedback can also reveal a broader failure. Once one reviewer explains what went wrong, it might be prudent to look for other production cases with the same patterns.
For regulated teams, keep the review history replayable. The original trace, evidence, reviewer, rubric version, decision, rationale, correction, and any later changes should remain connected. If someone asks why an AI output was approved, the team should not have to reconstruct the answer across several systems.
For teams using LLM judges, run the judge on the same cases the experts reviewed.
The complete judge-calibration process is covered in Calibrate Judges Against Expert Judgment.
FAQ
Frequently asked questions.
- Ground truth What is ground truth?
- Ground truth is reviewed data used as the trusted reference for evaluation. For a complex AI interaction, the human verdict should stay connected to the relevant context, rubric version, rationale, and correction.
- Prioritization Does every trace need human review?
- No. Technical failures can go directly to engineering, and trusted rules and calibrated judges can handle well-understood cases. Use domain experts for important, ambiguous, new, disputed, or high-risk decisions, and continue sampling automated verdicts to find new problems.
- Reviewer agreement What if human reviewers disagree?
- Read their reasoning before treating the verdict as ground truth. The rubric may be vague, evidence may be incomplete, or the task may allow more than one valid answer. Improve the rubric, provide more context, add another example, or keep the case marked as ambiguous.
- How many reviewed cases do we need?
- There is no useful universal number. Start with enough cases to cover normal behaviour, known failures, difficult edge cases, ambiguous cases, and mistakes that would be expensive to miss. If the first 200 cases all look alike, adding another 800 like them will not tell you much about the failures your evaluation misses.
- Review Copilot Can Review Copilot replace the expert?
- No. Review Copilot can bring together the relevant context and draft a rationale or a correction. The expert reviews, edits, and approves the final decision.
- Judge-Human Alignment How does expert review make automated evaluation more reliable?
- Experts review real cases against a shared rubric, and those decisions become ground truth. The team can then run an LLM judge on the same cases and compare its decisions with the experts. Cases where they disagree show where the judge, the rubric, or the reviewed dataset needs more work.
- Datasets and evaluations How do reviews stay current?
- Keep adding newly confirmed production failures, important disagreements, unusual inputs, and cases introduced by changes to models, prompts, documents, retrieval, and workflows. Rubrics and ground-truth datasets should keep learning from production.
Put scarce expert time where it helps.
Start free with a $5 credit. Begin with a small set of production cases that need real domain judgment. Give experts the context and rubric they need to review the work, then carry the approved decisions into future reviews, evaluation datasets, and judges.