Guide for: AI & ML Teams
For AI/ML engineers, product leads, evaluation, and platform teams who have a production AI system but struggle to answer when and why it fails.
Discover, Diagnose & Fix AI Failures at Scale
PROBLEM:
It is difficult to move from high volumes of raw traces and vague user
complaints to discovering repeated failure patterns, evidence-backed
root causes, and fixes.
One of our customers had an agent get stuck in an indefinite loop and sent hundreds of emails before the team was alerted by customers and shut it down.
We often see teams search thousands of traces manually, patch a prompt, and move on to the next bug or complaint. There are a few reasons why production AI is hard:
-
Production scale and changes
Production traffic is the real-world test. You need to know whether responses stay consistent, grounded, and trustworthy when real users interact with the system. Documents, prompts, or models change, and users ask questions in unexpected ways.
-
Known and unknown issues.
There are known issues, such as inaccuracies/hallucinations, weak RAG grounding, and failed tool calls. In addition, there are issues no one is looking for. More like, no one knows to look for them. For example, your AI agent might promise a refund when it should not. Or it might complete the task but skip a required check. Failures can be silent while the final answer still looks very confident.
"We keep getting feedback that a recommendation was wrong or an answer did not help. People often cannot explain why."
-
Diagnosis is hard.
Teams have the raw logs and observability data. Narrowing down where things went wrong can still be hard. Answering that and pinpointing the reason(s) can mean reading pages of long traces or logs to understand what the agent got wrong.
-
Fixes are hard to verify.
Finding a failure doesn't make the correct fix obvious. Teams often patch the prompt even when the cause is retrieval, bad state, a failed handoff, or an unclear policy. A change can fix one case and break another. Without rerunning the original failure, related cases, and the production signals that exposed it, teams can't know whether the fix worked or the problem simply moved.
SOLUTION:
We can tackle these challenges in three steps: discover quality issues
like accuracy failures & suboptimal answers, diagnose why, and fix in
a way we can verify.
What to do
The first goal is to find the failures or quality issues your current evals are missing and find out how often they happen.
Teams generally start with a limited eval set and a list of known problems, but production presents unseen challenges such as unexpected formats, old documents, incomplete inputs, unusual wording, or unexpected tool sequences. The answer "looks" confident, but it might be completely wrong.
This is why you need to look beyond predefined checks.
-
Start with real user signals and look for signs of where there may be a production AI problem. For example:
- Answers users ignore or rewrite
- Repeated questions
- Abandoned tasks
- Customer feedback
- Human escalations
- Retries and loops
- Unexpected tool calls
- Weak citations or grounding
One customer told us:
"It is not always a thumbs down. Closing the window or ignoring the response can tell us it was not good. Copying the response can be a positive signal."
-
Then look for repeated patterns across a set of traces.
One bad answer is an incident. The same failure across many traces is a much bigger product problem with potentially substantial business impact.
-
Track the newly discovered high-impact issues for regression.
Now that you've discovered unknown issues, and the scale of the known ones, it would be prudent to build checks that look out for regressions in future as there are many variables that change over time including prompts, models, documents, etc. And these can cause a resolved issue to become a regression.
What to do
Teams have the raw logs and observability data, but narrowing down where things went wrong can still be hard, especially for difficult cases. Answering that can mean reading pages of long traces to understand what the agent got wrong. Here's a statement from one of our customers that sums it up nicely:
"Observability gives us good raw traces, but it does not reduce the work of debugging at scale."
The visible error is often several steps away from the cause.
The root cause could be in any one, or a combination, of the following:
- Prompt used
- Retrieved context
- Retrieval quality
- Model calls
- Tool calls
- Agentic steps
- A bad handoff
- State and memory
An upstream agent can also pass the wrong state to the next agent. Every step after that may look reasonable while the workflow moves further away from the right result. We have to work backward from the end-user outcome and try to answer:
- How can we find similar traces? Do similar traces show the same pattern?
- What should have happened across them? What actually happened? What was the difference?
- Was the handoff wrong at a particular step?
- What evidence confirms the cause?
A useful diagnosis names the first causal failure. It shows the affected traces, the expected behavior, the actual behavior, and the evidence behind the conclusion. Suggested causes should remain suggestions until the evidence supports them.
What to do
The fix should follow the cause and evidence.
For instance, if retrieval failed, another prompt may not help. And similarly, tuning the judge may not help, if the rubric is unclear.
Depending on the root cause, the fix may include:
- Correct or update the source data
- Change parsing, chunking, or retrieval
- Clarify prompts or policies
- Validate state before an agent handoff
- Validate or block unsafe tool calls
- Add a required check or human escalation
- Fix the rubric or evaluation method
Some failures / quality issues need expert judgment from SMEs like security experts, doctors, nurses, underwriters, or analysts as they may be the ideal people to spot a problem that looks correct to an engineer or automated judge.
Use that feedback to confirm the expected behavior and whether the fix resolves the failure. Then rerun the original failure and related cases. Check that the change fixes the problem without creating new regressions. You have to continuously evaluate the agent while it is in production.
Save the trace, the relevant context and versions, the expected behavior, the confirmed cause, and the verified fix. Add it to your evaluation or regression dataset. Run it every time the next prompt, model, retrieval, or workflow changes.
Also watch the user signals, feedback, the abandoned task, the rework, or the escalation that first exposed the failure, and confirm it stops once the fix ships.
Optional 60–90 second product demo
See one production failure move through the full loop.
Show one user signal becoming a Finding, an evidence-backed investigation in Copilot, expert confirmation, and a reviewed regression case.
Video placeholder
User signal → repeated finding → Copilot diagnosis → expert review → evaluation
dataset
FAQ
Frequently asked questions.
- What do we need to start?
- Connect your production traces from tools like Langfuse or LangSmith. You can also send in user and workflow events so DataFramer can connect what happened for users to the traces behind it. Connect AI observability
- Does DataFramer replace our observability tools?
- No. Keep using your observability stack to capture traces. DataFramer uses them to find repeated problems, investigate failures, and send the right cases to review and evaluation.
- How does DataFramer find failures we did not define in advance?
- Findings looks at your production traces and groups similar behavior into patterns. You can search for known problems, or find patterns you did not know to look for. Discovery
- Does DataFramer figure out the root cause on its own?
- DataFramer points to a likely root cause and shows the traces that back it up. Based on analysis, evidence, or human insights, it also suggests a fix. Since it works from a sample of traces, an engineer or domain expert should check important conclusions. Tracking and suggested fixes
- How do we know if a fix worked?
- Rerun the tracked Finding after you ship the change and check if the cause is gone. DataFramer also watches new traces for the same pattern and can send a Slack alert if it comes back.
- How are engineers and domain experts involved?
- Flagged traces go to review queues with shared rubrics. Reviewers can grade each part, explain why, and write the correct answer next to the original trace. Human review workflow
- Can DataFramer stop known failures before they reach users?
- A calibrated DataFramer judge can be called through the API to grade a response before it is sent back. Every guardrail decision gets saved so you can review it later. Judge Guardrail API
- How do production failures turn into regression tests?
- Reviewed traces can go into an evaluation dataset, using the human verdict as ground truth. The same cases can also generate similar synthetic edge cases for wider regression coverage. Datasets and evaluations
Let's find what's breaking in your AI.
Start free, plug in your traces, and start discovering with a $5 credit.