What is DataFramer?

DataFramer helps you answer two questions about the AI in your product:

1. Is the AI accurate in the context of your business?
2. Does the complete AI-powered workflow deliver value to your users and your organization?

To answer them, it brings three kinds of evidence into one place: the AI traces your observability tool records (what each model or agent call did), the signals your users produce in your product (what they did with the result), and the verdicts your own experts give when they review a trace. Each trace is tied to the user journey it happened in, so you judge a model call in the context of what the user was doing.

​

How it works

DataFramer runs one loop, in five steps. The output of each step is the input of the next.

​

Capture & correlate

Connect Langfuse or LangSmith and DataFramer pulls your traces in. Add the Signals SDK to record what your users do around those traces: clicks, feedback, workflow events. Signals and traces carry a shared journey id, so DataFramer lines up a user’s actions with the AI calls that happened around them.

​

Measure user journeys

The Journeys page shows each journey as one timeline: user actions and AI traces in the order they happened, even across several agents. Then zoom out with Dashboards to analyze all journeys: completion, adoption, drop-offs, throughput, cycle time, and cost. That shows where AI helps or hurts the workflow and which improvements to prioritize, so you can tell whether the AI delivers value.

​

Discover & diagnose

Findings searches your traces for any behavior you describe in plain words and groups the matching traces into findings, including patterns you did not name. Pin the ones that matter to watch how often they come back. A finding marked as a failure also gets its root causes and suggested fixes.

​

Review with experts

Human Reviews sends selected traces to your own experts. A rubric sets the grading standard, reviewers grade against it in review queues, and the Review Copilot speeds them up. Every verdict is kept as ground truth for your judges and datasets, so the same judgment is reused on the next similar case instead of made again.

​

Evaluate at scale

A judge is an LLM that grades traces against the same rubric your reviewers use. Calibrate it on a dataset of reviewed traces until it agrees with your reviewers, then let it grade all your traffic to catch regressions, or call it from your own code as a guardrail.

​

Everything compounds

Journeys show where AI helps or hurts. Findings turns that into named patterns, Reviews turns patterns into expert verdicts, and Judges turn those verdicts into quality you can measure at scale. Your rubrics, your corrections, and the failures you name build up into a shared record of how your business judges quality.

​

Synthetic data generation

DataFramer also generates realistic, diverse synthetic datasets at scale. You can reach it from either end of the loop.

• Seeded by your traces. Wherever DataFramer lists traces, select the ones you care about and pick Add to dataset → Generation Seed Dataset. What you generate is then grounded in behavior you actually observed.
• On its own. Upload example data, or describe the dataset you want in text alone with no seed data at all (seedless mode). This is the path for fine-tuning sets, rare events your production data never contains, and drift scenarios.

See Core Concepts and the Quickstart.

​

Next steps