Datasets
An evaluation dataset is a set of real traces you use as a benchmark. You build one from your reviewed traces, so each trace carries the human verdict as ground truth. A judge runs against this set, and its scores are compared to what your reviewers said. Create a dataset here, or add traces straight from the Findings trace list, so a failure pattern you already found there becomes a benchmark in a few clicks and your judge is tested on the exact behaviors you care about. When you start an evaluation, DataFramer shows how many of the dataset’s traces already have human verdicts. Calibration is only as strong as that overlap, so the more reviewed traces a dataset has, the more the judge has to check itself against. To cover more cases, use the dataset’s own traces as seeds for synthetic generation: select rows on the dataset page, pick Add to dataset → Generation Seed Dataset, and generate additional inputs for regressions and rare edge cases from them. Run those through your app and review the resulting traces, so they combine with the datasets that already carry ground truth.Evaluations
An evaluation runs a judge across an evaluation dataset and scores every trace. Results break down per rubric dimension, with the judge’s grades and the human consensus side by side, so you can see where the judge agrees with your reviewers and where it still disagrees. This is how you check a judge at scale before you rely on it. You can also go trace by trace to see each judge verdict next to the human consensus, filter to just the disagreements, and open the original trace to dig in. This is how you find exactly where the judge and your reviewers part ways.Grading one interaction from your own code
Once a judge is calibrated, you can also call it directly on a single interaction and get the verdict back inline, rather than queueing a run over a dataset:messages transcript or a plain input/output pair. The call blocks until the judge has graded — usually a few seconds — which makes it usable as a runtime guardrail in front of a response, not just an offline measurement. A per-workspace concurrency cap applies; over it, the API returns 429 with a Retry-After header.
Every call made this way is recorded, and you can read the verdicts back under Evaluations → Guardrail API calls in the app.
Generating data to evaluate against
DataFramer can generate synthetic datasets from the failure patterns you found, useful for building regression sets or filling in thin edge cases. The Add to dataset → Generation Seed Dataset action is available on the dataset page and on an evaluation’s per-trace results, so the traces where your judge and reviewers disagreed can seed the next generation run directly. Generation also has a public API and Python SDK:Next steps
API & MCP
Programmatic access to datasets, specs, generation, and evaluations