> ## Documentation Index > Fetch the complete documentation index at: https://www.dataframer.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # Evaluate content with a judge > Grade content against a judge's rubric and get the verdict inline ## OpenAPI ````yaml POST /api/dataframer/judges/{judge_id}/evaluations/ openapi: 3.0.0 info: title: DataFramer API version: 0.1.0 description: '' termsOfService: https://www.aimon.ai/docs/privacy-policy.pdf contact: name: DataFramer Support email: info@dataframer.ai license: name: Proprietary x-logo: url: https://dataframer.ai/logo.png altText: DataFramer AI x-stainless: package-name: aimon-dataframer namespace: - aimon - dataframer servers: - url: https://df-api.dataframer.ai description: Production server security: - BearerAuth: [] externalDocs: description: Complete API Guide url: https://docs.dataframer.ai/dataframer paths: /api/dataframer/judges/{judge_id}/evaluations/: post: summary: Evaluate content with a judge description: >- Synchronously evaluate content with a hosted judge and return the verdict inline. **Synchronous operation**: unlike most create endpoints, this call blocks until the judge has graded the content — typically a few seconds, up to a couple of minutes for `-thinking` models on large inputs. Send either `messages` (a chat transcript in the standard role/content shape, with the candidate reply in `output`) or a plain `input`/`output` pair. Optional context fields (`name`, `generation_model`, `metadata`) are rendered as trace context for the judge — the same text shape judges see for traces pulled from observability platforms, so verdicts are directly comparable. Inline file attachments are supported as base64 `data:` URIs inside message content blocks; PDF and XLSX files are extracted to text before judging, images are not read. External URLs and observability-platform media references are rejected — inline the bytes instead. A per-workspace concurrency cap applies: excess concurrent evaluations receive `429` with a `Retry-After` header. Content exceeding the judge input budget is rejected with `422`. operationId: api_dataframer_judges_evaluations_create parameters: - name: judge_id in: path required: true schema: type: string format: uuid requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/JudgeEvaluationCreate' examples: guardrail_messages: summary: Grade a chat reply (guardrail shape) value: messages: - role: system content: You are a support assistant. - role: user content: Where is my refund? output: >- Your refund was issued yesterday and should arrive within 3 business days. plain_input_output: summary: Grade a plain input/output pair value: input: Summarize the attached contract for a non-lawyer. output: This agreement covers ... responses: '201': description: Verdict produced content: application/json: schema: $ref: '#/components/schemas/JudgeEvaluation' '400': description: Invalid payload or unknown judge model content: application/json: schema: type: object properties: error: type: string '404': description: Judge not found content: application/json: schema: type: object properties: error: type: string '409': description: Judge has no ready version (prompt still generating or failed) content: application/json: schema: type: object properties: error: type: string '422': description: >- Unprocessable content (unsupported attachment reference, attachment failure, or content over the judge input budget) content: application/json: schema: type: object properties: error: type: string '429': description: >- Per-workspace concurrent-evaluation cap reached; retry after the `Retry-After` interval content: application/json: schema: type: object properties: error: type: string '502': description: >- The judge call failed; the evaluation is persisted with `status: failed` and returned in the body content: application/json: schema: $ref: '#/components/schemas/JudgeEvaluation' x-codeSamples: - lang: Python source: |- import os from dataframer import Dataframer client = Dataframer( api_key=os.environ.get("DATAFRAMER_API_KEY"), # This is the default and can be omitted ) evaluation = client.dataframer.judges.evaluations.create( judge_id="182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e", messages=[{ "role": "system", "content": "You are a support assistant.", }, { "role": "user", "content": "Where is my refund?", }], output="Your refund was issued yesterday and should arrive within 3 business days.", ) print(evaluation.id) - lang: cURL source: >- curl https://df-api.dataframer.ai/api/dataframer/judges/$JUDGE_ID/evaluations/ \ -H 'Content-Type: application/json' \ -H "Authorization: Bearer $DATAFRAMER_API_KEY" \ -d '{ "messages": [ { "role": "system", "content": "You are a support assistant." }, { "role": "user", "content": "Where is my refund?" } ], "output": "Your refund was issued yesterday and should arrive within 3 business days." }' components: schemas: JudgeEvaluationCreate: type: object description: >- The content to grade. Provide either `messages` (with the candidate reply in `output`) or a plain `input`/`output` pair — `messages` and `input` are mutually exclusive, and at least one of `messages`/`input`/`output` is required. properties: messages: type: array nullable: true items: type: object description: >- Chat transcript in the standard role/content shape (the messages sent to your model). Content may be a string or a list of content blocks; inline files ride as base64 `data:` URIs in file blocks. input: nullable: true description: Plain input (string or JSON) — alternative to `messages`. output: nullable: true description: The candidate output to grade (string or message object). name: type: string nullable: true description: Name for this interaction (shown to the judge as context). generation_model: type: string nullable: true description: >- The model that produced the content (context for the judge — not the judge model). metadata: type: object nullable: true description: >- Free-form context shown to the judge (rendered as the interaction's trace metadata). model_override: type: string nullable: true description: >- Normally omitted — the judge already pins its grading model, and every evaluation uses it by default. Set this only to deliberately grade with a different judge model for this one call (public judge-model names; append `-thinking` for extended reasoning). JudgeEvaluation: type: object properties: id: type: string format: uuid judge_id: type: string format: uuid judge_version: type: integer nullable: true description: Version number of the judge version that graded this content. source: type: string enum: - api - monitor - ui status: type: string enum: - succeeded - failed model: type: string description: The judge model used. verdict: nullable: true allOf: - $ref: '#/components/schemas/JudgeVerdict' description: Null when status is `failed`. error: type: string nullable: true latency_ms: type: integer nullable: true created_at: type: string format: date-time JudgeVerdict: type: object properties: overall_score: type: number nullable: true description: Mean of the per-dimension normalized scores (0-1). confidence: type: number nullable: true description: The judge's self-reported confidence (0-1). reasoning: type: string description: The judge's overall reasoning. dimensions: type: array items: $ref: '#/components/schemas/JudgeVerdictDimension' JudgeVerdictDimension: type: object properties: dimension_id: type: string description: Stable machine id of the dimension. name: type: string scoring_model: type: string enum: - pass_fail - scale_1_5 - single_select - checklist label: type: string description: Chosen level's label (single-select scoring models). labels: type: array items: type: string description: Ticked levels' labels (checklist scoring model). score: type: number nullable: true description: >- Normalized 0-1 score for this dimension — identical normalization to the DataFramer UI. passed: type: boolean nullable: true description: >- For pass_fail dimensions only: whether the content passed. Null for other scoring models. notes: type: string description: The judge's note for this dimension. securitySchemes: BearerAuth: type: http scheme: bearer bearerFormat: API Key description: 'API Key authentication. Format: "Bearer YOUR_API_KEY"' ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.