Evaluation API
API surface for SOIT evaluation datasets, on-demand runs, reports, regression cases, and model replays.
The evaluation API supports repeatable tests for agents: datasets of cases, runs of a dataset on an agent version, the reports they record, and model replays.
Prefix
/api/v1/evaluationsEndpoint families
| Method | Path | Purpose |
|---|---|---|
| POST | /run | New in v1.5.0: run an agent's dataset now and answer with the report it records |
| GET | /regression-reports/latest | Latest machine-readable regression report |
| GET | /regression-reports/trend | Pass-rate series for an agent, optionally one dataset |
| POST | /regression-cases/from-run | Freeze a run as a regression case |
| POST | /model-replays | Replay agents' regression sets on a candidate model next to their current model |
| GET | /model-replays | List past model replays |
| GET | /model-replays/{replay_id} | One model replay with its agents |
Datasets
New in v1.5.0.
| Method | Path | Purpose |
|---|---|---|
| GET | /datasets | List the workspace's datasets, most recently changed first (subject_id, status of active or archived) |
| POST | /datasets | Create a dataset for an agent (subject_id, name, description) |
| GET | /datasets/{dataset_id} | One dataset |
| PATCH | /datasets/{dataset_id} | Change the description, or archive or restore it through status |
| DELETE | /datasets/{dataset_id} | Archive the dataset; nothing is deleted |
| GET | /datasets/{dataset_id}/cases | List the dataset's cases (q matches part of a name) |
| POST | /datasets/{dataset_id}/cases | Add a case |
| GET | /datasets/{dataset_id}/cases/{case_id} | One case |
| PATCH | /datasets/{dataset_id}/cases/{case_id} | Edit a case; it keeps its id |
| DELETE | /datasets/{dataset_id}/cases/{case_id} | Take the case out of the dataset; answers with the dataset at its new revision |
| POST | /datasets/{dataset_id}/import | Add the cases of a JSONL file ({"content": "…", "note": "…"}), all or none |
| GET | /datasets/{dataset_id}/export | The dataset's cases as JSONL (application/x-ndjson), in the format import reads |
| GET | /datasets/{dataset_id}/versions | Every revision's size, content hash, and change counts, newest first |
| GET | /datasets/{dataset_id}/versions/{revision} | One revision with the cases the dataset held at it |
Reports
New in v1.5.0.
| Method | Path | Purpose |
|---|---|---|
| GET | /reports | Reports newest first, without per-case results (subject_id, dataset, passed; with_total adds the count) |
| GET | /reports/{report_id} | One report with every case's result, its dataset revision, baseline, and the cases that regressed or were fixed |
Regression reports include per-case pass/fail with failure reasons, run_id, response_id, tool-call and citation counts, cost, and latency — the same shape the support-ticket release gate consumes.
Running a dataset
POST /api/v1/evaluations/run
{"subject_id": "agt_…", "subject_version_id": "ver_…", "dataset": "smoke", "model_ref": "model:openai:gpt-6", "max_cases": 50}Only subject_id is required: the published version and the default dataset are taken when nothing else is named. With model_ref the report is never a baseline. More than max_cases cases (default 50, at most 200), or no cases at all, is refused before anything runs. Running an archived dataset is refused with details.reason dataset_archived. See Running a dataset on demand.
Importing a dataset
A file with any invalid line imports nothing and answers 400 with every problem:
{"success": false, "code": "VALIDATION_ERROR",
"message": "2 of 3 lines are not valid cases; nothing was imported",
"details": {"error_count": 2, "errors": [{"line": 2, "message": "not valid JSON: Expecting value"}, {"line": 3, "message": "/expected_features: {} should be non-empty"}]}}Names that already exist in the dataset, or repeat within the file, are refused together with 409. A file holds at most 1,000 cases and 2 MB, and a dataset at most 5,000 active cases. The case format is in Importing and exporting cases as JSONL.
Related prefixes
/api/v1/agents,/api/v1/workflows— the subjects under evaluation/api/v1/runs— full runtime evidence for each evaluation case
Notes
- Reading datasets, cases, versions, reports, and exports needs workspace read access; every change and every run needs write access.
- Evaluation runs use deterministic test models (
model:test:*) where repeatability matters. - Historical runs can become regression cases, and agent publish can replay the
defaultdataset's active cases as a release gate — see Evaluation. - A model replay (since v1.3.0) runs at most 200 cases, answers when every case has run, and needs workspace write access — see Model replays.