Evaluation

Evaluation helps teams test whether agents, workflows, and retrieval paths keep producing acceptable results as models, prompts, knowledge, and tools change.

When to use evaluation

Use evaluation for regression suites, release gates, quality checks, and controlled comparisons between agent or workflow versions. The core question evaluation answers: "if I publish this version, does it still pass the cases that matter?"

How evaluation works

• Evaluations run against agents, workflows, knowledge-backed responses, or complete product paths.
• Runs use deterministic test models (model:test:*) where repeatability matters, so a failing case is a real regression rather than model noise.
• Every evaluation case produces full runtime evidence — the same runs, steps, citations, tool calls, costs, and audits as production execution — reviewable in Observe alongside real traffic.
• Reports are machine-readable: pass/fail per case with failure reasons, run_id, response_id, tool-call and citation counts, cost, and latency.

Evaluation as a release gate

The reference example is SOIT's own support-ticket regression suite: a fixed golden prompt set covering policy answers with citations and ticket workflows with tool calls, child workflow runs, audit, and cost evidence. It runs as a blocking gate — an agent version that fails it does not publish. Teams can apply the same pattern to their own agents: define golden cases, run them against a draft version, and gate the publish on a passing report.

The publish gate runs the agent's dataset named default. Cases frozen from a run always go there, and the dataset is created with the first one.

Evaluation datasets

New in v1.5.0. An agent's regression set is an object of its own: a dataset, a named, versioned list of cases for one agent. Running it produces a report that says which cases passed and, against the last comparable report, which ones a change broke. Datasets are the same regression sets the publish gate and model replays use; nothing about how those runs work changed.

• A dataset belongs to one agent and has a name, unique for that agent, and an optional description. The name cannot be changed: reports and the publish gate refer to a dataset by it.
• Cases are written in the console or through the API, imported from a JSONL file, or frozen from a past run with POST /api/v1/evaluations/regression-cases/from-run. A case frozen from a run keeps a link to it; the others have none.
• A case is never deleted. Removing one, or archiving a dataset, takes it out of every run, the publish gate included, and keeps the row so reports that name it still read. Restoring an archived dataset puts back exactly the cases archiving took out. An archived dataset can be exported and restored, but not run or edited.

What a case asserts

A case is an input and what a run must satisfy. expected_features holds at least one of:

A case that asserts nothing is refused, since it would pass whatever the agent says. A run that fails outright fails its case whatever it expected, and a case that asks for a judge fails when no judge is configured rather than passing unmeasured.

Importing and exporting cases as JSONL

One case per line, each a JSON object validated against server/app/kernel/specs/v1/dataset_case_spec.schema.json:

{"name":"refund-window","input":"How long do refunds take?","expected_features":{"minimum_output_terms":["14 days"],"max_latency_ms":4000}}
{"name":"angry-customer","input":{"messages":[{"role":"user","content":"This is the third time I am asking."}]},"expected_features":{"llm_judge":{"rubric":"Stays calm and offers a concrete next step","min_score":0.8}}}

• name: 1 to 255 characters, unique within the dataset.
• input: a string is the text sent to the agent. An object is stored as it is, and the agent is asked the content of the last user message of its messages, else its input_summary, else its input, else the object serialised as JSON.
• expected_features: as above; unknown keys are refused, so a misspelt expectation cannot quietly assert nothing.

An import is all or nothing: every line is checked first, and a file with any invalid line imports nothing and answers 400 with the line number and message of every problem. Names that already exist in the dataset, or repeat within the file, are refused together (409). A file holds at most 1,000 cases and 2 MB, and a dataset at most 5,000 active cases. An export writes the same format back, so exporting a dataset and importing the file into another reproduces the same cases and the same content hash.

Revisions and versions

Every change made through the dataset (adding, editing, removing, or importing cases) advances the dataset's revision and stores a version: the cases the dataset held at that revision, a SHA-256 content hash that does not depend on their order, and how many cases were added, removed, and changed since the revision before. Editing a case keeps its id, so reports that ran it still name it. Archiving and restoring change no revision.

A report records the revision it ran. The baseline a run is compared against is the latest report of the same dataset at the same revision, and reports on another model are never baselines. Change a dataset and the next run starts a new baseline.

Running a dataset on demand

The evaluation the publish gate makes can be run at any time, on the published version, on a draft, or on another model, and its report is recorded the same way:

POST /api/v1/evaluations/run
{"subject_id": "agt_…", "subject_version_id": "ver_…", "dataset": "smoke", "model_ref": "model:openai:gpt-6", "max_cases": 50}

Only subject_id is required: the published version and the default dataset are taken when nothing else is named, and an agent with no published version needs subject_version_id. Every case runs as a rehearsal before the request answers with the report. With model_ref the cases run on that model instead of the version's own; the report says so in its summary and is never a baseline. An evaluation that would run more than max_cases cases (at most 200), or has no cases to run, is refused before anything runs.

A report lists every case's result, its dataset, dataset_revision, and baseline_report_id, and the cases that regressed (passed in the baseline, fail now) and were fixed. A case that never passed is a known gap, not a regression.

In the console

Observe › Evaluations lists datasets with their agent, revision, case count, and how the latest report went, and whether the dataset has changed since it ran. New dataset creates one for an agent, optionally seeded from a JSONL file. A dataset opens to:

• Cases: add, edit, and remove cases, with an editor for each expectation above; Import JSONL and Export are in the header.
• Reports: pass rate per report, and for the report picked each case's result, the judge's score, and the cases that regressed or were fixed against its baseline.
• Versions: each revision's size, content hash, and change counts, with the cases it held.

Run evaluation runs the dataset on the published version, or a version named, optionally on another model, and opens the report it records. From a terminal or CI, soit dataset and soit eval run --fail-on-regression do the same.

Access and audit

Reading datasets, cases, versions, reports, and exports needs workspace read access; every change and every run needs write access. Datasets, cases, and reports are scoped to the workspace. Dataset and case changes, imports, and exports are written to the audit ledger as evaluation.dataset.created|updated|archived|restored|imported|exported and evaluation.case.created|updated|removed.

Upgrading to v1.5.0 creates one dataset, with one version holding its active cases, for every agent and dataset name that already had cases — see Migration and Upgrade.

Model replays

When a new model generation arrives, the question is whether your agents still do what their regression sets say they must. A model replay (since v1.3.0) answers that in one step: every regression case of every agent, or the ones you pick, runs on the model the agent uses today and on a candidate model, and the two are compared on pass rate, latency, and cost.

• An agent's regression cases are the cases of its datasets — the same cases that gate the agent's publishes.
• Each case runs twice against the agent's published version: once on the model that version binds, once on the candidate. Both run now, so neither side is measured against a stale report.
• Every run is a rehearsal (sandbox): tools are answered at the boundary and never called, and the runs are kept out of spend, dashboards, and alerts, like a publish gate's.
• Cases are judged exactly as in a publish gate: required output terms, latency and cost ceilings, and the LLM judge when a case asks for one. A run that fails outright fails its case.
• Agents without a published version, or without cases, are listed as skipped.

Start one under Build › Models › Regression replays, from the soit eval command, or through the API:

POST /api/v1/evaluations/model-replays
{"model_ref": "model:openai:gpt-6", "agent_ids": ["agt_…"], "dataset": "smoke", "max_cases": 50}

agent_ids and dataset are optional: without them every agent with cases, and every dataset, is replayed. A replay that would run more cases than max_cases (at most 200; each runs on both models) is refused before anything runs, with the count. A candidate that cannot serve at all (not configured, disabled, not allowed for the key) stops the replay with that error rather than failing every case. The request answers when every case has run, and needs workspace write access.

A replay records, per agent and dataset:

totals sums both sides over every agent, with the difference (candidate minus baseline) in pass rate, average latency, and cost, and lists the skipped agents.

A replay is evidence for a decision, not a gate: it never becomes the baseline a publish is compared against, and it changes no agent. Switching an agent to the new model is still a new version, which its own publish gate then checks.

Limits: the request runs every case before it answers, so keep replays within a few dozen cases or narrow them by agent and dataset; only agents' regression sets are replayed, as workflows have none yet.

Known limitations

• Datasets are for agents; workflows have none yet.
• An on-demand run, like a replay, answers when every case has run, so keep a dataset within a few dozen cases per run, or raise max_cases knowingly (at most 200).
• soit eval run runs an agent's dataset synchronously; scheduled evaluation is not included.
• A dataset's name cannot be changed.

How evaluation connects

• Evaluations target Agents and Workflows, typically comparing draft (preview) against published versions.
• Evidence lands in Tasks and Runs and is reviewable in Observe.
• Versioning makes comparisons meaningful — each evaluation records exactly which version it exercised.

API and UI surfaces

• API prefix: /api/v1/evaluations
• Related prefixes: /api/v1/agents, /api/v1/workflows, /api/v1/runs
• Console: Observe › Evaluations for datasets, cases, reports, and versions; Build › Models › Regression replays for model replays
• CLI: soit dataset, soit eval run, and soit eval
• API reference: Evaluation API