SOIT CLI
Install soit-cli from PyPI, sign in with an API key, run agents and evaluation datasets, move agents as files, replay regression sets on a new model, and export evidence from a terminal or CI.
soit is a small command-line client for the SOIT API, added in v1.3.0. It signs in with an API key, runs an agent, runs and manages evaluation datasets, moves an agent in and out as a file, replays regression sets on a candidate model, and takes evidence out.
Install
The CLI is published to PyPI as soit-cli since v1.5.1; the command it installs is soit:
uv tool install soit-cli # or: pipx install soit-cliThe v1.5.0 command line was not uploaded to PyPI, so install soit-cli 1.5.1 or later. To run a checkout instead, for example to try a change before it is released, install from the cli/ directory of the SOIT repository:
uv tool install ./cli # or: pipx install ./cliSign in
soit login --url https://soit.example.comsoit login asks for an API key (create one under Settings › API), checks it, and stores it with the address in your configuration directory, readable by you alone. soit whoami shows who the stored key signs in as, and soit logout forgets it.
In CI, set environment variables instead of signing in:
| Variable | Purpose |
|---|---|
SOIT_API_URL | The SOIT address |
SOIT_API_KEY | The API key |
SOIT_CONFIG | Another configuration file to use |
The environment wins over the stored file.
Commands
| Command | Does |
|---|---|
soit login [--url URL] [--api-key KEY | --api-key-stdin] | Check a key and remember it |
soit whoami | Show who the stored key signs in as |
soit logout | Forget the stored key |
soit run AGENT_ID "message" [--thread ID] [--json] | Run an agent's published version once; the answer on stdout, the run on stderr |
soit eval MODEL_REF [--agent ID]... [--dataset NAME] [--max-cases N] [--json] [--fail-on-regression] | Replay agents' regression sets on a model next to the one they use (model replays) |
soit eval run AGENT_ID [--dataset NAME] [--version ID] [--model MODEL_REF] [--max-cases N] [--json] [--fail-on-regression] [--fail-on-failure] | Run one dataset on an agent now (the published version unless --version), print each case's result and the report it records (evaluation datasets) |
soit dataset list [--agent ID] [--archived] [--json] | List the workspace's datasets with revision, case count, and latest report |
soit dataset export DATASET [--agent ID] [-o FILE] | Write a dataset's cases as JSONL (DATASET is its id or name; standard output by default) |
soit dataset import DATASET FILE [--agent ID] [--create] [--note TEXT] | Add the cases of a JSONL file, all or none; --create --agent ID creates the dataset first |
soit agent export AGENT_ID [-o FILE] | Write an agent and its published (or current) version's specification as YAML, without ids |
soit agent import FILE [--name NAME] | Create an agent, and its version as a draft, from a YAML or JSON file |
soit export evidence RUN_ID [-o FILE] | Save a run's evidence bundle, after checking it against the server's SHA-256 (evidence bundles) |
soit export runs|steps|costs|audit|events --since ISO [--until ISO] [--format jsonl|csv] [-o FILE|-] | Save one kind of ledger record for a window (ledger exports) |
Evaluation datasets
New in v1.5.0. soit dataset and soit eval run bring evaluation datasets to a terminal or CI:
dataset exportanddataset importname a dataset by id or by name; when several agents have a dataset of that name,--agentpicks one.dataset importreads-as standard input. A file with any bad line imports nothing: every bad line is listed asFILE:LINE: messageand the command exits1.eval runruns thedefaultdataset unless--datasetnames another. With--modelthe report is never a baseline.--max-casesrefuses a run with more cases (server default 50, at most 200).
A CI job can keep a dataset in the repository and evaluate every change against it:
soit dataset import refunds evals/refunds.jsonl --agent agt_123 --create
soit eval run agt_123 --dataset refunds --fail-on-regressionAgent files
New in v1.5.0. soit agent export writes an agent and its published (or, without one, current) version's specification as YAML, without ids, for version control or another workspace. soit agent import reads YAML or JSON (- reads standard input) and creates a new agent with its version as a draft; --name names the new agent instead of the file's. The model, tool, and knowledge refs the file names must exist in the importing workspace. See Agent files.
soit agent export agt_123 -o support-agent.yaml
soit agent import support-agent.yaml --name "Support agent (staging)"Exit codes
| Code | Meaning |
|---|---|
0 | Success |
1 | A refusal or failure |
2 | A usage error |
3 | eval --fail-on-regression found cases passing on the current model and failing on the candidate; eval run --fail-on-regression found a case that passed in the report's baseline and fails now; eval run --fail-on-failure found a failing case |
Exit code 3 lets a CI job stop a model switch:
soit eval model:openai:gpt-6 --dataset smoke --fail-on-regressionRelated pages
- Evaluation — datasets, regression sets, and model replays.
- Agents — agent files.
- Evidence and Exports — what the export commands save.
- OpenAI-Compatible Gateway — API keys and their limits.