OpenAI-Compatible Gateway

SOIT serves an OpenAI-compatible API under /v1, next to its own API under /api/v1. Any client built for OpenAI — the official SDKs, LangChain, the OpenAI Agents SDK, and most tools that accept a base URL — can call the models a workspace has configured, and every call is governed the way SOIT's own agents are: rate limits, quotas, budgets, content safety, cost recording, and audit. Anthropic clients work too: POST /v1/messages serves the Messages API, so Claude Code and the Anthropic SDKs can be pointed at SOIT the same way.

The gateway shipped in v1.2.0. Since v1.3.0 the same API keys also call the workspace's tools directly and from MCP clients (see Calling tools with the same keys). In v1.4.0 the gateway can run in processes of its own (see Running the gateway on its own), and image calls take quality (see Image options). v1.5.0 adds the Anthropic Messages API (see Anthropic Messages). In v1.6.0 a workspace can refuse calls with no price (see Calls with no price), and cost entries keep the provider's own ids.

When to use the gateway

• Your application already calls models from its own code and needs governance without a rewrite.
• A pipeline or partner system needs model access under its own identity and limits rather than a person's key.
• You want one model name that fails over across providers.

Connecting

Point an OpenAI SDK at SOIT and call it unchanged:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:9200/v1", api_key="sk_...")

reply = client.chat.completions.create(
    model="model:openai-main:gpt-5.5",  # a ref as GET /v1/models lists it
    messages=[{"role": "user", "content": "Say hello in five words."}],
)

GET /v1/models lists what the key may call, by the name a call uses. A model is named by its SOIT ref, not the provider's bare model name. From v1.5.0 each entry also carries Anthropic's fields (type, display_name, created_at), and the list has_more, first_id, and last_id, so the SDKs of either parse it.

Create keys under Settings › API or with POST /api/v1/api-keys. A key carries scopes (read, write, admin) that cap what it may do below its owner's role; calling models needs write. A key is shown once, at creation or rotation.

What is served

• POST /v1/chat/completions — whole or streamed, with tools, images in user messages, and structured output (response_format). n must be 1, so each choice stays one governed call.
• POST /v1/messages and POST /v1/messages/count_tokens — the Anthropic Messages API (new in v1.5.0); see Anthropic Messages.
• GET /v1/models — the active models and virtual models the key may call.
• POST /v1/embeddings — float or base64 vectors.
• POST /v1/images/generations and POST /v1/images/edits — n, size, response_format, background, output_format, and quality; see Image options. Before v1.4.0 a non-default quality was refused, and generations ignored background and output_format.

The Responses API (/v1/responses), Assistants, files, audio, batch, and fine-tuning are not served; the OpenAI Agents SDK therefore uses its chat-completions model. Of Anthropic's API, the Message Batches and Files APIs are not served. Fields SOIT does not know are ignored rather than refused, so clients that send newer OpenAI parameters keep working. Parameters, headers, and error codes are listed in the Gateway API reference.

From v1.4.0, OpenAI image parameters SOIT knows but does not send are the exception, since ignoring one would return and bill an image other than the one asked for: style, moderation, output_compression, partial_images, and stream (and input_fidelity on edits) are accepted at their OpenAI defaults, and any other value is refused with 400 naming the parameter. user is accepted and not sent.

Anthropic Messages

New in v1.5.0. POST /v1/messages answers the Anthropic Messages API, so Claude Code, the Anthropic SDKs, and other tools built for it call the workspace's models through the same governed path as /v1/chat/completions: one run per call (source=gateway), the same rate limits, quotas, budgets, content safety, and cost ledger. The model behind a call does not have to be Anthropic's: SOIT translates the request for whichever provider the model ref names.

For Claude Code:

export ANTHROPIC_BASE_URL=http://localhost:9200
export ANTHROPIC_API_KEY=sk_…
export ANTHROPIC_MODEL=model:anthropic:claude-sonnet-5-5

Claude Code uses a second, smaller model for background work; set it (ANTHROPIC_SMALL_FAST_MODEL, or ANTHROPIC_DEFAULT_HAIKU_MODEL in newer versions) to a ref the key may call, or those calls are refused. CI runs the official Anthropic Python SDK against /v1/messages, not Claude Code itself; if one of Claude Code's requests is refused, the 400 names the field.

With the Anthropic Python SDK:

from anthropic import Anthropic

# The SDK adds /v1/messages itself, so the base URL has no /v1.
client = Anthropic(base_url="http://localhost:9200", api_key="sk_...")

message = client.messages.create(
    model="model:openai-main:gpt-5.5",  # any ref as GET /v1/models lists it
    max_tokens=256,
    messages=[{"role": "user", "content": "Say hello in five words."}],
)

Anything SOIT cannot carry is refused with 400 naming it rather than ignored: a document or other content block, an image inside a tool_result, and tools Anthropic runs itself (web search, code execution, bash, the text editor, computer use), since dropping one would change what the model can do without saying so.

• A streamed call uses Anthropic's named events (message_start, ping, content_block_start, content_block_delta, content_block_stop, message_delta, message_stop). message_start carries SOIT's estimate of the prompt's tokens, and message_delta the provider's count once it is known. The gateway's outbound content check can hold text back, so a streamed reply may deliver its tool calls before its text.
• stop_reason follows what the provider reports (end_turn, max_tokens, tool_use, stop_sequence, refusal); a provider that reports a bare stop does not say whether a stop sequence ended the turn, so that case is end_turn, and stop_sequence is always null.
• POST /v1/messages/count_tokens returns {"input_tokens": n} for a prompt. It is SOIT's estimate, not the provider's tokenizer: it calls no model, opens no run, and is not billed.
• Refusals use Anthropic's error body; see Anthropic errors.

Image options

New in v1.4.0. Image calls, through /v1/images and /api/v1/images alike, take quality:

• quality is sent as given — gpt-image reads low, medium, and high; DALL-E 3 reads hd — except auto and standard, which leave the choice to the provider and are not sent. It must be a lowercase word.
• It reaches the provider on the routes that carry it: OpenAI, Azure OpenAI and Azure AI, the LiteLLM proxy, OpenAI-compatible providers, Xinference, CometAPI and ModelScope generations, OpenAI-style edits, and Bedrock Nova Canvas edits. Other routes refuse it with 422 and the reason route_cannot_carry.
• A model can be priced by quality, size, or both with image_variants in its pricing_json; see Image prices.

POST /v1/images/generations also honours background and output_format (png, jpeg, webp). background: auto is the provider's default and is not sent, and a transparent background with jpeg output is refused. POST /api/v1/images/generations accepts background (transparent, opaque) and output_format (png, webp) too, as its edit endpoint already did.

SOIT records which of background, output_format, quality, seed, strength, negative_prompt, mask, size, response_format, and more than one image (n above 1) each LiteLLM image route carries, and CI checks that record against the requests LiteLLM sends. An option the model's route does not carry is refused with 422 MODEL_IMAGE_CAPABILITY_UNAVAILABLE before a run opens, for an asynchronous job as for a synchronous call, and nothing is admitted or billed:

• On /v1 the error names the option in param.
• On /api/v1, details names it in param and capability, with reason route_cannot_carry and route, LiteLLM's name for the route, which is informational.
• size auto and output_format png ask for nothing a route could drop, so they are accepted and not sent where a route has no place for them.
• A route LiteLLM cannot generate or edit with at all, or a provider on the native adapter, which serves no images, is refused as a missing capability (MODEL_CAPABILITY_UNAVAILABLE, reason no_litellm_route or native_adapter), so a virtual model moves on to its next target.

The route-by-route table is in the SOIT repository's gateway guide. An image call that times out after the provider was asked is charged the images it asked for; see Charges SOIT estimates.

Every call is a run

Each call creates a run with mode=gateway, kind chat, embedding, or image, and source=gateway. Its subject is the API key, or the user when a signed-in session calls. The run id comes back in the x-soit-run-id response header and is the id of a chat completion (chatcmpl-{run id}) or of an Anthropic message (msg_{run id}).

The run records its model step, tokens, latency, and cost like any other run, and passes the same rate limits, quotas, credit checks, content safety, and cost recording as SOIT's own agents. Observe › Runs filters by entry and by key (source and api_key_id on GET /api/v1/runs and on every cost breakdown under /api/v1/runs/costs), and Settings › API links each key to its runs.

Per-key limits

Limits set on a key apply on top of its owner's own limits; an empty limit means none of that kind. Set them in Settings › API or with PATCH /api/v1/api-keys/{key_id}; rotation keeps them.

New in v1.4.0. A key's calls are its model calls and its tool calls through /api/v1/tools and /mcp, counted against one budget. Every tool request spends a call per minute, a poll or a replay with the same Idempotency-Key included; a tool call spends a call per 24 hours once, when it starts, so running it once approved spends no more. A refusal names the limit in details.quota (per_minute, daily_requests). In v1.3.0 these limits counted model calls only.

Behind a load balancer or reverse proxy, set TRUSTED_PROXIES (a JSON list of addresses or CIDR ranges, for example ["10.0.0.0/8"]). X-Forwarded-For is believed only from those proxies, read from the right, so an allowlist sees the real client rather than an address the client wrote.

Service principals

A pipeline or partner system should not borrow a person's key. A service principal is a non-human caller with a human owner and a workspace role (Viewer, Dev, or Admin), created by a workspace owner or admin under Settings › API or with POST /api/v1/service-principals. A key issued to it (principal_id on POST /api/v1/api-keys) authenticates as the principal, so runs, costs, and audit are attributed to it. It acts only in its own workspace, with the lower of its role and its owner's current role, and stops working when the principal is disabled or the owner leaves.

Virtual models

A virtual model is a workspace name, called as vmodel:{slug}, for an ordered list of up to eight model: refs. Edit them under Build › Models › Virtual models or with /api/v1/modelhub/virtual-models. /v1/models lists active virtual models, and a key's allowed models can name them.

A call tries the targets in order:

• A target that is unavailable (disabled, removed, or missing a capability the call needs) is skipped.
• A call that fails with a timeout, 408, 409, 429, a 5xx, or a lost connection moves to the next target once that target's own retries are spent.
• An invalid request or a policy refusal is not repeated elsewhere.
• A stream moves on only before its first chunk.
• An image call is never repeated on another provider, because the failed one may have billed it.

Each attempt is recorded on the run step, and cost is attributed to the provider that answered.

Budgets

Budgets limit spend for the workspace, one key, one member or service principal, or one agent. A budget with a hard stop refuses gateway calls once spent with 402 (insufficient_quota, code budget_exhausted), naming the budget, what it spent, and when it resets. See Budgets and Cost.

Each admitted call holds its budgets for the period's average call cost. In v1.4.0 checking a budget and holding it are one step, and the hold is released once the call's cost is committed or its run ends, instead of lasting a minute; see Holds for calls in flight.

Calls with no price

New in v1.6.0. A call to a model or tool with no price configured is recorded without an amount, so no budget counts it. The workspace setting Calls with no price (Settings › Security, or unpriced_call_policy on PATCH /api/v1/workspaces/{workspace_id}, for workspace admins) decides what such a call does: allow (the default), refuse_when_budgeted (refused while a hard-stop budget applies to the call), or refuse.

• A refused call is answered 403 (permission_error, code pricing_not_configured) naming the model or tool, before the provider or the tool is called, and written to the audit ledger as billing.unpriced.blocked.
• A virtual model moves on to a priced target.
• Give a local model an explicit zero price to keep it callable under either refusal.

See Calls with no price on the Budgets and Cost page.

Provider ids on cost entries

New in v1.6.0. A model call's cost entry keeps the provider's own ids, when it gave them: upstream_id, the response's id, and upstream_request_id, the request id from its response headers, for OpenAI, Anthropic, Gemini, and LiteLLM. Both filter GET /api/v1/runs/costs/entries, so a line of a provider's request log finds its entry; see Checking costs against a bill.

Content-free runs

When prompts and outputs must not be stored, set the workspace's content_capture to metadata_only (Settings › Security), or ask for it on one key. Run and step summaries, output previews, and error text are then stored as a length and a SHA-256 prefix; statuses, error codes, tokens, costs, and timings are recorded as before, and callers still get their answers.

• A key can tighten its workspace's mode but not loosen it.
• A workspace admin may switch a workspace to metadata_only; only a tenant admin may switch it back.
• Content capture is decided when a run starts; if the workspace setting cannot be read, content is withheld.

Evidence bundles keep no more content than the workspace records.

Streams

The response starts when the model's first chunk arrives, so a refusal before the stream (a limit, a budget, a policy) answers with its own status code. A failure part way through ends the stream with an OpenAI error event. A stream the client abandons fails its run with CLIENT_DISCONNECTED; in v1.4.0 SOIT also closes the provider stream and ends the model step as canceled (STREAM_ABANDONED).

New in v1.4.0. Every stream that reached the provider is charged, however it ends. A stream that ends without the provider's usage (the client left, the stream failed part way, or the backend never sends usage) is charged an estimate from the prompt and the text the model generated, flagged usage_estimated; see Charges SOIT estimates. The usage a response reports is always the provider's own count; only the ledger carries the estimate.

Content safety inspects streamed chat like a whole reply: the prompt is inspected before the stream opens, and the output is released a sentence at a time after inspection, so a redaction replaces text before the client sees it and a refusal ends the stream.

Calling tools with the same keys

The same keys call the workspace's tools by reference, with no agent in between (since v1.3.0):

• GET /api/v1/tools lists the tools the key may invoke — built-ins, installed plugin tools, and the tools of enabled MCP servers — with their argument schemas and whether they need approval.
• POST /api/v1/tools/{ref}/invoke calls one tool through the same tool gateway SOIT's agents use: secret injection, egress policy, rate limits, budgets, audit, and cost apply. The call is a run with mode=tool and source=gateway, named in x-soit-run-id.
• An Idempotency-Key header makes a call safe to retry, and a tool whose policy requires approval answers 202 until a reviewer decides in Govern › Approvals (see Approvals).
• MCP clients such as Claude Code and Cursor reach the same tools at /mcp; see MCP Server.

curl -s "http://localhost:9200/api/v1/tools/tool:function:time_now/invoke" \
  -H "Authorization: Bearer $SOIT_API_KEY" \
  -H "Idempotency-Key: nightly-2026-09-27" \
  -H "Content-Type: application/json" \
  -d '{"arguments": {}}'

Response shapes, idempotency rules, and approval handling are in the Gateway API reference.

Running the gateway on its own

New in v1.4.0. The gateway can run in processes of its own, scaled and exposed apart from the console's API. Start them with SOIT_ROLE=gateway and the same configuration, database, Redis, and storage as the rest of the deployment:

SOIT_ROLE=gateway uvicorn app.main:app --host 0.0.0.0 --port 9300

A gateway process serves /v1, /mcp with its discovery metadata under /.well-known, /api/v1/tools, and /health and /metrics; every other path answers 404. It runs none of the background work (outbox dispatch, usage aggregation, reapers, schedules, ingestion, the license heartbeat).

• Keep at least one process with the default role, SOIT_ROLE=all, beside the gateway processes: that is where a call's cost reaches its aggregates, budget thresholds, and alerts.
• SOIT_ROLE is validated at startup and takes all or gateway.
• Gateway processes send no telemetry.

Examples

The SOIT repository ships runnable examples in examples/gateway/ for curl, the OpenAI Python and Node SDKs, LangChain's ChatOpenAI, the OpenAI Agents SDK, the tools API, and, from v1.5.0, the Anthropic Python SDK (python_anthropic.py) and Claude Code (claude_code.md). Each script reads SOIT_BASE_URL, SOIT_API_KEY, and SOIT_MODEL. A compatibility suite in CI (server/tests/compat) drives the OpenAI Python SDK against the gateway: completions, streams, tool calls, structured output, base64 embeddings, model listing, image generation and edits, and error classes. From v1.5.0 it also drives the official Anthropic Python SDK against /v1/messages: messages, the stream helper, tool use and streamed tool input, token counting, the models list, and error classes.

New in v1.6.0. A gateway pilot kit in examples/pilot/ sets up a provider, models, a failover virtual model, a limited key, and a budget over the HTTP API, checks allowed and refused calls, failover, idempotency, cost reconciliation, and an evidence bundle, and tears it all down. A standard-library mock provider rehearses it without credentials.

Known limitations

As of v1.6.0:

• /v1/messages accepts thinking, cache_control, top_k, metadata, and service_tier and does not send them: prompt caching discounts and extended thinking are not available through the route, and responses carry no thinking blocks. count_tokens is SOIT's estimate, not the provider's tokenizer. The Message Batches and Files APIs are not served.
• The gateway does not serve OpenAI's /v1/responses.
• A call that fails, or records no cost, keeps its budget hold until its run ends (for a gateway call, when the call returns), or at most its timeout (LLM_TIMEOUT_SECONDS, or LLM_IMAGE_TIMEOUT_SECONDS for images) and a minute. A call still running after that, such as a very long stream, no longer holds its budget. While Redis is unreachable, replicas do not see each other's holds and can each overshoot a budget by about one call.
• Budget spend reads the daily aggregates for finished days and the cost ledger for today; a day's aggregate is rebuilt by the reconciler (USAGE_RECONCILE_INTERVAL_SECONDS, hourly by default).
• Content capture is decided when a run starts; if the workspace setting cannot be read, content is withheld.
• A key's calls-per-minute and daily limits count the tool calls made with it through /api/v1/tools and /mcp, but not the tool calls an agent or a workflow makes inside a run; those are held by the member's own tool limits, the key's allowed tools, and budgets.
• A direct tool call that waits for approval runs only when the caller sends it again; nothing runs it on the caller's behalf.

v1.4.0 lifts two v1.3.0 limitations: an abandoned stream now ends its model step and is charged, and a key's limits now count its direct and MCP tool calls. The full list of changes is in the changelog.

Related pages

• Gateway API reference — endpoints, headers, and error shapes.
• Budgets and Cost — spend limits, attribution, and checking costs against a bill.
• MCP Server — the same tools for MCP clients.
• Models — providers and model refs.
• Deployment — where gateway processes fit in the service topology.