Observability

SOIT observability starts with runtime evidence and extends to platform metrics and logs. The split matters: product observability answers "what did agents do", while platform observability answers "is the infrastructure healthy".

Product observability

The runtime ledger records this automatically for every execution — nothing to instrument:

• Runs and run steps with status, timings, and metrics.
• Task status with explicit transitions.
• Cost entries attributed to runs, steps, models, and tools.
• Retrieval citations linking answers to sources.
• Tool call evidence, including policy-gateway request/response audits.
• Approval and feedback state.
• Audit records for privileged operations.

Read it through the Observe views: /observe?range=24h for recent activity, /observe/runs for the filterable Run Explorer, /observe/runs/{run_id} for full replay, and /observe/audits?gateway_type=tool for gateway audit evidence.

Platform observability

• Structured logs carrying request_id and run_id, so infrastructure logs join to runtime evidence.
• OpenTelemetry-compatible tracing on the backend.
• Prometheus metrics where configured.
• Worker health and queue depth — knowledge ingestion and event dispatch fail quietly if workers die, so monitor them explicitly.
• Storage, vector, and secret backend health (MinIO, Milvus, Vault reachability).
• The outbox dead-letter table (dead_letter_events) — non-empty means events failed dispatch and need operator review or replay.

Trace export (OTLP)

New in v1.5.0. A trace — the runs sharing a trace_id and their steps — exports as an OTLP/JSON ExportTraceServiceRequest that an OpenTelemetry collector, Jaeger, or Tempo accepts on its OTLP/HTTP endpoint:

GET /api/v1/runs/trace/{trace_id}/otlp

The response is application/json, downloaded as trace-<id>.otlp.json. Export OTLP on a trace's page in the console downloads the same file.

• One resource (service.name: soit, soit.tenant.id, soit.workspace.id, soit.trace.id) and one scope named soit.
• One span per run, named run <kind>, whose parent is the parent run's span when that run is in the trace, and one span per step, named by its step type, under its run's span. A run without a parent is a SERVER span, a model, tool, or retrieval step a CLIENT span, everything else INTERNAL.
• Span ids are stable: the trace id is the stored id when it already is 32 hex characters, and otherwise a hash of it, and each span id is a hash of the run or step id, so an unchanged trace always gives the same document.
• Timings are nanosecond startTimeUnixNano and endTimeUnixNano; a run or step still open ends where it started. status.code is OK for succeeded, ERROR with the error code as the message for failed, and UNSET otherwise.
• Attributes are soit.run.* (id, mode, kind, status, subject, attempt, parent and source run, source, sandbox, error code), soit.step.* (id, step id, type, node id, status, error code), and each numeric top-level step metric as soit.step.metric.<name>.

The document holds identifiers, enumerations, timings, and numbers only: no run content — summaries, error text, tool arguments, and results are never in it, so a content-free workspace exports the same shape. The contract is server/app/kernel/specs/v1/otlp_trace_spec.schema.json. The export needs workspace read access, a trace with no run in the caller's workspace is 404, and each download is written to the audit ledger as trace.otlp_exported with the run and span counts.

Operational practice

Use Observe views to investigate product behavior — a bad answer, an expensive run, a failed workflow. Use infrastructure telemetry to diagnose service health, latency, queue delay, and external provider failures. The request_id/run_id pairing lets you cross from one to the other: an alert on API latency can be traced to the specific runs affected, and a suspicious run can be traced to the infrastructure state that shaped it.

Related pages

• Observe — the product pillar.
• Runtime Ledger — what gets recorded and why.
• Evidence and Exports — ledger exports and run evidence bundles.
• Troubleshooting — symptom-driven diagnosis.