Troubleshooting

Start troubleshooting by locating the failing layer: web, API, worker, database, Redis, vector store, object storage, Vault, provider adapter, or docs build.

Local stack checks

curl http://localhost:9200/health/ready
curl http://localhost:5000/
docker compose -f docker/docker-compose.yml ps

Expected: API ready, web responding, and all services (especially knowledge-ingest-worker) in a running state.

Common issues

API is unavailable

Check that the migrate job completed (docker compose logs migrate), the database is reachable, and the API service logs show a clean start. A fresh stack needs a few minutes for migrations and bootstrap before health/ready returns 200.

Web is unavailable

Check the web service logs and port conflicts on 5000. The web container depends on the API being reachable for sign-in.

Sign-in fails

Confirm BOOTSTRAP_ADMIN_EMAIL / BOOTSTRAP_ADMIN_PASSWORD in .env and that the bootstrap job ran successfully after migrations.

Knowledge ingestion stalls

Documents stuck in a pending status usually mean the worker path is broken. Check knowledge-ingest-worker health, MinIO and Milvus reachability, and Redis queue connectivity. Document records carry status, error_code, and retry_count — read them before restarting anything.

Model calls fail

Check provider configuration in ModelHub (/model), run the provider connectivity diagnostic, verify the secret reference resolves, and confirm egress policy allows the destination. Invalid credentials surface as a visible failure state on the provider.

A model server on a private or loopback address, such as a local Ollama, also needs its network in EGRESS_PRIVATE_NETWORKS (new in v1.4.0); without it every non-public address is refused. See Private networks.

Plugin tools are missing

Check that the plugin is enabled (GET /api/v1/plugins), then inspect the runtime registry with GET /api/v1/plugins/runtime/tools and re-sync with POST /api/v1/plugins/runtime/reload if needed.

Events not being processed

Check worker health and the outbox: pending rows in event_outbox mean dispatch is stalled; rows in dead_letter_events mean deliveries failed and need review or replay.

Docs build fails

Run npm run types:check in soit-docs and inspect MDX/frontmatter errors — a missing title or malformed frontmatter is the usual cause.

When evidence beats guessing

Most runtime questions are answerable from the ledger before touching infrastructure: open the failing run in /observe/runs/{run_id} and read the failing step's error evidence, metrics, and audit records. See Observability.