Knowledge

Knowledge gives agents and workflows governed access to workspace documents and retrieval results. SOIT uses knowledge bases, documents, chunks, indexes, embeddings, and citation evidence to make retrieval inspectable end to end. v1.5.0 adds connectors that keep a knowledge base in step with an S3-compatible bucket or a website. v1.6.0 adds restricted documents, and reads copies of retrieved content back by the reader's current access.

When to use knowledge

Use knowledge when agent answers or workflow decisions should be grounded in private documents, policies, runbooks, product manuals, or support content. For small durable context signals (user preferences, workspace facts), use Memory instead.

The ingestion and retrieval pipeline

1. Create a knowledge base with chunking, retrieval, and embedding settings (default_embedding_model_ref).
2. Ingest documents — sources are tracked by source_kind (upload | crawler | api | manual | connector), and each document is versioned with doc_key + version. Documents a connector syncs have source_kind connector.
3. Chunk and embed — the asynchronous ingestion worker splits documents into chunks, embeds them, and writes them to a named vector index (Milvus-backed).
4. Retrieve at runtime — retrieval appears as a retrieval run step with metrics (vector counts, latency), and answers carry citations linking output back to source chunks.

Ingestion is asynchronous and status-tracked — documents show status, error codes, and retry counts, so a stalled pipeline is visible rather than silent.

From v1.6.0 the keyword fallback of retrieval answers from the latest version of each document only, as the vector index does.

Knowledge connectors

New in v1.5.0. A source keeps a knowledge base in step with a system that already holds the documents: an S3-compatible bucket or a website. A sync lists what the remote has, ingests what is new, saves what changed as a new version, leaves what did not change alone, and, if the source says so, deletes the documents of items that went away.

Documents a source creates carry the remote identity in external_id and source_uri, and keep a stable document key per item, so a changed item becomes a new version of the same document and rollback works as for any other document. They are ingested through the same queue as uploads, so parsing, chunking, embedding, and their retries are the ingest worker's. A knowledge base needs an index before it can sync, as it does for uploads.

Setting up a source

1. For S3, create the secret the connector needs under Govern › Secrets.
2. In the knowledge base, open Sources › Add source, pick the connector, fill in its settings, and Test connection. The test lists a sample of what would sync and ingests nothing.
3. Save, then Sync now, or set a schedule.

Through the API:

curl -X POST "$SOIT/api/v1/knowledge/$KB/sources" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{
    "name": "handbook bucket",
    "connector_kind": "s3",
    "config": {"bucket": "handbook", "region": "eu-west-1", "prefix": "docs/",
               "include": ["*.pdf", "*.md"]},
    "secret_id": "sec_...",
    "schedule_cron": "0 2 * * *",
    "schedule_timezone": "Europe/Berlin",
    "delete_removed": false
  }'
curl -X POST "$SOIT/api/v1/knowledge/$KB/sources/$SOURCE/sync" -H "Authorization: Bearer $TOKEN"

A sync answers 202 with the queued run; a second request while one is queued or running answers 409.

S3-compatible storage (s3)

Works with AWS S3, MinIO, Cloudflare R2, and other services that speak the S3 API.

Credentials come from a secret whose value is JSON with access_key_id, secret_access_key, and an optional session_token. The source stores only the secret's id; the value is read when a test or sync runs and is never written to the source, a run, an audit event, or a response. Credentials placed in the source's configuration are refused.

Folder placeholder keys and files the knowledge parsers cannot read are counted as skipped. A listing of more than 500,000 objects under the prefix fails the run with a message to narrow the prefix or patterns.

Website crawl (web)

Starts at the seed URLs and follows links breadth-first.

• Links are followed only on the seeds' own hosts and ports, and a redirect to another host is not followed.
• robots.txt is obeyed, including Crawl-delay (up to 5 seconds), as are noindex and nofollow robots meta tags and rel="nofollow" links. The crawler identifies itself as SOITKnowledgeBot.
• No credentials, cookies, or custom headers are sent, so only public pages can be read. A secret on a web source is refused.
• HTML, plain text, Markdown, and PDF are accepted, up to 5 MB per page.
• A page that fails for a transient reason (a 5xx, a timeout) is reported as failed, never as removed. A crawl cut off by max_pages removes nothing, and a failing seed fails the run.

Change detection and removed items

For each item a sync decides, in order:

With delete_removed off (the default) a removed item's document is kept. With it on, every version of its document is deleted, like deleting the document in the console. Removal is decided only from a listing that finished: a run that failed part way, was canceled, hit one of its caps, or reported itself incomplete never removes anything.

The remote is the source of truth. A synced document deleted by hand is created again by the next sync while the item still exists remotely, and manual edits to a synced document's chunks are overwritten by its next version.

Scheduling and the sync worker

A source takes a five-field cron expression and a time zone (schedule_cron, schedule_timezone). Without one it syncs only on demand. A schedule that would fire more often than KNOWLEDGE_SYNC_MIN_INTERVAL_SECONDS (300 by default) is refused. A disabled source is neither scheduled nor synced on demand, and a firing that comes while the source's previous run is still going is skipped rather than stacked.

The sync worker runs wherever the knowledge ingest worker runs, because a sync hands its documents to it: in the API process when KNOWLEDGE_INGEST_WORKER_ENABLED is on, in scripts/ingest_worker.py, and in scripts/combined_worker.py. It is on by default; KNOWLEDGE_SYNC_WORKER_ENABLED=false turns the sync half off. A run whose worker died is reclaimed once its lease expires and starts over; a run interrupted KNOWLEDGE_SYNC_MAX_ATTEMPTS times (3 by default) is given up on.

Limits

Every sync is bounded. Each source may set its own caps (limits), within the deployment's ceilings:

Reaching the item or total cap ends the run early and marks it truncated ("cut off" in the console); nothing is removed on a truncated run.

Network access

Connectors reach the network only through SOIT's governed egress policy, the same policy that guards tool calls and web fetches: every request and every redirect hop is checked against the tenant and workspace allow and block lists. An address on a private or loopback network is refused unless its network is listed in EGRESS_PRIVATE_NETWORKS. For a MinIO on an internal network or an intranet documentation site, allow the host in the workspace egress policy and add its network to EGRESS_PRIVATE_NETWORKS. A refusal is reported on Test connection and on the run with the host and the setting to change.

Failures and audit

• A failing item makes the run partial and is listed in the run's outcomes.
• A listing or credential failure (rejected credentials, unreachable endpoint, egress refusal, a crawl seed that is gone) fails the run, and the source shows the error.
• A failed run appears in Observe's dead letters as knowledge_sync until a later run of the same source exists. Redrive queues a fresh run of the source; the failed run stays as the record.
• Creating, changing, and deleting a source, and each sync, are audited (knowledge.source.created, .updated, .deleted, knowledge.source.sync.started, .finished), with counts, ids, and names but never credentials.

Reading sources and runs needs read access to the knowledge base; creating, changing, testing, syncing, canceling, and deleting need update access. A private knowledge base's sources are as hidden as the base itself.

Connectors do not map the source system's permissions. Synced documents take the knowledge base's own visibility, so everyone who can retrieve from the knowledge base can retrieve synced documents that are not restricted. Keep content with different audiences in different knowledge bases, or restrict the documents that need it.

Not included: OAuth and SaaS connectors (Google Drive, Notion, Confluence), authenticated or JavaScript-rendered websites, and deleting a source's documents when the source is deleted (the documents stay in the knowledge base and are no longer kept in step).

Who can read knowledge

A knowledge base's visibility decides who reaches it at all:

Only the creator or a workspace Owner or Admin changes it.

Restricted documents

New in v1.6.0. A document can be kept from readers of its knowledge base. A restricted document is readable only by:

• the workspace's Owners and Admins,
• the knowledge base's creator,
• members and service principals holding a knowledge_document grant with the read action on it.

Everyone else meets it nowhere:

• It is left out of the document listing.
• The document, its content, its download, its chunks, and its versions answer 404.
• Retrieval leaves it out before ranking, reranking, or the keyword fallback see it, for every strategy (vector, keyword, hybrid, multi-index). An agent's RAG, a workflow's retrieve node, the knowledge_query tool, and MCP all retrieve through the same query, as the caller.

The retrieval step of a run records how many restricted documents it left out (restricted_documents_excluded in its metrics).

A restriction names the document by its doc_key, so it holds for every version: a new upload, a connector sync bringing a new version, and a rollback to an old one all stay restricted. It takes effect, and lifts, at once; no cache stands between a change and the next read.

Only a workspace Owner or Admin or the knowledge base's creator restricts a document or lifts a restriction, with PUT /api/v1/knowledge/{knowledge_id}/document-restrictions. Each change is written to the audit ledger as knowledge.document.restricted or knowledge.document.unrestricted. The restriction listing answers only the restricted documents the caller may read, so a reader is not told which documents are kept from them.

To open a restricted document to one member or service principal, a workspace Owner or Admin grants it with POST /api/v1/resource-grants, naming resource_type knowledge_document and resource_id "{knowledge_id}:{doc_key}"; each restriction in the listing carries that value as grant_resource_id. See the Knowledge API for the requests.

In the console, Access on a document's row in Build › Knowledge shows whether it is restricted, restricts it or lifts the restriction, and, once it is restricted, lists the members granted read and grants or revokes it.

Restrictions are stored in the knowledge_document_restrictions table (migration 20261002120000); nothing is restricted until someone restricts it. Downgrading below that migration drops every restriction, so restricted documents become readable to everyone their knowledge base admits.

Granting takes an Owner or Admin

From v1.5.2, granting or revoking access to any resource (POST and DELETE /api/v1/resource-grants) takes a workspace Owner or Admin; a member with the Developer role gets 403. Earlier, workspace write permission was enough, so a Developer could grant themselves every action on a private knowledge base. The same release requires a document's file_id to name a file under the same knowledge base's storage; any other is refused with 400. Both fixes are part of v1.6.0.

Copies kept in runs, responses, and threads

New in v1.6.0. Runs keep what retrieval returned: citations, knowledge_query results, source events, a thread message's citations and tool results. Reading them back obeys the reader's access now, not the access of whoever ran the retrieval or the access when it ran. A document the reader may not read — restricted from them, in someone else's private knowledge base, or in a base not shared with their workspace — is:

• left out of every list it appears in, with totals recounted,
• replaced by a withheld marker where it is a single item outside a list.

This applies to run detail (citations, tool calls, and response events), the response endpoints and their event stream, thread messages, tool result downloads, and a repeated tool call answered from its stored result. The stored records themselves are not changed, so lifting a restriction shows them again. A copy that names a deleted document or knowledge base is shown as it was.

Some copies are text that cannot be filtered item by item, so they are withheld whole (replaced by a marker giving their length and hash) when they came from a document the reader may not read:

• A step's output summary, in the step list, run detail, Observe replay, the evidence bundle, and a workflow's event stream. A knowledge_query step is judged by the result it kept, and a workflow step by the documents its output named, which the step now records in its knowledge_documents metric. A summary that is JSON is filtered like any other copy instead.
• An agent's retrieved context and conversation in a task's approval checkpoint, judged by the checkpoint's citations, in task detail, the task list, and the task handling drawer.

Steps recorded before v1.6.0 do not know which documents they came from and are judged conservatively: a knowledge_query step's summary is withheld when the reader may not read some document of the knowledge base it queried, and a workflow retrieve step's when the reader may not read some document anywhere in the workspace.

How knowledge connects

• Knowledge bases are workspace-scoped, like every SOIT resource.
• Agents bind knowledge bases as typed capabilities; retrieval settings come from the published agent version.
• Workflow retrieval nodes use the same governed retrieval path.
• Retrieval steps and citations land in the runtime ledger, so Observe can show which sources supported which answer.
• Embedding calls go through the model port and record cost entries.
• Resource grants and roles come from Identity and Access.

API and UI surfaces

• API prefix: /api/v1/knowledge (bases, documents, indexing, sources and sync runs, document restrictions, retrieval evidence)
• Related prefixes: /api/v1/runs, /api/v1/responses, /api/v1/resource-grants
• Web route: /knowledge (bases, documents, sources, analytics); Access on each document

Known limitations

As of v1.6.0:

• Text a model wrote from retrieved documents — a response's answer, assistant messages, and a workflow's later nodes that used it — is shown as written to every reader; only its citations are filtered.
• A run records how many documents retrieval left out, not which grant or role let it read each one.
• A knowledge_document read grant names one member or service principal; there are no groups.
• Connector-synced documents do not inherit the source system's permissions.