Skip to main content

Streaming & SSE

Keeptrusts supports streaming text responses over the public HTTP request families. Clients enable streaming by sending "stream": true in the request body.

Before testing, start a gateway with an agent binding, a provider credential, and a model that supports streaming. Use a separate application request token for the call.

Public route families​

Use streaming with the public text-family routes documented in Runtime Request Families:

  • POST /v1/chat/completions
  • POST /v1/responses
  • POST /v1/messages

Minimal provider config​

pack:
  name: streaming-sse
  version: 0.1.0
  enabled: true
providers:
  targets:
  - id: openai
    provider: openai
    model: your-streaming-model
    secret_key_ref:
      env: KEEPTRUSTS_OPENAI_API_KEY
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Example request​

Use an application API token, not the connected gateway's runtime token:

export KEEPTRUSTS_REQUEST_TOKEN="kt_..."

curl -N http://127.0.0.1:41002/v1/chat/completions \
  -H "Authorization: Bearer ${KEEPTRUSTS_REQUEST_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-streaming-model",
    "stream": true,
    "messages": [{"role": "user", "content": "Hello"}]
  }'

-N disables cURL output buffering so each SSE frame is visible as it arrives. For Chat Completions, a normal completed stream ends with the provider's terminal event (commonly data: [DONE]). Capture the HTTP status as well as the body when diagnosing a stream:

curl -N --fail-with-body \
  -D stream-headers.txt \
  http://127.0.0.1:41002/v1/chat/completions \
  -H "Authorization: Bearer ${KEEPTRUSTS_REQUEST_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"model":"your-streaming-model","stream":true,"messages":[{"role":"user","content":"Reply with one short sentence."}]}'

Practical guidance​

  • No extra gateway config is required beyond a normal provider target and a streaming-capable upstream model.
  • For chat-completions and Responses requests, input policies run before the upstream dispatch. Output policies that require accumulated content can make the gateway buffer chunks or interrupt the stream with a terminal policy error.
  • The Messages handler currently authenticates, resolves billing and routing, and proxies the stream, but does not run the chat/Responses policy-chain evaluation path. Do not treat /v1/messages as equivalent governance until that runtime path is implemented.
  • Provider fallback can choose another target before a successful stream begins. It does not restart on a different target after chunks have reached the client.

Verify and troubleshoot​

Tail decision events in a second terminal while sending one identifiable request:

kt events tail --since 10m --follow
  • 401 or 403 before any frame: check the application token, not the gateway runtime or provider credential.
  • A non-streaming JSON response: confirm the request body contains "stream": true and the selected model supports streaming.
  • Headers followed by an early terminal policy error: inspect the Event policy results; an output control may need accumulated content or may stop delivery.
  • A partial stream with a network error: do not assume the gateway retried on a second provider after the client received chunks.
  • No matching Event: verify event delivery, organization, region, and time filters before concluding that the request bypassed the gateway.

Next steps​