Skip to main content

Advanced Provider Features

These settings live under the top-level providers: section. This page states which keys affect current runtime behavior and which are metadata only.

Replace every your-...-model placeholder with a model ID enabled for that provider account before running the gateway.

Traffic mirroring​

Runtime-effective shadow routing; runtime_routing is not yet in the lint schema
runtime_routing:
  shadow_routing:
    enabled: true
    capture_mode: full_payload
providers:
  traffic_mirror:
    enabled: true
    mirror_target: openai-shadow
    sample_rate: 0.05
  targets:
    - id: openai-primary
      provider: openai
      model: your-openai-model
      secret_key_ref:
        env: KEEPTRUSTS_OPENAI_API_KEY
    - id: openai-shadow
      provider: openai
      model: your-shadow-openai-model
      secret_key_ref:
        env: KEEPTRUSTS_OPENAI_SHADOW_API_KEY

Both enable flags and capture_mode: full_payload are required before the runtime calls the mirror provider. With the default metadata_only mode, the runtime records an event but sends no shadow request; privacy-restricted mode also forces metadata-only behavior. Dispatch reads mirror_target and sample_rate; the sample rate is clamped to 0.0–1.0. Parsed log_mirror_response and timeout_ms values are not read by the runtime. The current lint schema does not admit top-level runtime_routing, so this combined runtime-effective snippet is intentionally a text fence.

A/B testing metadata​

providers.ab_test and its variants are parsed, but no dispatch path currently reads them to split or stick traffic. Do not use this block as an active A/B experiment control.

Model groups​

providers:
  model_groups:
    - name: fast
      aliases:
        - default-fast
      description: Low-latency chat targets
      targets:
        - openai-mini
        - openai-standard
      fallback_group: standard
    - name: standard
      targets:
        - openai-prod
        - openai-shadow
FieldTypeNotes
namestringRequired.
targetsstring[]Required and must reference known provider target IDs.
aliasesstring[]Optional alternate names.
descriptionstringOptional human-readable description.
fallback_groupstringOptional active target-chain continuation; must reference another known group and cannot create a cycle.

When a request model resolves to a group, the runtime orders that group's targets followed by each configured fallback_group. Normal provider-attempt rules determine whether execution advances along the chain.

Provider rate-limit metadata​

providers.rate_limits is parsed, but the current dispatcher does not enforce its per-key, per-user, per-team, global, or parallel-request values. Do not rely on it for runtime rate limiting.

Logging controls​

providers:
  logging:
    redact_message_bodies: true

redact_message_bodies is applied to gateway payload logging and defaults to false. redact_api_keys is parsed but has no runtime read, so it must not be treated as a credential-redaction control.

Context compression​

providers:
  context_compression:
    strategy: middle_out

Compression runs when a selected provider target declares max_context_tokens and the request exceeds it. The dispatcher reads only strategy (middle_out or oldest_first). It does not read enabled, the preserve_* fields, max_messages, or message_compression_strategy.

Zero-completion insurance​

providers:
  zero_completion_insurance:
    enabled: true

For buffered chat completions, enabled is the only field the dispatcher reads. A detected zero completion always advances to the next eligible provider. Parsed conditions, action, and retry_with_fallback values do not change detection, billing, retry, or fallback behavior.

Complete example​

pack:
  name: advanced-provider-controls
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: openai-prod
    provider: openai
    provider_type: openai
    format: openai
    model: your-openai-model
    max_context_tokens: 128000
    secret_key_ref:
      env: KEEPTRUSTS_OPENAI_API_KEY
  - id: openai-shadow
    provider: openai
    provider_type: openai
    format: openai
    model: your-shadow-openai-model
    secret_key_ref:
      env: KEEPTRUSTS_OPENAI_SHADOW_API_KEY
  model_groups:
  - name: standard
    targets:
    - openai-prod
    - openai-shadow
  logging:
    redact_message_bodies: true
  context_compression:
    strategy: middle_out
  zero_completion_insurance:
    enabled: true
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Best practices​

  • Use an enforced policy or platform rate-limit surface; provider-scope rate_limits metadata is not currently enforced.
  • Enable both runtime shadow routing and provider traffic mirroring, and set shadow capture mode to full_payload, before expecting provider calls.
  • Validate model-group target IDs carefully; unknown targets fail parsing.
  • Set target max_context_tokens when you expect context compression to run.

Next steps​