Skip to main content

Multi-Provider Fallback

Keeptrusts can route across multiple eligible provider targets. This is provider-target-level fallback, not automatic cross-region endpoint selection; each target uses its own configured provider, region, and endpoint.

The gateway advances to the next eligible target on a transport error. For an HTTP response, it advances only when the response is classified into one of the configured trigger_on classes. A successful response is returned without trying later targets.

Start with valid provider targets​

pack:
  name: multi-provider-routing
  version: 0.1.0
  enabled: true
providers:
  targets:
  - id: openai-primary
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: KEEPTRUSTS_OPENAI_API_KEY
  - id: azure-fallback
    provider: azure
    provider_type: azure-openai
    format: openai
    model: your-azure-openai-model
    base_url: https://replace-with-resource-name.openai.azure.com
    secret_key_ref:
      env: KEEPTRUSTS_AZURE_OPENAI_API_KEY
    azure_api_version: your-supported-api-version
    azure_deployment: your-azure-deployment
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Routing strategy​

Ordered routing​

providers:
  routing:
    strategy: ordered
  targets:
  - id: openai-primary
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: KEEPTRUSTS_OPENAI_API_KEY
  - id: azure-fallback
    provider: azure
    provider_type: azure-openai
    format: openai
    model: your-azure-openai-model
    base_url: https://replace-with-resource-name.openai.azure.com
    secret_key_ref:
      env: KEEPTRUSTS_AZURE_OPENAI_API_KEY
    azure_api_version: your-supported-api-version
    azure_deployment: your-azure-deployment

Latency-aware routing​

Runtime-effective adaptive routing; current lint schema uses stale field names
providers:
  routing:
    strategy: lowest_latency
    exploration_ratio: 0.1
    preferred_max_latency:
      ms: 500
      percentile: p90
  targets:
  - id: openai-primary
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: KEEPTRUSTS_OPENAI_API_KEY

The gateway's live metrics store currently uses a fixed 300-second window and five-sample threshold. Parsed measurement_window_seconds and min_sample_count values do not change that store. The parser reads ms and tps in SLA preference objects, while the current lint schema still accepts only stale value fields.

Fallback triggers​

providers:
  fallback:
    trigger_on:
    - rate_limit
    - server_error
    - timeout
    - context_length_exceeded
  targets:
  - id: openai-primary
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: KEEPTRUSTS_OPENAI_API_KEY
  - id: azure-fallback
    provider: azure
    provider_type: azure-openai
    format: openai
    model: your-azure-openai-model
    base_url: https://replace-with-resource-name.openai.azure.com
    secret_key_ref:
      env: KEEPTRUSTS_AZURE_OPENAI_API_KEY
    azure_api_version: your-supported-api-version
    azure_deployment: your-azure-deployment

Accepted trigger values are rate_limit, server_error, timeout, context_length_exceeded, content_filter, model_not_found, and auth_error. When trigger_on is omitted, the runtime defaults to every value in that list except auth_error. Authentication failures also have a dedicated one-attempt credential-cache refresh path.

Current enable and attempt-limit behavior

The current parser records fallback.enabled and max_fallback_attempts, but the dispatch loop does not use either value. Do not rely on them to disable or bound fallback. The loop considers all eligible targets; use a single target or an explicit provider pin when a request must not move to another target. You can also set providers.routing.allow_fallbacks: false to keep only the first eligible target in the candidate list.

Circuit breaker​

providers:
  circuit_breaker:
    enabled: true
    consecutive_failure_threshold: 5
    cooldown_seconds: 30
    half_open_successes: 1

What to validate​

  • Replace the example model placeholders with model IDs supported by the configured providers.
  • The target ids and provider secrets resolve correctly.
  • The routing strategy matches how you want the gateway to choose targets.
  • The fallback trigger list matches the failures you want retried automatically.
  • Circuit breaking is enabled only when you want unhealthy targets removed temporarily.

Circuit-breaker state is local to one gateway process. For streaming requests, fallback decisions happen from connection failures or the upstream HTTP status before a successful stream is passed through; the gateway does not restart a stream on another target after chunks have reached the client.

warning

Fallback can repeat a billable or side-effecting request if an upstream processed it before the gateway observed a transport failure. Use idempotency controls supported by your provider for operations where duplicate dispatch would be unsafe.

Next steps​