Skip to main content

Providers Configuration

The providers: section declares upstream LLM targets, routing, retry behavior, data-handling metadata, circuit breaking, and conditional traffic mirroring. This page distinguishes active dispatch controls from fields that are only parsed today.

Replace every your-...-model placeholder with a model ID enabled for that provider account before running the gateway.

Quick reference​

pack:
  name: config-providers-providers-1
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: primary
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: KEEPTRUSTS_OPENAI_API_KEY
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Provider targets​

Each target describes a single upstream endpoint. The gateway evaluates targets in order when using the ordered routing strategy.

Minimal target​

pack:
  name: config-providers-providers-2
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: openai-prod
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: KEEPTRUSTS_OPENAI_API_KEY
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Only id and provider are schema-required. A provider alias can infer base_url, API-key header, and request path defaults, but it never invents a credential or secret_key_ref. An ordinary auth-required target with no resolvable credential is inactive.

Full target reference​

pack:
  name: config-providers-providers-3
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: openai-prod
    provider: openai
    model: your-openai-model
    base_url: https://api.openai.com
    secret_key_ref:
      env: KEEPTRUSTS_OPENAI_API_KEY
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Credential resolution​

The parser permits an omitted secret_key_ref. Before dispatch, an ordinary auth-required target still needs a resolvable credential; otherwise the gateway marks it inactive. Auth-optional and self-credentialed provider types use their own documented authentication path. The public schema does not expose the runtime's internal per-target required flag.

Use explicit secret_key_ref values for ordinary env- or store-authenticated targets you want active, and remove targets you are not ready to configure yet.

pack:
  name: config-providers-credentials
  version: 1.0.0
  enabled: true
providers:
  routing:
    strategy: ordered
  targets:
  - id: openai
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: KEEPTRUSTS_OPENAI_API_KEY
  - id: azure
    provider: azure
    provider_type: azure-openai
    format: openai
    model: your-azure-openai-model
    base_url: https://replace-with-resource-name.openai.azure.com
    secret_key_ref:
      env: KEEPTRUSTS_AZURE_OPENAI_API_KEY
    azure_api_version: your-supported-api-version
    azure_deployment: your-azure-deployment
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Later eligible targets are considered without a providers.fallback.enabled switch. Use providers.routing.allow_fallbacks: false, a provider pin, or a single eligible target when the request must not move to another target.

Data policies​

Declare the data handling guarantees your provider contract includes. The data-routing-policy policy uses these to filter targets.

pack:
  name: config-providers-providers-4
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: openai-zdr
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: KEEPTRUSTS_OPENAI_API_KEY
    data_policy:
      zero_data_retention: true
      training_opt_out: true
      retention_days: 0
  - id: azure-standard
    provider: azure
    provider_type: azure-openai
    format: openai
    model: your-azure-openai-model
    base_url: https://replace-with-resource-name.openai.azure.com
    secret_key_ref:
      env: KEEPTRUSTS_AZURE_OPENAI_API_KEY
    azure_api_version: your-supported-api-version
    azure_deployment: your-azure-deployment
    data_policy:
      zero_data_retention: false
      training_opt_out: true
      retention_days: 30
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Token pricing​

Declare pricing for spend tracking, cost assertions, and the auto provider scoring.

pack:
  name: config-providers-providers-5
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: openai-4o
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: KEEPTRUSTS_OPENAI_API_KEY
    pricing:
      input_price_per_million: 0.15
      cached_input_price_per_million: 0.075
      output_price_per_million: 0.60
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Nested models​

A single target can serve multiple models. Each model can have its own aliases, pricing, and escalation routing.

pack:
  name: config-providers-providers-6
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: openai-prod
    provider: openai
    base_url: https://api.openai.com
    secret_key_ref:
      env: KEEPTRUSTS_OPENAI_API_KEY
    models:
      - model_id: your-openai-model
        aliases: [fast]
        pricing:
          input_price_per_million: 0.15
          output_price_per_million: 0.60
      - model_id: gpt-5.4
        aliases: [reasoning]
        escalation_routing:
          team_id: 11111111-1111-4111-8111-111111111111
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Escalation routing override​

Route escalations to specific teams or users per provider or model.

pack:
  name: config-providers-providers-7
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: healthcare-openai
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: KEEPTRUSTS_OPENAI_API_KEY
    escalation_routing:
      team_id: 11111111-1111-4111-8111-111111111111
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Health probes​

Active health probes run in the background and record each probe result as the provider's current health status.

pack:
  name: config-providers-providers-8
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: openai-prod
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: KEEPTRUSTS_OPENAI_API_KEY
    health_probe:
      endpoint: https://health.example.com/ready
      interval_seconds: 30
      timeout_ms: 2000
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Health probes send a bare GET; they do not attach the target credential or custom provider headers. Point endpoint at an operator-controlled, unauthenticated 2xx readiness route. The default {base_url}/health is useful only when that upstream actually exposes a public health endpoint.

OAuth2 authentication​

For providers requiring OAuth2 bearer tokens (Databricks, Snowflake Cortex, custom endpoints).

pack:
  name: config-providers-providers-9
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: databricks-prod
    provider: databricks
    model: your-databricks-model
    base_url: https://my-workspace.databricks.net
    oauth2:
      grant_type: client_credentials
      token_endpoint: https://accounts.example.com/oauth2/token
      client_id: keeptrusts-gateway
      client_secret_env: KEEPTRUSTS_DATABRICKS_CLIENT_SECRET
      scopes: [all-apis]
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Cloud-specific fields​

pack:
  name: config-providers-providers-10
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: azure-primary
    provider: azure-openai
    model: your-openai-model
    base_url: https://replace-with-resource-name.openai.azure.com
    secret_key_ref:
      env: KEEPTRUSTS_AZURE_OPENAI_API_KEY
    azure_api_version: your-supported-api-version
    azure_deployment: your-azure-deployment
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Routing strategies​

The routing.strategy field controls how the gateway selects a target for each request.

providers:
  routing:
    strategy: ordered
  targets:
  - id: openai-primary
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: KEEPTRUSTS_OPENAI_API_KEY
StrategyDescription
orderedTry targets in declaration order; first healthy target wins
round_robinCycle through targets evenly
weighted_round_robinCycle weighted by each target's weight field
randomRandom selection
simple_shuffleShuffle then iterate
lowest_latencyRank sampled targets by observed P50 time to first token
highest_throughputRank sampled targets by observed P50 throughput
least_connectionsSelect target with fewest in-flight requests
least_busySelect target with lowest current load
usage_basedSelect target with lowest cumulative usage
semanticMatch request embedding against target descriptions

Semantic routing changes declaration order only when the runtime can resolve an embedding target through routing.semantic_embedding_provider (or the configured cache embedding provider) and at least one candidate target has a non-empty description. It applies semantic_similarity_threshold; if no target reaches the threshold—or the embedding provider or descriptions are missing—the original candidate order is preserved.

Latency / throughput preferences​

Runtime-effective adaptive routing; current lint schema uses stale field names
providers:
  routing:
    strategy: lowest_latency
    exploration_ratio: 0.1
    preferred_max_latency:
      ms: 500
      percentile: p90
    preferred_min_throughput:
      tps: 50
      percentile: p50
  targets:
  - id: openai-primary
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: KEEPTRUSTS_OPENAI_API_KEY

The live gateway metrics store currently uses a fixed 300-second window and a five-sample threshold. Although measurement_window_seconds and min_sample_count are parsed, they do not configure that store. The SLA preference objects use ms and tps; the older value key is ignored. The current lint schema still accepts window_seconds and value instead of the runtime parser's field names, so it rejects the runtime-effective snippet.

Cost ceiling​

max_price is a dispatch-time estimate filter, not a hard spend ceiling. The current runtime does not pass prompt-token counts into this filter, so prompt cost is estimated as zero. Completion and request cost are also estimated as zero when the request omits max_tokens, and targets without a pricing block remain eligible. Use governed-token or wallet budget enforcement for a fail-closed monetary limit.

providers:
  routing:
    strategy: ordered
    max_price:
      prompt: 0.01
      completion: 0.05
      request: 0.1
  targets:
  - id: openai-primary
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: KEEPTRUSTS_OPENAI_API_KEY
    pricing:
      input_price_per_million: 0.15
      output_price_per_million: 0.60

Region and quantization filtering​

providers:
  routing:
    strategy: ordered
    require_region: eu
    require_quantizations:
    - fp16
    - int8
  targets:
  - id: openai-primary
    provider: openai
    model: your-openai-model
    region: eu
    quantizations: [fp16, int8]
    secret_key_ref:
      env: KEEPTRUSTS_OPENAI_API_KEY

require_region is fail-closed. If no targets match the configured region, the request is blocked. There is no cross-region fallback option.

Target filtering​

providers:
  routing:
    strategy: round_robin
    only:
    - openai-primary
  targets:
  - id: openai-primary
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: KEEPTRUSTS_OPENAI_API_KEY

Provider attempts and fallback behavior​

The gateway orders eligible targets, retries a target according to providers.retry_policy, and then considers later targets. A transport error advances to the next eligible target. An HTTP response advances only when its classification is in the runtime trigger list; a successful response stops the loop.

The default response triggers are rate_limit, server_error, timeout, context_length_exceeded, content_filter, and model_not_found. The runtime also recognizes auth_error, which has a one-attempt credential-cache refresh path.

To override the runtime trigger list, use this shape:

providers:
  fallback:
    trigger_on:
      - rate_limit
      - server_error
      - timeout
Runtime limitations

The parser records fallback.enabled, max_fallback_attempts, content_policy, and context_window, but dispatch does not read them. Do not rely on those fields to enable, disable, bound, replace, or reshape fallback. Use providers.routing.allow_fallbacks: false to restrict routing to the first eligible target, or pin the request to a provider.

See Multi-Provider Fallback for the exact attempt and streaming boundaries.

Context compression​

Context compression currently runs when the selected target declares max_context_tokens and the request exceeds that limit. Dispatch reads only context_compression.strategy (middle_out or oldest_first). It does not read enabled, the preserve_* fields, max_messages, or message_compression_strategy.

providers:
  context_compression:
    strategy: middle_out

Zero-completion handling​

For buffered POST /v1/chat/completions responses, the dispatcher currently checks only zero_completion_insurance.enabled. When its built-in detector finds a zero completion, the gateway always continues to the next eligible provider.

providers:
  zero_completion_insurance:
    enabled: true

The parser records conditions, action, and retry_with_fallback, but the dispatcher does not read them. Do not rely on those fields to select a detector, suppress billing, choose same-target retry, or control provider fallback.

Model groups​

Group targets into logical model groups with aliases and fallback chains.

providers:
  model_groups:
    - name: "fast-models"
      aliases: ["fast", "quick"]
      description: "Low-latency models for real-time use"
      targets: ["groq-llama", "openai-4o-mini"]
      fallback_group: "standard-models"
    - name: "standard-models"
      aliases: ["standard", "default"]
      targets: ["openai-4o", "azure-standard"]

fallback_group is active runtime behavior: it appends that group's target IDs to the ordered candidate chain after the current group's targets. The usual provider-attempt rules still decide whether execution advances along the chain.

Provider pipelines​

Orchestrate multiple provider targets behind one virtual model name. Use mode: sequence when one model's output should feed the next step, or mode: fan_out when you want multiple models to answer the same request and combine the successful outputs.

pack:
  name: config-providers-providers-25
  version: 1.0.0
  enabled: true
providers:
  pipelines:
  - name: draft-and-review
    aliases: [reviewed-answer]
    mode: sequence
    steps:
    - name: draft
      target: writer
    - name: review
      target: reviewer
      instruction: Review and improve the draft.
      input_mode: replace
      inject_as: user
  targets:
  - id: writer
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: KEEPTRUSTS_OPENAI_API_KEY
  - id: reviewer
    provider: azure
    provider_type: azure-openai
    format: openai
    model: your-azure-openai-model
    base_url: https://replace-with-resource-name.openai.azure.com
    secret_key_ref:
      env: KEEPTRUSTS_AZURE_OPENAI_API_KEY
    azure_api_version: your-supported-api-version
    azure_deployment: your-azure-deployment
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Pipeline rules:

  • Pipelines are invoked through the normal request model field, just like model_groups.
  • Each step must reference a target with an explicit providers.targets[].model. If you want to orchestrate different models from the same provider, declare separate targets for them.
  • Sequence steps after the first can append the previous step output to the original request (append) or replace the original request with it (replace).
  • fan_out runs every step against the original request and then either concatenates successful outputs in step order or returns the first successful output in config order.
  • Pipelines currently support POST /v1/chat/completions and POST /v1/responses. They are not supported for embeddings or moderations.
  • Direct request pinning with X-Keeptrusts-Provider or X-Keeptrusts-Model is intentionally rejected for pipeline models.

Circuit breaker​

Automatically remove unhealthy targets from the pool.

providers:
  circuit_breaker:
    enabled: true
    consecutive_failure_threshold: 5
    cooldown_seconds: 30
    half_open_successes: 1

Retry policy​

Configure retry behavior with backoff.

providers:
  retry_policy:
    max_retries: 3
    per_trigger:
      rate_limit: 5
    backoff:
      strategy: "exponential"        # exponential | fixed
      base_ms: 500

These retries happen against the same provider before the outer provider loop can advance. Current dispatch reads per-trigger limits for rate_limit and server_error; transport failures and response-body read failures use max_retries. Other parsed per_trigger entries are not consulted.

Provider-scope rate-limit metadata​

providers.rate_limits is parsed, but the current request dispatcher does not read or enforce its per-key, per-user, per-team, global, or parallel-request values. Do not use this block as a runtime rate-limit control.

Traffic mirroring​

Send sampled shadow traffic to a secondary provider without affecting the primary response. The current runtime discards the shadow response body and records only status/event metadata, so it does not perform response-content comparison. Both runtime_routing.shadow_routing.enabled and providers.traffic_mirror.enabled must be true.

Runtime-effective shadow routing; runtime_routing is not yet in the lint schema
runtime_routing:
  shadow_routing:
    enabled: true
    capture_mode: full_payload
providers:
  traffic_mirror:
    enabled: true
    mirror_target: openai-shadow
    sample_rate: 0.1
  targets:
  - id: openai-primary
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: KEEPTRUSTS_OPENAI_API_KEY
  - id: openai-shadow
    provider: openai
    model: your-shadow-openai-model
    secret_key_ref:
      env: KEEPTRUSTS_OPENAI_SHADOW_API_KEY

Both enable flags and capture_mode: full_payload are required before the runtime calls the mirror provider. The default metadata_only mode records a shadow-routing event without sending request content upstream; privacy- restricted mode also forces metadata-only behavior. Dispatch reads mirror_target and sample_rate, but not the parsed log_mirror_response or timeout_ms fields. The current lint schema does not admit top-level runtime_routing, so the combined runtime-effective snippet is shown as text rather than as a lintable YAML example.

A/B testing metadata​

providers.ab_test is parsed, but no dispatch path reads it to split or stick traffic between variants. Do not rely on it for an active experiment.

Logging controls​

providers:
  logging:
    redact_message_bodies: true

redact_message_bodies is applied to gateway payload logging. The parsed redact_api_keys field has no runtime read and must not be treated as a credential-redaction control.

Recognized provider aliases and authentication​

The gateway ships with 100+ provider aliases. The alias can infer base_url, api_key_header, and path_template; it does not infer credential material. secret_key_ref is optional in the schema, but ordinary auth-required targets need a resolvable env or store reference to become active. Auth-optional and self-credentialed targets follow their provider-specific guidance.

Alias recognition, a provider profile, or a wire-format translator proves only that the gateway can load that configuration path. It does not prove a complete production request contract. Check the owning guide in the Integrations section for current runtime status and required end-to-end verification before rollout.

ProviderAliasesAuth
OpenAIopenaiKEEPTRUSTS_OPENAI_API_KEY
Anthropicanthropic, claudeKEEPTRUSTS_ANTHROPIC_API_KEY
Google AI Studiogoogle, geminiKEEPTRUSTS_GEMINI_API_KEY
Google Vertex AIgoogle-vertex, vertexService account / OAuth2
Azure OpenAIazure-openai, azureKEEPTRUSTS_AZURE_OPENAI_API_KEY
AWS Bedrockaws-bedrock, bedrockSigV4 with AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY and optional AWS_SESSION_TOKEN
GroqgroqKEEPTRUSTS_GROQ_API_KEY
MistralmistralKEEPTRUSTS_MISTRAL_API_KEY
DeepSeekdeepseekKEEPTRUSTS_DEEPSEEK_API_KEY
Together AItogetherai, togetherKEEPTRUSTS_TOGETHER_API_KEY
Fireworks AIfireworks, fireworks-aiKEEPTRUSTS_FIREWORKS_API_KEY
CerebrascerebrasKEEPTRUSTS_CEREBRAS_API_KEY
PerplexityperplexityKEEPTRUSTS_PERPLEXITY_API_KEY
OpenRouteropenrouterKEEPTRUSTS_OPENROUTER_API_KEY
GitHub Modelsgithub, github-modelsKEEPTRUSTS_GITHUB_TOKEN
CoherecohereKEEPTRUSTS_COHERE_API_KEY
HuggingFacehuggingface, hfKEEPTRUSTS_HF_API_TOKEN
ReplicatereplicateKEEPTRUSTS_REPLICATE_API_TOKEN
DatabricksdatabricksOAuth2 / KEEPTRUSTS_DATABRICKS_TOKEN
Snowflake Cortexsnowflake-cortexOAuth2
Cloudflare AIcloudflare-aiKEEPTRUSTS_CF_API_TOKEN
Cloudflare AI Gatewaycloudflare-gatewayUnderlying provider credential; the OpenAI BYOK example uses KEEPTRUSTS_OPENAI_API_KEY
Alibaba / Qwenalibaba, qwen, dashscopeKEEPTRUSTS_DASHSCOPE_API_KEY
SambaNovasambanovaKEEPTRUSTS_SAMBANOVA_API_KEY
xAI (Grok)xai, grokKEEPTRUSTS_XAI_API_KEY
Vercel AI Gatewayvercel, vercel-aiKEEPTRUSTS_VERCEL_AI_GATEWAY_API_KEY

Wire format translation​

The gateway contains translators for the wire formats below and can infer a format from recognized provider profiles. Set format explicitly when the owning integration guide requires it. Translator presence alone does not prove that every required provider field or request shape is production-ready.

FormatProviders
openaiOpenAI, Azure, Groq, Mistral, Together, Fireworks, most OpenAI-compatible
anthropicAnthropic, Claude
cohereCohere
huggingfaceHuggingFace Inference Endpoints
replicateReplicate
watsonxIBM watsonx
google-geminiGoogle Gemini (AI Studio and Vertex)

Complete multi-provider example​

pack:
  name: openai-and-azure-routing
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: openai-primary
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: KEEPTRUSTS_OPENAI_API_KEY
  - id: azure-openai-fallback
    provider: azure-openai
    model: your-azure-openai-model
    base_url: https://replace-with-resource-name.openai.azure.com
    secret_key_ref:
      env: KEEPTRUSTS_AZURE_OPENAI_API_KEY
    azure_deployment: your-azure-deployment
    azure_api_version: your-supported-api-version
  routing:
    strategy: ordered
  circuit_breaker:
    enabled: true
    consecutive_failure_threshold: 5
    cooldown_seconds: 30
  retry_policy:
    max_retries: 3
    backoff:
      strategy: exponential
      base_ms: 500
policies:
  chain:
  - prompt-injection
  - pii-detector
  - quality-scorer
policy:
  prompt-injection:
    response:
      action: block
  pii-detector:
    action: redact
  quality-scorer:
    thresholds:
      min_aggregate: 0.8

In this example, retry_policy controls same-target retries. After that retry budget is exhausted, the normal provider-attempt loop can consider the next eligible target using its default trigger list; no explicit fallback block is needed.

Next steps​