Multi-Provider Gateway Setup

Configure your Keeptrusts gateway to route requests across multiple AI providers with automatic fallback, task-aware routing, and cost optimization.

Quick Start

Add providers to your policy-config.yaml:

providers:
  targets:
    - id: openai-default
      provider: openai
      model: gpt-5.4-mini
      required: false
      secret_key_ref:
        env: OPENAI_API_KEY

    - id: anthropic-default
      provider: anthropic
      model: claude-sonnet-4-20250514
      required: false
      secret_key_ref:
        env: ANTHROPIC_API_KEY

    - id: mistral-default
      provider: mistral
      model: mistral-large-latest
      required: false
      secret_key_ref:
        env: MISTRAL_API_KEY

    - id: gemini-default
      provider: gemini
      model: gemini-2.0-flash
      required: false
      secret_key_ref:
        env: GOOGLE_AI_API_KEY

    - id: github-models
      provider: github
      model: gpt-5.4-mini
      required: false
      secret_key_ref:
        env: GITHUB_TOKEN

  routing:
    strategy: ordered
    order: ["openai-default", "anthropic-default", "gemini-default", "mistral-default", "github-models"]
    allow_fallbacks: true

Provider Required vs Optional

By default, providers are optional (required: false). If credentials are missing, the provider is excluded from routing silently.

Set required: true for strict mode — the gateway will refuse to start if the credential is unresolved:

- id: openai-primary
  provider: openai
  model: gpt-5.4-mini
  required: true
  secret_key_ref:
    env: OPENAI_API_KEY

Secret Resolution Methods

Environment Variables (recommended for CI/CD)

secret_key_ref:
  env: OPENAI_API_KEY

Centralized Store (recommended for hosted gateways)

secret_key_ref:
  store: openai-production-key

macOS Keychain (local development only)

secret_key_ref:
  keychain: openai-api-key

Note: Keychain refs are rejected for hosted gateways.

Routing Strategies

Strategy	Description	Best For
ordered	Try providers in configured order	Predictable behavior
task_aware	Route by request task type	Mixed workloads
lowest_latency	Prefer fastest provider	Real-time apps
highest_throughput	Prefer highest tokens/sec	Batch processing
round_robin	Distribute evenly	Load balancing
weighted_round_robin	Distribute by weight	Cost optimization

Task-Aware Routing

providers:
  routing:
    strategy: task_aware
    task_profiles:
      - task_type: code_generation
        preferred_providers: ["anthropic-default", "openai-default"]
      - task_type: multilingual
        preferred_providers: ["gemini-default", "mistral-default"]
      - task_type: structured_output
        preferred_providers: ["openai-default"]

Budget Policy

providers:
  routing:
    budget_policy:
      soft_limit_action: downgrade  # Switch to cheaper model
      hard_limit_action: reject     # Reject request

Latency Optimization

providers:
  routing:
    latency_optimization:
      streaming_preferred_ttft_ms: 200
      batch_preferred_throughput_tps: 50.0

Validating Your Configuration

kt gateway check --config policy-config.yaml
kt secrets status --config policy-config.yaml