Rate Limits Configuration

Keeptrusts supports global, per-IP, per-user, and token-consumption limits, plus byte limits for selected LLM proxy paths. The current distributed backend coordinates only the global request counter; the IP, user, and token counters remain local to each gateway process.

Quick reference

global_rate_limit:
  max_requests: 1000
  window_seconds: 60

ip_rate_limit:
  max_requests: 100
  window_seconds: 60

user_rate_limit:
  max_requests: 30
  window_seconds: 60
  header_names: ["x-user-id"]

token_rate_limit:
  max_tokens: 500000
  window_seconds: 3600
  scope: "global"

size_limits:
  max_body_bytes: 1048576
  max_response_bytes: 10485760

Global rate limit

A single counter shared by the public proxy handlers that invoke the gateway's request-state checks. Health, config, and administrative endpoints are outside this counter.

global_rate_limit:
  max_requests: 1000        # required
  window_seconds: 60         # required

Field	Type	Required	Default	Description
max_requests	integer	yes	—	Maximum requests per window
window_seconds	integer	yes	—	Fixed window size in seconds

Runtime behavior: Atomic counter with epoch-based fixed window reset. Returns HTTP 429 Too Many Requests with Retry-After header when exceeded.

Per-IP rate limit

Independent counters per client IP address.

ip_rate_limit:
  max_requests: 100          # required
  window_seconds: 60          # required
  trust_proxy_depth: 1        # optional, default: 0

Field	Type	Required	Default	Description
max_requests	integer	yes	—	Maximum requests per IP per window
window_seconds	integer	yes	—	Window size in seconds
trust_proxy_depth	integer	no	0	Number of X-Forwarded-For hops to trust. 0 uses the direct connection IP

Behind a reverse proxy

When running behind nginx or a load balancer, set trust_proxy_depth to the number of trusted proxies in the chain:

# Gateway behind one nginx reverse proxy
ip_rate_limit:
  max_requests: 50
  window_seconds: 60
  trust_proxy_depth: 1

Per-user rate limit

Independent counters per user identity extracted from request headers.

user_rate_limit:
  max_requests: 30           # required
  window_seconds: 60          # required
  header_names:               # optional
    - "x-user-id"
    - "x-consumer-id"

Field	Type	Required	Default	Description
max_requests	integer	yes	—	Maximum requests per user per window
window_seconds	integer	yes	—	Window size in seconds
header_names	string[]	no	["x-user-id"]	Headers to extract user identity from. First non-empty value wins

Requests without a non-empty matching header bypass the per-user counter; they are not grouped into an unknown bucket. Require or inject a trusted identity header at the gateway boundary when this limit is intended to cover every request.

Token rate limit

Sliding-window limit on token consumption reported by buffered Chat Completions responses.

token_rate_limit:
  max_tokens: 500000         # required
  window_seconds: 3600        # required
  scope: "global"             # optional: global | per_key | per_ip

Field	Type	Required	Default	Description
max_tokens	integer	yes	—	Maximum tokens consumed per window
window_seconds	integer	yes	—	Sliding window size in seconds
scope	string	no	"global"	Bucketing scope: global, per_key, or per_ip

Runtime behavior: Uses a 6-sub-window sliding window on buffered POST /v1/chat/completions traffic. Tokens are recorded after a successful upstream response when its JSON body contains usage.total_tokens; the next Chat Completions request is checked against the remaining budget and receives HTTP 429 when the budget is exhausted. Streaming responses and other request families are not checked or recorded by this limiter.

Scope examples

# Global: one bucket for all accounted buffered Chat Completions traffic
token_rate_limit:
  max_tokens: 1000000
  window_seconds: 3600
  scope: "global"

# Per API key: each key gets its own budget
token_rate_limit:
  max_tokens: 100000
  window_seconds: 3600
  scope: "per_key"

# Per IP: each direct socket peer gets its own budget
token_rate_limit:
  max_tokens: 50000
  window_seconds: 3600
  scope: "per_ip"

token_rate_limit.scope: per_ip uses the direct socket peer address. It does not use X-Forwarded-For or ip_rate_limit.trust_proxy_depth; behind a reverse proxy, callers can therefore share the proxy's token bucket.

Size limits

Byte-level limits for the current LLM proxy implementation.

size_limits:
  max_body_bytes: 1048576        # 1 MB request body
  max_header_bytes: 8192         # 8 KB headers
  max_url_bytes: 4096            # 4 KB URL
  max_response_bytes: 10485760   # 10 MB response

Field	Type	Required	Default	Description
max_body_bytes	integer	no	unlimited	Maximum request body size
max_header_bytes	integer	no	unlimited	Maximum total header size
max_url_bytes	integer	no	unlimited	Maximum URL length
max_response_bytes	integer	no	unlimited	Maximum response body size

The current request-side check runs on POST /v1/chat/completions and evaluates body, headers, then URL. Exceeding one of those request limits returns HTTP 413 Payload Too Large. Do not assume these three request-side fields protect other gateway routes.

max_response_bytes is enforced separately on the buffered Chat Completions and Responses paths. An oversized buffered upstream response returns HTTP 502 Bad Gateway with error code response_size_exceeded. For a streaming response, the HTTP headers have already been sent; the gateway emits a terminal SSE error with that code and aborts the stream.

Consumer-group request limits are configured under consumer_groups.groups[].rate_limit.max_requests. The schema also accepts max_tokens there, but live consumer enforcement does not read it. Size limits are top-level gateway limits.

Distributed rate limiting

By default, rate limit counters are per-process and in memory. A Redis or Valkey backend can coordinate the global_rate_limit request counter across gateway instances. It does not distribute the per-IP, per-user, or token counters.

Inline configuration

distributed_rate_limit:
  backend: "redis"
  url_env: "KEEPTRUSTS_LLM_CACHE_REDIS_URL"

Field	Type	Required	Default	Description
backend	string	yes	—	redis or valkey (both use Redis wire protocol)
url_env	string	no	"KEEPTRUSTS_LLM_CACHE_REDIS_URL"	Environment variable containing the connection URL

global_rate_limit must also be present because its max_requests and window_seconds define the single shared counter. Keep the backend in the top-level distributed_rate_limit block shown above. The runtime loader contains parser paths for nested distributed blocks, but the published configuration schema rejects those placements, and dispatch does not create distributed IP or token counters from them.

Build and connection behavior

Distributed rate limiting requires a CLI built with the distributed feature. If a backend is configured without that feature, gateway startup fails with an error instead of accepting traffic.

With the feature enabled, an empty url_env value or a backend initialization failure produces a warning and starts the gateway with local-only state. Check startup logs and runtime readiness before treating a multi-instance limit as shared.

Environment variable wiring

Set the environment variable named in url_env to the Redis or Valkey connection string before starting the gateway.

Complete rate limiting example

pack:
  name: "rate-limited-gateway"
  version: "1.0.0"
  enabled: true

# Global ceiling
global_rate_limit:
  max_requests: 5000
  window_seconds: 60

# Per-IP protection
ip_rate_limit:
  max_requests: 100
  window_seconds: 60
  trust_proxy_depth: 1

# Per-user fairness
user_rate_limit:
  max_requests: 30
  window_seconds: 60
  header_names: ["x-user-id", "x-consumer-id"]

# Token budget
token_rate_limit:
  max_tokens: 1000000
  window_seconds: 3600
  scope: "global"

# Request size protection
size_limits:
  max_body_bytes: 2097152      # 2 MB
  max_response_bytes: 20971520 # 20 MB

# Shared global request counter
distributed_rate_limit:
  backend: "valkey"
  url_env: "KEEPTRUSTS_LLM_CACHE_REDIS_URL"

providers:
  targets:
    - id: "openai-prod"
      provider: "openai"
      model: "your-openai-model"
      secret_key_ref:
        env: "KEEPTRUSTS_OPENAI_API_KEY"

policies:
  chain:
    - "audit-logger"

Replace your-openai-model with a model identifier available to your OpenAI account.

Next steps

• Routes and Consumer Groups — Per-group rate limit overrides
• Security and Network Configuration — IP allowlisting and CORS
• Providers Configuration — Provider-level timeout settings
• Declarative Config Reference — Full schema reference