Skip to main content

Hugging Face Inference

Connect Keeptrusts to a Hugging Face inference endpoint with native model-path routing. Applications call the local Keeptrusts endpoint; the gateway owns the upstream credential and applies the configured policy chain.

Use this page when​

  • You already have access to Hugging Face Inference and need to route it through Keeptrusts.
  • You want one explicit provider target that can be linted and reviewed before rollout.
  • You need a stable integration contract without copying mutable prices, context limits, or retirement dates into your config.

Prerequisites​

  • Install the kt CLI.
  • Obtain the upstream credential and an enabled model or endpoint from Hugging Face Inference.
  • Obtain a Keeptrusts runtime API token for the gateway and a separate gateway key, access key, or personal API token for client requests. --agent uses only the runtime token.

Configure the provider​

Replace the replace-with-... values before starting the gateway. The example uses the current huggingface runtime contract.

pack:
  name: huggingface-integration
  version: 1.0.0
  enabled: true
policies:
  chain:
    - prompt-injection
    - pii-detector
    - audit-logger
providers:
  targets:
    - id: huggingface-primary
      provider: huggingface
      provider_type: huggingface
      format: huggingface
      model: "replace-with-organization/model"
      base_url: https://router.huggingface.co/hf-inference
      secret_key_ref:
        env: KEEPTRUSTS_HF_API_TOKEN

The provider credential is resolved inside the gateway process. It is not the credential that client applications send to Keeptrusts.

Start and verify​

export KEEPTRUSTS_API_TOKEN="replace-with-keeptrusts-api-token"
export KEEPTRUSTS_HF_API_TOKEN="replace-with-upstream-credential"

kt policy lint --file policy-config.yaml
kt gateway run \
  --agent huggingface-integration \
  --listen 127.0.0.1:41002 \
  --policy-config policy-config.yaml

KEEPTRUSTS_API_TOKEN authenticates the gateway runtime and control-plane synchronization. Do not reuse it as the client credential.

In another terminal, export the separate client token, confirm the gateway is healthy, and send one request:

export KEEPTRUSTS_CLIENT_TOKEN="replace-with-separate-client-token"

curl -fsS http://127.0.0.1:41002/healthz
curl -fsS http://127.0.0.1:41002/v1/chat/completions \
  -H "Authorization: Bearer ${KEEPTRUSTS_CLIENT_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"model":"replace-with-organization/model","messages":[{"role":"user","content":"Reply with one short sentence."}]}'

Use the same model identifier in the request and target unless you have configured an explicit multi-model route.

Current Keeptrusts contract​

SettingBehavior
providerhuggingface
Upstream requestHF Inference router endpoint at /hf-inference/models/{model}
Upstream authenticationBearer token from KEEPTRUSTS_HF_API_TOKEN
Client endpoint/v1/chat/completions on the Keeptrusts gateway

Use the exact repository model identifier accepted by HF Inference. The current adapter flattens chat messages into an inputs string, maps token and sampling options into parameters, and reads generated_text from the response. Select a text-generation model that accepts and returns that contract; other Hugging Face tasks and arbitrary custom-endpoint schemas are not supported by this target.

Model and production checks​

  • Verify the model ID, region, endpoint availability, and account permissions in the official provider surface before rollout.
  • Treat pricing, max_context_tokens, retention metadata, and certifications as operator declarations. Add them only after checking your current contract.
  • Validate streaming, tools, structured output, and other optional request features for the selected request family and model; provider-wide assumptions are unsafe.
  • Keep the upstream credential server-side. Bind production listeners only to the intended interface and protect them with your normal ingress controls.

Next steps​