Skip to main content

Third-Party AI Risk Assessment: Evaluating Provider Data Handling

Third-Party AI Risk Assessment: Evaluating Provider Data Handling

Third-party AI risk work goes off track when the entire assessment lives in a spreadsheet. Providers get marked green or yellow, legal teams collect contract language, and engineering still routes traffic however the config happened to be written months ago. Keeptrusts closes that gap by letting you declare provider-handling metadata on each target and enforce it with data-routing-policy. That means vendor due diligence can move from a procurement artifact to a runtime decision.

Use this page when​

  • You are comparing multiple AI providers and need a risk model grounded in actual routing controls.
  • You want provider-selection decisions to reflect retention, training, locality, and sanitization guarantees.
  • You need a practical bridge between vendor review and gateway enforcement.

Primary audience​

  • Primary: Technical Leaders
  • Secondary: Technical Engineers, AI Agents

The problem​

Most provider assessments stop too early.

Teams ask whether a provider offers zero retention, whether traffic is used for training, and whether local-only processing is available. Those are the right questions, but they do not change runtime behavior on their own. If the gateway still treats every provider target as equally eligible, a strong assessment result is just documentation.

There is also a maintenance problem. Provider guarantees differ by endpoint, contract, region, and deployment style. If those assumptions are not captured close to the routing layer, they drift. The security team may believe a target is zero-retention while the platform team quietly uses it as a general fallback.

Finally, risk assessment often forgets secret readiness. A target can be contractually acceptable and still be unusable because the credential is missing or the pilot environment references the wrong secret name. In practice, that kind of operational gap is one of the fastest ways teams fall back to a less-governed provider.

The solution​

Keeptrusts gives you an assessment model that maps directly to runtime fields.

Use Providers Configuration to declare each target and its data_policy metadata. That is where you record whether the target claims zero retention, training opt-out, in-memory-only processing, sanitized handling, tokenized-input support, internet egress, and local-only execution.

Use Data Routing Policy to enforce the minimum acceptable handling guarantees. This is the part that turns due diligence into fail-closed behavior.

Use Secret Management and provider preflight checks so an approved target is also operationally ready. A missing secret should not force teams onto a lower-trust fallback just because the safer path was never validated.

Finally, use Zero Retention Endpoints as a reference surface when you need a documented starting point for the stricter end of the provider pool.

Implementation​

This configuration scores three provider targets differently and then enforces the strictest path for regulated traffic.

pack:
  name: provider-risk-assessment
  version: "1.0.0"
  enabled: true

providers:
  routing:
    strategy: ordered
  targets:
    - id: openai-zdr
      provider: openai
      model: gpt-5.4-mini-mini
      required: true
      secret_key_ref:
        store: OPENAI_ZDR_KEY
      data_policy:
        zero_data_retention: true
        training_opt_out: true
        retention_days: 0
        in_memory_only: true
        sanitized: true
        accepts_tokenized_input: true
        allow_internet_egress: false
        local_only_processing: true

    - id: anthropic-standard
      provider: anthropic
      model: claude-sonnet-4-20250514
      required: false
      secret_key_ref:
        env: ANTHROPIC_API_KEY
      data_policy:
        zero_data_retention: false
        training_opt_out: true
        retention_days: 30
        in_memory_only: false
        sanitized: false
        accepts_tokenized_input: false
        allow_internet_egress: true
        local_only_processing: false

    - id: local-vllm
      provider: vllm
      model: meta-llama/Meta-Llama-3.1-8B-Instruct
      required: false
      secret_key_ref:
        env: VLLM_GATEWAY_TOKEN
      data_policy:
        zero_data_retention: true
        training_opt_out: true
        retention_days: 0
        in_memory_only: true
        sanitized: true
        accepts_tokenized_input: true
        allow_internet_egress: false
        local_only_processing: true

policies:
  chain:
    - data-routing-policy
    - audit-logger

policy:
  data-routing-policy:
    require_zero_data_retention: true
    require_no_training: true
    max_retention_days: 0
    require_in_memory_only: true
    sanitize_before_provider: true
    tokenize_sensitive_fields: true
    allow_internet_egress: false
    local_only_processing: true
    on_no_compliant_provider: block
    log_provider_selection: true

  audit-logger: {}

This is an assessment model disguised as configuration.

If a target cannot honestly declare the required data_policy fields, it should not stay eligible for the regulated route. If a target is acceptable only for lower-risk traffic, keep it in the config, but make sure the stricter path filters it out before routing begins.

Operational readiness matters just as much as contract language. Validate target readiness with the CLI before you depend on it:

kt secrets status --config policy-config.yaml
kt gateway check --config policy-config.yaml --verbose
kt events tail --since 24h --json

kt secrets status tells you whether the gateway can actually resolve the credentials for the provider set you approved. kt gateway check shows whether the configuration can start and which providers are active. kt events tail is useful after rollout because the event stream includes the provider and config version fields you need to confirm that the safer route is the route actually being used.

The key governance discipline is simple: do not let an unreviewed provider become an operational default just because it was easy to configure. The reviewed provider matrix must become the runtime provider matrix.

Results and impact​

Teams that use this model stop treating vendor review as a once-a-year questionnaire.

Provider risk becomes part of deployment quality. Missing metadata breaks the route. Missing secrets make the target visibly unavailable. Non-compliant targets are excluded before normal routing runs.

That creates a much healthier relationship between legal review, privacy review, and engineering. Everyone is now looking at the same set of facts in the same configuration surface.

Key takeaways​

  • Map provider review criteria directly to data_policy fields instead of keeping them in a separate spreadsheet only.
  • Use Data Routing Policy to turn provider risk decisions into fail-closed routing.
  • Validate credential readiness with Secret Management before relying on a provider target.
  • Keep lower-trust providers available only where they are intentionally allowed, not as accidental fallbacks.
  • Review and routing should share one source of truth.

Next steps​