Skip to main content

Policy Chain Ordering: How Execution Priority Affects Outcomes

In Keeptrusts, execution priority is the order of policies.chain, and that order is part of the policy meaning, not decoration. Earlier request-side controls see raw user input and can stop the model call entirely, while output-side controls such as Citation Verifier, Quality Scorer, Human Oversight, Financial Compliance, and Healthcare Compliance evaluate after generation. If a policy block exists but the kind is missing from policies.chain, it is configured but it does not execute.

Use this page when​

  • You are authoring policy-config.yaml and want the chain to express a clear enforcement strategy.
  • You are debugging why the gateway blocked, redacted, or escalated content earlier or later than expected.
  • You want to combine input controls, output quality checks, and human review without turning the chain into a random list.

Primary audience​

  • Primary: Technical Engineers and AI platform maintainers
  • Secondary: Technical Leaders reviewing rollout risk

Order is a control surface, not a formatting choice​

The easiest way to think about chain ordering is to ask a simple question for each policy: what is the best version of the content for this policy to inspect?

Some controls should inspect the raw request before anything leaves your environment. Prompt Injection Detection, PII Detector, and DLP Filter belong there because they protect the input boundary.

Other controls should inspect the generated answer before the caller sees it. That is where output policies belong. Citation Verifier checks grounding and citations. Quality Scorer evaluates response strength. Financial Compliance and Healthcare Compliance govern advice-like output. Human Oversight and flagged-review sit even later in the decision flow because they are review-oriented output controls.

When you mix those categories carelessly, the outcome changes.

A chain with clear intent​

This is a good example of a chain whose order communicates what should happen first:

pack:
  name: ordered-governance-stack
  version: 1.0.0
  enabled: true

policies:
  chain:
    - prompt-injection
    - pii-detector
    - dlp-filter
    - citation-verifier
    - quality-scorer
    - flagged-review
    - audit-logger

policy:
  prompt-injection:
    detection:
      attack_patterns:
        - "ignore.*previous.*instructions"
        - "reveal.*system.*prompt"

  pii-detector:
    action: redact

  dlp-filter:
    blocked_terms:
      - acquisition memo
      - board-only forecast
    action: block

  citation-verifier:
    require_sources: true
    require_source_match: true
    output_action:
      unverified_action: block

  quality-scorer:
    min_output_chars: 120
    thresholds:
      min_aggregate: 0.8

  flagged-review:
    mode: escalate

  audit-logger:
    retention_days: 365

The logic is readable from top to bottom.

  1. Stop obvious prompt attacks before the model sees them.
  2. Redact structured identifiers before sending the request upstream.
  3. Block organization-specific sensitive terms that should never leave the boundary.
  4. Check whether the generated answer is grounded.
  5. Check whether the grounded answer is also strong enough to ship.
  6. Escalate only the outputs that still need review after those automated gates.

That is much easier to reason about than a chain where review controls, request controls, and output controls are scattered without intent.

The most important ordering patterns​

Pattern 1: Protect the request boundary first​

Put request-side attackers and data-leak controls first.

Prompt Injection Detection is usually the first meaningful gate because it inspects the raw request boundary before the model call. If the input is trying to jailbreak the system, downstream sophistication does not matter.

PII Detector and DLP Filter usually follow because they handle data exposure. These controls make the most sense before the content is sent to an upstream provider.

That is also why pii-detector and dlp-filter are often paired. pii-detector handles built-in PII plus custom regexes and shared redaction, while dlp-filter handles the exact regex patterns and blocked terms that are specific to your organization.

Pattern 2: Put hard output blockers before human review​

If a response is clearly unacceptable, block it before it becomes reviewer work.

Citation Verifier can block unverified output when output_action.unverified_action: block. Financial Compliance and Healthcare Compliance can block configured substrings that should never appear in output.

Those are good first output gates because they remove the obviously non-compliant cases without burning reviewer time.

Pattern 3: Put quality and grounding before manual approval​

If a route ends in manual review, you still want the automated quality checks to run first. That is where Quality Scorer is useful. It can reject very short or weak responses, or replace them with a fallback message when failure_action.action: fallback is configured.

This matters because a reviewer queue should not become a trash collector for low-value model output. Human attention is expensive. Use the automated gates to narrow the queue to cases that actually require judgment.

Pattern 4: Do not assume a configured block executes​

A policy block under policy.<kind> is not enough. The kind must also appear in policies.chain.

This is a frequent source of false confidence. Teams add a detailed citation-verifier block or a strict human-oversight block, validate the YAML successfully, and then wonder why behavior did not change. The reason is simple: the block was configured but not executed.

Conditional chains let you keep one config without one chain for everyone​

Sometimes the right order depends on model, team, path, or gateway. That is what conditional chain entries are for.

Instead of maintaining separate configs, you can use object-form entries in policies.chain with when, stage, parallel, and targeting metadata.

For example, you might run Quality Scorer only on a high-value model route or add Healthcare Compliance only for clinical workloads. That keeps the global config readable while still applying stricter chains where they matter.

The important point is that conditional chains do not remove the need for thoughtful ordering. They just give you a way to express different orderings in one file.

How poor ordering creates bad outcomes​

The practical failures are usually operational, not theoretical.

If you put selective review too early, reviewers see cases that deterministic policies could have handled automatically.

If you put custom data classification too late, sensitive content may reach the upstream model before it is filtered.

If you omit a policy from policies.chain, you may think a control exists when it does not.

If you put all the output controls into the chain without deciding which ones are hard blockers and which ones are review signals, you end up with a system that is difficult to explain when something goes wrong.

That explanation burden matters. Governance controls are only useful when engineers, reviewers, and auditors can understand why a given response was blocked, redacted, or escalated.

A reliable authoring workflow​

Treat order as part of the design review, not the final formatting pass.

kt policy lint --file policy-config.yaml
kt policy test --json

Lint tells you whether the structure is legal. Testing tells you whether the ordering produces the verdicts you actually want. Those are different checks, and both matter.

When you review a chain, ask three questions:

  1. Which controls must see the raw request?
  2. Which controls should hard-block output before review?
  3. Which controls are there to escalate or log the small number of cases automation should not decide alone?

If the order does not answer those questions clearly, the chain is probably not ready.

Key takeaways​

  • policies.chain is the enforcement order, not a display list.
  • Request-side controls should usually come before provider calls.
  • Output-side hard blockers should usually come before human review.
  • A policy block that is missing from the chain does not run.
  • Conditional chains help you vary ordering by route or model, but they do not replace deliberate design.

Next steps​