Skip to main content

Quality Assertions Configuration

Use quality-scorer when you want the gateway to score model output against explicit checks before the response is accepted.

What the schema expects​

Each assertion is an object with:

  • type — the assertion kind.
  • Optional gate fields such as threshold, weight, mode, and severity.
  • config — the type-specific options.

The most common source of invalid configs is putting type-specific fields such as value or rubric at the top level of the assertion instead of under config.

Minimal full example​

pack:
  name: config-quality-assertions-example-1
  version: 1.0.0
  enabled: true
policies:
  chain:
    - quality-scorer
policy:
  quality-scorer:
    assertions:
      - type: contains
        config:
          value: disclaimer
      - type: llm-rubric
        threshold: 0.8
        config:
          rubric: Response must be factually accurate
    thresholds:
      min_aggregate: 0.8
    failure_action:
      action: block

Assertion object fields​

FieldTypeNotes
typestringRequired assertion kind.
namestringOptional human-readable label.
enabledbooleanDisable an assertion without deleting it.
thresholdnumberMinimum score for that assertion to pass.
weightnumberRelative weight in aggregate scoring.
configobjectType-specific options.
modestringenforce, audit, or shadow.
severitystringcritical, warning, or info.

Common assertion patterns​

Contains​

- type: contains
  name: has-disclaimer
  threshold: 1.0
  weight: 0.5
  mode: enforce
  severity: critical
  config:
    value: disclaimer

LLM rubric​

- type: llm-rubric
  threshold: 0.8
  config:
    rubric: Response must be accurate and directly answer the question

Word count​

- type: word-count
  config:
    min: 40

Moderation​

- type: moderation
  config:
    categories:
      - violence
      - hate

Multi-assertion quality gates​

Use several explicit assertions together when you want one quality gate to cover safety, grounding, and response structure.

pack:
  name: config-quality-assertions-example-52
  version: 1.0.0
  enabled: true
policies:
  chain:
    - quality-scorer
policy:
  quality-scorer:
    assertions:
      - type: moderation
        config:
          categories:
            - violence
            - hate
            - self-harm
      - type: is-refusal
        config:
          expected: false
      - type: context-faithfulness
        threshold: 0.8
      - type: contains
        config:
          value: source
      - type: llm-rubric
        threshold: 0.8
        config:
          rubric: Response must be helpful

Thresholds, failure action, and pass policy​

policy:
  quality-scorer:
    thresholds:
      min_aggregate: 0.8
      min_accuracy: 0.8
    pass_policy:
      strategy: weighted_average
      threshold: 0.8
    failure_action:
      action: block
      fallback_message: I cannot provide a sufficiently reliable answer.

Threshold keys activate their matching built-in scorer when the benchmarks block does not explicitly set that benchmark. For example, min_relevancy computes relevancy and includes it in the aggregate with a default weight of 1.0. min_coherence and min_completeness are also persisted in event quality scores when configured.

The schema accepts block, fallback, and retry, plus max_retries. The current runtime only special-cases fallback, which substitutes the configured fallback message and allows that response. retry and max_retries do not start a retry loop; when the scorer fails, retry currently ends as a blocked verdict just like block.

Judge-backed scoring​

Use judge when you want an additional model to attach a rubric-driven pass, warn, or fail result to quality details. The judge result is audit metadata; it does not by itself change the main quality-gate verdict.

policy:
  quality-scorer:
    judge:
      enabled: true
      endpoint: https://api.openai.com/v1/chat/completions
      model: gpt-5.4-mini
      secret_key_ref:
        env: KEEPTRUSTS_OPENAI_API_KEY
      threshold: 0.8
      warn_threshold: 0.6
      timeout_ms: 5000

The schema accepts regression_monitoring, but the current gateway scorer does not consume it. Use judge.sampling_rate to control how often the implemented judge call runs.

Complete quality gate example​

pack:
  name: quality-enforced
  version: 1.0.0
  enabled: true
policies:
  chain:
    - prompt-injection
    - quality-scorer
    - audit-logger
policy:
  prompt-injection:
    response:
      action: block
  quality-scorer:
    assertions:
      - type: contains
        config:
          value: source
      - type: word-count
        config:
          min: 40
      - type: llm-rubric
        threshold: 0.8
        config:
          rubric: Response must be accurate, helpful, and well structured
    thresholds:
      min_aggregate: 0.8
      min_accuracy: 0.8
    pass_policy:
      strategy: weighted_average
      threshold: 0.8
    failure_action:
      action: block
      fallback_message: Quality check failed.
    judge:
      enabled: true
      endpoint: https://api.openai.com/v1/chat/completions
      model: gpt-5.4-mini
      secret_key_ref:
        env: KEEPTRUSTS_OPENAI_API_KEY
      threshold: 0.8
      warn_threshold: 0.6
    benchmarks:
      ragas_faithfulness: true
      ragas_relevancy: true
  audit-logger: {}

Best practices​

  • Keep deterministic assertions simple and easy to explain in review.
  • Add judge-backed scoring only where rubric interpretation is worth the extra latency and cost.
  • Use audit or shadow mode when rolling out new assertions.
  • Prefer a small number of explicit assertions over an oversized quality gate that is hard to debug.

Next steps​