Quality Scorer

The quality-scorer policy evaluates provider responses after generation and decides whether the output should pass, be replaced with a fallback message, or fail the quality gate.

Phase and verdicts

• Phase: output
• Verdicts: allow on pass, block on failure by default, and allow with replacement content when failure_action.action: fallback. Although the schema accepts retry, the current final-decision path treats it as a block.

Configuration

pack:
  name: quality-scorer-example-1
  version: "1.0.0"
  enabled: true

policies:
  chain:
    - quality-scorer

policy:
  quality-scorer:
    min_output_chars: 80
    min_sentences: 2
    benchmarks:
      ragas_faithfulness: true
      ragas_relevancy: true
      bleu_score: false
      nli_entailment: false
    assertions:
      - type: is-json
        name: response-is-json
        threshold: 1.0
        mode: enforce
        severity: critical
        config: {}
      - type: llm-rubric
        name: concise-and-grounded
        threshold: 0.7
        weight: 0.5
        mode: audit
        severity: warning
        config:
          rubric: Explain the answer clearly and stay grounded in the supplied context.
    thresholds:
      min_aggregate: 0.7
      min_faithfulness: 0.8
      min_relevancy: 0.75
    weights:
      faithfulness: 0.5
      relevancy: 0.5
    failure_action:
      action: fallback
      fallback_message: I cannot provide a sufficiently accurate response right now.
    pass_policy:
      strategy: weighted_average
      threshold: 0.7

Supported top-level fields

Field	Type	Default	Notes
industry	string	""	Accepted metadata. Runtime profile selection uses request.keeptrusts.industry, not this field.
min_output_chars	integer	0	Fails when extracted output text is shorter than this value.
min_sentences	integer	0	Fails when the sentence count is below this value.
mock_scoring	boolean	false	Accepted by the schema but not used by the current gateway scorer.
providers / targets	array	[]	Accepted provider descriptors, but the current gateway scorer does not use them for provider selection.
benchmarks	object	{}	Supports ragas_faithfulness, ragas_relevancy, bleu_score, nli_entailment, coherence, and completeness. Threshold keys infer the matching benchmark when it is not explicitly configured.
bleu_reference	string	—	Used for BLEU when no request context is available.
assertions	array	[]	Assertion list evaluated against the output.
thresholds	object	{}	Supports min_aggregate, min_faithfulness, min_relevancy, min_bleu, min_accuracy, min_coherence, and min_completeness.
weights	object	{}	Supports faithfulness, relevancy, bleu, accuracy, coherence, and completeness. Enabled benchmarks default to weight 1.0 when no explicit weight is configured.
failure_action	object	{ action: "block" }	fallback is the only value that replaces the provider response inline.
pass_policy	object	{ strategy: "all", quorum: 0.5, threshold: 0.5 }	Controls how failing assertions are aggregated.
industry_profiles	object	{}	Applied only when the request carries keeptrusts.industry.
judge	object	—	Optional extra judge call whose result is attached to the score details blob.
regression_monitoring	object	—	Accepted monitoring metadata; the current gateway scorer does not consume it.

Assertion fields

Each assertion entry may contain:

Field	Type	Default	Notes
type	string	—	Required. Unsupported types are reported as assertion errors.
name	string	assertion type	Optional display name.
enabled	boolean	true	Disabled assertions are skipped.
threshold	number	type-specific	Compared with the assertion score when present.
weight	number	1.0	Used by weighted aggregation.
mode	string	enforce	audit and shadow never block.
severity	string	critical	Only enforce + critical failures contribute to blocking.
config	object	{}	Assertion-specific parameters.

Current behavior

1. Extracts text from the upstream response body.
2. Applies min_output_chars and min_sentences.
3. Computes enabled benchmark metrics.
4. Evaluates assertions.
5. Applies pass_policy using only blocking assertion failures (mode: enforce and severity: critical).
6. Applies threshold checks for aggregate and benchmark scores.
7. Sets the failure action.

Important behavior notes

• Threshold-only configs still compute quality scores. For example, thresholds.min_relevancy enables relevancy scoring unless benchmarks.ragas_relevancy is explicitly configured.
• industry_profiles are keyed from request.keeptrusts.industry, not from a top-level industry field in the policy config.
• Current industry-profile overrides apply to min_aggregate, min_faithfulness, min_relevancy, min_bleu, min_accuracy, min_coherence, and min_completeness.
• failure_action.action: fallback replaces the returned assistant text. The accepted retry value and max_retries field do not start a retry loop; a failed result configured as retry is blocked. block also suppresses the failing output.
• llm-rubric assertions are scored by the quality engine itself; the optional judge block is a separate additional judge call whose result is attached to details.

Minimal valid example

policy:
  quality-scorer:
    min_output_chars: 20
    assertions:
      - type: contains
        name: mentions-policy
        threshold: 1.0
        mode: enforce
        severity: critical
        config:
          value: policy

Next steps

• Config Quality Assertions
• Policy Controls Catalog
• Human Oversight