Compliance Testing for AI Systems

Regulatory compliance for AI systems requires demonstrable proof that governance controls are active, effective, and auditable. Keeptrusts policies map directly to regulatory requirements, and the event audit trail provides the evidence that auditors and regulators need.

Use this page when

You need to verify that governance policies map to and satisfy specific regulatory requirements
You are building automated compliance test suites that exercise each regulation-mapped policy
You want to generate structured audit evidence and verify audit trail integrity programmatically

Primary audience

Primary: Technical Engineers
Secondary: AI Agents, Technical Leaders

Policy-to-Regulation Mapping

Each governance policy should trace back to one or more regulatory requirements. Maintaining this mapping ensures complete coverage and simplifies audit preparation.

Mapping Structure

# compliance-mapping.yaml — policy to regulation traceability
mappings:
  - regulation: EU AI Act
    article: Article 14 — Human Oversight
    policies:
      - name: escalate-high-risk
        type: escalation
        description: "High-risk decisions escalated to human reviewer"
      - name: human-review-required
        type: topic_control
        description: "Employment and credit decisions require human approval"

  - regulation: HIPAA
    section: "164.530(j) — Record Retention"
    policies:
      - name: audit-log-retention
        type: retention
        description: "Audit events retained for 6 years minimum"
      - name: phi-redaction
        type: dlp
        description: "Protected health information redacted from AI outputs"

  - regulation: SOC 2 — CC6.1
    section: "Logical Access Controls"
    policies:
      - name: api-key-authentication
        type: auth
        description: "All gateway access requires valid API key"
      - name: rate-limiting
        type: rate_limit
        description: "Request rate limits prevent abuse"

Verifying Coverage

Build a script to check that every regulatory requirement has at least one policy:

#!/bin/bash
# check-compliance-coverage.sh

MAPPING="compliance-mapping.yaml"
POLICY_CONFIG="policy-config.yaml"

echo "=== Compliance Coverage Check ==="

# Extract all referenced policy names from the mapping
MAPPED_POLICIES=$(grep "^\s*- name:" "$MAPPING" | sed 's/.*name: //' | sort)

# Extract all policy names from the actual config
ACTIVE_POLICIES=$(grep "^\s*- name:" "$POLICY_CONFIG" | sed 's/.*name: //' | sort)

# Find mapped policies that are missing from the active config
MISSING=$(comm -23 <(echo "$MAPPED_POLICIES") <(echo "$ACTIVE_POLICIES"))

if [ -z "$MISSING" ]; then
  echo "PASS: All compliance-mapped policies are present in the active config"
else
  echo "FAIL: Missing policies referenced in compliance mapping:"
  echo "$MISSING"
  exit 1
fi

Automated Compliance Checks

Automate the verification that each compliance-relevant policy is correctly configured and enforced.

Policy Enforcement Verification

For each regulation, define test cases that exercise the corresponding policy:

[
  {
    "id": "hipaa-phi-redaction",
    "regulation": "HIPAA 164.530",
    "policy": "phi-redaction",
    "request": {
      "model": "gpt-4o",
      "messages": [{"role": "user", "content": "Patient John Smith, DOB 01/15/1980, MRN 12345678"}]
    },
    "expected": {
      "decision": "redact",
      "body_contains": "[REDACTED"
    }
  },
  {
    "id": "eu-ai-act-escalation",
    "regulation": "EU AI Act Art. 14",
    "policy": "escalate-high-risk",
    "request": {
      "model": "gpt-4o",
      "messages": [{"role": "user", "content": "Evaluate this candidate for the senior engineer role"}]
    },
    "expected": {
      "decision": "escalate"
    }
  },
  {
    "id": "soc2-auth-required",
    "regulation": "SOC 2 CC6.1",
    "policy": "api-key-authentication",
    "request_without_auth": true,
    "expected": {
      "http_code": 401
    }
  }
]

Running Compliance Test Suite

#!/bin/bash
# run-compliance-tests.sh

TESTS_FILE="compliance-tests.json"
GATEWAY="http://localhost:41002"
FAILURES=0

echo "=== Compliance Test Suite ==="

jq -c '.[]' "$TESTS_FILE" | while read -r test_case; do
  ID=$(echo "$test_case" | jq -r '.id')
  REGULATION=$(echo "$test_case" | jq -r '.regulation')
  EXPECTED_DECISION=$(echo "$test_case" | jq -r '.expected.decision // empty')
  EXPECTED_HTTP=$(echo "$test_case" | jq -r '.expected.http_code // empty')

  REQUEST=$(echo "$test_case" | jq -c '.request')

  RESPONSE=$(curl -s -w "\n%{http_code}" "$GATEWAY/v1/chat/completions" \
    -H "Content-Type: application/json" \
    -d "$REQUEST")

  HTTP_CODE=$(echo "$RESPONSE" | tail -1)

  if [ -n "$EXPECTED_HTTP" ] && [ "$HTTP_CODE" != "$EXPECTED_HTTP" ]; then
    echo "FAIL [$ID] ($REGULATION): Expected HTTP $EXPECTED_HTTP, got $HTTP_CODE"
    FAILURES=$((FAILURES + 1))
  elif [ -n "$EXPECTED_DECISION" ]; then
    EVENT=$(kt events list --last 1 --format json)
    DECISION=$(echo "$EVENT" | jq -r '.[0].decision')
    if [ "$DECISION" != "$EXPECTED_DECISION" ]; then
      echo "FAIL [$ID] ($REGULATION): Expected decision '$EXPECTED_DECISION', got '$DECISION'"
      FAILURES=$((FAILURES + 1))
    else
      echo "PASS [$ID] ($REGULATION)"
    fi
  else
    echo "PASS [$ID] ($REGULATION)"
  fi
done

if [ "$FAILURES" -gt 0 ]; then
  echo ""
  echo "=== $FAILURES compliance test(s) FAILED ==="
  exit 1
fi
echo "=== All compliance tests passed ==="

Evidence Generation

Auditors require concrete evidence that controls were active during the audit period. Keeptrusts events provide this evidence.

Exporting Audit Evidence

# Export events for the audit period
kt events list \
  --from "2025-01-01T00:00:00Z" \
  --to "2025-03-31T23:59:59Z" \
  --format json \
  --output audit-evidence-q1-2025.json

# Generate summary statistics
jq '{
  total_events: length,
  blocked: [.[] | select(.decision == "block")] | length,
  redacted: [.[] | select(.decision == "redact")] | length,
  escalated: [.[] | select(.decision == "escalate")] | length,
  allowed: [.[] | select(.decision == "allow")] | length
}' audit-evidence-q1-2025.json

Evidence Report Structure

Organize evidence by regulation:

#!/bin/bash
# generate-compliance-report.sh

EVENTS_FILE="audit-evidence-q1-2025.json"
REPORT_DIR="compliance-report"
mkdir -p "$REPORT_DIR"

# HIPAA evidence — PHI redaction events
jq '[.[] | select(.policies_applied[] | .name == "phi-redaction")]' \
  "$EVENTS_FILE" > "$REPORT_DIR/hipaa-phi-redaction-evidence.json"

# EU AI Act evidence — escalation events
jq '[.[] | select(.policies_applied[] | .name == "escalate-high-risk")]' \
  "$EVENTS_FILE" > "$REPORT_DIR/eu-ai-act-escalation-evidence.json"

# SOC 2 evidence — auth failures
jq '[.[] | select(.http_code == 401)]' \
  "$EVENTS_FILE" > "$REPORT_DIR/soc2-auth-evidence.json"

echo "Compliance report generated in $REPORT_DIR/"
ls -la "$REPORT_DIR/"

Audit Trail Verification

The audit trail itself must be trustworthy. Verify its integrity and completeness.

Completeness Check

Every gateway request should produce an event. Verify no events are missing:

# Compare gateway access logs with event count
GATEWAY_REQUESTS=$(wc -l < gateway-access.log)
EVENT_COUNT=$(kt events list --from "24h" --format json | jq 'length')

echo "Gateway requests: $GATEWAY_REQUESTS"
echo "Recorded events: $EVENT_COUNT"

if [ "$GATEWAY_REQUESTS" -ne "$EVENT_COUNT" ]; then
  echo "WARN: Event count mismatch — potential event loss"
fi

Immutability Verification

Events should not be modified after creation. Verify that event timestamps and content are consistent:

# Export events and verify ordering
kt events list --from "24h" --format json | \
  jq '[.[] | .created_at] | sort == .' 
# Should return true — events are in chronological order

# Verify no gaps in event sequence
kt events list --from "24h" --format json | \
  jq '[.[] | .id] | length as $count | 
      if $count > 0 then "Events present: \($count)" 
      else "WARNING: No events found" end'

Compliance Testing in CI

# .github/workflows/compliance-gate.yml
jobs:
  compliance:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Check compliance coverage
        run: ./scripts/check-compliance-coverage.sh

      - name: Validate policy config
        run: kt policy lint --file policy-config.yaml

      - name: Start test gateway
        run: |
          kt gateway run --policy-config policy-config.yaml --port 41002 &
          sleep 3

      - name: Run compliance tests
        run: ./scripts/run-compliance-tests.sh

      - name: Upload evidence
        uses: actions/upload-artifact@v4
        with:
          name: compliance-evidence
          path: compliance-report/

Key Takeaways

Map every governance policy to its regulatory requirement for traceability
Automate compliance tests that verify each policy is correctly enforced
Use kt events list to export audit evidence for specific time periods
Generate structured compliance reports organized by regulation
Verify audit trail completeness and immutability as part of regular testing
Integrate compliance gates into CI to prevent non-compliant deployments

For AI systems

Canonical terms: compliance mapping, policy-to-regulation traceability, audit evidence, audit trail integrity, compliance gate, EU AI Act, HIPAA, SOC 2
Config file: compliance-mapping.yaml maps regulations → policies
CLI commands: kt events list, kt events export, kt config export
Key patterns: each regulation article maps to one or more policy names in policy-config.yaml
Related pages: Compliance Infrastructure, Regression Testing, Test Automation

For engineers

Create a compliance-mapping.yaml that maps each regulatory article to specific policy names in your active config
Build check-compliance-coverage.sh that verifies every mapped policy exists in the active policy-config.yaml
For each mapped policy, write a test case that sends a prompt triggering the policy and asserts the expected decision (block, redact, escalate)
Export audit evidence with kt events export --filter 'result=block OR result=escalate' --from <date> --to <date>
Verify audit trail integrity: confirm events are immutable, timestamps are sequential, and no gaps exist
Integrate compliance gates into CI so non-compliant deployments fail before reaching production

For leaders

Policy-to-regulation mapping provides auditor-ready traceability for SOC 2, HIPAA, EU AI Act, and GDPR
Automated compliance tests replace manual audit preparation — evidence is always current
CI compliance gates prevent deploying policy configs that break regulatory coverage
Structured compliance reports organized by regulation simplify board reporting
Continuous verification (not just point-in-time audits) satisfies SOC 2 Type II continuous monitoring requirements

Next steps

Set up Compliance Infrastructure for retention, export, and SOC 2 control mapping
Use Regression Testing to detect when policy changes break compliance coverage
Build end-to-end pipelines with Test Automation for continuous compliance verification

Use this page when​

Primary audience​

Policy-to-Regulation Mapping​

Mapping Structure​

Verifying Coverage​

Automated Compliance Checks​

Policy Enforcement Verification​

Running Compliance Test Suite​

Evidence Generation​

Exporting Audit Evidence​

Evidence Report Structure​

Audit Trail Verification​

Completeness Check​

Immutability Verification​

Compliance Testing in CI​

Key Takeaways​

For AI systems​

For engineers​

For leaders​

Next steps​