Tutorial: Real-Time Event Tailing for Debugging

This tutorial shows you how to use kt events tail to stream gateway decision events in real time, filter events by policy and status, parse structured output with jq, and pipe events to external log aggregators.

Use this page when

• You need to stream gateway decision events in real time for debugging.
• You want to filter events by policy name, status (blocked/modified/flagged), or consumer group.
• You are piping structured event JSON to jq or an external log aggregator.
• You need to correlate a specific request with the policies that evaluated it.

Primary audience

• Primary: Platform engineers debugging policy behavior and request routing
• Secondary: SRE teams monitoring gateway health in real time; security analysts investigating blocked requests

Prerequisites

• kt CLI installed (first-run tutorial)
• A running Keeptrusts gateway with traffic flowing through it
• A running Keeptrusts API instance (for event storage)
• jq installed for JSON processing

Step 1: Basic Event Tailing

Start tailing events from your gateway in real time:

kt events tail

Expected output (events stream as requests flow through the gateway):

[2026-04-23T10:30:01Z] REQUEST id=evt_a1b2c3 policy=content-filter action=flag result=pass model=gpt-4o-mini tokens=142 latency=285ms
[2026-04-23T10:30:03Z] REQUEST id=evt_d4e5f6 policy=pii-redaction action=redact result=modified model=gpt-4o-mini tokens=98 latency=312ms
[2026-04-23T10:30:05Z] BLOCKED id=evt_g7h8i9 policy=injection-defense action=block result=blocked model=gpt-4o-mini latency=15ms

Press Ctrl+C to stop tailing.

Step 2: View Recent Events

Instead of streaming, fetch the most recent events:

# Last 10 events
kt events tail --last 10

# Last 50 events
kt events tail --last 50

Step 3: Filter by Policy

Show only events from a specific policy:

kt events tail --policy pii-redaction

Output shows only PII redaction decisions:

[2026-04-23T10:30:03Z] REQUEST id=evt_d4e5f6 policy=pii-redaction action=redact result=modified model=gpt-4o-mini tokens=98 latency=312ms
[2026-04-23T10:30:15Z] REQUEST id=evt_j1k2l3 policy=pii-redaction action=redact result=pass model=gpt-4o-mini tokens=65 latency=290ms

Step 4: Filter by Status

Show only blocked or flagged events:

# Only blocked events
kt events tail --status blocked

# Only modified events (e.g., PII redacted)
kt events tail --status modified

# Only flagged events
kt events tail --status flagged

Combine filters:

# Blocked events from the injection defense policy
kt events tail --policy injection-defense --status blocked

Step 5: Filter by Consumer Group

Track events for a specific team:

kt events tail --consumer-group engineering

Output:

[2026-04-23T10:31:01Z] REQUEST id=evt_m4n5o6 consumer_group=engineering policy=content-filter result=pass model=gpt-4o tokens=320 latency=510ms
[2026-04-23T10:31:04Z] REQUEST id=evt_p7q8r9 consumer_group=engineering policy=content-filter result=pass model=gpt-4o-mini tokens=85 latency=220ms

Step 6: JSON Output with jq

Use --format json for structured output that you can process with jq:

kt events tail --last 5 --format json | jq .

Full event structure:

[
  {
    "id": "evt_a1b2c3",
    "timestamp": "2026-04-23T10:30:01Z",
    "request_id": "req_x1y2z3",
    "consumer_group": "engineering",
    "model": "gpt-4o-mini",
    "provider": "openai",
    "policies": [
      {
        "name": "content-filter",
        "type": "content_filter",
        "action": "flag",
        "result": "pass",
        "duration_ms": 8
      },
      {
        "name": "pii-redaction",
        "type": "pii_detector",
        "action": "redact",
        "result": "pass",
        "duration_ms": 12
      }
    ],
    "usage": {
      "prompt_tokens": 22,
      "completion_tokens": 120,
      "total_tokens": 142
    },
    "cost": 0.008,
    "latency_ms": 285,
    "status": "completed"
  }
]

Useful jq recipes

Extract just blocked events with their reason:

kt events tail --last 100 --format json | jq '.[] | select(.status == "blocked") | {id, policy: .policies[] | select(.result == "blocked") | .name, timestamp}'

Calculate total tokens by consumer group:

kt events tail --last 100 --format json | jq 'group_by(.consumer_group) | .[] | {group: .[0].consumer_group, total_tokens: [.[].usage.total_tokens] | add, requests: length}'

Expected output:

{"group": "engineering", "total_tokens": 12500, "requests": 42}
{"group": "marketing", "total_tokens": 3200, "requests": 15}

Find the slowest requests:

kt events tail --last 100 --format json | jq 'sort_by(-.latency_ms) | .[:5] | .[] | {id, model, latency_ms, consumer_group}'

Show PII redaction details:

kt events tail --policy pii-redaction --format json | jq '.[] | select(.policies[].result == "modified") | {id, entities: .policies[].details.redactions}'

Step 7: Correlate with Request IDs

Every request through the gateway gets a unique request_id. Use it to trace a specific request:

kt events tail --format json | jq 'select(.request_id == "req_x1y2z3")'

The request_id is also returned in the gateway response headers:

curl -s -D- http://localhost:41002/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Hello"}]}' \
  2>&1 | grep X-Request-Id

X-Request-Id: req_x1y2z3

Use this to look up the exact decision event:

kt events tail --last 100 --format json | jq '.[] | select(.request_id == "req_x1y2z3")'

Step 8: Pipe to External Log Aggregators

Pipe to a file

kt events tail --format json >> /var/log/keeptrusts/events.jsonl &

Pipe to a syslog endpoint

kt events tail --format json | while read -r line; do
  echo "$line" | logger -t keeptrusts -p local0.info
done

Pipe to a webhook (e.g., Slack, PagerDuty)

Stream only blocked events to a webhook:

kt events tail --status blocked --format json | while read -r event; do
  POLICY=$(echo "$event" | jq -r '.policies[] | select(.result == "blocked") | .name')
  ID=$(echo "$event" | jq -r '.id')
  curl -s -X POST "$WEBHOOK_URL" \
    -H "Content-Type: application/json" \
    -d "{\"text\": \"🚨 Request $ID blocked by policy: $POLICY\"}"
done

Pipe to Elasticsearch / OpenSearch

kt events tail --format json | while read -r line; do
  curl -s -X POST "http://localhost:9200/keeptrusts-events/_doc" \
    -H "Content-Type: application/json" \
    -d "$line" > /dev/null
done

Step 9: Continuous Monitoring Dashboard Query

For a live dashboard, combine tail with watch:

# Refresh every 5 seconds — show last 10 events summary
watch -n 5 'kt events tail --last 10 --format json | jq ".[] | {time: .timestamp, status, policy: (.policies | map(.name) | join(\",\")), latency: .latency_ms}"'

Advanced: Debug a Specific Issue

When debugging why a request was blocked or modified, follow this workflow:

# 1. Get the request ID from the client-side error
REQUEST_ID="req_x1y2z3"

# 2. Look up the event
kt events tail --last 1000 --format json | jq --arg rid "$REQUEST_ID" '.[] | select(.request_id == $rid)'

# 3. Check which policy triggered
kt events tail --last 1000 --format json | jq --arg rid "$REQUEST_ID" '.[] | select(.request_id == $rid) | .policies[] | select(.result != "pass")'

# 4. Review the detection details
kt events tail --last 1000 --format json | jq --arg rid "$REQUEST_ID" '.[] | select(.request_id == $rid) | .policies[] | select(.result != "pass") | {name, result, details}'

For AI systems

• Canonical terms: Keeptrusts gateway, event tailing, decision events, kt events tail, policy status, consumer group filter.
• CLI commands: kt events tail, kt events tail --policy <name>, kt events tail --status blocked, kt events tail --consumer-group <name>, kt events tail --last <n>, kt events tail --json.
• Filter flags: --policy, --status, --consumer-group, --model, --last, --json, --since.
• Best next pages: Export Compliance Evidence, Gateway Health Monitoring, PII Redaction.

For engineers

• Prerequisites: kt CLI, running gateway with traffic, running Keeptrusts API, jq.
• Quick check: kt events tail --last 5 shows the most recent 5 events without streaming.
• Filter blocked: kt events tail --status blocked isolates policy violations instantly.
• JSON mode: kt events tail --json | jq '.policy, .result' enables programmatic filtering.
• Pipe to aggregator: kt events tail --json | your-log-shipper forwards events to Datadog, Splunk, etc.

For leaders

• Event tailing provides real-time visibility into every AI request and policy decision flowing through the gateway.
• Enables rapid incident response — security teams can immediately see blocked injection attempts or data leaks.
• Per-consumer-group filtering supports usage accountability and team-level debugging.
• JSON streaming integrates with existing observability stacks (SIEM, log aggregators) without additional tooling.

Next steps

• Set up PII redaction and monitor redaction events
• Block prompt injections and track blocked attempts
• Test policies in CI/CD to catch issues before they reach production

Troubleshooting

Symptom	Cause	Fix
No events appearing	Gateway not connected to API	Check api.url in config
kt events tail hangs	No traffic through gateway	Send a test request via curl
JSON parse errors with jq	Trailing newlines or mixed output	Use --format json explicitly
Events delayed	High API latency	Check API health at $KEEPTRUSTS_API_URL/health
--policy filter returns nothing	Policy name misspelled	Run kt events tail --last 1 --format json | jq '.policies[].name' to check names