Skip to main content

Content Extractor

The content-extractor policy fetches content from URLs found in request messages and records extracted text in policy details, with strict host allowlisting and size controls.

Configuration​

policy:
  content-extractor:
    allow_hosts:
    - docs.example.com
    - support.example.com
    timeout_ms: 2000
    max_bytes: 65536
    fetch_urls: true
    action_on_error: warn
pack:
  name: content-extractor-example-1
  version: 1.0.0
  enabled: true
policies:
  chain:
  - content-extractor

Fields​

FieldTypeDescriptionDefault
allow_hostsstring[]Permitted fetch hostnames. Only URLs matching these hosts will be fetched. An empty list means no URLs are allowed — hosts must be explicitly listed. Supports exact hostnames only, not wildcards.[]
timeout_msinteger (min: 1)Fetch timeout per URL in milliseconds. Requests that exceed this timeout are treated as errors.2000
max_bytesinteger (min: 1)Maximum response body size in bytes. Responses exceeding this limit are truncated.65536 (64 KB)
fetch_urlsboolAutomatically detect and fetch URLs found in request content. When disabled, the policy is effectively a no-op.true
action_on_errorstringAction when extraction returns a blocked_reason (for example invalid URL syntax, unsupported scheme, host denial, DNS/private-address failures, fetch errors, or body-read failures): "warn" allows the request to continue, "block" rejects it. Oversize bodies are truncated and do not trigger this setting."warn"

Use Cases​

Audit allowlisted URL fetches​

Fetch referenced documentation URLs from approved hosts and record the extracted text in the policy result details.

pack:
  name: rag-content-fetch
  version: 0.1.0
  enabled: true
policies:
  chain:
  - content-extractor
  - prompt-injection
  - audit-logger
policy:
  content-extractor:
    allow_hosts:
    - docs.company.com
    - confluence.company.com
    - github.com
    timeout_ms: 5000
    max_bytes: 131072
    fetch_urls: true
    action_on_error: warn
  prompt-injection: {}
  audit-logger: {}

Fail-closed allowlisted fetches​

Fetch linked documents from approved hosts with strict size limits and block the request when extraction cannot complete safely.

pack:
  name: "doc-summarizer"
  version: "0.1.0"
  enabled: true

policies:
  chain:
    - content-extractor
    - safety-filter

policy:
  content-extractor:
    allow_hosts:
      - "storage.googleapis.com"
      - "s3.amazonaws.com"
    timeout_ms: 10000
    max_bytes: 524288
    fetch_urls: true
    action_on_error: "block"

  safety-filter:
    action: "block"

Verify that URLs referenced in requests are reachable and serve permitted content, blocking requests with broken or disallowed links.

pack:
  name: "link-verifier"
  version: "0.1.0"
  enabled: true

policies:
  chain:
    - content-extractor
    - dlp-filter

policy:
  content-extractor:
    allow_hosts:
      - "api.example.com"
    timeout_ms: 3000
    max_bytes: 1024
    fetch_urls: true
    action_on_error: "block"

  dlp-filter:
    action: "redact"
    detect_patterns:
      - "sk-[a-zA-Z0-9]{48}"

How It Works​

  1. URL detection — When fetch_urls is enabled, the gateway scans request message content for URLs using standard URL pattern matching.
  2. Safety checks — Each detected URL must use http or https, match allow_hosts, and resolve only to public IPs. Private, loopback, link-local, and other internal addresses are rejected.
  3. Fetch execution — Permitted URLs are fetched via HTTP GET with the configured timeout_ms. Responses larger than max_bytes are truncated at the byte limit.
  4. Policy details capture — Successfully fetched content is stored in the policy result details.extracted_text array alongside the detected URLs.
  5. Error handling — The extractor stops at the first invalid or failed URL and sets blocked_reason. With action_on_error: "warn", the request continues with an allow verdict; with "block", the request is rejected.

Best Practices​

  • Always specify allow_hosts explicitly. An empty list blocks all fetches by default. This is a security control — never use wildcards or overly broad host lists.
  • Set conservative max_bytes limits. Large fetched documents consume LLM context window tokens. Start with 64 KB and increase only for specific use cases like document summarization.
  • Use "block" for critical pipelines. If the fetch itself is a required precondition, set action_on_error to "block" so requests fail closed on invalid URLs, host denials, DNS issues, or fetch errors.
  • Review extracted content through policy details or audit logs. The extractor records fetched text in policy result details instead of rewriting the forwarded request body.
  • Monitor timeout settings. A 2-second default is appropriate for internal documentation. Increase timeout_ms for external hosts or large documents, but be mindful of the impact on end-to-end latency.

Next steps​