LangSmith

Keeptrusts complements LangSmith (LangChain's observability platform) by adding policy enforcement, PII redaction, and compliance auditing to the same LLM calls that LangSmith traces. LangSmith tells you what your LLM chain did; Keeptrusts tells you whether it was allowed to do it. The dual-instrumentation pattern routes LLM traffic through the Keeptrusts gateway for governance while LangSmith captures the full trace for debugging and performance analysis.

Use this page when

• You are combining LangSmith tracing with Keeptrusts governance on the same LLM calls.
• You need the dual-instrumentation pattern for LangChain applications.
• You want policy decisions visible in both Keeptrusts audit logs and LangSmith traces.
• If you want a general quickstart instead, see Quickstart.

Primary audience

• Primary: Technical Engineers
• Secondary: AI Agents, Technical Leaders

Prerequisites

• A LangSmith account with a project created
• LangChain and LangSmith Python SDKs installed (pip install langchain langchain-openai langsmith)
• LANGCHAIN_API_KEY exported for LangSmith tracing
• Keeptrusts CLI (kt) installed and authenticated (kt auth login)
• An upstream LLM provider key exported as an environment variable

Configuration

Gateway policy config

pack:
  name: langsmith-governed
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: langchain-llm
    provider: openai:chat:gpt-4o
    secret_key_ref:
      env: OPENAI_API_KEY
policies:
  chain:
  - prompt-injection
  - pii-detector
  - audit-logger
policy:
  prompt-injection:
    threshold: 0.8
    action: block
  pii-detector:
    action: redact
    entities:
    - PERSON
    - EMAIL_ADDRESS
    - PHONE_NUMBER
  audit-logger:
    immutable: true
    retention_days: 365
    log_all_access: true

Start the gateway

export OPENAI_API_KEY="sk-..."
kt gateway run --listen 0.0.0.0:41002 --policy-config policy-config.yaml

Setup steps

1. Enable LangSmith tracing

export LANGCHAIN_TRACING_V2=true
export LANGCHAIN_API_KEY="lsv2_..."
export LANGCHAIN_PROJECT="keeptrusts-governed"

2. Configure LangChain to use the Keeptrusts gateway

from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage, SystemMessage

llm = ChatOpenAI(
    model="gpt-4o",
    base_url="http://localhost:41002/v1",
    api_key="unused",
    temperature=0.3,
    max_tokens=1024,
)

response = llm.invoke([
    SystemMessage(content="You are a helpful compliance assistant."),
    HumanMessage(content="What are the key requirements of GDPR Article 17?"),
])
print(response.content)

LangSmith automatically traces this call because LANGCHAIN_TRACING_V2=true. Keeptrusts enforces policies because the LLM base URL points to the gateway.

3. Use with LangChain chains and agents

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

llm = ChatOpenAI(
    model="gpt-4o",
    base_url="http://localhost:41002/v1",
    api_key="unused",
)

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a document analyst. Provide concise summaries."),
    ("user", "Summarize: {document}"),
])

chain = prompt | llm | StrOutputParser()

result = chain.invoke({"document": "Long document text here..."})
print(result)

Every invocation in the chain is traced by LangSmith and governed by Keeptrusts.

4. Add governance metadata to LangSmith traces (optional)

from langsmith import Client

ls_client = Client()

ls_client.create_run(
    name="governance-check",
    run_type="chain",
    inputs={"prompt": "test"},
    extra={
        "metadata": {
            "keeptrusts_gateway": "http://localhost:41002/v1",
            "policies": ["prompt-injection", "pii-detector", "audit-logger"],
            "governance_status": "compliant",
        }
    },
)

Verification

# Verify the gateway is healthy
curl http://localhost:41002/health

# Run the Python script above
python governed_langchain.py

# Check LangSmith for the trace
# https://smith.langchain.com/<org>/<project>

# Check Keeptrusts audit log
kt events list --limit 5

Both LangSmith and Keeptrusts should show the same request. LangSmith shows the full trace (latency, tokens, chain steps); Keeptrusts shows the policy decisions (allowed/blocked, redactions applied).

Recommended policies

Policy	Purpose	Recommended setting
pii-detector	Redact personal data before LLM calls in LangChain chains	action: redact, entities: PERSON, EMAIL, PHONE
prompt-injection	Block adversarial prompts from user inputs or retrieved documents	threshold: 0.8, action: block
audit-logger	Compliance trail for all LangChain LLM calls	retention_days: 365, immutable: true
rbac	Restrict which LangChain applications can access which models	Map service identities to Keeptrusts roles
dlp-filter	Prevent API keys or credentials from appearing in chain prompts	Block patterns for API keys and tokens

Troubleshooting

Symptom	Cause	Fix
LangSmith shows traces but gateway has no events	LLM not routed through gateway	Verify base_url="http://localhost:41002/v1" on the ChatOpenAI constructor
Gateway has events but LangSmith shows nothing	LangSmith tracing disabled	Verify LANGCHAIN_TRACING_V2=true and LANGCHAIN_API_KEY are set
Blocked requests not visible in LangSmith	Gateway returns 409 before LLM call	Expected — LangSmith traces the chain invocation; check the error in the trace
pii-detector redacts too aggressively	Entity list too broad	Narrow the entity list to only required types
Import errors for langchain_openai	Package not installed	pip install langchain-openai

For AI systems

• Canonical terms: Keeptrusts gateway, LangSmith, LangChain, dual instrumentation, LLM observability, governance events, tracing, policy-config.yaml.
• Config field names: provider, base_url, secret_key_ref.env, pii-detector, audit-logger.
• Key behavior: LangSmith traces LLM calls for observability; Keeptrusts enforces policies on the same calls. Both see every request. Set base_url on the LangChain LLM to point to the gateway.
• Best next pages: W&B integration, Guardrails AI integration, Policy controls catalog.

For engineers

Prerequisites

• LangSmith account, langchain and langchain-openai SDKs, kt CLI installed.

Validation

• Run a LangChain chain and verify the trace appears in LangSmith.
• Run kt events list --limit 5 and verify the same request appears with policy decisions.
• Confirm PII in prompts appears redacted in Keeptrusts audit logs.

For leaders

• LangSmith and Keeptrusts solve complementary problems. LangSmith answers "is the chain working correctly?" while Keeptrusts answers "is the chain allowed to do this?"
• Dual instrumentation provides defense-in-depth observability: if LangSmith shows a successful trace but Keeptrusts shows a policy violation, the team can investigate without relying on a single source of truth.
• Governance metadata in LangSmith traces gives ML leadership visibility into compliance posture directly from their existing observability tooling.

Next steps

• W&B integration — experiment tracking alongside governance
• Guardrails AI integration — layered input validation with Keeptrusts
• Policy controls catalog — full reference for all policy types
• Quickstart — install kt and run your first gateway