Procurement Guide: AI Vendor Evaluation

As a Procurement professional responsible for AI vendor selection, you need to evaluate providers on cost, capability, compliance, and risk — and then enforce contractual commitments through technical controls. Keeptrusts gives you the data infrastructure to benchmark vendors, calculate true costs, and verify SLA compliance with objective evidence rather than vendor self-reporting.

Use this page when

• You are evaluating AI/LLM vendors on cost, capability, compliance, and reliability
• You need actual usage data for TCO analysis and cost comparison across providers
• You are negotiating AI vendor contracts and need verifiable SLA enforcement
• You want to benchmark providers using real performance data (quality scores, latency, error rates)
• You are building a vendor risk scoring framework backed by operational data

Primary audience

• Primary: Technical Leaders (Procurement Managers, Vendor Managers, IT Sourcing)
• Secondary: IT Directors, Finance, Legal Counsel, CTOs

Vendor Assessment Framework

Assessment Dimensions

Evaluate AI vendors across six dimensions, using Keeptrusts for ongoing measurement:

Dimension	Assessment Method	Keeptrusts Evidence
Capability	Model quality and task suitability	quality-scorer metrics per provider
Cost	Per-token pricing, total cost of ownership	Console Cost Center data
Compliance	Regulatory certifications, data handling	Policy enforcement rates per provider
Reliability	Uptime, latency, error rates	Gateway health and event logs
Security	Data protection, prompt injection resilience	Security policy trigger rates
Support	SLA responsiveness, issue resolution	Incident tracking records

Vendor Evaluation Scorecard

Criteria	Weight	Scoring (1-5)	Evidence Source
Model quality	25%	Quality scorer average per provider	Events API
Cost per interaction	20%	Actual cost data from Cost Center	Console
Data residency compliance	15%	Regional gateway routing verification	Event logs
Security posture	15%	Content filter and injection detection rates	Events API
Uptime and reliability	15%	Gateway error rates per provider	Health monitoring
Contract flexibility	10%	Vendor terms evaluation	Procurement review

TCO Analysis

True Cost Components

LLM vendor cost goes beyond per-token pricing. Use Keeptrusts data for accurate TCO calculation:

Cost Component	Source	Measurement
API usage (tokens)	LLM provider billing	Console Cost Center — actual spend
Governance platform	Keeptrusts license	Per-user or per-team pricing
Infrastructure	Gateway hosting	Compute and networking costs
Integration labor	Development team	Hours to integrate (minimal with proxy pattern)
Compliance overhead	Legal/security review	Hours per vendor assessment
Incident response	Security/operations	Cost per security incident

Pulling Cost Data for TCO Modeling

# Get actual spend per provider for the evaluation period
curl -H "Authorization: Bearer $API_TOKEN" \
  "https://api.keeptrusts.com/v1/events?since=90d&group_by=provider"

# Export detailed cost data for analysis
kt export create \
  --type events \
  --format csv \
  --since 90d \
  --description "TCO analysis — provider cost comparison"

Cost Comparison Framework

Provider	Tokens Processed	Actual Cost	Cost/1K Tokens	Quality Score	Cost-Adjusted Quality
Provider A	From events	From Cost Center	Calculated	From quality-scorer	Quality / cost ratio
Provider B	From events	From Cost Center	Calculated	From quality-scorer	Quality / cost ratio
Provider C	From events	From Cost Center	Calculated	From quality-scorer	Quality / cost ratio

Contract Negotiation

Key Contract Terms for AI Vendors

Ensure these terms are in every AI vendor contract, with Keeptrusts-verifiable enforcement:

Contract Term	Why It Matters	Verification Method
Data processing location	Regulatory compliance	Regional gateway routing + event logs
Data retention limits	Privacy compliance	Vendor terms + Keeptrusts event retention
SLA guarantees	Service reliability	Gateway error rates per provider
Security certifications	Risk management	Vendor documentation + security event rates
Audit rights	Compliance verification	Event export capabilities
Price protections	Budget predictability	Cost Center historical trends
Termination terms	Vendor flexibility	Multi-provider gateway configuration

Avoiding Vendor Lock-In

Keeptrusts enables multi-provider deployments by design, reducing lock-in risk:

pack:
  name: procurement-providers-1
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: openai
    provider: 
    secret_key_ref:
      env: OPENAI_API_KEY
  - id: anthropic
    provider: 
    secret_key_ref:
      env: ANTHROPIC_API_KEY
  - id: azure-openai
    provider: 
    base_url: https://your-instance.openai.azure.com
    secret_key_ref:
      env: AZURE_OPENAI_API_KEY
policies:
  chain:
  - audit-logger
policy:
  audit-logger:
    immutable: true
    retention_days: 365
    log_all_access: true

Applications connect to the Keeptrusts gateway, not directly to providers. Switching providers requires only a configuration change — no application code modifications.

# Validate a configuration change when switching providers
kt policy lint --file updated-provider-config.yaml

SLA Requirements

Minimum SLA Requirements for AI Vendors

SLA Component	Minimum Requirement	Measurement
Availability	99.9% monthly uptime	Gateway error rate monitoring
Latency	< 500ms p95 for API responses	Event timestamp analysis
Error rate	< 0.1% server errors	Provider error events
Support response	< 4 hours for P1, < 24 hours for P2	Vendor support tracking
Data deletion	< 30 days upon request	Vendor confirmation + audit
Incident notification	< 1 hour for P1 incidents	Vendor SLA

Monitoring Vendor SLA Compliance

Use Keeptrusts event data to independently verify vendor SLA claims:

# Check provider error rates (SLA compliance)
curl -H "Authorization: Bearer $API_TOKEN" \
  "https://api.keeptrusts.com/v1/events?since=30d&group_by=provider"

# Export SLA compliance evidence
kt export create \
  --type events \
  --format csv \
  --since=30d \
  --description "Monthly vendor SLA compliance verification"

Vendor Risk Scoring

Risk Scoring Model

Assign risk scores to each AI vendor based on objective data:

Risk Factor	Weight	Scoring Criteria	Data Source
Data handling practices	25%	Certifications, DPA terms	Vendor documentation
Security incident history	20%	Past breaches, vulnerability disclosures	Public records + vendor disclosure
Content safety performance	20%	Content filter trigger rates per provider	Keeptrusts events
Financial stability	15%	Revenue, funding, market position	Financial analysis
Concentration risk	10%	% of traffic to single provider	Console events by provider
Regulatory alignment	10%	Compliance certifications relevant to your industry	Vendor documentation

Calculating Concentration Risk

Avoid over-reliance on a single provider:

# Provider distribution analysis
curl -H "Authorization: Bearer $API_TOKEN" \
  "https://api.keeptrusts.com/v1/events?since=90d&group_by=provider"

Target: No single provider should handle more than 60% of total AI traffic.

Risk Mitigation Through Multi-Provider Strategy

Risk Level	Action
Low (score < 30)	Standard monitoring
Medium (score 30-60)	Quarterly review, diversification plan
High (score > 60)	Active migration to alternative provider

Benchmarking

Provider Benchmarking Process

Run structured benchmarks through Keeptrusts to compare providers on equal footing:

Step 1: Configure multi-provider gateway

providers:
  targets:
  - id: openai
    provider: 
    secret_key_ref:
      env: OPENAI_API_KEY
  - id: anthropic
    provider: 
    secret_key_ref:
      env: ANTHROPIC_API_KEY
policies:
- name: benchmark-quality
  type: quality-scorer
  min_score: 0.0
  action: allow
  enabled: true

Step 2: Run benchmark workloads

# Deploy benchmark gateway
kt gateway run --policy-config benchmark-policy.yaml --port 41002

Step 3: Collect results

# Export benchmark data
kt export create \
  --type events \
  --format csv \
  --since 7d \
  --description "Provider benchmark results"

Step 4: Analyze

Compare providers on:

• Quality scores per task type
• Response latency
• Cost per interaction
• Content safety trigger rates

Procurement Workflow with Keeptrusts

Task	Frequency	Tool
Vendor cost review	Monthly	Console Cost Center
SLA compliance verification	Monthly	Event exports by provider
Vendor risk scoring update	Quarterly	Risk scoring model + events
Contract renewal preparation	Per renewal	90-day export for performance evidence
New vendor evaluation	As needed	Benchmark gateway + quality scoring
Concentration risk review	Quarterly	Provider distribution analysis

RFP Template: AI Governance Requirements

Include these Keeptrusts-relevant requirements in your AI vendor RFPs:

Requirement	Priority	Verification
OpenAI-compatible API endpoint	Must have	Integration test
Data processing in specified regions	Must have	Regional gateway routing
SOC 2 Type II certification	Must have	Vendor documentation
SLA with 99.9% uptime guarantee	Must have	Keeptrusts monitoring
Content filtering capabilities	Should have	Quality and safety benchmarks
Usage-based pricing with volume discounts	Should have	Cost Center analysis
Custom model fine-tuning support	Nice to have	Capability assessment

Success Metrics for Procurement

Metric	Target	Source
Cost per AI interaction	Decreasing trend	Console Usage
Vendor SLA compliance	> 99.9% for all vendors	Event-based SLA tracking
Vendor concentration	No provider > 60% of traffic	Provider distribution
Contract renewal savings	Measurable year-over-year	Negotiation outcomes
Time to onboard new vendor	< 2 weeks	Configuration change + validation
Vendor risk score	All vendors below medium threshold	Risk scoring model

For AI systems

• Canonical terms: Keeptrusts, vendor evaluation, TCO analysis, contract negotiation, SLA verification, vendor risk scoring, benchmarking
• Key surfaces: Console Usage (per-provider spend), Events API (group_by=provider), Console Dashboard, Export API
• Commands: kt export create, kt events list
• Assessment dimensions: Capability (quality-scorer), Cost (Usage reporting), Compliance (policy enforcement rates), Reliability (error rates), Security (injection/filter trigger rates), Support (incident tracking)
• TCO components: API usage tokens, governance platform, infrastructure hosting, integration labor, compliance overhead, incident response
• Best next pages: IT Director Guide, Gateway Configuration, Exports Guide

For engineers

• Pull per-provider spend: GET /v1/events?since=90d&group_by=provider
• Export cost comparison data: kt export create --type events --format csv --since 90d --description "TCO analysis - provider comparison"
• Quality-scorer metrics per provider feed directly into vendor evaluation scorecards
• Gateway error rates and latency data (from events) provide objective SLA compliance evidence
• Multi-provider providers block in gateway config enables parallel evaluation without application changes

For leaders

• Keeptrusts provides actual cost-per-token and quality-per-provider data from production traffic — vendor evaluations are based on measured performance, not marketing claims
• TCO calculation goes beyond per-token pricing to include governance platform, infrastructure, integration labor, compliance overhead, and incident response costs
• Contract terms (data residency, retention, SLA guarantees, audit rights, price protection) can each be verified with event data rather than relying on vendor self-reporting
• Running multiple providers through the gateway gives negotiating leverage: you can demonstrate willingness to shift traffic based on cost-adjusted quality ratios
• Vendor performance review data is automatically generated — export quarterly reports for procurement governance cadence

Next steps

• Manage vendor portfolio: IT Director Guide
• Configure providers: Gateway Configuration
• Export vendor data: Exports Guide
• Understand architecture: Architecture Overview