TokenTuner is a self-hosted AI Gateway built in Go. Semantic caching slashes your LLM bills with well under 2ms of proxy overhead — and an opt-in Egress Shield filters malicious AI output before it reaches the user's terminal.
✅ Upstream TTFT: 342ms
💰 Billed: $0.0012 (125 prompt, 45 completion tokens)
# same intent, asked a different way
🔥 SEMANTIC CACHE HIT (Score: 0.97)
# EGRESS_SHIELD_ENABLED=true — agent reads a page with a hidden instruction
✅ Upstream TTFT: 890ms
🛡️ GHOSTJACKING BLOCKED: malicious egress detected (Prompt Injection Detected)
Drop-in replacement. No SDKs. Your prompts never leave your server.
Opt-in output filtering that catches an AI agent acting on hidden instructions in content it read, before that action reaches your terminal. Adds ~1-1.3s of shield inference when enabled — which is exactly why it's off by default.
Understands context — "What is X" and "Tell me about X" trigger a cache hit at 95% similarity, so repeated intents skip the LLM call entirely. Includes a Purge API and per-request TTLs for RAG apps.
Built in Go with a transparent, streaming proxy design — no code changes, just swap your Base URL. The proxy and cache path itself adds well under 2ms; optional security scanning has its own separate, clearly-labeled latency budget.
Bring Your Own Key (BYOK). We never see, log, or store your API keys. Embeddings and security models run locally.
ClickHouse backend tracks exact token usage, TTFT, cache hits, and estimated costs, with rows self-purging after 90 days.
Dynamically clear cached responses when your database updates. Prevents stale LLM answers in RAG workflows.
Spin up the Go Proxy, Vector DB, Analytics DB, and AI Sidecars locally or inside your VPC.
Point your OpenAI SDK from api.openai.com to localhost:8080. That's it.
TokenTuner handles the rest: streams responses, filters malicious AI output when Egress Shield is enabled, and logs every saved penny.
We are launching TokenTuner Cloud soon. Zero infrastructure management. Just point your API key and pay per token saved. Join the early access list.
No spam. Unsubscribe at any time.