Inferock Bench

What is Inferock Bench?

inferock-bench is a local diagnostic proxy for metered LLM API traffic. You point your app or coding agent at it on localhost, it forwards the call to the provider using your saved key, and it writes an independent per-call receipt from its own stored event records. The receipt separates four things it refuses to add together: provider spend observed, bill-bounded money loss, time loss, and invoice-check exposure. It exists because the company that charges you also decides what counts as a failure and keeps the only detailed records; this puts a second, local record in your hands. Built for engineering and AI platform teams who are accountable for inference spend and reliability and want per-request evidence rather than a monthly total.

Key features

• Runs as a local proxy (default http://127.0.0.1:4318) with a token-gated dashboard; you redirect traffic by changing a base URL, not by rewriting your app.
• Measures four provider planes: OpenAI, Anthropic, the Gemini Developer API, and pinned OpenRouter endpoints spanning meta-llama, deepseek, mistral, moonshot/kimi, z-ai/glm and qwen on observed hosts.
• Detects delivery failures that can affect a bill or audit trail: billed-empty output, refusals, truncation, token-recount mismatches, duplicate request IDs, cache-discount-at-risk evidence, and provider-fault retries.
• Receipt headline keeps unlike numbers apart — 'spent', 'money loss', 'time loss' and 'invoice-check exposure' are reported separately, and cache discount at risk is shown as a 'verify your invoice' detail line rather than as money loss or a refund claim.
• Reports a coverage state per surface — watched-clean, signal, or not-openable — so checks that were never opened stay visible instead of being counted as clean.
• Labels observations (status codes, measured latency, provider-reported token counts, detector-flagged calls) separately from interpretations (dollar figures computed from those observations under published assumptions).
• Provider keys stay on your machine: stored under ~/.inferock-bench/ with owner-only file permissions, shown back only masked, attached only to provider requests, and not sent to Inferock. Calls use a separate local-only 'ibl_' bench key.
• Documented integrations for Claude Code (set ANTHROPIC_BASE_URL to the local proxy and use the local bench key) and the OpenAI SDK (set apiKey and baseURL); a built-in 'npx inferock-bench test' flow shows estimated tokens, estimated dollars and a spend cap before it makes any provider call. Requires Node.js 22+.

Who it's for

• Auditing an LLM bill call by call so a dispute with a provider is backed by per-request records rather than a monthly total.
• Measuring what a coding agent's metered traffic actually costs by routing Claude Code through the local proxy.
• Answering 'was I billed for a call that failed?' — truncated answers, empty replies, refusals, and provider-fault retries that were still charged.
• Checking whether a cache discount you expected actually showed up, before assuming the savings are real.
• Establishing a local reliability and latency baseline for a provider before committing production traffic to it.

When not to use it

It can only measure traffic that actually passed through the local proxy — it cannot audit calls that bypassed it, cap spend on calls it never saw, reconcile a monthly invoice without the matching bill, or reach web-subscription usage such as claude.ai, which exposes no per-call detail.

FAQ