Caching

Two different caches touch your traffic, and they behave nothing alike. The first is the gateway's own response cache, which answers a request out of IQ's storage without calling a provider at all. The second is the provider's prompt cache, which still calls the provider but charges you a discounted rate for the part of the prompt the provider already had in memory. A gateway hit costs you nothing. A provider prompt-cache read costs less than a fresh input token but is not free. Conflating the two is the single easiest way to misread your bill, so this page keeps them apart throughout.

Gateway response caching

When the gateway can serve a request from its own store, it does, and no upstream call happens. Three behaviours are live.

Single-turn traffic is matched by an exact-match layer first and a semantic-similarity layer second, so a near-identical prompt from the same team returns the stored response instead of paying for a fresh generation. Both layers are scoped per team and per requested model, so one team's traffic never serves another's responses and a response stored against one model id or alias is never replayed for a request that named a different one. The semantic layer matches by storing a numeric fingerprint (an embedding) of each cacheable prompt alongside the response, so a hit on this layer holds a fingerprint of your prompt as well as the response itself. Entries age out roughly seven days after they are written. With Zero Data Retention on, the gateway does not cache your organization's traffic at all; see Caching and zero data retention below.

Multi-turn agent traffic is covered by a conversation-scoped layer, keyed on the whole transcript so far rather than on the last message, with a short TTL of five minutes by default and strict org isolation. This is the layer that makes a repeated agent loop cheap: an agent that re-derives the same state on the way to a slightly different next step pays for the new step, not for the whole prefix again.

Streaming requests hit the cache too. A cached streaming response is replayed frame by frame in the original wire shape, so your client sees a normal stream and cannot tell a hit from a fresh call other than by how fast it arrives.

A gateway cache hit is billed at zero. It is not exempt from your limits, though: the rate-limit and budget checks run ahead of the cache lookup, so a hit still counts against your per-minute request allowance. Size a replay-heavy workload against the request limit, not just the spend. Latency lands in the single-digit to low-double-digit millisecond range. A hit skips the provider round trip entirely, so it returns faster than the original request did. The request row records the hit, so you can see exactly which traffic was served from cache and what it saved you. The same signal rides on the response itself, request by request: the cache_hit field in the x-iq-routing header is true on a hit, so your own code can branch on it without a dashboard round-trip. See /docs/sdks/curl for a worked example of reading the header.

What still bypasses the gateway cache

A request that carries an image attachment is never looked up and never written. The cache key is built from text, so a content-blind key would let one image's description answer a question about a different image. Error-shaped responses are refused too: a reply with neither text nor a tool call is never stored, so a single upstream failure cannot poison a key for the rest of its TTL. Gateway-generated notices are refused on the same grounds.

Turns that carry tool calls are not excluded. An agent step whose output is a function call is a cacheable turn like any other, because the alternative was that agent traffic structurally never hit the cache at all. If a step in your loop carries a side effect, make the tool call idempotent on your side and treat a cached replay the way you would treat any retry.

Invalidation

Owners and admins can clear all of your organization's cached responses at once with Clear my org's cache in the dashboard's settings, on every plan. Otherwise, entries age out on their own schedule, seven days for the single-turn layers and five minutes by default for the conversation-scoped layer, and are evicted under memory pressure. The conversation layer's TTL is an org-level setting bounded at an hour, and setting it to zero turns that layer off for the org entirely.

The semantic layer matches on a fixed similarity cutoff rather than a per-org dial, so how close a paraphrase has to be to count as a hit is the same for every team. The cutoff is deliberately tight: a near-miss falls through to a fresh provider call rather than answering a different question.

Provider prompt caching

Provider prompt caching is a different mechanism owned by the upstream provider, not by IQ. The gateway passes it through and accounts for it per request, splitting cached input tokens out from ordinary input tokens on every row. The economics are asymmetric: writing a prompt into the provider's cache costs more than an uncached input token, and reading it back costs substantially less, so a cache that is written often and read rarely can cost you more than no cache at all.

The practical consequence is that a provider prompt-cache read shows up on your bill at a discounted rate, while a gateway cache hit shows up at zero. If you see cost on a request you believed was cached, check which cache you were counting on. The accounting detail, including how the four-way token split is reported and how to tell whether your cache strategy is paying for itself, lives on the billing page.

Caching and zero data retention

Your organization's requests, responses and routing results are never shared with another organization, including anything IQ Routing caches for speed. With zero data retention on, IQ Routing does not cache them at all: the gateway neither reads from nor writes to its response caches for your organization. Entries cached before you turned ZDR on are no longer served; they expire on their normal schedule, or an owner or admin can remove them at once with Clear my org's cache. The full boundary, including what is and is not persisted with ZDR on, is on the security page. Loop-detection metadata is covered by the retention boundary on the security page.

Cache hit rate

The cache hit rate on your dashboard is the share of input tokens served from a cache instead of being processed again.