Caching

The gateway runs a semantic cache against every chat-completion request. A near-identical prompt from the same team returns a cached response instead of a fresh upstream call, so repeated or closely-worded requests skip the provider round-trip. A per-team similarity threshold controls how close a prior prompt has to be to count as a hit; tighten it when your prompts are sensitive to small wording shifts.

What gets cached

The cache is scoped per team and per model family, so one team's traffic never serves another's responses and a cheap-model reply never satisfies a flagship request. A cache hit returns the prior response verbatim, with served_from_cache=true in the request row and cost_usd=0. Cache hits do not consume the team's token budget or the per-key rate limits.

What does not get cached

Streaming responses, tool calls, and images bypass the cache. Tool calls in particular have side effects (database writes, search queries) so a verbatim replay is unsafe by default. Streaming bypass exists because the cache stores the final response shape, not the chunked stream.

Invalidation

The cache has no manual invalidation surface. Entries age out after a period of inactivity and are evicted under memory pressure. Tighten the threshold per team in /teams/<id>/settings if your prompts are sensitive to small wording shifts.