Prompt token optimizer

The token optimizer is a payload-size pass on the user-side messages and the system prompt before the request reaches the upstream provider. A 30 percent compression on a Sonnet call is roughly 30 percent off the input token cost on that request. The optimizer is a FinOps lever, not a quality lever; it does not rewrite prompts to be "better."

The optimizer fails open. Any error, timeout, or floor breach ships the uncompressed prompt instead, and the request proceeds normally. The hot path never blocks on the optimizer.

Three modes

| Mode | What it does | Latency budget | When to use | |---|---|---|---| | off | Pass-through. No compression. | 0 ms | Default. Use when payload size is not a cost driver. | | safe | Deterministic, lossless cleanup of redundant formatting in conversational text. | 250 ms budget | Default-on recommendation. Lossless for the model; observable token reduction on chat-heavy prompts. | | aggressive | Learned token-importance compression at the org's threshold. | 250 ms hard cap; fails open above. | Cost-driven workloads. Gated by an explicit opt-in so the trade-off is a conscious choice, not a default. |

What the optimizer never touches

Tool-call payloads. Function arguments and structured outputs pass through unchanged. The compression scope is conversational text, not machine-readable JSON.
Assistant messages. The optimizer compresses what you send; what the model said back stays as-is for the conversation history.
Code fences. Triple-backtick blocks pass through whitespace- preserving even in safe mode. Code semantics depend on indentation.

Picking a mode

Settings → Token optimizer → mode picker. The aggressive option is a deliberate control: an explicit opt-in checkbox turns it on at the org's threshold, so choosing more compression for cost-driven workloads is a conscious choice rather than a default.

One composition to know about: an org whose routing focus mode is set to quality runs safe even when the optimizer dial says aggressive. An operator who asked for quality on the routing side does not silently get lossy compression on the prompt side.

The dashboard surface lives at /settings#token-optimizer.

The pane also exposes:

Threshold (aggressive only). The retain-or-drop threshold the aggressive pass uses. A lower threshold means more aggressive compression and a higher quality risk.
Compression stats. A 30-day rollup of tokens saved, cost saved, average compression ratio, and the number of requests the optimizer ran on. Tokens saved is the number the ROI case rests on.

Safe mode

Safe mode is a deterministic, lossless cleanup of redundant formatting in conversational text. It tidies the kind of structure that costs tokens without carrying meaning for the model, and it preserves the content of the prompt verbatim. Code fences pass through whitespace-preserving so indentation-sensitive content is never touched. The result is a smaller payload with no change to what the model reads as substance.

Aggressive mode

Aggressive mode applies learned token-importance compression: it scores the prompt and drops the lowest-value tokens down to the org's threshold, trading a small quality risk for a larger token reduction. The aggressive mode:

Loads lazily. The compression model loads on first use, not at gateway boot. A cold-start aggressive request pays the load cost; subsequent requests reuse the loaded model.
Fails open on a latency cap. A hard wall-clock cap on the compression step. If the compression exceeds the cap, the aggressive pass aborts and the uncompressed prompt ships.
Falls back to safe. If aggressive compression is unavailable gateway-wide, orgs set to aggressive automatically downgrade to safe mode rather than losing the optimizer entirely.
Disabled in bundle mode. The standalone gateway bundle does not ship the aggressive-mode weights, to keep the binary size sane. Bundle installs cap at safe mode.

Floor and fail-open semantics

Both modes carry a compression floor: safe mode aborts if the result would fall under 50 percent of the original size, aggressive mode's floor is 30 percent. Below the floor, the pass aborts and the uncompressed prompt ships, because an over-aggressive reduction on a coherent prompt almost always means the optimizer mis-scored a load-bearing section, and falling open is safer than shipping a degraded prompt. A floor breach records a compression_threshold_breached audit event you can subscribe a webhook to, and it contributes no saving to the compression-stats rollup because the uncompressed prompt is what shipped.

Rollback

Rollback is a mode change, not a deploy. If safe mode misbehaves on your traffic, set the org's optimizer mode to off in the dashboard and the pass-through path restores the prior behaviour on the next request. If aggressive mode misbehaves, drop the org back to safe from the same picker; the aggressive pass stops running immediately and nothing else about the request path changes.

Bundling for offline use (Enterprise)

The standalone gateway bundle is the Enterprise on-prem option, arranged with us rather than downloaded; there is no self-hosted distribution on the self-serve plans.

The bundle does not ship the aggressive-mode weights and does not load them at runtime, on any host, regardless of network access. An org running the bundle that is set to aggressive runs safe instead -- the same gateway-wide fallback described in Aggressive mode, above. safe mode is unaffected: it is pure Python with no model to load, so it runs identically in the bundle and on the hosted gateway.