Prompt token optimizer

The token optimizer is a payload-size pass on the user-side messages and the system prompt before the request reaches the upstream provider. A 30 percent compression on a Sonnet call is roughly 30 percent off the input token cost on that request. The optimizer is a FinOps lever, not a quality lever; it does not rewrite prompts to be "better."

The optimizer fails open. Any error, timeout, or floor breach falls back to the uncompressed prompt and logs token_optimizer_failed_open. The hot path never blocks on the optimizer.

Three modes

| Mode | What it does | Latency budget | When to use | |---|---|---|---| | off | Pass-through. No compression. | 0 ms | Default. Use when payload size is not a cost driver. | | safe | Deterministic, lossless cleanup of redundant formatting in conversational text. | < 5 ms | Default-on recommendation. Lossless for the model; observable token reduction on chat-heavy prompts. | | aggressive | Learned token-importance compression at the org's threshold. | Hard latency cap; fails open above. | Cost-driven workloads. Gated by an explicit opt-in so the trade-off is a conscious choice, not a default. |

What the optimizer never touches

Tool-call payloads. Function arguments and structured outputs pass through unchanged. The compression scope is conversational text, not machine-readable JSON.
Assistant messages. The optimizer compresses what you send; what the model said back stays as-is for the conversation history.
Code fences. Triple-backtick blocks pass through whitespace- preserving even in safe mode. Code semantics depend on indentation.

Picking a mode

Settings → Token optimizer → mode picker. The aggressive option is a deliberate control: an explicit opt-in checkbox turns it on at the org's threshold, so choosing more compression for cost-driven workloads is a conscious choice rather than a default.

The dashboard surface lives at /settings#token-optimizer.

The pane also exposes:

Threshold (aggressive only). The retain-or-drop threshold the aggressive pass uses. A lower threshold means more aggressive compression and a higher quality risk.
Shadow mode (any mode). Run the optimizer alongside the live path for one out of every N requests, log the dual-path metrics, and do not apply the compressed prompt. Useful for measuring the compression delta before a mode flip.
Compression stats. Last-7-day rolling averages of input-token reduction, latency overhead, and fail-open rate. The data backs the ROI tile on the board deck.

Safe mode

Safe mode is a deterministic, lossless cleanup of redundant formatting in conversational text. It tidies the kind of structure that costs tokens without carrying meaning for the model, and it preserves the content of the prompt verbatim. Code fences pass through whitespace-preserving so indentation-sensitive content is never touched. The result is a smaller payload with no change to what the model reads as substance.

Aggressive mode

Aggressive mode applies learned token-importance compression: it scores the prompt and drops the lowest-value tokens down to the org's threshold, trading a small quality risk for a larger token reduction. The aggressive mode:

Loads lazily. The compression model loads on first use, not at gateway boot. A cold-start aggressive request pays the load cost; subsequent requests reuse the loaded model.
Fails open on a latency cap. A hard wall-clock cap on the compression step. If the compression exceeds the cap, the aggressive pass aborts and the uncompressed prompt ships.
Falls back to safe. If ENABLE_AGGRESSIVE_OPTIMIZER=false flips globally, orgs with mode='aggressive' automatically downgrade to safe for the rest of the deploy cycle.
Disabled in bundle mode. The standalone gateway bundle does not ship the aggressive-mode weights, to keep the binary size sane. Bundle installs cap at safe mode.

Floor and fail-open semantics

Both modes carry a compression floor. If a pass would remove the bulk of the original tokens, it aborts and the uncompressed prompt ships, because an over-aggressive reduction on a coherent prompt almost always means the optimizer mis-scored a load-bearing section, and falling open is safer than shipping a degraded prompt. The floor breach surfaces as token_optimizer_floor_breach in the logs and on the optimizer pane's last-7-day fail-open rate tile.

Rollback

Safe mode misbehaves. Set org.token_optimizer_mode='off' per-org via the dashboard or via SQL UPDATE organisations SET token_optimizer_mode='off' WHERE id=.... The optimizer pass-through restores the prior behaviour.
Aggressive mode misbehaves. Set ENABLE_AGGRESSIVE_OPTIMIZER=false env globally; the aggressive mode's _load short-circuits and the dashboard mode picker disables the aggressive option for new opt-ins.

Bundling for offline use

The standalone gateway bundle (iq-gateway) keeps the aggressive-mode weights out of the binary to keep the download small. The bundle uses the lazy-download path by default: an operator running the bundle on a host with outbound internet pays the model download cost once on first aggressive request, and the next boot finds the cached weights and skips the download.

For air-gapped deployments, pre-stage the model into the cache at build time and set the IQ_AGGRESSIVE_MODEL_PRELOADED=true environment variable on the bundle host. With the env set, the gateway loads the model from the local cache only, and an empty cache surfaces as a clean disabled error rather than a hub fetch attempt.

# Stage the model cache on a host with outbound internet, then ship the
# cache directory alongside the bundle. On the bundle host:
export HF_HOME=/path/to/staged-cache
export IQ_AGGRESSIVE_MODEL_PRELOADED=true
./iq-gateway serve

The default bundle does not preload the aggressive model; the download is opt-in. Operators who let the lazy-download run leave the env unset; operators who pre-stage at build time set the env so the gateway reads the cache directly.