Operations

This page covers the cross-cutting operations primitives: the cron task state persistence primitive that closes the worker-restart duplicate-fire hazard, the dep-drift reconciliation contract that pinned the nine out-of-pin packages, plus a reference to the v3 operator handover runbook.

The audience is a gateway operator running a production deploy. Each section names the on-disk surface, the operator step (if any), plus the verification command from the re-verification matrix in the operator handover runbook.

Cron task state persistence

The cron scaffolding at app/workers/cron.py ran the daily Merkle rollup plus other registered tasks via an in-process _RegisteredTask.last_run_day dict. A worker restart mid-day cleared the dict and let the next slot fire a task that had already run earlier in the same UTC day. The DB layer protected against duplicate writes through the ON CONFLICT (org_id, day) DO NOTHING clause on the Merkle table, but the S3 write fired twice plus the per-org compute path ran twice. A persistent state table now gates the cron firing across worker restarts.

The cron_task_state table

Migration 0040_phase14_cron_task_state creates the new table:

CREATE TABLE cron_task_state (
  name          TEXT PRIMARY KEY,
  last_run_day  DATE NOT NULL,
  last_run_at   TIMESTAMPTZ NOT NULL DEFAULT now()
);

The name column matches the task name registered via register_daily_task so the dispatcher can read by name in one round trip. The last_run_day column is the dedup gate the dispatcher checks on every minute poll; the last_run_at column is the wall-clock timestamp surfaced in operator dashboards plus the §11 health endpoint.

The persistent state gate

The dispatcher at app/workers/cron.py:_dispatch_due_tasks walks every registered task on every minute-poll. For a task whose (hour, minute) matches the current UTC time and whose in-process last_run_day does not match today, the dispatcher reads cron_task_state.last_run_day via _read_persistent_last_run_day. If the persistent value reads today, the dispatcher primes the in-process cache and skips the fire. Otherwise the dispatcher fires the task body and UPSERTs (name, last_run_day=today, last_run_at=now()) via _write_persistent_last_run_day.

The in-process cache stays as the fast path because a long-lived worker that has already fired today reads the in-process value on every minute-poll and short-circuits before the DB read. The persistent gate only fires on the cross-restart path; the steady state is one DB read on the first dispatch attempt of the day plus one DB UPSERT on the fire itself.

The worker-restart behaviour

A worker that restarts at 03:30 UTC after the 03:00 UTC Merkle rollup has already fired today reads the empty in-process cache on the first 03:00 UTC slot check (which fires on the very next day; the 03:30 boot misses the 03:00 window entirely so the dispatcher reads the persistent state on the next 03:00 UTC slot the following day). The persistent state reads last_run_day = today (yesterday from the new day's perspective), the dispatcher primes the in-process cache, plus the task fires exactly once for the new day.

A worker that restarts inside the 03:00 UTC fire window (the boot lands at 03:00:30 with the task already running in the prior process) hits the persistent gate immediately. The dispatcher reads last_run_day = today (the prior process wrote the UPSERT before its kill signal), the in-process cache primes, plus the dispatcher skips the fire. The prior process's task body completes on a separate path; the new process never duplicates the work.

A task body that raises still lands the UPSERT because the dispatcher writes the persistent state unconditionally after the try/except block. A perpetually-failing task does not retry-storm on every minute-poll; the per-task body owns its own retry contract (the Merkle rollup leans on the ON CONFLICT at the DB layer).

Verification

The §15 row 21 verification command in the operator handover runbook walks the persistent state:

psql "$PROD_DATABASE_URL" -c "SELECT name, last_run_day, last_run_at FROM cron_task_state ORDER BY name;"

Expected output: one row per registered cron task plus a recent last_run_at timestamp. A row with last_run_day more than one UTC day behind the current date is a signal the task is failing or the worker is not running.

Dep-drift reconciliation

A dependency audit named nine pinned packages out of sync with the .venv installed versions: cryptography (48.0.0 installed vs 43.0.0 pinned), rich, coloredlogs, flatbuffers, hiredis, onnxruntime, protobuf, sse-starlette, typer, plus humanfriendly. The cryptography package carried the largest API-surface area; the others were transitive drift. A parity smoke check runs across the cryptography-touching surfaces (Argon2id verify, HMAC chain compute, key-vault encrypt and decrypt) at both versions and the regression set passes at the installed version.

The converged state

The pinned versions in requirements.txt are reconciled to match the installed versions in .venv. The named packages now read at parity:

• cryptography==48.0.0 (the Fernet plus HMAC plus Argon2id verify surface).
• rich==15.0.0 (CLI rendering, pure Python).
• typer==0.24.1 (CLI framework on top of click).
• sse-starlette==2.1.3 (EventSourceResponse helper for the feed endpoint).
• onnxruntime==1.20.0 (Presidio transformer-model recogniser).

The remaining transitive packages (coloredlogs, flatbuffers, hiredis, protobuf, humanfriendly) reconciled via the parity smoke; no explicit pin bump was required because the named first-tier packages pulled the transitive deps to the converged state.

The per-package contract

A future dep bump touching any of the named packages must run the parity smoke at the new version before the pin bump lands:

• cryptography bump: run the Argon2id verify regression (tests/test_phase14_argon2_parity.py) plus the HMAC chain compute regression plus the key-vault encrypt and decrypt regression at the new version.
• rich bump: run the CLI rendering smoke at the new version; the package is pure-Python so the smoke is fast.
• typer bump: run the CLI command surface smoke at the new version; the package rides on click so a bump can ripple into the CLI's flag-parsing contract.
• onnxruntime bump: run the Presidio transformer-model smoke at the new version; the package is the largest native dep and a bump can break the DPI lazy load.

The dep-drift smoke at tests/test_phase14_dep_drift_smoke.py carries five regression pins covering the named first-tier packages. A future bump that breaks one of the five pins must land a fixed dep version or a code-side accommodation before the pin bump lands.

Runbook v3 polish reference

The v3 operator handover runbook lives at iq-routing-dashboard/docs/operator-handover-runbook.md (1378 lines, +241 over the v2 baseline). The runbook is the canonical operator artefact for the production deploy; this page points to the v3-specific additions plus surfaces no v3-only operator step that lives outside the runbook.

The v3 baseline carries fifteen pre-existing sections (§1 through §18) plus the v3-specific additions:

The v3-specific additions

The v3 polish absorbed twelve surfaces into the existing section structure without rewriting §1 through §15:

• §1 pre-flight gained one new bullet covering the production gateway 503 callout. The boot-loop traces to four Fly secrets queued for the operator cycle (items 8, 9, 10, 18 in the §15 matrix).
• §3.1 audit-chain primitive carries the new caveat that migration 0036 (the NOT NULL flip) is parked. The chain skipped 0035 -> 0037; the --check-only invocation pre-flight stays as the canonical verification.
• §3.1 also picks up the anomaly types extension covering migration 0039's five new heuristics (request_rate_spike, cost_rate_spike, failed_auth_burst, key_vault_reveal_burst, webhook_failure_streak) plus the existing thinking_budget_excess.
• §3.2 cross-org Merkle extension covers migration 0037 plus the owner-only verifier endpoint at /admin/compliance/audit-merkle-cross-org-verify.
• §3.3 covers the new DASHBOARD_ALERT_CENTRE_HMAC_KEY env at §18.
• §3.7 (new sub-block, 21 lines) covers the GDPR DSAR pack endpoint plus the dashboard download button. The synchronous endpoint pattern matches the SOC2 plus HIPAA endpoints; an operator who hits the endpoint sees either the ZIP or the 4xx error in one round trip.
• §3.8 (new sub-block, 24 lines) covers the HIPAA BAA pack endpoint plus the dashboard download button. The 6-year cap matches HIPAA §164.316(b)(2)(i).
• §9 cron_task_state extension covers migration 0040 plus the persistent state primitive documented above.
• §15 re-verification matrix gained four new rows: row 18 (migration 0036 PARKED), row 19 (cross-org Merkle), row 20 (anomaly types extended), row 21 (cron task state).
• §16 covers the queued Fly secrets plus the operator cycle status.
• §17.3 PII columns extension covers migration 0038's three new per-org columns (pii_redaction_latency_budget_ms, pii_redaction_dpi_enabled, pii_redaction_reinject_policy).
• §17.7 (new section) plus the renamed §17.8 cover the new surfaces with no operator step at handover.
• §18 gained one new row for the DASHBOARD_ALERT_CENTRE_HMAC_KEY env.

When to use the v3 runbook versus this page

The v3 runbook is the operator handover artefact. An operator running the production deploy reads the runbook top to bottom plus runs every §1 through §14 step in order, then walks the §15 matrix for re-verification. This page covers the cross-cutting primitives the runbook references but does not duplicate; an operator who needs the deeper context on cron task state persistence or dep-drift reconciliation reads this page after the runbook walk.

The §3.7 plus §3.8 sub-blocks align with the synchronous endpoint shape of the GDPR DSAR and HIPAA BAA pack endpoints. The runbook's acceptance check for both GDPR plus HIPAA exports walks the org_audit_events row write directly because the synchronous endpoint guarantees the audit row lands before the StreamingResponse opens; an operator who triggers a download from the dashboard plus runs the SELECT sees the count increment without a polling delay.