Webhook capacity planning

The webhook deliverer runs inside each gateway worker process. It dequeues pending deliveries, signs the payload with HMAC-SHA256, POSTs to the receiver, and marks the delivery success, failed, or dead_lettered. Throughput depends on two settings: the connection pool size and the worker concurrency. This page walks you through both knobs, what they trade off, and how to size them for a real workload.

Defaults

The deliverer maintains a connection pool with these defaults:

max_connections=50 -- caps total in-flight TCP sockets. A request beyond the cap blocks until a slot frees, then errors.
max_keepalive_connections=20 -- caps idle sockets kept warm between deliveries. Reusing a warm connection skips the TCP handshake and TLS round trip.

These defaults assume a typical tenant: a handful of receivers per org, sustained delivery rate well under 50 per second, and receivers that respond inside the 10-second read timeout. Most installs never need to touch them.

Concurrency knob

WEBHOOK_DELIVERER_CONCURRENCY controls how many worker threads the deliverer spawns. The default is 8, clamped to the range [1, 64].

Each worker independently claims a pending delivery without colliding with peers. Raising concurrency drains the queue faster when receivers are fast and the queue depth is high. Lowering it caps the load you put on slow receivers and reduces pool contention.

The relationship between the three numbers is:

WEBHOOK_DELIVERER_CONCURRENCY is the upper bound on simultaneous in-flight deliveries from one worker process.
max_connections must be at least equal to concurrency, otherwise workers stall on pool timeouts waiting for a socket.
max_keepalive_connections should track the number of unique receiver hosts you talk to. Past that, you pay handshake cost on every dispatch.

If you run multiple gateway worker processes, each process has its own connection pool and its own worker threads. The per-org rate limit (60 deliveries per minute, see /docs/webhooks) is enforced upstream of dequeue, so horizontal scaling does not bypass the receiver-side cap.

Three dimensions to scale

Sustained delivery rate

Estimate the rate by reading the webhook_deliveries table at peak. A query like SELECT count(*) FROM webhook_deliveries WHERE created_at > now() - interval '5 minutes' divided by 300 gives a rough deliveries-per-second figure. Multiply by your peak-to-average ratio (2x is a reasonable default) to get the design rate.

The rule of thumb: pick WEBHOOK_DELIVERER_CONCURRENCY such that each worker handles between 5 and 15 deliveries per second when receivers are healthy. At 100 deliveries per second sustained:

Concurrency 16, max_connections 100. One worker process at this setting handles roughly 6 to 7 dispatches per coroutine per second, which sits inside the healthy band.
Concurrency 8 (default), max_connections 50. Same workload pushes each coroutine to 12 dispatches per second; healthy if receivers are fast, but margin shrinks.

If your design rate exceeds 200 per second on one process, run multiple gateway workers rather than pushing one process past concurrency 32. Per-process concurrency above 32 starts contending on the _dequeue_delivery row lock more than it parallelises work.

p95 receiver latency

Slow receivers absorb pool capacity. A delivery in flight for 2 seconds holds one connection slot for 2 seconds. Little's Law gives the pool size you need:

in_flight_at_steady_state = throughput * average_latency

Use p95 as the latency input rather than mean, because the pool sizes to the worst-case occupancy not the average. Worked sketch:

p95 = 2.0 s
sustained = 50 req/s
in_flight = 50 * 2.0 = 100
required max_connections = 100

A 50 deliveries-per-second design rate against a receiver that takes 2 seconds at p95 needs max_connections=100. With the default 50, half the workers stall on pool=5.0 and time out; the deliveries reschedule on backoff and the queue grows.

The same arithmetic flips around: if your receiver is fast (p95 = 100 ms), 50 sustained per second only needs 5 in-flight, and the default 50 has 10x headroom.

Concurrent receiver count

max_keepalive_connections=20 covers about 20 unique receiver hosts warm. Beyond that, the pool evicts the oldest idle connection on every new dispatch, and the next delivery to the evicted host pays a fresh TCP handshake plus TLS round trip. On a transcontinental link that overhead is 100 to 300 ms per delivery.

If you have 30 unique receiver hostnames and cold-start latency matters (for a tight SLO on api_key_revealed deliveries to a SIEM, say), bump max_keepalive_connections=30. The cost is steady-state memory for 30 idle TLS sockets per worker process, which is small (roughly 20 KiB per socket).

If your install funnels every webhook through one ingress (one receiver host that fans out internally), the default 20 is plenty and lowering keepalive does nothing useful.

Failure modes

When the queue lags, work through these in order:

Deliveries falling behind. Watch webhook_deliveries row count with status='pending' AND scheduled_at <= now() over time. A growing tail means dispatch rate is below arrival rate. Either raise WEBHOOK_DELIVERER_CONCURRENCY (and max_connections to match) or fix the slow receiver. A receiver at p95 = 5 seconds is usually the bottleneck even at default concurrency.
TCP handshake overhead too high. Look for connect_ms p50 above 50 ms in the deliverer logs. If unique receiver count exceeds max_keepalive_connections, raise the keepalive cap.
Memory pressure. Each in-flight delivery holds the request body, the response body excerpt (capped at 4 KiB), and the socket buffer. At concurrency 64 with large payloads, one process can pin tens of MiB. Either lower concurrency or shard across more worker processes.

Worked example: 1000 per minute, 5 receivers, p95 = 500 ms

Walk through end to end. The arrival rate is 1000 / 60 ≈ 17 deliveries per second sustained. Apply a 2x peak-to-average ratio for the design point: 34 deliveries per second.

Required in-flight from Little's Law: 34 * 0.5 = 17. So max_connections needs at least 17 with margin; the default 50 is plenty. No change to _HTTPX_LIMITS needed.

Concurrency: 34 deliveries per second across 8 worker coroutines is about 4 per coroutine per second, which is inside the healthy band. Default WEBHOOK_DELIVERER_CONCURRENCY=8 is fine.

Keepalive: 5 unique receivers fits inside 20 with room to spare. Default max_keepalive_connections=20 is fine.

Conclusion: ship the default config. The small spike margin and the fast-receiver assumption mean nothing needs tuning.

If the same workload had p95 = 3 seconds instead, in-flight rises to 34 * 3.0 = 102, which overflows the default 50. Bump to max_connections=120 and WEBHOOK_DELIVERER_CONCURRENCY=16 so the queue drains without pool timeouts.

What not to do

Raising max_connections past WEBHOOK_DELIVERER_CONCURRENCY * unique_receivers_in_flight is wasted memory. The pool never opens more sockets than coroutines asking for them, multiplied by host count. Setting max_connections=500 with concurrency 8 gives the same throughput as max_connections=100 with concurrency 8.

Setting WEBHOOK_DELIVERER_CONCURRENCY above the receiver-side rate limit guarantees error-budget burn. If a receiver caps at 30 requests per second and you push 60, half the deliveries get a 429 or 5xx and reschedule on the 1s, 5s, 25s, 125s backoff. The queue cycles without making progress and the dead-letter rate climbs.

Setting max_keepalive_connections higher than the unique receiver count does not improve anything. The pool cannot keep more idle sockets than it has hosts to keep them for. Match the keepalive cap to your actual receiver count, no higher.

Verification

Operators monitor deliverer health through three surfaces that exist today: the GET /admin/webhooks/{id}/deliveries admin endpoint (last 100 deliveries per destination), a SQL queue-depth query against webhook_deliveries, and the webhook_deliverer_ structured log stream.

The GET /admin/webhooks/{id}/deliveries endpoint returns the last 100 deliveries for one destination, with per-delivery status, response code, and timing. Pair it with a SQL query against webhook_deliveries for aggregate queue depth:

SELECT status, count(*) FROM webhook_deliveries
WHERE scheduled_at > now() - interval '15 minutes'
GROUP BY status;

A growing pending count at fixed success count is the canonical "deliveries falling behind" signal. A rising dead_lettered count signals a misconfigured receiver or a concurrency setting above the receiver's rate limit.

For per-worker detail, the deliverer emits structured logs at webhook_deliverer_loop_started, webhook_deliverer_worker_cancelled, and per-delivery success or failure events; grep for webhook_deliverer_ in the gateway log stream to read current state. The httpx pool's internal socket counts are not exposed separately, so the delivery log plus the queue-depth query are the authoritative occupancy signals.