An outbound webhook delivery service in Rust.
Applications hand Relay an event. Relay signs it, queues it, and delivers it to every subscribed endpoint — retrying with backoff, tripping circuit breakers on dead endpoints, rate limiting slow ones, and recording every attempt.
Sending one HTTP POST is trivial. Everything that makes this a system is what happens when the POST fails — and against thousands of servers on the open internet, it fails constantly.
| Failure | Response |
|---|---|
| Endpoint down for hours | Retry with jittered exponential backoff |
| Endpoint never returns | Dead letter queue, replayable |
| Large event bursts | Durable queue with a worker pool |
| Slow endpoint | Per-endpoint token bucket rate limiting |
| Dead endpoint, deep backlog | Circuit breaker with probing recovery |
| Forged requests | HMAC-SHA256 signatures over the raw body |
| Duplicate delivery | Idempotency keys and a stable delivery id |
| Delivery disputes | Append-only attempt log with status and latency |
- At-least-once delivery with a delivery id that is stable across retries, so receivers can deduplicate. Exactly-once over an untrusted network is not achievable; retries are made safe rather than rare.
- Postgres as the queue —
FOR UPDATE SKIP LOCKEDwith leases and a reaper for crash recovery. One transactional store provides durability and atomic state without a separate broker. - Two processes — an ingest API that accepts and returns
202immediately, and a dispatcher that drains the queue. They interact only through the database, and either can restart independently. - Pure core — logic that can be free of I/O (signing, backoff, breaker state
machine, outcome classification) lives in a crate with no
asyncand no database, so it is deterministic and fast to test.
crates/
domain/ pure logic, no I/O — signing, backoff, breaker state machine
store/ sqlx repositories and migrations
dispatcher/ worker pool, reaper, HTTP sender
api/ axum ingest and admin endpoints
testkit/ configurable receiver for integration tests
Requires a recent stable Rust toolchain.
docker compose up -d # Postgres on 5433
cargo test # unit tests plus end-to-end against a real databaseThree processes: the ingest API, the dispatcher, and a configurable receiver standing in for a customer endpoint.
export DATABASE_URL=postgres://relay:relay@localhost:5433/relay
RELAY_TESTKIT_SECRET=whsec_demo cargo run -p relay-testkit # :9090
cargo run -p relay-api # :8080
cargo run -p relay-dispatcherRegister an endpoint and send it an event:
curl -X POST 127.0.0.1:8080/v1/endpoints \
-H 'content-type: application/json' \
-d '{"url":"http://127.0.0.1:9090/verify","event_types":["order.paid"]}'
curl -X POST 127.0.0.1:8080/v1/events \
-H 'content-type: application/json' \
-d '{"type":"order.paid","amount":4999}'
# 202 Accepted, with the event id and one delivery id per subscribed endpointThe receiver exposes failure modes for testing delivery behaviour:
/always500, /slow?ms=, /flaky?pct=, /429?retry_after=, and /verify,
which checks the signature and the freshness window.
Modelled on Stripe's scheme, so existing receiver implementations apply.
Relay-Timestamp: 1700000000
Relay-Signature: v1=<hex>,v1=<hex> # current and previous secret
Relay-Delivery-Id: <uuid> # stable across retries
The signed string is exactly <timestamp>.<raw body bytes>. Signing the raw
bytes matters: re-serialising JSON can reorder keys and invalidate every
signature. Two signatures are sent during secret rotation so endpoints can
migrate without failed deliveries.
MIT