Lesson 17 — Webhooks and third-party APIs

Stack: DRF + Celery (preview) · Project: TicketFlow Status: Published — closing the APIs module Prerequisite: Lesson 16 — gRPC/GraphQL/WS/SSE

Objectives

1. Receive the gateway's webhook safely: signature verification, replay and ordering.
2. Answer 2xx fast and process later (the pattern that avoids retry storms).
3. Design your outgoing webhook for others (idempotency, retries, signatures) — the craft's symmetry.

1. The incoming webhook: zero trust

The gateway (Stripe/PayPal/simulated) calls your public URL when the payment changes state. Your handler cannot trust the body: anyone can POST {"status": "SUCCEEDED"}. Defenses, in order:

1. HMAC signature: the provider signs the payload with a shared secret; you recompute over the raw body and compare (constant-time). No match → 401/400 and log.
2. Anti-replay: the payload carries a timestamp; older than 5 minutes, reject (a stolen body isn't valid forever). Stripe signs timestamp.payload and the timestamp is part of the HMAC.
3. Idempotency by event_id: the provider retries (at-least-once, Lesson 14): look up before creating; duplicate → silent 200.
4. Ordering not guaranteed: payment.succeeded can arrive before payment.pending (retries). Your processing must be tolerant: the payment's state is governed by the state machine (00b), not by arrival order.

@api_view(["POST"])
@authentication_classes([])  # the webhook doesn't use your auth: it uses its signature
@permission_classes([AllowAny])
def gateway_webhook(request):
    signature = request.headers.get("X-Signature", "")
    if not verify_hmac(request.body, signature, settings.GATEWAY_SECRET):
        return Response(status=401)
    payload = json.loads(request.body)
    if abs(time.time() - payload["timestamp"]) > 300:
        return Response(status=400)
    process_gateway_event.delay(payload)   # enqueue: Lesson 29
    return Response(status=200)            # immediate ACK

2. Answer fast, process later

Why delay() and not process right there? Because the provider retries with a timeout (typically 5-30s) and exponential backoff when you fail. If your handler does: verify payment in DB, send email, generate PDF tickets... and takes 40s, the provider marks a failure and retries while your first processing is still alive → double processing + storm. The pattern: ACK 2xx on receipt (validated + enqueued), processing in a worker (Celery, Lesson 29), worker idempotency by event_id.

If you don't have Celery yet: synchronous but bounded processing (only update state + create the DB record, <1s) is acceptable to start; the queue is the adult form.

3. Calling third-party APIs: you are a client too

When YOU call the gateway to create the payment (before the webhook):

• Explicit timeout (connect+read, e.g. 3s/10s): the library default of 60s+ = your worker hanging.
• Retries with backoff only for idempotents: GET yes; payment POST with the provider's Idempotency-Key (whenever they offer it) — so the retry doesn't charge twice (14).
• Circuit breaker (preview of 54): if the gateway has 30 failures, cut it off and fail fast instead of piling up timeouts.
• Never secrets in logs (05) and per-environment credentials (27).

4. Your outgoing webhook (when others consume you)

TicketFlow will notify organizers ("you sold 50 tickets"): the same discipline in reverse — HMAC signature of the body + timestamp, retries with backoff (5 attempts, 1m/5m/30m/2h/6h), idempotency with event_id, and a failed-deliveries panel (an outgoing-webhook health endpoint: last delivery, code, scheduled retry). An outgoing webhook without retries is a promise you don't keep.

5. Incoming vs outgoing webhooks: the asymmetry that rules

Two houses live in TicketFlow and are not governed the same:

The detail connecting with 25/30: outgoing webhooks are NOT sent directly by the service — they are written to the outbox and a poller publishes them (the same mechanism as domain events): the transactional guarantee ("we only notify what really happened") is the same for both uses. And the other side's receiver of our webhooks deserves the same generosity we demand: retries with backoff, a verifiable signature and a documented window — the badly designed webhook is the one forcing the other side to poll.

6. Incoming endpoint security: the fortress checklist

The endpoint receiving webhooks is public by definition (the gateway must reach it). The full armor: (1) HMAC verification ALWAYS (raw body, per-provider key, Lesson 27's dual rotation); (2) anti-replay window (5 min) + guard for out-of-window timestamps; (3) dedup by the provider's event_id (30: at-least-once IS the norm); (4) immediate 200 response + work in queue (the ACK is not the work: it's the receipt); (5) per-origin rate limit (23: the gateway doesn't need 1000 rps; an attacker does); (6) the secret never in logs (23: the Authorization header with the HMAC leaks in proxy logs: filter it in 45); (7) invalid-signature monitoring (46: spikes of invalid signatures = someone forging). With all seven, TicketFlow's incoming webhook is as auditable as the rest of the system.

1. Why is the signature verified over the raw body and not over the already-parsed request.data?
2. What double duty does the signed timestamp serve (anti-replay), and why 5 minutes?
3. Why ACK before processing, and what exactly happens if you process slowly without a queue?
4. The webhook arrived out of order (succeeded before pending): which component of your model absorbs the disorder?
5. Your outgoing webhook failed 5 times for an organizer. What's next? And if the organizer's URL now returns 410?

Continue with the exercises. The solutions only after trying it yourself.