The Professor
Index Course index

Module 6 · Async processing and messaging

Lesson 32 — Eventual consistency and sagas

The payment, the reservation and the email: three systems, one coherent story.

Published
In this lesson
  1. Objectives
  2. 1. Strong where it matters, eventual where it doesn't
  3. 2. The purchase saga: orchestration over choreography
  4. 3. The remote boundary: yes, no and DON'T KNOW
  5. 4. Compensations: the undo that does exist
  6. 5. The complete diagram and the resulting guarantees
  7. Self-assessment

Stack: Django/DRF · Project: TicketFlow Status: Published — closing the async processing module Prerequisite: Lesson 31 — Scheduled tasks and batches

Objectives

  1. Distinguish strong consistency (local transaction, 10) from eventual, and define WHAT ends up consistent in TicketFlow.
  2. Orchestrate the purchase saga: payment + reservation + email with compensations when a remote step can no longer be undone with a rollback.
  3. Handle the remote boundary's three states (yes/no/don't-know) with timeouts and recognition by event.

1. Strong where it matters, eventual where it doesn't

The Postgres transaction (10) gives STRONG consistency inside one DB: reservation + seats + outbox are atomic. But the payment lives at the gateway and the email in SMTP: there is no rollback in other people's systems. The architecture decision is choosing what is strong and what is eventual: money and inventory are strong (local transaction + locks); notifications and reports are eventual (the email arrives 30 s later; the dashboard, 2 min). "Eventual" is not "wait and pray": it is a guarantee with defined convergence — if there is no permanent failure, the state converges (29-30's retries); if there is, the alert (46) triggers human repair.

The beginner's trap: making eventual the things the user waits for (availability at purchase time) or strong the impossible things (confirming the reservation in the SAME transaction that charges the gateway — two resources, no distributed transaction; that is why the saga exists).

2. The purchase saga: orchestration over choreography

A saga is the sequence of local transactions with compensations: every remote step has its undo (or its acceptance of the fact). Two styles: choreography (each service reacts to events, nobody commands) and orchestration (an orchestrator knows the story and sends commands). TicketFlow picks orchestration for the purchase (the story has states and legal deadlines: the user sees "your reservation will be confirmed shortly") and choreography for the periphery (analytics, informational emails react to 30's stream).

STATES:   PENDING → PAID → CONFIRMED
                     ↘ PAYMENT_FAILED → (compensation) → EXPIRED/CANCELLED

Orchestrator (a Django service, local transactions):
1. reserve (local TX: seats+outbox)                       [strong, 10]
2. charge at the gateway (remote command, idempotent intent.id, 14)
3. on receiving PaymentSucceeded (event, 25) → CONFIRMED + email + webhooks (on_commit, 29)
4. if PaymentFailed → compensate: free seats + "try another method" email

The orchestrator is NOT a microservice: it is a Django service with a state table (PurchaseSaga: saga_id, reservation, intent_id, state, attempts). Persistent, not in memory: the process can die and another one picks it up (the table is the truth, 55's stateless style).

3. The remote boundary: yes, no and DON'T KNOW

The remote step has three outcomes, not two: success, clear failure, and don't know (timeout: did it charge or not?). Handling the "don't know" is the difference between a grown-up backend and a support nightmare:

python
def ejecutar_paso(self, comando) -> Outcome:
    try:
        return self.gateway.cobrar(comando)           # SUCCEEDED | DECLINED
    except GatewayTimeout:
        # DON'T KNOW: do NOT compensate yet (it may have charged)
        return Outcome.UNKNOWN

UNKNOWN rules: (1) never compensate blind — freeing the seats of a reservation whose charge DID go through is giving away money and a seat; (2) ask: GET /charges/{intent.id} (the gateway answers the final state — that is why intent.id is idempotent, 14); (3) if it remains unknown, retry the query with backoff (30) until the maximum deadline, and only then compensate with an auditable record (08's ledger leaves the trail). The timeout is ALWAYS defined per operation (54 systematizes it): without a timeout, the "don't know" is eternal.

4. Compensations: the undo that does exist

The compensation doesn't erase the past: it adds an inverse fact. The charge isn't "deleted": it is REFUNDED (ledger: two entries, 08); the reservation isn't un-confirmed: it goes to CANCELLED with reason saga_compensated. Compensation rules:

  • Idempotent (they may run twice: the refund with the same intent/key doesn't duplicate, 14).
  • In reverse order of execution (if you charged and reserved, compensate by refunding and freeing — the stack).
  • Recorded as events (RefundIssued, ReservationExpired) so the peripheral consumers (emails, analytics) converge without orchestration (30).
  • With a deadline: a pending compensation is a visible state (compensation_pending) that the daily job (31) chases and alerts on if it ages.

5. The complete diagram and the resulting guarantees

                ┌────────────────────────────────────────────┐
                │  PurchaseSaga (table: state, attempts)     │
                └────────────────────────────────────────────┘
   command         │ event (stream, 30)           command
        ┌──────────┴──────────┐               ┌───────────┴──────────┐
   Gateway ──PaymentSucceeded──▶ CONFIRMED │  Gateway (charge)    │
        │        (webhook 17 / poll)         │  timeout → UNKNOWN   │
        ▼                                    │  → GET intent → fix  │
   PaymentFailed ──▶ compensate (refund + free + email)             │

Guarantees the system offers the business: reservation and seats are strong (10); the payment converges (the intent answers yes/no with a reconciliation query); emails/webhooks are eventual with at-least-once + dedup (29/30); the whole saga is observable (every transition is event + metric, 46) and resumable (the table survives restarts). What it does NOT offer: global atomicity between DB and gateway — nobody offers that; the saga is the honest answer.

Self-assessment

  1. What do you choose to make strong and what eventual in TicketFlow, and what does "defined convergence" mean for the eventual parts?
  2. Orchestration vs choreography: why is the purchase orchestrated and the emails choreographed?
  3. The remote boundary's three outcomes: what is NEVER done on UNKNOWN, and what is the correct sequence?
  4. Why is the compensation an inverse fact and not a deletion? List its 4 rules.
  5. What does the PurchaseSaga table guarantee that process memory doesn't, and what happens when the worker dies mid-saga?

Continue with the exercises. The solutions only after trying it yourself.

^ top