Henry Hoang

Distributed Transactions & SAGA Pattern - Slides

Instructor slide content for Unit 4: managing distributed transactions across microservices using choreography and orchestration SAGA patterns

SLIDE DECK: MODULE 04 - SAGA PATTERN - V2.0 (PRODUCTION-READY)

Total Duration: 360 minutes (180 min Part 1 + 180 min Part 2) Audience: Fresher/Employee (Completed Module 03: Async & RabbitMQ)

Slide 1: Title Page

  • Content:
    • (Company / Training Unit Logo)
    • MODULE 04: SAGA PATTERN
    • Managing Distributed Transactions
    • "How do you 'Rollback' in a Microservice World?"
    • Trainer: (Your Name)
    • Date: (Training Date)
  • Visualization:
    • Key visual: A chain of 3 "dominoes" falling.
    • Next to it, a "reversed" image: A "domino" (Compensating Transaction) is triggered to push back the fallen dominoes, representing a "rollback."
  • Instructor Script:
    • "Welcome to Module 04. In Module 03, we learned how to send one message safely and reliably. But that was just one step."
    • "In the real world, a single business operation (like 'Place Order') requires 3, 4, or 5 services to work together. What happens if the third service fails? How do we 'rollback' the work that services 1 and 2 already did?"
    • "Today, we are solving the hardest problem in microservices: Distributed Transactions. We will learn a pattern called SAGA to ensure our business operations 'either succeed all together, or fail all together'."

Slide 2: Session Agenda (Updated)

  • Content:
    • AGENDA (PART 1 & 2)
    • P1. The Problem: The "Distributed Transaction" Disaster
    • P2. Theory: Why 2PC Fails & Welcome "BASE vs. ACID"
    • P3. Intro to SAGA Pattern & State Machines [Upgraded]
    • P4. [Part 1] Pattern #1: Choreography
    • P5. [Part 2] Pattern #2: Orchestration
    • P6. Comparison: Choreography vs. Orchestration
    • P7. [Part 2] The "Hard Parts" (Production-Ready) [Upgraded]
      • Correlation (Saga ID) & Headers
      • Transactional Outbox & Idempotency
      • Timeouts, Retries & DLQ
    • P8. Implementation (Python) & Lab Prep (Assignment 04) [Upgraded]
  • Visualization:
    • An 8-step timeline. Parts P4 (Choreography), P5 (Orchestration), and P7 (Hard Parts) are highlighted as the three core sections.
  • Instructor Script:
    • "This is our upgraded agenda, covering both Part 1 and Part 2."
    • "We'll go from the 'disaster' (P1) to the 'philosophy' (P2) and the 'solutions' (P3-P6) as before."
    • "But [EMPHASIZE] the key difference today is P7: 'The Hard Parts.' These are the 4 critical 'potholes' that 90% of SAGA demos online ignore: How do you link messages? How do you avoid double-processing? How do you not lose a message on publish? And how do you handle timeouts? This is the 'production-ready' part that I (as the SME) demand."
    • "Finally, we'll review the upgraded sample code and the new rubric for Assignment 04."

Slide 3: Learning Objectives (Updated)

  • Content:
    • OBJECTIVES (AFTER THIS MODULE, YOU WILL BE ABLE TO...)
    • 1. Explain: The problem of distributed transactions and the difference between ACID vs. BASE philosophies.
    • 2. Design:
      • A SAGA State Machine.
      • The corresponding Compensating Transactions (rollback actions).
    • 3. Implement: A SAGA Choreography (Happy/Failure path) using RabbitMQ.
    • 4. Apply: Critical "production-ready" techniques:
      • Correlation ID (for tracking).
      • Idempotent Consumers (to prevent duplicates).
      • Transactional Outbox (to prevent 'dual-write' failures).
  • Visualization:
    • 4 icons: [Icon: Brain (Explain)], [Icon: State Machine Diagram (Design)], [Icon: Happy/Failure Path (Implement)], [Icon: Shield/Anvil (Apply)].
  • Instructor Script:
    • "Our objectives have been upgraded. You won't just 'implement'; you must be able to 'design' the state machine and 'apply' the 3 'immortality' techniques for SAGAs: Correlation ID, Idempotency, and the Outbox Pattern. This is the requirement to pass Assignment 04."

Slide 4: Module 03 Recap

  • Content:
    • RECAP: WHERE ARE WE?
    • Module 03: We learned to send ONE message (Event) safely.
      • User Service (Producer) -> [RabbitMQ] -> Email Service (Consumer)
    • We Mastered:
      • durable=True, delivery_mode=2 (No message loss on restart)
      • auto_ack=False, basic_ack (No message loss on consumer crash)
      • DLX/DLQ (No message loss on code error)
    • NEW QUESTION: "How do we chain MULTIPLE messages into a single business workflow?"
  • Visualization: *
    • Simple diagram: [Producer] -> [RabbitMQ] -> [Consumer] with the "safety" checklists we learned.
    • Below is the "New Question" with an image: [Svc A] -> ? -> [Svc B] -> ? -> [Svc C]
  • Instructor Script:
    • "Let's remember Module 03. We learned how to 'send one letter' (message) and ensure it's 'immortal' (durable, persistent) and 'processed' (manual ack, DLQ)."
    • "But that was a single operation. Now, the harder problem: How do we coordinate a business operation that requires 3 letters? Service A sends letter 1, Service B receives it, processes, and must send letter 2. Service C receives, processes... How do we manage this entire chain?"

Slide 5: Section Intro - P1 & P2 (Understanding the Problem)

  • Content:
    • PART 1 & 2
    • UNDERSTANDING THE PROBLEM
    • In this section, we will cover:
      • The "Disaster" Scenario: Distributed Transactions
      • Why Traditional Solutions Fail (2PC)
      • Philosophy Shift: ACID vs. BASE
      • Eventual Consistency & SLA
  • Visualization:
    • A section divider slide with icon: [Icon: Warning/Disaster]
    • Background showing broken chain links symbolizing failed distributed transactions
    • Clean layout with the section topics listed
  • Instructor Script:
    • "Before we dive into solutions, we must first understand the problem deeply."
    • "In Parts 1 and 2, we'll explore the 'disaster scenario' that happens when distributed transactions fail, why old solutions like 2PC don't work in microservices, and the fundamental philosophy shift from ACID to BASE."
    • "Understanding this foundation is critical. You can't design a proper SAGA if you don't understand why we need it in the first place."

Slide 6: P1 - The Problem: The "Distributed Transaction" Disaster

  • Content:
    • THE "DISASTER" SCENARIO: PLACING AN ORDER
    • A PlaceOrder operation requires 3 services to work:
      1. Order Service: Create Order, status PENDING. (Local DB Transaction 1 - OK)
      2. Payment Service: Charge the credit card. (Local DB Transaction 2 - OK)
      3. Inventory Service: Deduct stock. (Local DB Transaction 3 - FAILED! (Out of stock))
    • RESULT = CORRUPTED SYSTEM (INCONSISTENT STATE)
      • Order is PENDING.
      • The customer WAS CHARGED.
      • No product was shipped.
    • This is a failed Distributed Transaction. We must "rollback" (compensate) Transaction 2 (refund the money).
  • Visualization: *
    • A 3-step diagram:
      1. [Order Service (DB)] -> [Icon: Check (OK)]
      2. [Payment Service (DB)] -> [Icon: Check (OK)]
      3. [Inventory Service (DB)] -> [Icon: Cross (FAILED)]
    • Below this is an [Icon: Angry customer] with "Card Charged!" and "No Product!" speech bubbles.
  • Instructor Script:
    • "This is the 'disaster'. The classic scenario. A user places an order."
    • "Service 1 (Order) creates a Pending order, OK. Service 2 (Payment) charges the card, OK. Service 3 (Inventory) checks the stock... and 'OUT OF STOCK.' Fail!"
    • "What's the result? The customer was charged, but gets no product. Our system is now in an 'inconsistent' state. This is the most critical failure in system design. We just stole from our customer."
    • "We must have a way to 'undo' step 2 (refund the customer). But Service 2 already 'committed' its transaction. How?"

Slide 7: P2 - Theory: ACID vs. BASE (UPGRADED)

  • Content:
    • Why not use old "Distributed Transactions" (2PC)?
    • Two-Phase Commit (2PC): A "coordinator" locks all databases (DB1, DB2, DB3). Asks "All ready?". If all OK -> "Commit."
    • Why it fails in Microservices:
      1. Blocking & Slow: It's synchronous. It 'locks' resources. Violates 'Performance'.
      2. Tight Coupling: Services are 'locked' together. Violates 'Independence'.
    • WE ABANDON "ACID", WE EMBRACE "BASE"
    • ACID (Atomicity, Consistency, Isolation, Durability):
      • The philosophy of Monoliths & Relational DBs. Very Consistent but Tightly Coupled.
    • BASE (Basically Available, Soft state, Eventual consistency):
      • The philosophy of Microservices.
      • Basically Available: The system is always available (no 'locks').
      • Soft State: We accept a "temporary" inconsistent state (Order is PENDING).
      • Eventual Consistency: The system will eventually become consistent.
      • UPGRADE: "Eventual" does not mean "whenever." It is an SLA (Service Level Agreement). e.g., "Order consistency will be final under 10 seconds in 99.9% of cases."
  • Visualization: *
    • A comparison diagram:
      • ACID: An [Icon: Steel Block (Rigid)] - Instantly Consistent.
      • BASE: A [Icon: Chain Link (Flexible)] - Eventually Consistent.
    • Add text: "Eventual Consistency = SLA (e.g., < 10s)".
  • Instructor Script:
    • "...(Keep 2PC part)..."
    • "We accept a new philosophy. We abandon 'ACID' and embrace 'BASE'."
    • "ACID (from Monoliths) says: 'I must be consistent RIGHT NOW.' BASE (for Microservices) says: 'I accept that the order might be 'Pending' for 5 seconds (Soft state), as long as it eventually becomes either 'Complete' or 'Failed' and the money is refunded. I trade instant consistency for performance and resilience'."
    • "[UPGRADE] And 'eventually' doesn't mean 'someday.' It's an SLA! As an SME, you must define it: is 'eventual' 5 seconds, 10 seconds, or 1 minute? This is a 'metric' you must measure."

Slide 8: P3 - Intro to SAGA Pattern & State Machine (UPGRADED)

  • Content:
    • P3: INTRO TO SAGA PATTERN
    • A SAGA is the mechanism to achieve "Eventual Consistency" (BASE).
    • Definition: A SAGA is a sequence of "Local Transactions."
    • Step 1: Design the State Machine
      • Before you code, you must draw the SAGA's states.
      • Example (Order SAGA):
        • PENDING -> (on OrderCreated)
        • AWAITING_PAYMENT -> (on PaymentSuccess)
        • AWAITING_INVENTORY -> (on InventoryFailed)
        • AWAITING_REFUND -> (on RefundSuccess)
        • FAILED
        • COMPLETED
    • Step 2: (Advanced) Saga Log
      • You should have a DB table or stream (Kafka) to log every step of the SAGA (sagaId, step, eventIn, eventOut, result) to replay/recover.
  • Visualization: *
    • A clear state machine diagram for the Order:
      • START -> PENDING
      • PENDING --(OrderCreated)--> AWAITING_PAYMENT
      • AWAITING_PAYMENT --(PaymentSuccess)--> AWAITING_INVENTORY
      • AWAITING_INVENTORY --(InventorySuccess)--> COMPLETED
      • AWAITING_INVENTORY --(InventoryFailed)--> AWAITING_REFUND
      • AWAITING_PAYMENT --(PaymentFailed)--> FAILED
      • AWAITING_REFUND --(RefundSuccess)--> FAILED
  • Instructor Script:
    • "And the 'way' we achieve BASE is the SAGA Pattern."
    • "[UPGRADE] Before you write a single line of SAGA code, you must do Step 1: Draw the State Machine. [Point to diagram] You must define all possible states: Pending, Awaiting_Payment, Awaiting_Refund, Failed, Completed."
    • "If you can't draw this, you will get lost in your code. Every complex SAGA starts with a simple state diagram."

Slide 9: Section Intro - P4 to P6 (SAGA Patterns)

  • Content:
    • PART 4, 5 & 6
    • SAGA PATTERNS: TWO APPROACHES
    • In this section, we will cover:
      • Pattern #1: Choreography (Event-Driven Dance)
      • Happy Path & Failure Path
      • Compensating Transactions
      • Pattern #2: Orchestration (Centralized Conductor)
      • Head-to-Head Comparison
  • Visualization:
    • A section divider slide with two icons side by side:
      • Left: [Icon: Dancing figures] - Choreography
      • Right: [Icon: Conductor/Orchestra] - Orchestration
    • Clean layout showing both approaches visually
  • Instructor Script:
    • "Now that we understand the problem and the BASE philosophy, it's time to learn the solutions."
    • "In the next three parts, we'll explore the two main patterns for implementing SAGAs: Choreography and Orchestration."
    • "Choreography is like a dance crew where everyone knows their moves. Orchestration is like an orchestra with a conductor. Both have their strengths and weaknesses, and we'll learn when to use each one."

Slide 10: P4 - Pattern #1: Choreography (Happy Path)

  • Content:
    • PATTERN #1: CHOREOGRAPHY
    • Philosophy: "There is no boss. Each service knows what to do."
    • Services "listen" for each other's events and "react."
    • "HAPPY PATH" SCENARIO:
      1. Order Service: (Creates Order) -> Publishes order.created event
      2. Payment Service: (Listens for order.created) -> (Charges card) -> Publishes payment.succeeded event
      3. Inventory Service: (Listens for payment.succeeded) -> (Deducts stock) -> Publishes inventory.succeeded event
      4. Order Service: (Listens for inventory.succeeded) -> (Updates Order: COMPLETED)
  • Visualization: *
    • An "event circle," with Routing Keys on the arrows:
    • [Order] --(order.created)--> [Payment]
    • [Payment] --(payment.succeeded)--> [Inventory]
    • [Inventory] --(inventory.succeeded)--> [Order]
  • Instructor Script:
    • "The first way to do a SAGA is 'Choreography.' Imagine a dance crew. There's no 'manager'; each dancer (service) 'listens' to the 'music' (event) and knows their next 'move' (action)."
    • "This is the 'Happy Path.' [Point to diagram] Order publishes order.created. Payment 'hears' it, charges the card, and publishes payment.succeeded. Inventory 'hears' that, deducts stock, and publishes inventory.succeeded. Finally, Order 'hears' that, and updates itself to COMPLETED."
    • "It's beautiful, elegant, and has no single point of failure (no boss)."

Slide 11: P5 - The Problem: Failure Path

  • Content:
    • BUT... WHAT IF THE "DANCE" FAILS?
    • "FAILURE PATH" SCENARIO:
      1. Order Service: Publishes order.created
      2. Payment Service: Publishes payment.succeeded (MONEY IS TAKEN!)
      3. Inventory Service: (Listens for payment.succeeded) -> (Checks stock... OUT OF STOCK!) -> Publishes inventory.failed event
    • SYSTEM IS "STUCK" (INCONSISTENT):
      • Money is taken, but there is no stock. The SAGA is "stuck."
      • We need a "reverse dance."
  • Visualization: *
    • Similar diagram, but step 3 fails:
    • [Order] --(order.created)--> [Payment] --(payment.succeeded)--> [Inventory]
    • [Inventory] --(inventory.failed - RED)--> ???
  • Instructor Script:
    • "But this is where choreography gets tricky. [Point to diagram] Steps 1 and 2 are OK. The money is taken."
    • "Step 3, Inventory is out of stock. It publishes a 'failure' event: inventory.failed."
    • "Now what? The money is gone, but there's no product. The Payment Service has no idea; it already 'danced' its part. We need a 'compensation' mechanism."

Slide 12: P5 - Solution: Compensating Transactions

  • Content:
    • SOLUTION: COMPENSATING TRANSACTIONS
    • Definition: A "local transaction" whose purpose is to undo the effect of a previous successful transaction.
    • Principles:
      • Easy: CreateUser -> Compensate: DeleteUser
      • Easy: DeductStock -> Compensate: AddStockBack
      • Hard: ChargeCard -> Compensate: RefundCard (A new transaction)
      • Very Hard: SendEmail -> Compensate: SendApologyEmail (Cannot be truly undone)
    • In a SAGA: A service must listen for "failure" events and trigger its own compensating transaction.
  • Visualization: *
    • A 2-column comparison table:
      TransactionCompensating Transaction
      ReserveStockReleaseStock
      ChargeCreditCardRefundCreditCard
      UpdateStatus('Pending')UpdateStatus('Failed')
      SendEmail (Non-compensable)SendApologyEmail (A corrective action)
  • Instructor Script:
    • "This is the 'undo button' for a SAGA. A 'Compensating Transaction'."
    • "[Point to table] Every 'forward' action (Transaction) must have a 'backward' action (Compensation). Deduct stock -> must have 'add stock back.' Charge card -> must have 'refund card'."
    • "Sending an email is the hardest. You can't 'un-send' an email. You can only 'send an apology email.' This is why 'non-compensable' steps (like sending email) must be pushed to the very end of the SAGA, when you are 100% sure it has succeeded."

Slide 13: P5 - Choreography (Failure Path) - Fixed

  • Content:
    • THE "COMPENSATION DANCE"
    • FIXED SCENARIO:
      1. ...
      2. Inventory Service: Publishes inventory.failed (Red)
      3. Payment Service: (Listens for inventory.failed) -> Triggers Compensation (RefundCard) -> Publishes payment.refunded (Red)
      4. Order Service: (Listens for inventory.failed OR payment.refunded) -> (Updates Order: FAILED)
    • RESULT = EVENTUAL CONSISTENCY
      • After a few seconds, the system "heals itself":
      • Order status: FAILED
      • Money: Was refunded.
      • Stock: Was not changed.
      • -> The system is consistent again.
  • Visualization: *
    • The complete "dance" diagram:
    • [Order] --(OK)--> [Payment] --(OK)--> [Inventory]
    • [Inventory] --(inventory.failed)--> [Payment] --(payment.refunded)--> [Order]
    • The "backward" arrows (in red) show the compensation chain.
  • Instructor Script:
    • "And this is the 'reverse dance.' [Point to diagram] Step 3, Inventory publishes inventory.failed."
    • "Step 4, [EMPHASIZE] Payment Service must also listen for this failure event. It 'hears' inventory.failed, understands the SAGA is broken. It triggers its own compensating transaction, RefundCard(). When done, it publishes payment.refunded."
    • "Step 5, Order Service 'hears' the failure, and updates the Order to FAILED. The system has 'healed itself.' It took 5 seconds, but eventually it became consistent. This is the BASE philosophy."

Slide 14: P6 - Pattern #2: Orchestration

  • Content:
    • PATTERN #2: ORCHESTRATION
    • Philosophy: "There is a 'Boss' (Orchestrator). Other services are 'workers' and only do as commanded."
    • We create a new service (e.g., OrderOrchestrator).
    • "HAPPY PATH" SCENARIO:
      1. Client calls OrderOrchestrator.
      2. Orchestrator -> Sends Command cmd.payment.charge to Payment Service.
      3. Payment Service (Processes) -> Sends Reply Event evt.payment.succeeded back to Orchestrator.
      4. Orchestrator (Gets Reply) -> Sends Command cmd.inventory.deduct to Inventory Service.
      5. Inventory Service (Processes) -> Sends Reply Event evt.inventory.succeeded back to Orchestrator.
      6. Orchestrator (Gets Reply) -> Updates Order COMPLETED.
  • Visualization: *
    • A "Star" topology diagram:
    • [Orchestrator] is in the middle.
    • Arrow (Command) from Orchestrator -> [Payment]
    • Arrow (Reply) from [Payment] -> Orchestrator
    • Arrow (Command) from Orchestrator -> [Inventory]
    • Arrow (Reply) from [Inventory] -> Orchestrator
    • Note: Payment and Inventory do not know about each other.
  • Instructor Script:
    • "Pattern 1 (Choreography) is nice, but when your SAGA has 10 steps, it becomes a 'spaghetti' of events. Very hard to debug."
    • "The alternative is 'Orchestration.' We create a new 'boss,' the OrderOrchestrator."
    • "[Point to diagram] This 'boss' will 'command' each service via a private channel (e.g., cmd.payment.charge). Payment finishes, it 'replies' to the 'boss' (e.g., evt.payment.succeeded). The 'boss' gets the reply, issues the next 'command'..."
    • "All the SAGA 'logic' is centralized in the 'boss.' The 'worker' services (Payment, Inventory) become very 'dumb'—they just do as they're told and report back. They don't need to know who their 'coworkers' are."

Slide 15: P6 - Orchestration (Failure Path)

  • Content:
    • HOW THE "ORCHESTRATOR" HANDLES FAILURE
    • "FAILURE PATH" SCENARIO:
      1. ...
      2. Orchestrator -> Sends command cmd.inventory.deduct.
      3. Inventory Service (Processes) -> Sends Reply evt.inventory.failed (Red) back to Orchestrator.
      4. Orchestrator (Gets 'Failed' Reply):
        • It knows the SAGA has failed.
        • It issues a compensating command cmd.payment.refund to Payment Service.
      5. Payment Service (Processes) -> Sends Reply evt.payment.refunded.
      6. Orchestrator -> Updates Order FAILED.
  • Visualization: *
    • The same "Star" topology.
    • The Reply arrow from Inventory is evt.inventory.failed (Red).
    • The Orchestrator (after receiving failure) shoots a new "Command" arrow (Red) cmd.payment.refund to the Payment Service.
  • Instructor Script:
    • "And this is why Orchestration is 'easier to understand.' When a failure happens..."
    • "[Point to diagram] Inventory 'reports' to the 'boss': 'Sir, out of stock!'"
    • "The 'boss' (Orchestrator) is the only one who holds the 'big picture.' It knows, 'Ah, I already charged the card in the previous step.' It immediately issues a 'compensating command': 'Payment service, issue a refund, now!'."
    • "The failure-handling logic is completely centralized in the 'boss.' Much easier to read and debug."

Slide 16: P7 - Comparison: Choreography vs. Orchestration

  • Content:
    • HEAD-TO-HEAD: "DANCE" VS. "CONDUCTOR"
    • (A detailed comparison table)
      CriteriaChoreographyOrchestration
      Philosophy"Events & Reactions""Commands & Replies"
      SAGA LogicDistributed in all servicesCentralized in Orchestrator
      Service CouplingIndirect (Loose) (Knows Events)Very Loose (Knows only Orchestrator)
      ProsSimple (for 2-3 steps), No SPOF*Easy to understand, Easy to debug, Easy to add/remove steps
      ConsHard to debug (Where did my event go?), "Spaghetti" EventsNew service to build/monitor, "Anemic" worker services
      When to use?Simple SAGAs, few steps, need for speedComplex SAGAs (>3-4 steps), need for clarity
  • Visualization:
    • A clear, 2-column comparison table filling the slide.
  • Instructor Script:
    • "So, which one do we choose?"
    • "Choreography is very nice, very 'pure' microservices. But if your SAGA has 10 steps, you will go 'insane' debugging it. You have no idea which service is listening to which event."
    • "Orchestration is more 'pragmatic.' All the logic is in one place, easy to read, easy to modify. But, you have to build and maintain that extra 'boss' service."
    • "My rule: 2-3 step SAGA? Use Choreography. 4+ steps? Think about Orchestration. In Assignment 4, we will use Choreography."

Slide 17: Section Intro - P7 (Production-Ready Hard Parts)

  • Content:
    • PART 7
    • PRODUCTION-READY: THE HARD PARTS
    • In this section, we will cover:
      • Hard Part #1: Correlation ID & Message Headers
      • Hard Part #2: Transactional Outbox & Idempotency
      • Hard Part #3: Timeouts, Retries & DLQ
      • Why 90% of SAGA demos fail in production
  • Visualization:
    • A section divider slide with icon: [Icon: Shield/Anvil/Fortress]
    • Background showing armor or defensive elements symbolizing production-ready system
    • Three main topics highlighted as critical pillars
  • Instructor Script:
    • "Congratulations, you now understand both SAGA patterns. But here's the truth: what you've learned so far is 'demo code.'"
    • "In this section, we tackle the 'Hard Parts'—the four critical problems that 90% of online SAGA tutorials ignore."
    • "These are not optional. Without solving these four problems—Correlation, Outbox, Idempotency, and Timeouts—your SAGA will fail in production. This is what separates a demo from a real production system."

Slide 18: P7 (Continued) - "Hard Part" #1: Correlation & Headers (NEW)

  • Content:
    • "HARD PART" #1: CORRELATION
    • Problem: When Payment Service gets 1000 OrderCreated events, how does it know which event belongs to which SAGA?
    • Solution: Correlation ID (or Saga ID).
    • Rule:
      1. The first step of the SAGA (Order Service) must generate a unique sagaId (e.g., UUID()).
      2. This sagaId must be passed in the Message Headers of EVERY event related to that SAGA.
    • Standard Message Headers: [Upgraded]
      • sagaId: The ID of this specific SAGA instance.
      • messageId: The unique ID of this specific message (for deduplication).
      • causationId: The ID of the "parent" message that caused this one (for debugging).
      • eventType: "payment.succeeded"
      • eventVersion: 1.0
  • Visualization: *
    • An image "dissecting" a message packet:
      • Headers (Metadata): sagaId: "abc-123", messageId: "xyz-789"
      • Body (Payload): { "orderId": 1, "total": 100 }
  • Instructor Script:
    • "Now we get to the 'hard parts.' Problem 1: 'Ghost Messages'."
    • "Without a 'Correlation ID,' your system is chaos. Imagine you're the Payment Service, you get 1000 OrderCreated and 500 InventoryFailed events. Which message belongs to which order?"
    • "The solution is a sagaId. The first service (Order) generates an ID (abc-123) and stuffs it into the message 'Header'."
    • "EVERY other service, when it receives this message and publishes a new event, must copy that sagaId into its new message. Now, we can 'stitch together' all events belonging to the same SAGA."
    • "We also need messageId (for deduplication) and eventType + eventVersion (for contract management)."

Slide 19: P7 (Continued) - "Hard Part" #2: Outbox & Idempotency (NEW)

  • Content:
    • "HARD PART" #2: "DUAL WRITE" & DUPLICATE PROCESSING
    • 1. The "Dual Write" Problem:
      • Order Service try...
        1. db.save(order) (OK)
        1. rabbit.publish(event) (FAILED! - RabbitMQ is down)
      • Result: Order is saved, but the event is lost. The SAGA never starts.
    • Solution: Transactional Outbox Pattern
        1. Inside ONE Local Transaction (ACID):
        1. db.save(order)
        1. db.save(event_message) (Save the message to an OUTBOX table in the same DB)
        1. db.commit()
        1. A separate "Relay" process reads the OUTBOX table and actually sends the message (guaranteeing at-least-once).
    • 2. The "Duplicate Delivery" Problem:
      • Because of "At-least-once," a Consumer might get the same message twice.
    • Solution: Idempotent Consumer
      • The Consumer must be "idempotent": processing the same message 5 times is the same as processing it 1 time.
      • How: Save the messageId (from Header) into a PROCESSED_MESSAGES table (with a UNIQUE index). If INSERT fails (duplicate) -> Ignore and ack() the message.
  • Visualization: * *
    • Diagram 1 (Outbox): [Service] -> [DB (Order + Outbox)] (1 Transaction). A separate [Relay] service reads [Outbox] and sends -> [RabbitMQ].
    • Diagram 2 (Idempotent): [RabbitMQ] -> [Consumer] -> Check [Processed DB] -> Process -> Save [Processed DB].
  • Instructor Script:
    • "Hard Part #2. What happens if you save to the DB successfully, but the publish to RabbitMQ fails? Your SAGA 'dies before it's born'."
    • "The solution is the 'Transactional Outbox.' [Point to Diagram 1] You don't call RabbitMQ directly. Instead, you 'save' the message into your same database with the Order, in the same transaction. Commit once, save both. Then, a separate 'bot' (Relay) reads this 'Outbox' table and sends the message. Now it's 'immortal'; if sending fails, it will retry."
    • "The reverse is 'Idempotency.' Because RabbitMQ is 'at-least-once,' you will get duplicates. Your code must be 'immune.' [Point to Diagram 2] The easiest way: Get the messageId (which we added in the last slide), and save it to a table. If saving fails (unique key violation) -> it means you've processed it. Ignore it, ack, and move on."

Slide 20: P7 (Continued) - "Hard Part" #3: Timeouts & Retries (NEW)

  • Content:
    • "HARD PART" #3: TIMEOUTS & TEMPORARY ERRORS
    • The "Infinite Wait" Problem:
      • Order Service publishes OrderCreated.
      • Payment Service gets the message, but it... crashes and never publishes PaymentSuccess or PaymentFailed.
      • Result: The SAGA is "stuck" in state AWAITING_PAYMENT forever.
    • Solution: SAGA Timeout (Watchdog)
      • When the Orchestrator (or Service) starts the SAGA, it also sends a "delayed" message (using RabbitMQ TTL + DLX) to itself.
      • Example: Send message CheckPaymentTimeout (delay 20 seconds).
      • If 20s pass, the "delayed" message is "activated."
      • The service receives its own CheckPaymentTimeout message. It checks the DB, sees SAGA is still AWAITING_PAYMENT -> Triggers Compensation (Rollback).
    • Solution: "Temporary Errors" (Retries):
      • If a Consumer fails (e.g., OptimisticLockException), it can nack(requeue=True) to retry.
      • Danger: Beware of "Poison Messages" (infinite loop).
      • Better: Use Exponential Backoff (Retry after 1s, 5s, 30s) and then send to DLQ.
  • Visualization: *
    • Diagram:
      1. [Service] -> (Start SAGA) -> [DB (PENDING)]
      2. [Service] -> (Publish Command)
      3. [Service] -> (Publish DELAYED Message 'TimeoutCheck' 20s)
      4. (If Reply arrives before TimeoutCheck -> OK)
      5. (If TimeoutCheck arrives before Reply -> Trigger Compensation)
  • Instructor Script:
    • "Final 'hard part': What if the Payment Service just 'dies'? Your SAGA is 'stuck' forever."
    • "The solution is a 'Timeout.' [Point to diagram] When the SAGA starts, it also 'sets an alarm' (by sending a 20-second delayed message to itself)."
    • "If 20 seconds pass and it hasn't heard a 'reply' (PaymentSuccess), it will receive its own 'alarm' message. It 'wakes up,' sees the SAGA is still 'waiting' -> It knows the SAGA has timed out and triggers a rollback."
    • "Similarly, when retrying, don't 'requeue' immediately; that creates a loop. Use 'backoff' (retry after 1s, 5s, 30s) and finally, give up and send it to the DLQ (from Module 3)."

Slide 21: Section Intro - P8 (Implementation & Lab Prep)

  • Content:
    • PART 8
    • IMPLEMENTATION & LAB PREPARATION
    • In this section, we will cover:
      • Python/Pika Implementation Patterns
      • "Consumer is Producer" Architecture
      • Assignment 04 Requirements & Rubric
      • Grading Criteria: Happy Path, Failure Path, Reliability
  • Visualization:
    • A section divider slide with icon: [Icon: Code/Terminal/Lab Flask]
    • Background showing code editor or development environment
    • Emphasis on practical implementation
  • Instructor Script:
    • "We've covered all the theory and patterns. Now it's time to see how this works in practice."
    • "In this final section, we'll look at the actual Python implementation using Pika, and I'll explain the requirements for Assignment 04."
    • "This is where everything comes together—you'll build a complete SAGA choreography with both Happy and Failure paths, implementing all the production-ready techniques we've discussed."

Slide 22: P8 - Implementation (Python/Pika) (UPGRADED)

  • Content:

    • IMPLEMENTATION: "CONSUMER IS A PRODUCER" (UPGRADED)
    • SAGA Choreography = Module 3 Skills + "The Hard Parts".
    • A service will be both a Consumer (listens to previous event) and a Producer (publishes next event).
    • payment_service.py (Upgraded pseudo-code):
    import pika, json, uuid
from idempotency_store import has_processed # (Mock service)

# Both a Consumer (listens for OrderCreated)
def on_order_created_callback(ch, method, props, body):
    msg = json.loads(body)
    headers = props.headers
    saga_id = headers['sagaId']
    message_id = headers['messageId']

    # UPGRADE: Idempotency Check
    if has_processed(message_id):
        ch.basic_ack(method.delivery_tag)
        return # Already processed, just ignore

    try:
        process_payment(msg['order_id']) # Local Transaction

        # UPGRADE: Pass Headers
        new_headers = {'sagaId': saga_id, 'messageId': str(uuid.uuid4()), 'causationId': message_id}
        event = {'order_id': msg['order_id'], 'payment_id': 'xyz'}

        # (This should use Outbox Pattern, this is an example)
        publish_event(exchange='saga_exchange',
                      routing_key='payment.succeeded',
                      body=event, headers=new_headers)

        ch.basic_ack(method.delivery_tag)

    except Exception as e:
        # (Publish 'payment.failed' event with headers)
        ch.basic_nack(method.delivery_tag, requeue=False) # Send to DLX

  • Visualization:

    • A code block, highlighting the "UPGRADES":
      1. Reading headers (sagaId, messageId).
      2. Calling has_processed(message_id) (Idempotency).
      3. Creating new_headers (Correlation).

  • Instructor Script:

    • "This is the 'upgraded' pseudo-code for Payment Service. [Point to code]"
    • "First, it reads the headers to get the sagaId and messageId."
    • "Next, the 'Idempotency Check': 'Have I processed this messageId before?' If yes, ack it and return immediately."
    • "If not, it processes the payment. Then, it transforms into a Producer."
    • "When it publishes, it creates new_headers, and copies the sagaId over. This is 'Correlation.'"
    • "(Ideally, the 'process_payment' and 'publish_event' steps should be wrapped in a 'Transactional Outbox' as we discussed on Slide 16)."

Slide 23: P8 - Lab Prep: Assignment 04 (UPGRADED)

  • Content:
    • IMPLEMENTATION LAB (ASSIGNMENT 04)
    • Mission: Build Order Service (Task 3) and complete the SAGA (Happy/Failure).
    • Provided:
      • docker-compose.yml (RabbitMQ + Mgmt UI).
      • "Failure Switch" (ENV): INVENTORY_FAIL_RATE=1.0 (100% fail) to test Failure Path.
    • GRADING RUBRIC (UPGRADED):
    • A. Architecture & Flow (40%)
      • [ ] Choreography Happy Path works (Order → Payment → Inventory → Order).
      • [ ] Failure Path: Inventory fail (using Switch) -> Triggers Payment refund -> Order status FAILED.
    • B. Reliability & Safety (35%)
      • [ ] Inherits Module 3: All Exchanges/Queues durable, messages persistent, consumers manual ack + prefetch + DLQ.
      • [ ] SAGA ID: sagaId is generated and passed in all message headers.
      • [ ] Idempotency: Consumers have a mechanism to prevent duplicate processing (e.g., check messageId).
    • C. Observability (15%)
      • [ ] All service logs must print the sagaId and orderId (for debugging).
      • [ ] (Bonus) Propagate traceId (OTel) with sagaId -> Prepares for Module 5.
    • D. Code Quality (10%)
      • [ ] Clear separation of business logic, SAGA logic, and RabbitMQ logic (publisher/handler).
  • Visualization: *
    • The lab's architecture diagram, showing both flows (Happy in green, Failure in red).
    • A detailed checklist (Rubric A, B, C, D) takes up half the slide.
  • Instructor Script:
    • "This is Assignment 04, the 'capstone' project for SAGA. You will code both flows."
    • "I will provide a 'Failure Switch' to simulate errors. Your job is to turn it on and prove your system 'heals itself' (issues the refund)."
    • "And this is the new Rubric. [Point to Rubric] I will be checking Section B (Reliability) very closely. I want to see sagaId in your logs. I want to see your 'Idempotency' code. Just making the 'Happy Path' work is not enough to pass."

Slide 24: Case Study #1 - E-commerce Order SAGA (Real-World)

  • Content:
    • REAL-WORLD: E-COMMERCE ORDER PROCESSING
    • Scenario: A customer places an order on an e-commerce platform (like Amazon/Shopee).
    • 5 Services Involved:
      1. Order Service: Creates order (status: PENDING)
      2. Payment Service: Charges credit card → PAYMENT_AUTHORIZED
      3. Inventory Service: Reserves stock → STOCK_RESERVED
      4. Shipping Service: Creates shipment → SHIPMENT_CREATED
      5. Notification Service: Sends confirmation email (non-compensable)
    • SAGA Flow (Choreography):
      • Each service publishes events: OrderCreated → PaymentAuthorized → StockReserved → ShipmentCreated → OrderCompleted
    • Failure Scenario: Shipping service fails (no available courier).
      • Compensation Chain:
        • ShippingFailed → Inventory releases stock
        • → Payment refunds customer
        • → Order status = FAILED
        • → Notification sends apology email
    • Key Lessons Learned:
      • ✅ Non-compensable steps (email) placed at the end.
      • ✅ Each service maintains its own "SAGA state" table (for recovery).
      • ✅ All events carry sagaId (for tracing across 5 services).
      • ✅ Timeout: 60-second timeout for the entire SAGA (if any service hangs).
  • Visualization:
    • A flowchart showing 5 services in sequence (Happy Path in green arrows)
    • Below it, the "Compensation Path" in red arrows (failure at Shipping → propagates backward)
    • Highlight: Email is at the end (marked as "Non-Compensable")
  • Instructor Script:
    • "Let's look at a real-world example: e-commerce order processing."
    • "[Point to diagram] This is a 5-step SAGA. Order → Payment → Inventory → Shipping → Notification."
    • "The tricky part: What if the Shipping service fails? Maybe no couriers are available."
    • "[Point to red arrows] The compensation chain kicks in. Shipping publishes ShippingFailed. Inventory 'hears' it, releases the stock. Payment 'hears' it, refunds the money. Order updates status to FAILED. Finally, Notification sends an apology email."
    • "Notice: The email step is last. You can't 'unsend' an email, so we only send it after we're 100% sure the order succeeded."
    • "This is how real companies like Amazon handle millions of orders per day."

Slide 25: Case Study #2 - Banking Money Transfer SAGA

  • Content:
    • REAL-WORLD: CROSS-BANK MONEY TRANSFER
    • Scenario: Transfer $100 from Bank A (Account X) to Bank B (Account Y).
    • 3 Services Involved:
      1. Transfer Service: Creates transfer request (status: PENDING)
      2. Bank A Service: Debits $100 from Account X → DEBITED
      3. Bank B Service: Credits $100 to Account Y → CREDITED
    • SAGA Flow (Orchestration - Better for Financial Compliance):
      • TransferOrchestrator controls the flow:
        • Step 1: Send command DebitAccount to Bank A → Wait for reply DebitSuccess
        • Step 2: Send command CreditAccount to Bank B → Wait for reply CreditSuccess
        • Step 3: Update Transfer status to COMPLETED
    • Failure Scenario: Bank B's system is down (cannot credit).
      • Compensation:
        • Orchestrator receives CreditFailed from Bank B.
        • Orchestrator sends compensating command RefundAccount to Bank A (credit $100 back to Account X).
        • Transfer status = FAILED
    • Key Lessons Learned:
      • ✅ Orchestration used (not Choreography) because:
        • Regulatory compliance requires centralized audit trail.
        • Easier to implement complex retry logic (e.g., retry Bank B 3 times before giving up).
      • ✅ Transactional Outbox critical: Bank A must save "Debit" and "Debit Event" in the same DB transaction.
      • ✅ Idempotency critical: If Bank B receives CreditAccount command twice (due to network retry), it must process only once.
      • ✅ SLA: Transfer must complete or fail within 30 seconds (regulatory requirement).
  • Visualization:
    • A "Star" topology: TransferOrchestrator in the center, sending commands to Bank A and Bank B
    • Show both Happy Path (green) and Failure Path (red, with Refund command)
    • Add icon: [Icon: Bank/Finance/Shield] - Emphasize compliance
  • Instructor Script:
    • "Another critical use case: banking money transfer."
    • "[Point to diagram] This is a 3-step SAGA, but it uses Orchestration, not Choreography. Why?"
    • "Because in finance, compliance and auditability are everything. The regulator wants to see a centralized log: 'Who authorized this transfer? What was the exact sequence of events?'"
    • "With Orchestration, the TransferOrchestrator has the full picture. It logs every command, every reply, every failure."
    • "[Point to Failure Path] If Bank B's system is down, the Orchestrator knows immediately. It sends a compensating command to Bank A: 'Hey, refund that $100 back to Account X.'"
    • "And notice the SLA: 30 seconds. If the transfer doesn't complete in 30 seconds, it's automatically rolled back. This is a regulatory requirement in many countries."
    • "This is how real banks handle billions of dollars every day without losing a single cent."

Slide 26: Best Practices & Common Pitfalls

  • Content:
    • SAGA BEST PRACTICES
    • ✅ DO:
      1. Design the State Machine First: Before writing code, draw all states (Pending, Awaiting, Failed, Completed).
      2. Use Correlation ID (sagaId): Every message must carry sagaId for tracing.
      3. Implement Idempotency: Every Consumer must be idempotent (check messageId before processing).
      4. Use Transactional Outbox: Never publish events directly; save them in the DB first.
      5. Set Timeouts: Every SAGA step must have a timeout (e.g., 20 seconds). Use "delayed messages" (TTL + DLX).
      6. Log Everything: Log sagaId, orderId, step, eventType in every log statement (for debugging).
      7. Push Non-Compensable Steps to the End: (e.g., sending email, SMS) - only execute after 100% success.
      8. Test Failure Paths: Don't just test Happy Path. Use "Failure Switches" (like INVENTORY_FAIL_RATE=1.0) to simulate failures.
    • ❌ DON'T:
      1. Long-Running SAGAs: Don't create SAGAs with 10+ steps. Break them into smaller SAGAs or use Orchestration.
      2. Missing Compensating Transactions: Every forward action must have a backward compensation (or be non-compensable).
      3. Ignoring Idempotency: Without idempotency, duplicate messages will corrupt your data (double-charging, double-refund).
      4. Dual-Write: Never do db.save() + rabbit.publish() separately. Use Outbox Pattern.
      5. No Timeouts: Without timeouts, a failed service will cause your SAGA to "hang" forever.
      6. No Correlation ID: Without sagaId, you cannot debug failures in production (you won't know which messages belong to which SAGA).
    • PRODUCTION CHECKLIST:
      • [ ] State Machine documented?
      • [ ] All messages have sagaId and messageId?
      • [ ] Idempotency implemented (check messageId table)?
      • [ ] Transactional Outbox implemented?
      • [ ] Timeouts configured for all SAGA steps?
      • [ ] Compensating Transactions tested (failure path)?
      • [ ] Non-compensable steps pushed to the end?
      • [ ] Logs include sagaId for tracing?
  • Visualization:
    • A 2-column layout:
      • Left: ✅ DO (Green checkmarks, Best Practices)
      • Right: ❌ DON'T (Red X marks, Common Pitfalls)
    • Bottom: Production Checklist (with checkboxes)
  • Instructor Script:
    • "Before we wrap up, let's talk about Best Practices and the mistakes I see every single time in production."
    • "[Point to DO list] These 8 practices are not optional. They are the difference between a SAGA that works in production and a SAGA that causes outages at 3 AM."
    • "Number 1: Design the state machine before you code. I've seen teams skip this, and they end up with 'spaghetti SAGAs' that no one can debug."
    • "Number 3: Idempotency. I cannot emphasize this enough. Without it, you will double-charge customers. I've seen it happen."
    • "[Point to DON'T list] And these are the 'land mines' you must avoid."
    • "Number 4: Dual-Write. This is the most common mistake. Developers do db.save() and then rabbit.publish(). If the publish fails, your SAGA is 'dead on arrival.' Use the Outbox Pattern."
    • "[Point to Checklist] This is your production checklist. Before you deploy, go through this list. If you check all 8 boxes, your SAGA is production-ready."

Slide 27: Decision Framework - When to Use SAGA?

  • Content:
    • DECISION FRAMEWORK: SAGA vs. ALTERNATIVES
    • 1. When to Use SAGA?
      • ✅ You have multiple microservices that need to work together for a single business operation.
      • ✅ You cannot use a single database transaction (because services have separate DBs).
      • ✅ You can tolerate "eventual consistency" (a few seconds of delay is OK).
      • ✅ You can define compensating transactions for every step.
    • 2. When NOT to Use SAGA? (Use Alternatives)
      • ❌ Use 2PC (Two-Phase Commit) if:
        • You are in a monolith or tightly-coupled system (all services share one DB).
        • You must have instant consistency (no delay allowed).
        • (But remember: 2PC doesn't scale well in distributed systems.)
      • ❌ Use Event Sourcing if:
        • You need a full audit trail of every state change (e.g., financial systems, blockchain).
        • You need to "replay" events to rebuild state.
        • (But Event Sourcing is more complex than SAGA.)
      • ❌ Just Use Local Transactions if:
        • Your operation involves only one service (no need for SAGA).
    • 3. Choreography vs. Orchestration?
      • Use Choreography if:
        • ✅ SAGA has 2-3 steps (simple flow).
        • ✅ Services are independent and loosely coupled.
        • ✅ No strict compliance/audit requirements.
      • Use Orchestration if:
        • ✅ SAGA has 4+ steps (complex flow).
        • ✅ You need centralized control and easy debugging.
        • ✅ You have regulatory requirements (e.g., banking, healthcare).
    • 4. Scalability & Maintenance Considerations:
      • Choreography: Scales horizontally very well (no SPOF*), but hard to debug/maintain (events fly everywhere).
      • Orchestration: Easier to understand/maintain, but the Orchestrator becomes a bottleneck (must scale it carefully).
      • Rule of Thumb: Start with Choreography for simple SAGAs. Move to Orchestration when complexity grows.
  • Visualization:
    • A decision tree flowchart:
      • "Multiple services?" → No → "Use Local TX" / Yes → "Can define compensations?" → No → "Consider 2PC/Event Sourcing" / Yes → "Use SAGA"
      • Below SAGA: "Simple (2-3 steps)?" → Yes → "Choreography" / No (4+ steps) → "Orchestration"
    • Icons: [Icon: Decision tree, Scales, Complexity meter]
  • Instructor Script:
    • "Finally, let's talk about when to use SAGA and when not to."
    • "[Point to decision tree] Start here: Do you have multiple services? If no, just use a local transaction. No need for SAGA."
    • "If yes, ask: Can you define compensating transactions? If you can't (e.g., sending physical letters, irreversible actions), then SAGA won't work. You might need 2PC or Event Sourcing."
    • "If yes, use SAGA. Now, Choreography or Orchestration?"
    • "[Point to lower part of tree] If your SAGA is simple (2-3 steps), Choreography is beautiful. But if it's complex (4+ steps, many failure paths), Orchestration will save your sanity."
    • "And remember the trade-off: Choreography scales better but is harder to debug. Orchestration is easier to understand but you must scale the Orchestrator carefully."
    • "In Assignment 04, you'll use Choreography because it's a 3-step SAGA. But in your real projects, if you find yourself with 10 events flying around, it's time to consider Orchestration."

Slide 28: Module 4 Summary (UPGRADED)

  • Content:
    • MODULE 4 SUMMARY
      1. Ditch ACID/2PC; Embrace BASE (Eventual Consistency = SLA).
      1. SAGA is the pattern to achieve BASE (using Local TX + Compensation TX).
      1. Choreography (decentralized events) vs. Orchestration (centralized commands).
      1. A "Production-Ready" SAGA MUST HAVE:
      • State Machine: (To design).
      • Correlation ID: (To debug & trace).
      • Transactional Outbox: (To prevent lost events on publish).
      • Idempotent Consumers: (To prevent duplicate processing).
      • Timeouts & DLQ: (To prevent infinite errors/waits).
  • Visualization:
    • Summary bullet points. Item #4 (Production-Ready) is highlighted in red.
  • Instructor Script:
    • "To summarize Module 4. We ditch ACID, we embrace BASE. We use SAGA."
    • "But a 'demo' SAGA and a 'production' SAGA are two different things. A 'production-ready' SAGA [Point to item 4] must solve all 4 problems: State, Correlation, Outbox, and Idempotency. Miss one of these, and your system will fail at scale."

Slide 29: Q&A

  • Content:
    • Q & A
    • Questions & Answers
  • Visualization:
    • A clean, minimal slide. Just the large letters "Q&A".
  • Instructor Script:
    • "This is the most complex topic in the course. I am sure there are many questions about 'how to guarantee,' 'what if this fails,' etc. Please, ask away."

Slide 30: Thank You & Next Module

  • Content:
    • THANK YOU!
    • (Your Contact Info: Email, LinkedIn, etc.)
    • COMING UP (MODULE 6):
    • Module 6: Performance (Redis Caching)
    • Our system is now 'Safe' (SAGA), how do we make it 'Fast' (Caching)?
  • Visualization:
    • A "teaser" for Module 6.
    • The Redis logo.
  • Instructor Script:
    • "Thank you. You have just completed the hardest part of this course: making a distributed system 'safe' and 'consistent' (SAGA)."
    • "But 'safe' isn't enough. Users demand 'fast.' In our next module (Module 6), we will learn how to make our system 100x faster using Caching with Redis. See you then."

Asynchronous Communication with RabbitMQ - Slides

Instructor slide content for Unit 3: building production-ready async messaging with RabbitMQ, including durable queues and manual ACKs

Performance Patterns with Redis Caching - Slides

Instructor slide content for Unit 5: implementing Cache-Aside, Read-Through, and Write-Through patterns with Redis

On this page

SLIDE DECK: MODULE 04 - SAGA PATTERN - V2.0 (PRODUCTION-READY)