Event-driven design: API Gateway, SQS, SNS, EventBridge and Step Functions

Why decouple?

Imagine a food delivery app where the order API does everything itself, one step after another: save the order, charge the card, tell the restaurant, send an SMS, update the analytics. If the SMS provider is slow, the customer stares at a spinner. If analytics is down, the whole order fails. The parts are tightly coupled: each one depends on all the others being up and fast right now.

Decoupling means putting a message in between. The API saves the order, drops a message saying "order placed" into a queue, and answers the customer at once. Workers pick up the message when they are ready. If a worker crashes, the message waits and is tried again. Each part can be scaled, changed or fixed without touching the others.

API Gateway: the front door

Amazon API Gateway gives your backend a public HTTPS address, checks requests, and passes them to Lambda or another service. It comes in three types: REST APIs, HTTP APIs and WebSocket APIs.

SQS: queues

Amazon SQS (Simple Queue Service) stores messages until a consumer reads them. A producer sends a message; a consumer polls the queue, processes the message, and then deletes it. If the consumer never deletes it, SQS assumes the work failed.

• Standard queues: almost unlimited throughput, at-least-once delivery (a message can occasionally arrive twice) and best-effort order.
• FIFO queues: strict order within a message group and exactly-once processing within a deduplication window. The name must end in .fifo, and throughput is lower.

The visibility timeout is the key idea. When a consumer receives a message, SQS hides it from other consumers for that many seconds (30 by default, up to 12 hours). If the consumer deletes it in time, it is gone. If not, it becomes visible again and another consumer gets it. With Lambda reading the queue, AWS recommends a visibility timeout of at least six times the function timeout.

A message that keeps failing is called a poison message. A dead-letter queue (DLQ) catches it: in the source queue's redrive policy you set maxReceiveCount, say 5. After a message has been received 5 times without being deleted, SQS moves it to the DLQ, where a human can inspect it and later redrive it back.

{
  "VisibilityTimeout": "60",
  "MessageRetentionPeriod": "345600",
  "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:ap-south-1:111122223333:orders-dlq\",\"maxReceiveCount\":\"5\"}"
}

aws sqs create-queue --queue-name orders-dlq
aws sqs create-queue --queue-name orders --attributes file://orders-queue.json
aws sqs send-message --queue-url https://sqs.ap-south-1.amazonaws.com/111122223333/orders \
  --message-body '{"orderId":"A1001","city":"Pune"}'
aws sqs receive-message --queue-url https://sqs.ap-south-1.amazonaws.com/111122223333/orders \
  --wait-time-seconds 20

SNS: publish once, deliver to many

Amazon SNS (Simple Notification Service) is publish/subscribe. A publisher sends one message to a topic; SNS delivers a copy to every subscriber: SQS queues, Lambda functions, HTTPS endpoints, email or SMS. This is called fan-out. Subscribers can use filter policies to receive only the messages they care about. A common pattern is SNS in front of several SQS queues, so each subscriber gets its own durable copy.

EventBridge: an event router and scheduler

Amazon EventBridge has event buses. AWS services, your own apps and some SaaS partners put events on a bus. Rules match events by pattern and send them to targets such as Lambda, SQS or Step Functions. EventBridge Scheduler runs targets on a cron or rate schedule, with time zone support, which replaces many old "cron on a server" jobs.

{
  "source": ["foodapp.orders"],
  "detail-type": ["OrderPlaced"],
  "detail": { "city": ["Pune", "Mumbai"] }
}

Step Functions: workflows

AWS Step Functions runs a state machine: a workflow of steps written in JSON (Amazon States Language). Each step can call Lambda or another AWS service, with built-in retries, error catching, branches, waits and parallel steps. Standard workflows can run for up to a year; Express workflows are for short, high-volume runs of up to 5 minutes.

Putting it together: an order on a food delivery app

1. The app sends POST /orders to API Gateway, which calls a small Lambda function.
2. That function checks the order, saves it to DynamoDB, sends a message to the orders SQS queue, and answers 202 Accepted within a fraction of a second.
3. A worker Lambda reads the queue in batches, charges the payment and updates the order status.
4. When payment succeeds, the worker publishes OrderConfirmed to an SNS topic. Subscribers: a queue for the restaurant tablet service, a function that sends the SMS and push notification, and a queue for analytics.
5. A message that fails 5 times goes to the DLQ, and a CloudWatch alarm on the DLQ size tells the on-call engineer.