Amazon DynamoDB streams: Everything you need to know.
Last week in The Cloud Handbook, we talked about how docker works and containerized sample react applications with docker. In this week’s episode, let’s discuss DynamoDB streams. What is it? How is it useful? Patterns and use cases.
Introduction
If you are building serverless apps on AWS, chances are you will likely use the Amazon DynamoDB database. It integrates well with other AWS services like Lambda, API Gateway, etc.
Amazon DynamoDB is a fully managed, No-SQL, key-value store database that can achieve single-digit millisecond performance. It is used for some of the most critical services like Amazon dot com’s checkout services. It is well suited for OLTP applications where real-time data is used.
If you looked at Jeff Bar’s blog Prime Day by numbers from last year, DynamoDB peaked at 126M requests per second on Prime Day 2024.
One of my favorite features of DynamoDB is DynamoDB streams. In this issue, we discuss it in detail.
Let’s get started.
What are DynamoDB Streams?
Imagine this scenario…
Let’s say you have use cases where you need to send a notification as soon as there are some changes in the DynamoDB table. Or need some real-time data.
DDB has features that capture every change and make it available to consume. It is just a basic example, but you could do much with it.
DynamoDB Streams is a feature of DDB that captures data modifications in DynamoDB tables and makes them available for processing. The capture is a time-order sequence at the item level.
When you enable a stream on a table, DynamoDB captures information about every modification to data items in the table.
How does it work?
When enabled, DynamoDB can stream changes of data in its table and make it available to use through its web API.
It can capture item-level changes.
Here are the changes we can capture.
Key attributes only —Only the key attributes of the changed item.
New image—The entire item as it appears after it was changed.
Old image—The entire item as it appeared before it was changed.
New and old images—Both the new and old images of the changed item.
Two models of capturing Change data in DynamoDB
DynamoDB Streams
Amazon Kinesis Data Streams
Use Cases & Patterns
Let’s discuss some of the places where we can utilize DynamoDB streams.
1. Messaging/Notification System
DynamoDB Streams can trigger AWS Lambda functions, enabling real-time processing of changes. This is useful for tasks like updating views and sending notifications to users. It can be done by integrating AWS Lambda with other services like SNS, and SES.
2. Real-time Monitoring/Alerts
Streams can be used to monitor database activity in real-time, enabling alerts based on specific patterns or thresholds.
3. Event-Driven Architecture
As we already mentioned, DynamoDB Streams can trigger AWS Lambda functions, enabling real-time processing of changes. This is useful for tasks like updating views, notifications, or synchronizing data between systems.
4. Auditing and Logging
DynamoDB Streams can be used to keep a detailed audit log of all changes made to the database. This is important for compliance, debugging, and historical analysis. By capturing all changes, streams can help maintain versions of data, allowing rollback or reconstruction of past states.
5. Microservice communication
Decoupling Services: DynamoDB Streams can be used to decouple microservices by enabling event-driven communication. One service can update a DynamoDB table, and other services can react to those updates via streams.
6. Analytics
Changes captured by streams can be processed and sent to analytics platforms like Amazon Redshift or Elasticsearch for real-time analytics and monitoring.
How to use it?
Enabling the DynamoDB stream is easy. The easiest way is through the AWS Management Console. First, select the table you want to enable streams, and then on the ‘Export and Stream’ section you will find the enable option.
If you want to use Amazon Kinesis Stream.
Based on your requirements, you can choose what kind of data changes you want to capture from your table.
Once it’s turned on, you will be able to access it through the DynamoDB streams API.
Wrapping things up!
For further reading:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html
https://aws.amazon.com/blogs/database/dynamodb-streams-use-cases-and-design-patterns/
Over to you, what’s your experience? How are you using this feature? I would love to know.