The AWS data engineering exam, which I was studying for and have just passed, covers roughly 100 to 150 different technologies that interact with each other in different ways. Some of them are made by Amazon Web Services, while others are general technologies that have become something like the founding principles of working with data. As someone who loves maps and diagrams, studying any subject has always meant drawing a lot of diagrams for me, on paper or in diagramming tools. But today, with Claude Code and a good JavaScript imagination, I thought of building a small interactive platform for studying. Over the last two weeks I worked with Claude Code on Data Stack Atlas, a platform that covers hundreds of technologies: the AWS data tools, some general data engineering tools, and products from other companies such as Microsoft, as well as open-source tools. More importantly, it holds the patterns of connection between those technologies, architecture patterns, hundreds of scenarios for connecting them and changing their settings, and screens that show how different settings affect the price and the performance of each technology.
However useful hands-on practice in AWS is, as engineers we care, or should care, about the scientific and technical principle behind a technology, and that is what I tried to focus on. You hear and read about the difference between Glue and EMR in theory, and you know that EMR inherited its name from MapReduce, yet today it runs modern engines such as Spark, Flink or Trino. Then you wonder: what is left of the historical MapReduce? And why does AWS sell Spark here and there? Hands-on practice will make you understand it, and so will the “How it runs” diagrams in the atlas.
Every tool on one map
More than 550 tools and features are in the side menu on the left, with each feature shown as a branch of the main tool it belongs to. When you pick one, its full details come up in the panel on the right, and the canvas fills with every possible connection of the tool: lines for real-time connections and for batch transfers, as well as the connections that control tools (control), the ones that pass events on as they happen (events), and the ones used to reach a tool (access). The lines show these connections the way they should look, in an expressive way (and a beautiful one, I hope!).
Behind the atlas there are 800 documented connections, and each one carries a mark of one to three bars showing whether the connection is common or not, based on the scenarios and questions provided for the tool. Apart from the questions, scenarios and data that came from Amazon, every one of those connection lines was reviewed by Claude agents working in parallel to check that each line is correct. Each line also carries extra information: the file formats it can carry and the kind of encryption it uses, and each of these was a separate task for Claude Code. My own checking of all this happened while I studied, but even after all that I can’t guarantee it. Later, I also updated the direction of the connections, which used to be one-way only.
What a tool is not, and how it compares
It is easy to read about almost any technical tool, mostly from its marketing pages or its tutorials, or to try it out to learn what it does. But knowing what it doesn’t do, or what it isn’t meant for, is something exams do better. In the interest of technical honesty, I am not a user of Firehose or Kinesis Data Streams, and I don’t think these tools have many users, but having “Kinesis” in both names creates a lot of misunderstanding before you start using them and trying them out. This is where an important field I created in the atlas comes in, in the panel on the right, built from the scenarios in the tools’ documentation, from the questions and from the different architecture patterns. If you don’t plan to try these tools, you can learn from it that Athena doesn’t store data, that Firehose doesn’t need a job to be scheduled for it, and that it doesn’t move data in real time.
There are also comparisons with the tools closest in function, the ones that could be confused with each other, and the comparisons focus on four main dimensions: price, ease of getting started, the expertise needed, and the speed of the tool.
Architecture patterns
There are 77 architecture patterns, which I gathered first from the reference architecture patterns of AWS and Azure, and then from the scenarios around the exam and the ones mentioned in the tools’ documentation. When you choose a pattern, you can see all its tools on the canvas, with the data moving in its directions and in its own way of moving: real time, batch and so on.
How it runs: labs
When you press “How it runs”, you will find a variety of small labs in which data is drawn as particles moving quickly between the different parts of a system. These screens focus on the abstract side of how technologies work, as engineered tools made of different pieces that serve different purposes. You can see how memory fills up when you do a certain thing, or how the price or the run time grows with certain choices, and all of it is hypothetical, of course.
There are labs for Glue, EMR, Redshift, Athena, Lambda, Kinesis, Step Functions, EBS, Hadoop, SQL engines, NoSQL databases, Kafka and Flink, and more can still be added. This is one of the talents of AI that I don’t claim credit for: these tools were built in JavaScript, and I was only the person with the idea behind them, not the programmer, as with the rest of the atlas. The numbers, as I said, are hypothetical and illustrative, not literal. And the cases shown revolve mainly around the exam and its questions.
How it runs: engines step by step
Part of the labs is a set of interactive, step-by-step drawings for some technologies that are hard to picture when you only read about them. For Spark, for example, its diagram shows how a job starts, how it is planned, how the cluster manager manages the different executors and the tasks that read from the partitions, and it also shows memory growing and how storage is used.
Guides
Some topics about the tools seemed hard for me to remember, or I simply found that they were fertile material for a beautiful diagram. Sometimes making a diagram tempts me even more than the learning it is for. One example is the windows in Flink, my favourite example of this, which you can see below.
On the phone
Part of my studying for the exam happened while I was exercising: solving questions or reading things about the exam. And in general I use reels, in all their kinds and names, on YouTube, TikTok or Instagram, so why not have educational cards (reels)? If you open the site on your phone, you will find an option for it. But I admit this part went through my checking less than the other parts, since I built it on the last day before I passed the exam and didn’t use it much.
How I used it to study
After building the basic form of the atlas and adding the question scenarios to it, it became clear which parts weigh more in the exam, and that guided me on where to study and focus more. There is a selector at the top of the atlas where you can choose the exam; for now you will only find the Amazon exam there, and I may add more with future exams. The questions don’t only show what the tools are, but what they don’t do and what they are not: “this is not…”, “this tool doesn’t…” and so on.
The canvas also holds many explanations of security and networking features, and there are interactive panels dedicated to security and networking.
This tool includes many Amazon tools and links to Amazon’s documentation, but I built it completely independently, and it is not affiliated with Amazon.
What comes next
Maybe another exam, and maybe I will work on the tool more to turn it into a working design tool, not only a tool for studying for exams. At heart, it already helps with design.
You can try the tool on a computer or on a phone (with the cards): omarmeriwani.com/data-atlas
Comments