Four steps for describing cloud anchitectures on a whiteboard

2 04 2015

If you work at a company that owns a lot of cloud services, it’s inevitable that you’ll have some high level meetings with other teams regarding the architecture of their systems. Usually people start drawing boxes while saying, “we own this service called Foo which talks to this service called Bar, which then talks to Pineapple and….”.


Five minutes later, everyone is lost. The diagram and the descriptions made while drawing it transfer nothing except the knowledge that they own webservices, which have names (probably unrelated to what they do), and at some undefined point in time communicate something with eachother. Everyone spends the rest of the meeting checking their email in between squinting at the whiteboard and wondering when they can go back to writing code.

To try and make describing a distributed system easier, I ask four questions.

1. What are the entities?

Your system does stuff with entities (SKUs, Reports, Documents, Processing Requests, Marketplaces, Orders, Resources…). Telling me all about your service that processes SKU Merger Requests is going to be Chinese to me unless you start with telling me what the heck a SKU Merger Request is. Systems deal with entities that are usually abstractions around business concepts and client functionality. Start with a non-technical explanation of what the system does, why it exists in the first place, and most importantly what are the entities it deals with.

2. Where is the state/persistence?

There are stateless services and services with state. If the service has state, where is it persisted? Oracle databases? DynamoDb? Elastic search clusters? Encoded on top of the Quantom superposition of hydrogen atoms? I don’t care much about the details of how things are encoded in the persistence layer (SQL, JSON, text files…). I just care about where on the diagram of distributed stuff the state of the system resides.

3. What triggers synchronous chains of events?

Drawing arrows between service boxes doesn’t represent when those arrows are exercised. In distributed systems, there are lots of triggers which start some chain of synchronous events. These triggers should be described. For example, a user on a webpage submits an order. This is a trigger to some chain of events. Perhaps that chain of events will end with a message being put on a queue. An asynchronous agent pops the message off that queue, which begins another chain of synchronous events, that perhaps ends with a database call. The database happens to have a post-commit trigger which starts another chain of events. You get the picture; describe the triggers and make sure you draw distinctions between different chains of events.

4. The functional bits: APIs, inputs, and outputs.

Finally, start breaking down the details of the interactions between system components. All software takes input and produces output. This can be hard to draw on a whiteboard, but can be described while drawing at least.


The symbols to draw on the whiteboard? It doesn’t matter. People are adaptable. Jeff can draw his chains of events using different colored markers, Bob can draw his chains of events using squiggly lines, and Jessica can draw hers with numeric labels. The whiteboard is just temporary storage to sketch out systems that would otherwise escape short term memory before they can be fully understood. It’s a scaffolding that must be filled in with descriptions and discussions during the drawing process.

Just remember to hit all of the high level talking points. What entities does your system deal with? Where are they stored? What triggers their mutation? What are the inputs and outputs to your APIs? This is usually enough to sketch out the high level workings of complex systems filled with workflows, asynchronous queues, distributed state, and  all sorts of common cloud computing patterns. Really the only thing missing is conditionals, and my answer that? Don’t draw a bunch of crazy conditional paths on the same diagram. Make one diagram follow a single path through code and draw different diagrams for the alternative flows when authorization is denied, processing is cancelled, or whatever other branching and exception cases you’re trying to communicate.

Deep Thoughts from a Lone Traveler

31 12 2013

In front of me is the beach of Ventura, California. The sun is slowly setting to my right, somewhat by design, as I commanded my aching legs to briskly walk here again before the sky separated itself into beautiful colors one last time before I travel home. I was compelled to write, and sitting here watching a dog run up and down the sand in front of the sunset seemed the place to maximize inspiration.

The locals are perplexed about why I’m here. I’ve given up trying to explain it, because in the process I realized I don’t really know myself. “Nothing happens in Ventura. You should have kept driving North to Santa Barbara”, they tell me. “You should have went to Vegas”, a man named Matt tells me at a local dive bar. “Very random”, a girl at the beach tells me, as I play with her dog and she waits for someone she knows to finish surfing. I wanted a simple vacation. Biking, hiking, beaches and bars. A little town in California that I fell in love with when I spent a night here seemed the perfect place. I don’t know why I fell in love with it; love is like that, you can’t help who you fall in love with. The locals are perplexed nonetheless.

The first night I ran across a man playing the Star Wars imperial march on a guitar and a homeless beggar with a sign saying, “bad advice: $1”. I found a bar where they ask, “boot or barrel?”, and give you a glass shaped like a boot or a bigger one shaped like a barrel. I eventually ended up at a tiny dive bar with chandeliers made out of bras and only a single beer on tap. I like terrible bars. People bring their girlfriends to nice bars. They bring their families, they go with coworkers after work, they bring their friends. People don’t bring their friends to terrible bars. That’s what makes them the best. You can go there alone, and strike up conversations with the other patrons around you, also alone, not just going to a bar to talk to the same old people but with a different view.

I feel as if every day I’ve been here I’ve had a realization. Answered a question not by logical thinking, but more an emergence of an idea as my mind wandered without direction, the reigns removed for a few days to give it the freedom to try and figure out life, instead of being commanded to focus it’s attention on some specific problem. My body hasn’t had as much of a vacation. I can’t help but feel it a separate entity, as it seems to have a will of it’s own, wanting to move slower, to stop sooner, to rest, but I compel it to carry on anyway. In the last two days I’ve told it to hike 8 miles across an island in the pacific, bike untold distances across hilly terrain, carry my curious mind to the shops and museums of downtown, and drag my laptop to this beach where I sit.

And I feel compelled to write. The words starting to form in my mind before they had anything to spill into. I wonder if this is why great poets like Walt Whitman, men with soft hearts and hard bodies, speak so highly of the open road. Allons, the road is before us!

The first day’s realization was had walking down the beach. I was planning to take it easy that day, in preparation for a trek across an island the next, but before you know it I had walked for hours along the beach collecting pockets full of sea shells. I like walking to the end of places. The beach is seemingly never ending. Seeing others at the beach gave me the slight desire to stop and talk to someone. I eventually ran across a red headed girl in jeans and a sweater, sitting high up on the sand alone. It was on my way back and she had been there for a while. As I walked up, I hesitated a bit, trying to think of what to say, and then trying to muster the courage to say it. I envisioned myself bolding sitting beside her and saying, “Hello. You look like you could use some company. Or maybe I’m just saying that because I could.”. I was nearly done hesitating, but the moment I decided to walk up, she got up and started making her way back toward the houses above the beach. I couldn’t help thinking, what would have happened if I had come back 10 seconds earlier? I would have met someone new. A local to the area, someone independent that enjoyed walking down to the beach and watching the sunset alone. I would have asked her about good places to go in the area, places to eat, maybe if I was feeling bold enough I’d ask her to have dinner with me. Maybe she would have said no, or maybe we’d just talk for a while and I’d never ask, but the conversation alone would surely have changed the events of my day. Maybe it could have changed the course of my life.

So I had the first realization. I’ve had it before, maybe it’s common sense, but in that moment I saw it with more clarity than ever before. The entire course of my life could have changed if I had been somewhere 10 seconds earlier or if I didn’t hesitate. All of our lives can be traced back to events that could have happened entirely different if you had been somewhere at just a slightly different time. As I walked back up the beach, my mind searched for these moments and thought about how things would have been different, or at least realized they wouldn’t have been like they were.

I once walked into a bar called Mr. G’s and got a drink when I was only a bit over 21 years old. If I had sat two seats further down the bar, I never would have met a man that bought me a drink and told me about being an aircraft mechanic, a traveler, and a cyclist. If I had sat two seats away from where I did, I wouldn’t have had such a great conversation that I felt like going back to that bar. I wouldn’t’ have become a regular, learned how to play pool, met some of my neighbors, and felt such loss when that little bar closed down. I wouldn’t have started to develop the confidence to walk into a bar alone and be able to meet new people. I never would have made a lot of friends that I did. If I had sat two seats down at the bar that day, it’s possible I would have never taken a liking to bars at all, and traveling alone would have been a bit more dull. It’s quite likely I wouldn’t be sitting at this beach feeling the cool ocean breeze, listening to the sound of the water crashing against the sand, and feeling the night creep upon the beach with its ever so cold fingers.

I wouldn’t have the career I do now if it wasn’t for random chance that, as a ham radio operator, I turned on my radio late one night when I was supposed to be sleeping and heard some people joking about morse code. I wouldn’t have found out that the people I heard actually had a youth ham radio club not too far from my house. I wouldn’t have met a bunch of intelligent electrical engineers, software engineers, and people that become roll models. I wouldn’t have had anyone to ask how to make a webpage, and anyone to tell me to learn HTML to do it. I wouldn’t have had anyone to ask how to program, and I wouldn’t have been told by a random friend to learn the obscure language TCL. I later wouldn’t have gotten an internship at Emerson Network Power doing TCL scripting. If I hadn’t turned on a radio that one night, I probably wouldn’t have become a Software Engineer. And if my grandfather hadn’t showed me all of his equipment one day, I never would have become a ham radio operator. I need to thank him for that one day. I can really trace everything in my career back to a conversation I had with him when I was 12 years old, and I can’t possibly imagine how much life would be different if that hadn’t happen. It wasn’t a deep profound conversation either, it was just a random event, the everyday kind of thing you don’t realize will be root of a branching tree that will change the course of your life forever.

The sun has set now, the light nearly entirely gone. The temperature is dropping, my body starts having a will of it’s own again, urging me to find shelter, find heat, to stop commanding my slowing fingers to keep typing on this cold beach. I need to buy a better jacket.

The second day I had a different realization, my mind satisfied for a while with answering the question of how I got here, done tracing events back to their sources. Off the coast of Ventura, over an hour by boat, there’s a chain of islands called the Channel Islands. The biggest of which is called Santa Cruz island, and there’s a road that runs 7+ miles from a place called Scorpion Beach to a place called Smugglers Cove on the other side of the island. I hiked across it, against the will of my complaining legs.

The cold got to me. I’ve retreated to the hotel lobby. Interrupted by bodily concerns mid story, how rude. I could keep the action and thought to myself, but this is a story that’s still happening. It’s a story of past and present intertwined, it’s a story of thoughts without purpose. The lobby is actually somewhat crowded. Three girls sit across the way; I’m tempted to ask them what they’re doing for New Year’s, they look like they’re going out soon. It’s about 6 hours until 2013 ends in this little town. No idea what I’m doing tonight. They have glow stick crowns. Where did they get glowsticks? Ah, they’re leaving. Another moment of hesitation. A world that could have been. A world maybe not better, but different. An older couple is asking if there will be fireworks tonight. It appears the answer is no. Nothing much happens in Ventura. The lobby is busy though, maybe some overheard conversation will give me some tidbit of information about what to do tonight.

Where was I? Ah yes, the second realization. After hiking around the island, I was on the boat ride back. I brought a book, but I found it too difficult to pull myself away from living in the moment and retreat to a world of fiction, however unremarkable the moment was.

I moved to the very front of the boat and stood with some others at the edge, holding onto the railing, the cold wind blowing hard against us, watching the texture of the ocean ahead. The ocean has a texture to it; I never knew that. I’m not sure what causes it. Some parts were smooth, some parts were rough. Some sections were darker than others. Some glistened like a perfect undulating puddle of liquid glass, other sections were filled with floating plants and algae. The texture would change quickly. You could see it coming, see the boundary, and then see where it went back to the old texture.

There are chips and salsa? I think the lobby might have beer too. Explains why it’s so crowded here. Some guys just walked past to their rooms with a cooler of beer. I guess they gave up on the idea of going out, nothing interesting happens in Ventura after all. A girl also walked by with a painting. An older couple just gave a girl a glowstick halo. Her boyfriend just came back, looked confused, and the old man said “did we de something wrong?”. The man replied, “Nah! If anything you did something amazingly right!”. The older couple laughed and asked if he wanted a glowstick too. Now we know where the glowsticks came from.

So there I was, standing on the edge of the boat, watching the texture of the ocean. I saw another man, alone, doing the same. Noticing him made me notice how few solo travelers there were on this boat. The boat had dozens of couples, families, groups of friends. Only a handful of people going alone. Maybe only the two of us. Why do so many people insist on doing everything with others? It seems woman are the worst. Most of them would never dream of going to a movie, going to a bar, or going on a road trip by themselves. I spent the rest of that day with the question in the back of my mind. I rejected ideas like people wanting someone near by to talk to or genuine concerns for safety. You don’t talk in movies and they’re quite safe, yet people still fear going to them alone. I think it runs deeper than fear of being different by being there alone too.

Eventually that night the answer came to me. I’m not sure if it’s the correct answer, but it seems to have a sufficient amount of explanatory power. I think we’re conditioned to be dependent from birth. We spend the first 18 years of our lives only doing things with family and friends. Do kids ever go to the movies alone? Go to dinner alone? Go to a party where they don’t know people? No, parents would never allow that, and it really just doesn’t come up to begin with. From a combination of parental fear and social tradition, we spent our childhood only doing things with family and family approved friends. Then we move out. We live on our own. We become independent. But most people still want to do things with others. To find friends to cling to, to find significant others to take with them. Maybe our entire culture of dating has actually come about from people not being independent enough to do anything on their own. Why should people be expected to drastically change the way they do things when they move out? It seems to explain the situation. It’s a hypothesis.

The third realization dawned on me today, when I picked a random direction and biked. No destination, just a direction, and eventually I had to give up on that direction and go somewhere else. There are mountains to the north I wanted to bike in, but I kept finding dead ends. One road north dead ended at a farm, another to a logging camp, and a third into a gated community that I nearly got stuck in. I tried a few codes at the gate, found out that 1234 worked, and went through. Then I realized it didn’t go anywhere interesting, and there was no keypad on the outside. The fence itself was the kind created from spear like sharpened pieces of metal. I found a way back eventually, but the ordeal turned me off on the direction of North. I decided South was a better idea.

At some point I stumbled across miles and miles of strawberries. Strawberries as far as the eye can see. Strawberries farms are actual rather bland scenery, especially after miles and miles of them. What world would have transpired if I had biked East? Too many variables to ever know, but I would’t be writing about strawberries.

A girl with a black cowboy hat walks past on her way to her room. I can’t help but stopping to smile. I think I like hotel hobbies. I never knew hotel hobbies were at all interesting. Imagine the world that would have been if I had brought a better jacket was still at the beach.

On my way back, I saw an old farmer on the side of the road selling strawberries and avocados on the side of the road I was biking on. I decide to stop and buy a box of strawberries. Best decision of the day. They were the best strawberries I’ve ever had in my life. Maybe that experience removed the stigma in my mind regarding eating food from some guy selling it on the street. Oh imagine the world that may now be! Maybe one day as an old man I’ll die from eating a bad breakfast burrito from a man on the side of the road, and it’ll all trace back to choosing to bike South instead of East today.

When we were sailing out to Santa Cruz island, we ended up getting there nearly an hour late, because we kept stopping to observe things. First, we stopped to see some sea lions sunning themselves on a big orange buoy floating out in the ocean. Second, we saw a huge flock of birds, and after heading in that direction we found that they had gathered because a huge pack of dolphins were hunting in the area. We slowed down and let the dolphins follow along in the wake of boat. Finally, when we had almost made it to the island, we came across the four grey whales swimming past. We stopped and floated for quite a while, watching them come to the surface for a few moments. They would come up every ten seconds or so three or four times, taking a breath each time, and then dive deeper, staying under for around five minutes.

The next day, sitting on the side of a road in a completely unremarkable location, eating a box of strawberries, I had my third realization. I was comparing the experience on the boat with the experience of seeing dolphins and whales at Sea World. The experience on the boat was amazing, and the strength of that experience was partly because of the serendipitous nature of it. There hadn’t been a whale spotting for days, and the fact we ran across four of them like that was somewhat rare on this particular boat trip. We live in an on-demand culture. We want entertainment now. We want to see a whale now. We want to hear a story now. We want to micromanage and plan our lives. The best experiences are the ones that were unplanned though. The best experiences are the ones where you can’t guarantee you’ll see something, or that something will happen, but you place yourself in a situation where something could happen, and if that something does, it’s so much better than the canned on-demand version. Oh serendipity! How so many of the good things in our lives can be traced back to it.

It’s 5 hours until the end of 2013 now. I’ve apparently been typing for an hour. Nothing terribly interesting is happening in the lobby now. Someone took the chips and salsa away. The impulse to write has subsided, the beast well fed for now. Maybe these realizations can be combined to form something more coherent later. Something about how the tiniest events can change the courses of our lives, how the best things in life are when the courses of our lives are changed not from some on-demand desire being fulfilled, but because of patience and putting ourselves into situations setup to make interesting things possible. And of how traveling alone, parental upbringings and social conditioning be damned, is the best kind of travel.

An older lady that works at the lobby restaurant asks if I’m okay, since I’ve been rather quiet. I smile and assure her I’m fine. As I get up to leave, she says, “no you don’t have to leave!”. I tell her that’s okay, I’m going to wander downtown to see if anything interesting is happening. Probably not, nothing interesting happens in Ventura. She asks me if I’ll be back at the lobby for breakfast tomorrow. I tell her yes, and she wishes me a safe night.

Unseen sacrifices of becoming more efficient

12 01 2012

Sometimes you feel like you have no time, and start trying to shave off minutes and hours here and there in order to save time. In the beginning of college, when I had far more free time and energy, I was a hardcore mountain biker that would bike 110 miles around Tuscon every November in less than 6 hours. Despite living nearly 20 miles from campus, I would actually grab my mountain bike and commute that way when the weather was nice. Eventually I felt that I didn’t have the free time to kill 2 hours a day biking, and basically fell out of the mountain bike scene. After that my mode of transportation turned strictly into public transportation: bus and light rail. I still ended up killing nearly 2 hours a day commuting, but now I could spend my time reading and doing homework on the bus. Doing homework on the bus is a terrible idea, as it usually requires internet access, multiple books in front of you, and possibly trying to scribble something on paper while the bus is hitting bumps in the road and making your chickens scratch degrade to a quality you can’t even read yourself. Instead, I spent the time reading science fiction, lot of it. I read my way through most of the Ender’s Game universe, a huge part of Asimov’s universe, and dozens of stand alone novels and trilogies. I would usually kill a paperback novel in less than 2 weeks, and I’d be in the library all the time queuing up more books for my commute.

Despite getting a ton of reading done, when I got busy with school and work the buses would be an endless source of frustration and near rage for me. I was wasting 2 hours a day, sometimes more if I’d miss buses or they’d end up late. Eventually when I had enough money saved up, this resulted in my moving closer to campus and buying a car. Did this make me more efficient? No, not really. Instead of waking up at 7:30am to make it somewhere by 9:00am, I’m likely to wake up at 8:30am and get some extra sleep. I didn’t magically gain 2 hours of time, and even if I did I probably waste those 2 hours most of the time doing something less productive than exercising or reading a good book. I actually miss how much reading I used to get done, because now, it ironically feels like I don’t have time to read a lot of the time (plus it’s hard to just get the motivation).

The moral of the story is, trying to become fixed to schedules and saving time in your life may have more consequences than you would think. The time spent taking a bus, getting groceries, or even watching TV might provide benefits besides the obvious. I actually sleep better when I watch TV for a while before going to bed. Otherwise my mind ends up racing and thinking about the day rather than calming down and going to sleep; and most of the thoughts are just repetitive and useless since you’re not getting any new input to that internal state machine laying in bed staring at the ceiling. Something that seems like a complete waste of time might be something you miss after you quit doing it, and for good reason.

Newegg Build Part 3: Software and Benchmarking

24 02 2011

Now for the tests! All the marketing fluff of the computer hardware aside, how does the new machine really perform? Freaking awesome, that’s how.

Windows Experience Index

Well, that’s what I get for not going for a solid state drive.

CPU Stress Temperature Testing

The DH-14 certainly does its job well, with idle temps hovering around 35C and peek temps around 48C.

Game Benchmarks: Starcraft 2

Plenty of performance to handle even the big 4v4 battles with every graphic setting at ultra with my 1440 x 900 monitor. Most 1v1 battles are over 60FPS, only dipping down to 30FPS in team games (2v2, 3v3, 4v4). Getting 70-100 FPS for macroing.

Benchmarking Program: Far Cry 2

Awesome performance on Far Cry 2, even on ultra settings it stays well above 60FPS.

Benchmarking Program: Heaven Benchmark

Cranked all the way up with Direct X 11, anti-aliasing at 8x, and 1440×900 resolution, handles quite well except a small glitch on stage 18 that makes it hang for a bit and ruins the min FPS measurement.

Newegg Desktop Build: Part 2

18 02 2011

It’s alive! After some slightly painful events during the construction process.

First off, I realized that I had accidentally gotten one of the SATA power cables pinned between the case and motherboard when I was installing it. Not a big problem, except that the motherboard was entirely installed and plugged in by that point. I decided it would be easier to back up, remove the video card, and unplug a lot of cables anyway to get the CPU cooler on, but in the process I realized why bottom mounted power supplies are often a bad idea. Gravity. Of course while removing the motherboard to unpin the cable, I dropped a screw down into the PSU, which then made me need to take the PSU out to turn it over and shake it out…

So, after a few irritating but minor setbacks, everything turned on and powered up fine. Enjoy some more hardware pics, next post I’ll move into the software and performance side.

The LGA1366 Socket with the protective covers removed.

I7-950 being installed.

The Noctua NH-D14 CPU Cooler.

Installing the heatsink for the NH-D14. A fairly easy process provided you have enough room in the case to get behind the motherboard to mount the back plate. It would have been best to install the cooler before putting the motherboard in, but it worked out.

Fans popped onto the heat sink, video card put back in, and first power on attempt!


Front view, case panels still missing while I do some initial tests.


Once you go dual monitors, you never go back, especially as a programmer.First bootup of Windows 7.



More pics from the panasonic ZS5

24 11 2010

Another round of pictures playing with the new camera. I left them at full size this time, though it always bothers me that high megapixel camera pictures actually look fuzzy and grainy when you view them at full size. The reason being full size is something like 4,000×3,000 pixels and when you shrink them down to a decently normal size like 1920×1080 they look great. Most browsers automatically shrink pictures to whatever your current window size is though, and most people also have high speed connections now, so no worries.


Maybe I’ll start a photo blog somewhere…

Memorial Union South Sign



Dual Geysers



Statue near art building


Computer Engineering Levels of Abstraction


Web Developers are too forgiving

13 11 2010

Computer language: a set of characters, symbols, and syntactic rules for their combination and use.


The purpose of a computer language is to clearly convey our instructions to the computer (or compiler) so that it may perform the operations we desire. There’s no room in computer languages for ambiguity, it must always work on any computer. An assembly program should be able to run on any processor that supports the instruction set architecture, and a higher level language should run on any interpreter or compiler that’s designed for that language’s specification. Much work is put in by language developers to ensure that ambiguous parse trees can never come about, and error handling is implemented for almost every processor, compiler, and interpreter.

Someone forgot to tell this to the writers of web browsers. Start tags? End tags? Proper tag nesting? Mostly optional and harmless. Who decided that HTML rules are really HTML suggestions?

It’s amazing how many HTML syntax rules you can ignore, break, and bend, but still have your webpage display more or less properly without the user viewing it knowing any better. The result of this is a huge collection of webpages on the internet that are ignoring HTML specifications. If these pages become popular enough, browser writers will continue to try and make them render even when they’re riddled with holes and depreciated features.

Unfortunately, it’s hard to solve this given the current situation of the web. The average user doesn’t want to see obscure errors about missing tags and depreciated features, and the average web developer doesn’t do anything to test his webpages but run them on a variety of browsers and see if they work. Web developers have no reason at all to learn about new features and proper style/syntax when their old pages will continue to work just fine in the state they’re already in.

Even worse than that, perfectly valid HTML will render differently on different browsers. Oh the fun of web development…