Adding Simplicity - An Engineering Mantra

Extending the Architectural Life

In my last article I introduced the concept of Architecture Shelf Life. I also put forth the postulate that for most patterns and technologies it is approximately 5 years. If you buy that as well as the premise that architectures can only survive 2-3 shelf life refreshes before they become dated and impractical, then you must be asking, how can I hedge against that. I want to extend the life of my architecture in some way.

Most critical for any architecture that is to survive is the ability to be refreshed with new technologies in an incremental fashion. The way this can be most readily achieved is by following standard architectural principles of good component design with the loosest possible coupling between components. This obviously allows components to be implemented independent of each other and also lets them follow their own technology curve for being replaced. Unfortunately though architectures rarely have good components that are truly loosely coupled. The reasons for this are the results of organic growth of the system. I have noticed some specific patterns emerge though.

Poor Protocols

This one is probably one of the most common. The components are reasonable and the interfaces have decent semantics. But then the implementation choice couples the components to a specific technology or implementation strategy. In most cases the decision is made based on the desire to optimize the interface speed. And in the majority of these cases it was probably an unnecessary optimization that bought a little speed but at the expense of long term architectural flexibility.

Interfaces between components should be text based (e.g. XML, JSON) as they offer the maximum flexibility for migrating components to alternate implementation technologies. The state of the art for standard text format parsers and generators has reached sufficient efficiency that the overhead introduced is nominal compared to the processing time of most components. We have now reached a point in text formats that we can move past the efficiency debate and focus on the architectural benefits we gain from decoupling our interface protocols from an implementation strategy.

Seeds of Destruction

The business needs a simple new feature, say a trivial fraud check. So it's added to an existing component. It works great but now they need a few more capabilities. So the component is enhanced. This cycle continues for 18 months and suddenly you have what really amounts to two components, but they are implemented as one, badly coupled and intertwined. Unfortunately the problem is usually worse than that though as the customers have come to expect a behavior that will be hard to maintain if you decouple. Either because of latency or availability or both, you find yourself with an architectural conundrum that could have been avoided if you had maintained a separation of concerns, regardless of the relative size of the two concerns involved.

Unintentional Vendor Lock

How can it be unintentional? You know, that's a question I often have asked as well. But the reality is that it can be done very easily. I have nothing against Oracle per se, but it will serve to illustrate my point. Oracle provides a few unique features that are in some cases tempting and in other cases unavoidable. Anonymous PL/SQL blocks are stored procedures that are stored in your code, delivered at run time. This is great from an application maintenance perspective because you get the benefits of a stored procedure without the revision management issues. But this feature is unique to Oracle and as far as I know, one competitor (Enterprise DB) which means if you choose to leverage it, you have to devise solutions if you want to migrate to another database.

But that feature is more overt and you can choose to not use it. A more covert feature is how Oracle manages concurrency. It relies on rollback segments that provide read committed concurrency while avoiding row level locks for the duration of the transaction. This provides excellent performance, especially in high concurrency situations. If your update patterns are not particularly contentious then the difference in performance between normal row level locks and rollback segments may be nominal but if you do have records that receive high update loads, you will experience a drop in performance.

The point of this though is it's important to understand the behavior of your vendor's products and understand the implications of how they will impact your architecture, short and long term. It's not just the over features you can avoid, but the more subtle implementation details that you can't avoid but may come to rely upon.

Persistence Binding

This one is a bit harder to avoid but is worth the investment to minimize. Until recently most architectures considered persistence to mean database. Many architects would apply best practices to avoid vendor lock but little to no effort to avoid database lock. And why would you be worried about database lock? Well because for many classes of persistence, a database is not the most cost effective storage. It may be the easiest to initially implement but as other forms of persistence mature, you may want to be able to take advantage. Yet if you've assumed you have a database with SQL available you may find this difficult to do.

One of the best ways to minimize persistence lock is to separate your access paths in your resource tier. The primary access path to any entity should be via primary key. All other access paths should be added with care and carefully delineated within the implementation of the resource tier. This allows the alternate paths to be managed in other forms of persistence in the future.

I am sure there are other suggestions from my readers. The general theme, as you can see, is to minimize coupling. This keeps your architecture flexible and leaves the door open for integrating newer technologies and patterns as they emerge.

Technorati Tags: architecture, asynchronous, database, engineering, java, messaging, performance, programming, protocol, scalability, services, soa, software, to_read, toread

Architectural Shelf Life

Architectures are often thought of having an useful life but that life is usually hard to predict. Most of the time it is determined by how much pain an organization is experiencing in trying to extend it to the current business needs. After giving this some thought, I think I can state that an architecture typically has a 10, to at most 15 year useful life. To explain this idea though, I need to introduce another concept:

Architectural Shelf Life - The duration that a collection of patterns and technology are applicable when starting a new system design.

So to elaborate, if you have the chance to start over, complete green field, what architectural patterns and technology do you use. I argue that these change about every 5 years. And from that I derive that any architecture should be replaced in part or in whole about every 2-3 generations of this shelf life. Not convinced of the shelf life argument?

Roll the clock back to 1990 and look at how enterprises would deliver services to their customers. Most offered no access from the customer's location. Some had basic communications in place via email. The more progressive may offer forms via Compuserve, Prodigy, Delphi, or AOL. Applications were implemented monolithically, scaled vertically, and most likely in languages that are waning in popularity or possibly already dead. Databases were also monolithic, using mid tier to mainframe servers with direct attached storage.

By 1995, your customers could begin to find you on the web. The form screens were literally translated to web forms. The application stack and data storage had not changed much. The web was primarily about user interface. Upstart competitors are challenging that with applications written in C++ or some fledgeling scripting languages. Their databases were still largely the same platform as 1990.

If you weren't too busy fixing Y2K bugs coming in to 2000, you would see a significant acceleration in change. Pure web architectures are now common, though the web fronting legacy enterprise applications is still prevalent. Applications are now multi-tiered although still largely monolithic deployments. Scale out architectures are emerging as the preferred way to scale out business logic. Mid tier servers are taking over the job of databases from mainframe and SANs become the preferred way to manage storage. Applications were being written in Java, VB, and new language C#.

Over the past 8 years, databases have evolved to horizontal scaling through sharding. Services have emerged as the preferred design pattern for integrating tiers and components. Java and C# frameworks have matured to provide dramatically better productivity and have been joined by Ruby on Rails and Django to name a few. Distributed storage on low cost unreliable devices is gaining popularity.

Of course some of you will immediately jump to correct my memory or facts. That's not the point but rather the dramatic shift in architectural patterns and technology is what you should take from this. Additionally, looking at the business players that have emerged and those that have faded due to the shifts in technology gives a chilling picture into how much technology disrupts the world of business. The improvements in developer efficiency and the lowering costs of deployment makes it possible to build and operate products cheaper every 5 years. Of course there is a lot more to a successful business than the operating cost but facing competition that has a better cost structure is not a desirable situation.

The challenge you face then is how to continually evolve your platform to adopt new patterns and technologies as they emerge. This can be a daunting when you have a well established customer base and a overflow of business features in the pipeline. Operations will be concerned about maintaining availability through any transition and the business wants enhanced functionality to the product. As long as those two goals can be met with your current architecture, it takes considerable momentum to cause a shift. And in fact, even when it becomes questionable as to the effectiveness of the current architecture, it is not uncommon to still meet resistance to making a major architectural shift.

The fallacy in this situation is that a new architecture will be to disruptive and costly. What is missed is that the new patterns and platforms can be used to disrupt your business anyway. If you have a successful business, many others want a share. And technology becomes the tool they can use to go after your business. And the disruption they can cause by offering your services at lower cost or with more compelling features is far greater than any disruption that will be incurred for internal architectural shifts. In simple terms, if you don't use technology to disrupt your current way of doing business, somebody else will. Therefore, incorporating architectural shelf life and therefore architectural life span into your business is a necessary investment to remain current.

Technorati Tags: architecture, engineering, java, programming, services, soa, software, to_read, toread, web

Shard Lessons

No, not SHARED lessons, I mean SHARD lessons. I have to admit that until about a year ago I didn't really know the term shards in relation to databases. Now don't confuse that with not understanding how databases can be horizontally scaled. I was introduced to that concept and helped to define the various ways it can be done but we just called it splits. Regardless of what you call it, there are some interesting challenges that are introduced. The well known challenges of consistency are discussed ad nauseam, even by me, so I'm not going there with this article. But besides that, there are some other lessons to learn when applying the pattern to your data.

Lesson 1: Right Size Your Shards

Sounds like a fast food commercial when I put it in those terms but the idea is actually the same. Determining the initial number of shards can be tricky. You don't have infinite resources and no matter how good your tools are, more shards are more problematic to manage than fewer. Yet you also want to select a number that going to last you for a while. I would recommend picking a number that will give you 18-24 months of growth with a margin for safety.

Simple enough, right? Well, not quite yet. I also wouldn't start with a single digit number of shards even if that will last you for two years. Why start with 4 when the incremental overhead of 12 is manageable. You allow a longer growth path and by hosting multiple databases on the same physical machine, you limit your hardware expenses. Of course then why not 24 or more? Well, at some point you add unnecessary overheads and get diminishing returns.

Lesson 2: Use Math on Shard Counts

If you notice the numbers I selected in the previous lesson, they are multiples of 12. Why that and not multiples of 10 or 7 or just whatever number happens to be your lucky number? Well, let's say you are going to run multiple shards per physical machine. You're a small organization. You pick 10 as your number of shards and spread it on two machines. Five each, things are perfect. Now you find your growth means you need more capacity. Would be nice to add one machine, but that leaves you with a 3, 3, 4 split and one machine has 33% more load than the other two. In fact, your next physical growth is 5, which may not be terrible if you are running your databases on commodity hardware, as you should!

But had you started with 12, you find your growth options to have fewer cost step functions. You start with 2 boxes, 6 shards each. Move to 3 with 4 shards, then 4 with 3 shards, and finally 6 with 2 shards. If you had projected 12 shards would allow you to scale for 2 years, you can see how you are able to grow your hardware requirements in a smoother fashion than with 10 shards.

Lesson 3: Carefully Consider the Spread

The way you spread the data across the shards needs to be determined based on what you are hoping to optimize. Certainly the simplest strategy is a uniform distribution which can be achieved using simple modulo math on the primary access key. This approach requires very little to implement and works well when the primary goal is to evenly distribute the load and you have very few access paths.

There may be times however where you want different locality policies. For example, you may want to keep all members of a group on the same shard. This can improve performance for applications that tend to operate upon a group. It also introduces challenges in balancing loads as some groups may be larger or more active than others. The benefits of clustering by a group must be weighed against the challenges of balancing work loads.

Lesson 4: Plan for Exceeding Your Shards

No matter what number you pick, the possibility of reaching capacity of your databases is very real. You selected a shard pattern to allow for a scale out strategy. So as you reach the limits of your current hardware you really don't want to shift to s a scale up on those shards. That's why it is important to bake into your shard strategy your scale out plan.

From Lesson 3, if you are clustering by group, adding more hosts to support new groups is probably straightforward. This helps when your scaling challenge is a growth in the number of groups and not just a few groups growing larger. Shards that cluster based on a group will probably need to support a scheme to migrate groups to allow capacity to be more readily managed.

Uniform spreads can be expanded through another trick of math. Each shard can be treated as a root node in a shard tree. When it is time to grow the capacity, another layer of shards can be added to grow capacity geometrically. For simplicity, let's say you originally started with 4 shards. You calculate the shard number with the following formula:

shard = key % 4

Now you need to scale. Let's say we will expand from 4 to 16 hosts. The technique for computing the root shard and final shard is:

root = key % 4
leaf = (key / 4) % 4

You now have 4 * 4 or 16 possible shard combinations using root.leaf as the shard identifier. Furthermore migration can be done online simply by following the scheme of checking root.leaf first and if you do not find your record, check root. Appropriate locking mechanisms will obviously be required by migration scripts to insure that no data is lost or corrupted. And this technique does require a temporary increase in the number of database instances to (old shard + new shard) hosts (in this example, 20 hosts total).

Lesson 5: Shard Early and Often

No matter how disciplined you think or hope your team is, if you give them a monolithic schema, and tight deadlines, they will eventually create a mission critical query that assumes two rows are on the same physical host. And, in my experience without fail, those two rows can't possibly live on any sane shard spread strategy. So if you have any hint at all that you will need to scale out a schema, do it early. The longer you wait, the more your application will depend upon a single schema instance and the harder it will be to migrate to a shard schema.

Care to share lessons you have learned from shard patterns? I am sure there are more that are worth knowing!

Technorati Tags: architecture, database, engineering, java, performance, programming, scalability, shard, software, to_read, toread

Availability Enlightenment

The air in the room was still and slightly cool, some might complain too cool. Sitting near the window, providing light filtered by vertical shades partially drawn was a man. Hair streaked with glints of gray that indicated either age or stress. Yet the hair was somewhat betrayed by the bright Hawaiian shirt that fit loosely over his shoulders. The room had sufficient shadows to cause the glow of the LCD screen to light his face.

Approaching the room at a hurried pace was a young engineer. Stress was clear upon his face. He was obviously in need of guidance, help with some crisis. As he approached the desk, the older engineer turned and met him with a calm steady stare. Our young engineer quickly introduces himself, "Hello, my name is Bill, they say you are the fellow". The older man nods, and offers Bill a seat with a motion of his hand and offers, "You can call me TF for short." And our story begins.

Bill: "I have a terrible crisis on my hands. Several months ago we were asked to achieve 99.999% availability."

TF simply returns Bill's gaze.

Bill: "That's only 5 minutes of downtime per year!"

TF nods, "How did you approach it?"

Bill: "Well, we sought out the most reliable components we could buy. We surveyed all of the vendors and selected them carefully. We deployed our applications on this new configuration."

TF: "And your availability?"

Bill: "Well", glancing at his shoes now, "it actually went down."

TF nods again, "You were lead to believe that reliable components result in a reliable system, weren't you?"

Now it was Bill's turn to simply nod.

TF: "And nobody expressed any concern that this might not be right?"

Bill shook his head slowly, almost with shame.

TF: "Of course not as the wisdom of the many is that reliability comes from reliable vendors who purvey their products as being the solution to the availability they so desperately desire. But alas, this is not the path to reliability, only the road to stress."

Bill nods, "But then was it the secret to availability."

TF: "Well, if reliable components aren't the answer, what do you think?"

Bill looks puzzled.

TF: "What is the opposite of reliable components?"

Bill: "Unreliable components?"

TF: "Exactly!'

Bill: "Wait, to get better availability, I should begin with components that break more often?"

TF nods.

Bill: "I don't follow. If my components are failing, then how do I keep my system up?"

TF: "What do you think you should do if a component is down but your system still needs to be up?"

Bill ponders it for a moment, "Design around component failures?"

TF smiles broadly and nods, "You are a chosen one. Assume nothing in your system can be counted upon to work. Everything breaks, regularly. With that as a baseline, now you can design your system to compensate for failures. Through this approach you can achieve new levels of availability."

Bill: "But I can only design the software, what about failures in other components."

TF shakes his head disappointed. "Bill, you are building a system, not a collection of parts. It has to be designed from the customer all the way to the silicon and ferrous atoms or you will never achieve the nirvana you seek."

Bill's face suddenly brightens. "Wait, if I assume nothing is reliable, then the argument for vendor supported software loses a lot of credibility. I can choose less expensive open source solutions."

TF nods. "I assume you know that open source doesn't imply that software is less reliable. In fact in may cases it can be argued it is more reliable."

Bill: "Yes, I know but there are chains of approval that have to be convinced."

TF nods again.

Bill: "But still, five 9's. Can it be done on inexpensive components including open source?"

TF: "It already has been done Bill. I helped build it. Actually this approach can be used to achieve almost any level of availability. You simply decide what component failures you'll design around and which you'll accept as having the ability to bring your system down. It gives you significant control over selecting cost vs. availability decisions."

Bill looks a bit shocked. "So let me see if I can summarize the path to high availability. Assume nothing works, design for all possible failures, and design the system holistically."

TF: "Yes Bill, you are on the path to availability enlightenment."

Bill: "I wish I had come to you first."

TF: "The path to enlightenment is often only visible after failure, if at all."

Technorati Tags: architecture, engineering, java, performance, scalability, services, soa, software, to_read, toread, web