Adding Simplicity - An Engineering Mantra

Focus on the Cloud, not the Clouds

There are a lot of very good conversations going on about the challenges with cloud computing. Storage is just beginning to mature in the cloud and there are many interesting issues around privacy, SOX, and PCI compliance. Nobody has clear answers yet on the security and compliance related issues. But I think for many enterprises, that may be a pointless dialog because it isn't federating their capacity into the clouds that is important but rather how to leverage the concepts and technologies from the cloud to improve their operational efficiency.

Before getting into complex discussions about the security challenges of using the cloud, I believe most organizations need to take a good look at how well their applications can operate in any utility compute model. Does your application currently support all of the following concepts easily?

Loose Coupling to Platform

Are your applications platform independent? Have you made them completely agnostic to the processors they require? Do they scale well from a single core to several cores? Have you optimized your memory footprint, making it fit well into a modest memory model? What assumptions do your applications make about operating systems? Directory paths? Available local disk space?

Ultimately, can you run your applications in a collection of virtual containers in your environment right now. The answer for many organizations is no. And until that is fully addressed, their ability to enter in cloud is largely non-existent. Forget about your PCI issues, your JVM footprint is keeping you stuck on your own hardware.

Crisp Packaging

Do you have a repeatable build and release process? Are your build targets well defined and self contained? Do they externalize their dependencies for tools to detect deployment requirements? Are the artifacts independent of their deployment environment (development, testing, production)? Are all environmental dependencies externalized and injected into your application at run time?

To leverage a utility compute model, you have to have components that are compatible. Everyone likes to draw parallels to the electricity model when discussing clouds. But imagine if every electronic component you purchased had different voltage needs. Failed to clearly identify their current needs. Forget about the obvious each coming with their own style of plug. We wouldn't have the ubiquitous electric utility model we have now. Yet few organizations are far off this analogy when it comes to the state of their applications.

Well Specified Management Interfaces

Do you have standard interfaces for expressing state of your application? How about for changing runtime behavior? Or detecting exceptional conditions? Or tracking activity for both debugging and analysis?

This is something I've covered in past articles but is worth emphasizing again. Operational control of the application requires a set of interfaces that follow the same rigor as your other API's. They need to be designed and evolved using the same principles and processes. And without such interfaces you will have a difficult time operating components, in your own data center much less in a compute cloud.

In Conclusion

The interesting part of this situation to me is these are all things you should be focused on accomplishing whether you are looking to leverage a vendor's cloud or not. These allow you to achieve lower cost of operation and better utilize your internal resources. And they do that without having to bring into question the challenges of privacy and security.

Technorati Tags: architecture, engineering, performance, programming, scalability, services, soa, software, to_read, toread, web

Are Data Warehouses Dinosaurs?

As anybody that follows my blog knows, I am not a fan of vertical scaling. I don't like solutions that can only be implemented in a single address and storage space. Unfortunately, there are analytical problems that need a holistic view of data. This is very typical of data warehousing applications. As a result, data warehouses are expensive, often out of the reach of smaller organizations. But there may be an alternative that is less expensive and horizontally scalable. What is this great revelation? Processing streams of events using an Event Stream Processor (ESP) solution.

ESP analyze streams of events using a language similar to SQL. In the same manner that databases and data warehouses use SQL to perform analysis of data tables, ESP use their query language to analyze streams of events. The simplest way to understand ESP is to think of events as rows in a table and the attributes of an event as the columns. Each event type is the equivalent of a table. From this perspective, it becomes straightforward to see how ESP works. But how does this relate to replacing data warehouses?

Data warehouse analysis involves aggregating information along a variety of axis as well as inverting relationships in the data. The goal is to provide the business with different perspectives on what the customers are doing. In order to do this, data is loaded into the warehouse periodically. Typically daily ETL processes are performed on the production databases to keep the warehouse fresh. This process though has a couple issues beyond the cost of the warehouse infrastructure. First, the ETL places a significant load on your production databases. If your business has nice offline windows for the ETL, that's great, but if not, managing the scale becomes a challenge. Second, the freshness of the warehouse is typically 24 hours behind or more. As your business grows this lag will grow as well.

ESP address this by analyzing the changes to your data as it occurs. Rather than doing batch ETL's, you stream business events as the state of your data changes. This creates a more manageable scaling model for your production system. The business analytics extracts are spread throughout the transaction day. ESP can also be horizontally scaled, providing a more cost effective solution for your business. And since ESP is performing the analysis in real time, the business metrics can be current and remain that way as the business grows.

Does this spell the end of data warehouses? Well, maybe but there is one challenge with the ESP approach. While it is able to provide analytics cost effectively, it does not provide the ability to perform historical analysis. If you know what you want, then ESP will deliver the results from the current point in time forward. But what if you want a different perspective on your business activity and you want it over the past 3 months. One solution is to create a framework for capturing and replaying transactions but this can be expensive. This becomes a matter of deciding the business value of performing the historical analysis.

Whether you choose to use a data warehouse or not, ESP is definitely worth investigating as a way of delivering business analytics more cost effectively.

Technorati Tags: architecture, database, engineering, performance, scalability, services, software, to_read, toread, web

Free Energy

The first law of thermodynamics tells us that energy in a system is constant. Energy is neither created nor destroyed. And thus the fantasy of so many that you can get more energy out than you put in is dashed. Of course there is the Adam Savage comment that there is really only free energy to you. Meaning of course, it's free if you don't have to put your energy into extracting it. Of course there are ways to get free energy without siphoning your neighbor's gas tank at night or covertly tapping their power lines. I have a few favorites that illustrate the concept.

Regenerative brakes use electric generators to provide resistance and slow a car. Or in some cases, they compress hydraulic fluid but whatever the technique, the idea is the same. Rather than converting the vehicle's momentum into heat and annoying brake dust that cakes on your wheels, capture it and store it for reuse. Is it free? No, but it was otherwise wasted. In this particular application, the regeneration process doesn't even have to be especially efficient. Even if it only captures 10% of the vehicles kinetic energy, it has reclaimed something, not to mention reducing brake dust. Have I mentioned I'm not fond of brake dust?

There are two intriguing proposals that I have also seen. One involves a floor mat that contains coil inductors. Stepping on the mat depresses causes a small amount of electrical energy to be generated. The volume is relatively small until you consider an application such as a train station or airport terminal where thousands of travelers pass by everyday. The efficiency isn't great but the cost of the generation source is largely free. It's not clear if it can be manufactured cost effectively enough but the concept is spot on. Find a source of energy that is otherwise being wasted and capture it. Along a similar vein is a system that generates electricity from the heat put off by people in office buildings. Again, find a source of energy that is otherwise discarded and convert it into usable energy.

By now you're probably wondering what any of this has to do with architecture. Well, power efficiency is definitely one of the challenges we currently face in our architectures. As I have said before, power consumption is a software architecture problem. One of the interesting challenges I have been pondering of late is how to find largely idle resources in the data center and put them to use. Gain some more business benefit from the capital expenditures and electricity bill companies are already paying.

One of the idle resources I've observed is disk capacity. The traditional web/services architecture follows a multi-tier design with application servers and database servers. The storage is of course on the database tier. But what about the drives in the application servers? Disk capacities have grown to almost ludicrous sizes with most 1U servers arriving with 500GB or more of storage. Even with virtualization, the operating system and applications rarely occupy more than 100GB, but let's be generous and make it 200GB. So you have your application servers spinning drives with 300GB of unused capacity. Even a smallish site of 100 application servers has 30TB of idle capacity sitting in their application servers.

So let's think about that capacity for a minute. The capacity isn't reliable as application servers are typically stateless. So any use of the storage must take that into consideration. Not only can application servers go down, but they can die and be completely replaced, the contents of the drive lost forever. Additionally, if your application servers are located in a single data center, not just one server's storage but the entire storage farm could be lost. Let's assume though that we're only going to worry about losing a server and not the entire storage farm.

There are a lot of uses where losing storage is completely acceptable. Log file storage, temporary space for back office analytical processes, and test data sets are just a few examples. But what if you ran Hadoop across these storage nodes with 2 or even 3 replicas. You would still have 10TB of storage available and you can now tolerate the loss of a single node without any impact to the availability of the data. Most companies have storage needs that could easily be met by such storage configuration and it is almost free. At the extreme end of this concept is Wuala which allows users to not only access a distributed storage network but trade idle capacity on their computers for additional capacity in the network.

Another idle resource I've observed is off peak processing power. A typical web service has a peak that is approximately 2.5 times the average traffic. Ignoring disaster recovery capacity and peak to off-peak, that means that there are windows throughout the day when 50% of the companies available compute resources are lying idle. This is capacity that has already been purchased and is occupying data center space and consuming power.

Again, there is an opportunity to harness this capacity for other business functions that can live in the off-peak processing windows. What if business reporting activities could be moved into another virtual container? Or how about business analytics functions being performed as a map/reduce operation during off-peak cycles? Every organization has dozens if not hundreds of such processing tasks that are traditionally assigned to dedicated hardware that could easily be completed on otherwise idle resources.

And so bringing this full circle, just like regenerative braking exploits energy you've already paid for to reclaim benefits that would otherwise require additional new energy, harnessing idle capacity from your application tier can let you reclaim business benefits for resources you already fund.

Technorati Tags: architecture, engineering, services, soa, software, to_read, toread, web

Extending the Architectural Life

In my last article I introduced the concept of Architecture Shelf Life. I also put forth the postulate that for most patterns and technologies it is approximately 5 years. If you buy that as well as the premise that architectures can only survive 2-3 shelf life refreshes before they become dated and impractical, then you must be asking, how can I hedge against that. I want to extend the life of my architecture in some way.

Most critical for any architecture that is to survive is the ability to be refreshed with new technologies in an incremental fashion. The way this can be most readily achieved is by following standard architectural principles of good component design with the loosest possible coupling between components. This obviously allows components to be implemented independent of each other and also lets them follow their own technology curve for being replaced. Unfortunately though architectures rarely have good components that are truly loosely coupled. The reasons for this are the results of organic growth of the system. I have noticed some specific patterns emerge though.

Poor Protocols

This one is probably one of the most common. The components are reasonable and the interfaces have decent semantics. But then the implementation choice couples the components to a specific technology or implementation strategy. In most cases the decision is made based on the desire to optimize the interface speed. And in the majority of these cases it was probably an unnecessary optimization that bought a little speed but at the expense of long term architectural flexibility.

Interfaces between components should be text based (e.g. XML, JSON) as they offer the maximum flexibility for migrating components to alternate implementation technologies. The state of the art for standard text format parsers and generators has reached sufficient efficiency that the overhead introduced is nominal compared to the processing time of most components. We have now reached a point in text formats that we can move past the efficiency debate and focus on the architectural benefits we gain from decoupling our interface protocols from an implementation strategy.

Seeds of Destruction

The business needs a simple new feature, say a trivial fraud check. So it's added to an existing component. It works great but now they need a few more capabilities. So the component is enhanced. This cycle continues for 18 months and suddenly you have what really amounts to two components, but they are implemented as one, badly coupled and intertwined. Unfortunately the problem is usually worse than that though as the customers have come to expect a behavior that will be hard to maintain if you decouple. Either because of latency or availability or both, you find yourself with an architectural conundrum that could have been avoided if you had maintained a separation of concerns, regardless of the relative size of the two concerns involved.

Unintentional Vendor Lock

How can it be unintentional? You know, that's a question I often have asked as well. But the reality is that it can be done very easily. I have nothing against Oracle per se, but it will serve to illustrate my point. Oracle provides a few unique features that are in some cases tempting and in other cases unavoidable. Anonymous PL/SQL blocks are stored procedures that are stored in your code, delivered at run time. This is great from an application maintenance perspective because you get the benefits of a stored procedure without the revision management issues. But this feature is unique to Oracle and as far as I know, one competitor (Enterprise DB) which means if you choose to leverage it, you have to devise solutions if you want to migrate to another database.

But that feature is more overt and you can choose to not use it. A more covert feature is how Oracle manages concurrency. It relies on rollback segments that provide read committed concurrency while avoiding row level locks for the duration of the transaction. This provides excellent performance, especially in high concurrency situations. If your update patterns are not particularly contentious then the difference in performance between normal row level locks and rollback segments may be nominal but if you do have records that receive high update loads, you will experience a drop in performance.

The point of this though is it's important to understand the behavior of your vendor's products and understand the implications of how they will impact your architecture, short and long term. It's not just the over features you can avoid, but the more subtle implementation details that you can't avoid but may come to rely upon.

Persistence Binding

This one is a bit harder to avoid but is worth the investment to minimize. Until recently most architectures considered persistence to mean database. Many architects would apply best practices to avoid vendor lock but little to no effort to avoid database lock. And why would you be worried about database lock? Well because for many classes of persistence, a database is not the most cost effective storage. It may be the easiest to initially implement but as other forms of persistence mature, you may want to be able to take advantage. Yet if you've assumed you have a database with SQL available you may find this difficult to do.

One of the best ways to minimize persistence lock is to separate your access paths in your resource tier. The primary access path to any entity should be via primary key. All other access paths should be added with care and carefully delineated within the implementation of the resource tier. This allows the alternate paths to be managed in other forms of persistence in the future.

I am sure there are other suggestions from my readers. The general theme, as you can see, is to minimize coupling. This keeps your architecture flexible and leaves the door open for integrating newer technologies and patterns as they emerge.

Technorati Tags: architecture, asynchronous, database, engineering, java, messaging, performance, programming, protocol, scalability, services, soa, software, to_read, toread

Architectural Shelf Life

Architectures are often thought of having an useful life but that life is usually hard to predict. Most of the time it is determined by how much pain an organization is experiencing in trying to extend it to the current business needs. After giving this some thought, I think I can state that an architecture typically has a 10, to at most 15 year useful life. To explain this idea though, I need to introduce another concept:

Architectural Shelf Life - The duration that a collection of patterns and technology are applicable when starting a new system design.

So to elaborate, if you have the chance to start over, complete green field, what architectural patterns and technology do you use. I argue that these change about every 5 years. And from that I derive that any architecture should be replaced in part or in whole about every 2-3 generations of this shelf life. Not convinced of the shelf life argument?

Roll the clock back to 1990 and look at how enterprises would deliver services to their customers. Most offered no access from the customer's location. Some had basic communications in place via email. The more progressive may offer forms via Compuserve, Prodigy, Delphi, or AOL. Applications were implemented monolithically, scaled vertically, and most likely in languages that are waning in popularity or possibly already dead. Databases were also monolithic, using mid tier to mainframe servers with direct attached storage.

By 1995, your customers could begin to find you on the web. The form screens were literally translated to web forms. The application stack and data storage had not changed much. The web was primarily about user interface. Upstart competitors are challenging that with applications written in C++ or some fledgeling scripting languages. Their databases were still largely the same platform as 1990.

If you weren't too busy fixing Y2K bugs coming in to 2000, you would see a significant acceleration in change. Pure web architectures are now common, though the web fronting legacy enterprise applications is still prevalent. Applications are now multi-tiered although still largely monolithic deployments. Scale out architectures are emerging as the preferred way to scale out business logic. Mid tier servers are taking over the job of databases from mainframe and SANs become the preferred way to manage storage. Applications were being written in Java, VB, and new language C#.

Over the past 8 years, databases have evolved to horizontal scaling through sharding. Services have emerged as the preferred design pattern for integrating tiers and components. Java and C# frameworks have matured to provide dramatically better productivity and have been joined by Ruby on Rails and Django to name a few. Distributed storage on low cost unreliable devices is gaining popularity.

Of course some of you will immediately jump to correct my memory or facts. That's not the point but rather the dramatic shift in architectural patterns and technology is what you should take from this. Additionally, looking at the business players that have emerged and those that have faded due to the shifts in technology gives a chilling picture into how much technology disrupts the world of business. The improvements in developer efficiency and the lowering costs of deployment makes it possible to build and operate products cheaper every 5 years. Of course there is a lot more to a successful business than the operating cost but facing competition that has a better cost structure is not a desirable situation.

The challenge you face then is how to continually evolve your platform to adopt new patterns and technologies as they emerge. This can be a daunting when you have a well established customer base and a overflow of business features in the pipeline. Operations will be concerned about maintaining availability through any transition and the business wants enhanced functionality to the product. As long as those two goals can be met with your current architecture, it takes considerable momentum to cause a shift. And in fact, even when it becomes questionable as to the effectiveness of the current architecture, it is not uncommon to still meet resistance to making a major architectural shift.

The fallacy in this situation is that a new architecture will be to disruptive and costly. What is missed is that the new patterns and platforms can be used to disrupt your business anyway. If you have a successful business, many others want a share. And technology becomes the tool they can use to go after your business. And the disruption they can cause by offering your services at lower cost or with more compelling features is far greater than any disruption that will be incurred for internal architectural shifts. In simple terms, if you don't use technology to disrupt your current way of doing business, somebody else will. Therefore, incorporating architectural shelf life and therefore architectural life span into your business is a necessary investment to remain current.

Technorati Tags: architecture, engineering, java, programming, services, soa, software, to_read, toread, web

Shard Lessons

No, not SHARED lessons, I mean SHARD lessons. I have to admit that until about a year ago I didn't really know the term shards in relation to databases. Now don't confuse that with not understanding how databases can be horizontally scaled. I was introduced to that concept and helped to define the various ways it can be done but we just called it splits. Regardless of what you call it, there are some interesting challenges that are introduced. The well known challenges of consistency are discussed ad nauseam, even by me, so I'm not going there with this article. But besides that, there are some other lessons to learn when applying the pattern to your data.

Lesson 1: Right Size Your Shards

Sounds like a fast food commercial when I put it in those terms but the idea is actually the same. Determining the initial number of shards can be tricky. You don't have infinite resources and no matter how good your tools are, more shards are more problematic to manage than fewer. Yet you also want to select a number that going to last you for a while. I would recommend picking a number that will give you 18-24 months of growth with a margin for safety.

Simple enough, right? Well, not quite yet. I also wouldn't start with a single digit number of shards even if that will last you for two years. Why start with 4 when the incremental overhead of 12 is manageable. You allow a longer growth path and by hosting multiple databases on the same physical machine, you limit your hardware expenses. Of course then why not 24 or more? Well, at some point you add unnecessary overheads and get diminishing returns.

Lesson 2: Use Math on Shard Counts

If you notice the numbers I selected in the previous lesson, they are multiples of 12. Why that and not multiples of 10 or 7 or just whatever number happens to be your lucky number? Well, let's say you are going to run multiple shards per physical machine. You're a small organization. You pick 10 as your number of shards and spread it on two machines. Five each, things are perfect. Now you find your growth means you need more capacity. Would be nice to add one machine, but that leaves you with a 3, 3, 4 split and one machine has 33% more load than the other two. In fact, your next physical growth is 5, which may not be terrible if you are running your databases on commodity hardware, as you should!

But had you started with 12, you find your growth options to have fewer cost step functions. You start with 2 boxes, 6 shards each. Move to 3 with 4 shards, then 4 with 3 shards, and finally 6 with 2 shards. If you had projected 12 shards would allow you to scale for 2 years, you can see how you are able to grow your hardware requirements in a smoother fashion than with 10 shards.

Lesson 3: Carefully Consider the Spread

The way you spread the data across the shards needs to be determined based on what you are hoping to optimize. Certainly the simplest strategy is a uniform distribution which can be achieved using simple modulo math on the primary access key. This approach requires very little to implement and works well when the primary goal is to evenly distribute the load and you have very few access paths.

There may be times however where you want different locality policies. For example, you may want to keep all members of a group on the same shard. This can improve performance for applications that tend to operate upon a group. It also introduces challenges in balancing loads as some groups may be larger or more active than others. The benefits of clustering by a group must be weighed against the challenges of balancing work loads.

Lesson 4: Plan for Exceeding Your Shards

No matter what number you pick, the possibility of reaching capacity of your databases is very real. You selected a shard pattern to allow for a scale out strategy. So as you reach the limits of your current hardware you really don't want to shift to s a scale up on those shards. That's why it is important to bake into your shard strategy your scale out plan.

From Lesson 3, if you are clustering by group, adding more hosts to support new groups is probably straightforward. This helps when your scaling challenge is a growth in the number of groups and not just a few groups growing larger. Shards that cluster based on a group will probably need to support a scheme to migrate groups to allow capacity to be more readily managed.

Uniform spreads can be expanded through another trick of math. Each shard can be treated as a root node in a shard tree. When it is time to grow the capacity, another layer of shards can be added to grow capacity geometrically. For simplicity, let's say you originally started with 4 shards. You calculate the shard number with the following formula:

shard = key % 4

Now you need to scale. Let's say we will expand from 4 to 16 hosts. The technique for computing the root shard and final shard is:

root = key % 4
leaf = (key / 4) % 4

You now have 4 * 4 or 16 possible shard combinations using root.leaf as the shard identifier. Furthermore migration can be done online simply by following the scheme of checking root.leaf first and if you do not find your record, check root. Appropriate locking mechanisms will obviously be required by migration scripts to insure that no data is lost or corrupted. And this technique does require a temporary increase in the number of database instances to (old shard + new shard) hosts (in this example, 20 hosts total).

Lesson 5: Shard Early and Often

No matter how disciplined you think or hope your team is, if you give them a monolithic schema, and tight deadlines, they will eventually create a mission critical query that assumes two rows are on the same physical host. And, in my experience without fail, those two rows can't possibly live on any sane shard spread strategy. So if you have any hint at all that you will need to scale out a schema, do it early. The longer you wait, the more your application will depend upon a single schema instance and the harder it will be to migrate to a shard schema.

Care to share lessons you have learned from shard patterns? I am sure there are more that are worth knowing!

Technorati Tags: architecture, database, engineering, java, performance, programming, scalability, shard, software, to_read, toread

Availability Enlightenment

The air in the room was still and slightly cool, some might complain too cool. Sitting near the window, providing light filtered by vertical shades partially drawn was a man. Hair streaked with glints of gray that indicated either age or stress. Yet the hair was somewhat betrayed by the bright Hawaiian shirt that fit loosely over his shoulders. The room had sufficient shadows to cause the glow of the LCD screen to light his face.

Approaching the room at a hurried pace was a young engineer. Stress was clear upon his face. He was obviously in need of guidance, help with some crisis. As he approached the desk, the older engineer turned and met him with a calm steady stare. Our young engineer quickly introduces himself, "Hello, my name is Bill, they say you are the fellow". The older man nods, and offers Bill a seat with a motion of his hand and offers, "You can call me TF for short." And our story begins.

Bill: "I have a terrible crisis on my hands. Several months ago we were asked to achieve 99.999% availability."

TF simply returns Bill's gaze.

Bill: "That's only 5 minutes of downtime per year!"

TF nods, "How did you approach it?"

Bill: "Well, we sought out the most reliable components we could buy. We surveyed all of the vendors and selected them carefully. We deployed our applications on this new configuration."

TF: "And your availability?"

Bill: "Well", glancing at his shoes now, "it actually went down."

TF nods again, "You were lead to believe that reliable components result in a reliable system, weren't you?"

Now it was Bill's turn to simply nod.

TF: "And nobody expressed any concern that this might not be right?"

Bill shook his head slowly, almost with shame.

TF: "Of course not as the wisdom of the many is that reliability comes from reliable vendors who purvey their products as being the solution to the availability they so desperately desire. But alas, this is not the path to reliability, only the road to stress."

Bill nods, "But then was it the secret to availability."

TF: "Well, if reliable components aren't the answer, what do you think?"

Bill looks puzzled.

TF: "What is the opposite of reliable components?"

Bill: "Unreliable components?"

TF: "Exactly!'

Bill: "Wait, to get better availability, I should begin with components that break more often?"

TF nods.

Bill: "I don't follow. If my components are failing, then how do I keep my system up?"

TF: "What do you think you should do if a component is down but your system still needs to be up?"

Bill ponders it for a moment, "Design around component failures?"

TF smiles broadly and nods, "You are a chosen one. Assume nothing in your system can be counted upon to work. Everything breaks, regularly. With that as a baseline, now you can design your system to compensate for failures. Through this approach you can achieve new levels of availability."

Bill: "But I can only design the software, what about failures in other components."

TF shakes his head disappointed. "Bill, you are building a system, not a collection of parts. It has to be designed from the customer all the way to the silicon and ferrous atoms or you will never achieve the nirvana you seek."

Bill's face suddenly brightens. "Wait, if I assume nothing is reliable, then the argument for vendor supported software loses a lot of credibility. I can choose less expensive open source solutions."

TF nods. "I assume you know that open source doesn't imply that software is less reliable. In fact in may cases it can be argued it is more reliable."

Bill: "Yes, I know but there are chains of approval that have to be convinced."

TF nods again.

Bill: "But still, five 9's. Can it be done on inexpensive components including open source?"

TF: "It already has been done Bill. I helped build it. Actually this approach can be used to achieve almost any level of availability. You simply decide what component failures you'll design around and which you'll accept as having the ability to bring your system down. It gives you significant control over selecting cost vs. availability decisions."

Bill looks a bit shocked. "So let me see if I can summarize the path to high availability. Assume nothing works, design for all possible failures, and design the system holistically."

TF: "Yes Bill, you are on the path to availability enlightenment."

Bill: "I wish I had come to you first."

TF: "The path to enlightenment is often only visible after failure, if at all."

Technorati Tags: architecture, engineering, java, performance, scalability, services, soa, software, to_read, toread, web

Software Is A Craft

I've been writing a lot of code for the past 6 months. I have to admit, it is a welcome change from Word documents and PowerPoint presentations. Java, unlike text and graphics has a clearer definition of correct and finished. But, like text and graphics, the definition of done isn't finite. There is usually a bit more polishing you can do. Another way to factor the code to make it a bit more tidy. And that's something I've been doing a lot of in this 6 month period. Factoring, rearranging, trying to follow the spirit of my blog, add simplicity. It's done when there's nothing left to remove.

I've felt for a long time now that we don't view software engineering correctly. Even that term, "software engineering", illustrates a root cause of the problem. Engineering, in all disciplines, is viewed as the opposite of artistic. I've had these conversations with my "creative-type" friends and it's a stereotype that is common. Yet I view all engineering disciplines as art with constraints. Just like a photographer is bounded by the limits of the lens and the film (or digital sensor), software engineers are constrained by the boundaries of the programming language and the processing power of the target system. But that doesn't make software engineering any less creative or artistic.

Given that though, we teach software like we teach math. Artists are taught not only about the mechanics of their craft but also the art. We stop software engineering after the mechanics. Languages are covered in detail. Code factoring is taught with precision, not as the soft skill that it is. And then we enter the workforce. If we're lucky, we run into some mentors that help us learn our craft. If we're not, we bang out code to project deadlines and while that gives us experience, it doesn't give us skill.

Getting a chance to focus on writing software for the past few months, gave me a chance to understand this reality. I do not make clay pottery but I have friends that do. And what they relate is a feel that you develop for the clay. You come to know when the clay is the right consistency. How much pressure to apply to achieve the desired thickness. And this can only be learned by doing. This is the way software is for me. There's a feeling when the software is coming together well. It isn't something that can be expressed in precise terms, codified into a set of steps. It's an emotional response to the components falling into place cleanly. In other words, it's art.

I thought about how I learned to write software. I was fortunate enough to be exposed to some very good software engineers. They guided me early in my career. Helped me understand the difference between okay, good, and great software. In other words, I spent time as a journeyman. Given problems to solve, code to write, and usable feedback on how I could make my software better. Of course I also learned a lot the hard way. Ideas that seemed good at the time, only to discover they had a serious shortcoming. And of course, I rarely settle on a solution to a problem until I'm completely satisfied with it. We all see the same problem repeatedly through our career and I take that as an opportunity to continue to adjust and refine the solution until I know it is solved as cleanly as possible.

And of course, that is what an artist does. Express oneself in a way that provides satisfaction. Refine the expression until it is succinct and yet powerful. This is what software should be. Concise solutions to business problems that can illicit a positive emotional response from others in your trade. If you don't have emotional responses to software, perhaps this isn't your craft. For that is the only way you can move past just creating code to crafting code.

Technorati Tags: architecture, careers, engineering, java, programming, software, to_read, toread

What Metadata?

I'm positive that is what operations teams often think developers are asking themselves. And what exactly is metadata? Well, the simple answer is any information that is outside the main data flows that is used to control or monitor the behavior of the application. And with an answer that vague, it's little wonder it's often not part of the initial design. I break metadata into two major categories. Configuration data that controls the behavior of the application. And telemetry data that is generated by the application.

Configuration Data

Configuration data is usually given little if any attention during application development. Just throw whatever you need in a property or perhaps an XML file. The file is picked up from somewhere in the file system and that's that. Unfortunately, this fails to take into consideration several factors that can impact the manageability of the component. Let's take a look at some of the common pitfalls.

Configuration vs. Code

This is one of the biggest issues I see when designing configuration data. People get confused that properties and XML files may contain what is essentially code. Anything that would only be changed by a developer, is code, regardless of what file format is used to express it. Making files that are essentially code available for potential modification as configuration in a production deployment is a formula for disaster. Code must be tested before released to production. You can't test code that you allow to be changed in production. Therefore, files that are essentially code should be delivered embedded in your deliverable (e.g. inside the jar file) and not with configuration files you might expect to be modified in production.

Configuration Resource

Where to put the configuration information and how to format it is a subject of continuous debate. Files are the obvious choice for most applications but this brings up several interesting questions. First is the format. Java properties? XML? Something else (probably not)? Fortunately this issue can usually be resolved by looking at the structure of the information. The more structured the more likely XML is the right choice.

Files bring up the interesting problem of where do you put the file. If you deploy it inside your WAR, then it becomes a challenge for operations to locate the file. It's buried somewhere under your application server directory. You can put it in some other distinguished location but if you do this, you need to allow operations to override that location at server start or you will constrain their ability to manage the number of instances of the application per server.

Delivering the configuration with the application has other issues though. Invariably, there will be values of configuration that must be set to reflect the production environment. Once operations has modified the file, how do you deliver your subsequent versions. You can't overwrite the version of the file on the production server, yet if you've added necessary configuration values, how do these get merged into the production file.

Another challenge with file based configuration is that it does not scale well. It works well enough for O(10^2) servers but beyond that it is unwieldy. Attempting to manage files on thousands of servers requires appropriate tools to have any chance at all. Even with the best tools, there is a tendency for files to be missed and very confusing production errors result.

So what is the alternative? Centralizing configuration into services such as LDAP or a configuration database (CDB) can alleviate the challenges associated with distributing files. This definitely scales better to a large number of servers. But there are some challenges with this approach as well. The primary challenge is providing developers with a usable environment for testing. Developers typically require a local version of their resources with the ability to change the content at any time. This is relatively straightforward with files but much more difficult with LDAP or CDB. Still, the scalability offered to production may well be worth this hassle.

Telemetry

Okay, logging, but I have actually borrowed this term from professional racing (or NASA, take your pick) for a good reason. From Wikipedia:

Telemetry is a technology that allows the remote measurement and reporting of information of interest to the system designer or operator.

Thinking of the information that a component can send to logs in terms of telemetry instead of just logs, gives a better perspective on what belongs in the stream for operators. Developers tend to only think of logs as tools for themselves to help in debugging when in fact they are a necessity to properly monitor the health of a running application.

The question of course is what kind of information should be in the telemetry? Considering a common web service, at a minimum, I'd expect to find the following information easily in the stream:

• The request URI including parameters
• Basic parametric information about the request
• External resource interactions performed by the service. These should include status and timings.
• The result status of the request.
• The processing time of the request.

If this information is made available to operations in real time, there is several types of monitors that can be created. Alarms can be set on thresholds for error status ratios or dramatic drops in request rates. Operational graphs can be made for average response time with potential alerts for response time drifting out of SLA. Dependency graphs can be constructed that will help operations correlate resource failures to client impacts. And this is from the small amount of information proposed above. Additional telemetry can provide even more operational monitoring capabilities.

The Java logging facility can meet the needs of generating telemetry although you may want to separate out telemetry into its own logger name. For small scale deployments, this telemetry can simply be sent to log files. Scripts that regularly scrape the logs can be used to extract the relevant bits of information. For larger scale operations however, a central logging scheme is more relevant. Logging using the socket handler may be sufficient although for very large scale installations, it may be desirable to move to a less reliable but more scalable transport such as multicast.

Summary

I know that I've only scratched the surface metadata issues. The point of this posting wasn't to give you an exhaustive guide of configuration and telemetry, but rather to bring up some issues and initiate a dialog. As always, comments most welcome.

Technorati Tags: architecture, asynchronous, engineering, java, messaging, performance, programming, scalability, services, soa, software, to_read, toread, web

Gartner's Top 10 Technologies for 2008

Network World has published an article on Gartner's Top 10 Technologies for 2008. I love predictions although I will admit that October is a bit early. But why not get yours out early and beat the holiday rush, right? Actually as predictions go these are not bad. Like most, they range from the obvious to the "as-if" but it's risky to stick your neck out and we shouldn't be harsh with any organization that does. I do have a few comments on the predictions though.

Starting off in the #1 spot is green computing. I think they are right, especially for large organizations. It's no surprise to anybody trying to run a large data center that power is now the constraining resource. Unfortunately, the message is that green computing is a hardware problem. They also fall short of tying #5, virtualization, to the challenges of green computing. But if we get people to use virtualization to drive up utilization without realizing it's also saving power, I'm okay.

I'm a bit more dubious about #2. Not that I wouldn't love to have some unification of my communication world, but for me it seems my communications are diverging, not converging. I have far too many mailboxes, phone numbers, and calendars now. Every social network a friend joins just brings another communication channel for me to manage. At some point we may all cry "Uncle!" but right now it feels like we are moving in any direction but unified communications.

The next one that captured my attention was #4, meta data. This is definitely a problem that needs more attention. It is usually left as an afterthought to the overall architecture and yet any system of scale is virtually unusable without a good meta data system, designed in, not added on. I couldn't agree more that this needs to become serious focus in the year to come.

Virtualization is #5 on their list. Personally I'd move this higher. Abstracting deployments from physical instances is incredibly powerful. It is the next step in the logical progression of abstractions (machine language, high level language, virtual machines, and now virtual OS containers). Fortunately there is little for application developers to do on this one but if you aren't designing your applications to live happily inside a virtual container, start now.

It is interesting that the broke out their definition of computing fabrics into a separate item, #8. If my execution platform is sufficiently abstract, then a logical extension is that the hardware platform will be able to gain flexibility in how it allocates physical resources like processors and memory across boundaries within blades. The more interesting fabric to me is a better abstraction of systems like Beowulf so programming becomes more practical. There are an increasing number of social problems that scale far beyond the confines of one system but also don't split will due to the network nature of the data. Environments that let mere mortals program to loosely coupled clusters would help tremendously.

As always with predictions, it's fun to have a look back at the end of the year and see what you got right and what went far astray. I'm sure 2008 will follow several of these trends but will also bring us some interesting surprises. Feel free to share your predictions with comments!

Technorati Tags: architecture, engineering, messaging, programming, scalability, security, services, soa, software, to_read, toread, web