Adding Simplicity - An Engineering Mantra

NoSQL, SQL, Is that the Question?

Is it NoSQL or NOSQL (Not Only SQL)? Or is it SQL on MySQL, Postgres, or Oracle? Or maybe, just files? My simple answer is most likely any and all, once you go through the process of defining your requirements at a higher level. In many ways, I find the debate about SQL and the flavors of other non-sql stores to be a little silly as they both have a role as do files. The question has to start with what qualities do you need from your persistence for your platform. Even that will uncover that you probably have different needs for different bits of data that you need to keep.

That said, it is very exciting to me to have some serious choice. Roll the clock back a scant 5 years and your only practical choice was a SQL database (either commercial or open source) with your data most likely stored on a SAN. The simple problem of matching importance of the data to the cost of storage was very constrained, with perhaps only 50% cost difference separating your most expensive SAN space from your least expensive. Now we have a range of solutions that give us some real choice when matching requirements to solutions. But what are those requirements?

Life Cycle - All data has a meaningful life cycle. Very little data really is meaningful forever. Data you use to run your business needs to stick around for at least 7 years, but beyond that has diminishing value. Shopping carts may only be meaningful for a few days or weeks. How long the data is meaningful is often one of the drivers in the next requirement...

Business Importance - Ultimately if you're putting data on disk, that means it has a use for the duration of it's life cycle. The importance of the data is directly proportional to the business impact should it be lost. If you lose your order history, that will be very bad for business. If you lose a shopping cart, that might be irritating to the impacted users, but the business impact will be considerably less.

Availability - Obviously if you stored the data, you'd like to get back to it. Again, it's important to understand the impact to your application should the data be temporarily unavailable. Data that is always available is costly to achieve and comes with other interesting challenges (i.e. CAP theorem applies).

Scaleability - What volume of transactions will the data need to support? And what is the mix of read to write? The volume of data plays a role here as well, but in most scale problems it's the transaction rate more than the data volume that presents challenges in scaling.

Access Patterns - Clearly the types of access that are required to find the data is important. This is the center of the debate about SQL vs NoSQL. The business value of the perceived access patterns should be carefully reviewed when defining these requirements. Patterns that lend marginal business value, may not really be requirements at all.

Each storage platform has strengths and weakness in each of these areas. What is important is to understand these as well as your application requirements. For each piece of data that you will want to store, match these requirements and the answer of how to store the data becomes clear in most cases. The question really isn't SQL vs NoSQL, but as the NOSQL camp likes to state, it's about selecting the tool that best meets your requirements for each piece of data you need to persist.

One requirement I would like to discuss a bit further is business importance as I feel it's only with the introduction of serious alternatives to SQL on SAN that we have an opportunity to better match costs of storage to value to the business. SAN's allow the raw storage they control to be configured as RAID 0 through 6, although some RAID controllers will support a subset of these options. Most organizations rely on either RAID 1 or 5 for protecting themselves against the loss of a disk drives within the SAN. This leads to an overhead that reduces the protected storage vs the raw storage in the SAN.

RAID 1 is accomplished via mirroring and while 2 drives provides protection, 3 drives are required to ensure you are never running without protected storage. This means that you need to purchase 3X the raw storage that you want for protected capacity. RAID 5 improves the situation considerably allowing protection overhead that is as little as 125% raw storage for protection, although many organizations use double parity protection in RAID 5 which results in a 40% overhead. Unfortunately SAN's bring their own cost overhead with them due to the cost of controllers and other support hardware. SAN storage can range from $1,000 to $2,000 per TB for raw capacity. In the world of dropping storage costs, this is still a very high cost per TB for raw storage, especially when direct attached drives can be as little as $300/TB.

NoSQL (and to a large extent, even MySQL) platforms can take advantage of this inexpensive storage and protect data through replication, much like RAID 1. But since the cost of storage is dramatically less, the cost of protected storage is also less. There is also more flexibility available, allowing the number of copies of data retained to be easily matched to the value the data represents to the business. Mission critical data may have 6-8 copies, spread across two geographically diverse data centers. Low value data, may only have 2 copies in one data center. The opportunity to tune your costs to your business value has finally become practical and at lower price points than have been previously available.s

This is currently a hot area for discussion and debate. I encourage and welcome feedback and comments on the ideas that I have put forth here. Let me conceded now that I am not a RAID expert, so if you would like to elaborate and correct any mistakes I've made, please do.

architecture, database, engineering, nosql, programming, scalability, services, shard, soa, software, to_read, toread, sql, cassandra, mysql, mongodb, oracle, postgres, postgresql

Architecture, What Is It?

This shouldn't start any spirited debate. Attempting to define software architecture. It's only been debated and discussed for as long as the title "Software Architect" has existed, but recently, it's become a curious role. As software processes compress, relying on agile iterations and continuous deployment, the boxes in the process charts that were typically where the architect reigned supreme are shrinking and disappearing. What then becomes the role of the architect? Do they disappear completely?

The real question is what is the role of the architect in the larger view of the organization. I believe that architects are responsible for ensuring both technology and processes will meet the business needs as effectively and efficiently as practical. Practicality of course is an ever present constraint because no company has infinite time and resources to apply to their platforms. They need to ship capabilities that bring value to customers or there is no business.

That is certainly simple enough to state in broad, general terms, but what does it mean in practical, actionable terms. For me, it comes down to two major jobs.

1. Provide consulting and guidance on the iterations to ensure they are staying within parameters that will let the organization evolve over time.
2. Be strategic and far enough out in front of the current product iterations that you can guide the teams as they evolve.

This is a shift from the way architecture used to work. Architects need to be come more agile as well, viewing themselves as a service to the engineers rather than mandating approaches. Let's explore each job a bit further.

Architects as Consultants

The role of architect in most organizations involves working on big designs, setting the top level component structure, and reviewing the work of engineers to ensure that the solution conforms to a set of company standards. The architect lives in the upper portions of the organization which places them in areas of broader authority than the engineers responsible for implementation. The architect also has more responsibility and accountability, so broader authority would seem appropriate. This works reasonably well when an organization is following a traditional waterfall methodology. But, it's become clear that waterfall doesn't serve any constituent of the business particularly well, which is why it's being abandoned in favor of agile.

With agile, you've conceded you won't think of everything up front, so solve the problems you do understand, and iterate. This changes the job of the architect drastically. Now the job changes from ensuring the perfect architecture up front to identifying the boundaries of the architecture and ensuring iterations stay within those bounds.

In the past I always viewed architecture as laying out the components and their dependencies. The result is a nice box that everything fits in neatly and allows a system to be built that lives within the box. Now I view architecture as a pair of fences that stretch on infinitely. My job is to set the fences far enough apart to let software engineers be successful during their iterations but close enough together to protect the present and future of the business. I worry less about what's between the fences because that will change frequently as the platform evolves through iteration.

Shifting to that perspective changes the role from one of governance and mandates to one of consulting. The architect works with the engineers to help guide iterations to stay between the fences, but relies on the engineers to define components based on the best knowledge they have at the time. Ideally components are relatively persistent and evolve over time, but in some cases, they may be combined or split as the product evolution dictates. What used to be considered pure architecture now iterates as well and the architect is most effective as a consultant instead of commander during these iterations.

Architects as Strategists

Carrying my visualization further, if the architect is responsible for laying the fences that engineers iterate between, they have to know where to place the fences. The only way to be successful at this job is to know where the business is heading. More than just knowing the current business strategy, it's important to follow technology trends to help the business understand how new technology might influence strategy.

It is important therefore for architects to constantly seek and assimilate knowledge. The rate at which technology changes has accelerated in the past 20 years dramatically. The evolution of technology requires refreshing solutions every 3 to 5 years or organizations run the risk of being out paced by competitors. Design patterns and processes evolve quickly to keep pace with emerging technology. Architects need to ensure they are keeping the business current and relevant in their market.

architecture, careers, engineering, programming, scalability, software, to_read, toread