Data is no longer relational

Ok, so the headline may be a little of an over statement for effect. Perhaps I could have said that data that people are interested in is no longer relational but then it wouldn't have been nearly so pithy.

With much respect to Edgar Codd and his invention of the relational model for database storage I think it is time to move forward. Relational databases are great for things like financial models, personnel data and other Enterprise Systems as well as many other standard, repetitive data. It gave a solid reference point to learn data structures and modeling to several generations of budding Computer Scientists.

When I say it is time to move forward I am not saying we should immediately move all data systems to Object Oriented Databases or otherwise induce data chaos. What I do want to push is the idea of unstructured data. Computers are great with rules, with structure and with fundamentally binary relationships. Algorithms are starting to mature around unstructured data (for an example go search google.) but it is still not widespread or well understood.

New exciting algorithms such as Amazon's Dynamo (Werner Vogels is one of my favorite speakers and bloggers on distributed tech, if you are not familiar with him in this space you should be) database are showing in real world situations that distributed systems and distributed data are a reality.

In a lot of system designs because people are so familiar with relational data structures and systems we find Object Models that look like a relational database design. When asked why it looks like this the answers are fairly consistently things like "this is how the database stores it, for speed we need to do the same" or "it just made sense when we pulled the DBA in to help us with the model."

Objects are not relational! They are objects. Then when you get into full structures of objects or object trees there are relationships but it is not the same as a relational database. Especially as we start to build and mature distributed system algorithms it doesn't make sense to use a centralized data store. If the data can be broken up, distributed and stored with the algorithms that will use it performance will improve.

In fact I would argue that the elusive SLA of a system response can begin to be discussed if we can tie the data to the processing. Granted there are new complexities in this model for synchronization, segmentation and consistency but there are ways to solve them. Similarly consistent access to the same servers is also possible.

What other great examples of distributed computing and distributed data storage have you seen?

You wouldn't move without a destination for your stuff...

Consider this story. You gets a transfer to a new city. It's a great opportunity, great job, benefits and salary. The new company offers you a relocation package where they will help you with housing, selling your current home and everything. It all just falls into place, your home sells in short order and you are ready to go. You pack up all of your belongings to move out to your new city. Couches, Televisions, Cars, Toys etc. Everything is picked up by the movers put into the 18 wheeler with care and zooms off down the road. You get on a plane and head to your new city, ready to get on with your life. You land and are ready to head to your new home and realize you don't have one. You never set anything up. No new home purchased, no apartment, nothing. Oops. Your belongings have no where to go. You call your company and tell them they need to get you a house and... what do you know... they just laugh.

Ok... so this story is a little strange and something you are probably thinking "I wouldn't do that. What kind of idiot would pack everything, be ready to move and not have a destination?" Not many right...

Actually you would be surprised. While not a house and a couch we get requests all the time for a home for an application. An application that has to be delivered in just two weeks! This is when the inside voice starts to emphatically state "Failure to plan on your part does not constitute and emergency on mine." Of course that is the inside voice and not the outside voice.

What can you do about it? If something is being built, know where it is going to go. Don't just assume, know. Know what the requirements are up front, what the application will run in, on and through. It seems obvious that you would want a planned home for a product but when you get busy coding and building those types of things are easy to loose track of or to assume that "someone" is taking care of it.

The principles of Tao Teh Ching in Software Development

You may be asking if I drank a little too much Green Tea to be comparing Software development to the principles of Tao Teh Ching but please bear with me on this one. Honest, it makes sense. (At least I hope it will after I write it down and it's outside of the world inside my head) First, the basic principles. (there is a lot more in the book as its a compilation of a good deal of Chinese Philosophy)

That which remains, is easy to handle.

That which is not yet developed is easy to manage.

That which is weak is easy to control.

That which is still small is easy to direct.

Deal with little troubles before they become big.

Attend to little problems before they get out of hand.

For the largest tree was once a sprout, the tallest tower started with the first brick, and the longest journey started with the first step.

Just as these wise words say, things are easier to handle early on. Once they get rolling or get bigger though many times there is too much momentum to control it. Just like a small snowball at the top of a mountain can become a massive avalanche so too can a small problem ignored early in development become a costly, potentially project killing problem later in the development cycle.

Releaseing early and often during software development makes sense. By focusing our teams on the first things that really matter we can focus our efforts on the things that make a difference. By focusing on the things that we expect to be hard we can ensure we have enough time to manage the risk associated with them. Identifying things early allows us to adapt our plans and make the best set of decisions.