The Cathedral and the Bazaar and the Ignorance of Crowds


CIO.com recently did an interview with Linus Torvalds and the rules he lays out for quality code development are very good. It goes well with my theory of each development team needing a Benevolent Dictator role.


Some time ago I was forwarded a great article on Innovation as it relates to Open Source. Several of the key observations were saved for the last page. I thought it would have been good to draw some suggested next steps so I thought I took a crack at it and the items below are the result.

The article by itself has a good deal of value. But for complete understanding you probably need to have the previous context (inferred from the article is probably sufficient for basic understanding) of The Cathedral and the Bazaar. Wikipedia has a good simple summary of the paper. At it's simplest though it is a paper written by Eric Raymond about how "given enough eyeballs, all bugs are shallow." This means that it becomes easier to drill to quality because more people with disparate backgrounds means better test coverage and bug finding. (And a lot more but that's not what this post is about.)

My actual observations from the Strategy and Business article: I have historically referred to the needed role to make any product development work, whether Bazaar or Cathedral, as the Benevolent Dictator role. It is someone with the vision and strength of personality to push or pull those around them into a unified functional reality of a product. Without that role you get a lack of direction and vision and problems ensue. We have seen this need prove out over and over with the hardening work that we have done. Those teams that had a Benevolent Dictator that laid down tenets for development, surrounded themselves with a “Wizard Enclave” of smart folks for enforcement and saw it through generally had much better systems. This showed in test coverage, uptime, ability to adapt and change and many other areas.

I am a big fan of Open Source developed products and have been for some time. It has certainly proven to be the case that those that are successful have this Wizard/Leader/Benevolent Dictator that surrounds themselves with the Wizards that get there via meritocracy. From this “Brain” squad comes unification and initial innovation. Ongoing innovation can be pulled from the Chaos of the Crowd but to become real will need to be driven by the Benevolent Dictator.
When it comes to our own efforts, Internal Open Source and Product Development need to be set up with this type of role and if we can’t identify someone to fill this role then we can’t honestly expect to move that product forward.

Data is no longer relational

Ok, so the headline may be a little of an over statement for effect. Perhaps I could have said that data that people are interested in is no longer relational but then it wouldn't have been nearly so pithy.

With much respect to Edgar Codd and his invention of the relational model for database storage I think it is time to move forward. Relational databases are great for things like financial models, personnel data and other Enterprise Systems as well as many other standard, repetitive data. It gave a solid reference point to learn data structures and modeling to several generations of budding Computer Scientists.

When I say it is time to move forward I am not saying we should immediately move all data systems to Object Oriented Databases or otherwise induce data chaos. What I do want to push is the idea of unstructured data. Computers are great with rules, with structure and with fundamentally binary relationships. Algorithms are starting to mature around unstructured data (for an example go search google.) but it is still not widespread or well understood.

New exciting algorithms such as Amazon's Dynamo (Werner Vogels is one of my favorite speakers and bloggers on distributed tech, if you are not familiar with him in this space you should be) database are showing in real world situations that distributed systems and distributed data are a reality.

In a lot of system designs because people are so familiar with relational data structures and systems we find Object Models that look like a relational database design. When asked why it looks like this the answers are fairly consistently things like "this is how the database stores it, for speed we need to do the same" or "it just made sense when we pulled the DBA in to help us with the model."

Objects are not relational! They are objects. Then when you get into full structures of objects or object trees there are relationships but it is not the same as a relational database. Especially as we start to build and mature distributed system algorithms it doesn't make sense to use a centralized data store. If the data can be broken up, distributed and stored with the algorithms that will use it performance will improve.

In fact I would argue that the elusive SLA of a system response can begin to be discussed if we can tie the data to the processing. Granted there are new complexities in this model for synchronization, segmentation and consistency but there are ways to solve them. Similarly consistent access to the same servers is also possible.

What other great examples of distributed computing and distributed data storage have you seen?

Monkeys down a fire pole

How many monkeys can slide down a fire pole in 5 minutes?

Ok, so this question doesn't seem to have a lot to do with technology or engineering when you fist consider it. But give me a little bit of patience and let's see where we go.

I have recently begun to undertake a personal quest for education and education provision to others about threading and how to do it well and efficiently. With the introduction of multi-core processors and processor companies no longer creating faster and faster single core CPUs with clocking, over-clocking and over-over-clocking, it has become more important to actually understand threads.

Even as recently as when Java first came out the only places that you really ran into threads in open systems was in Solaris (or other Unix) systems with big multi-CPU backplanes. These were usually reserved for big 'ol database boxes though so you could optimize your system to really what amounted to one big fire pole. If we imagine each processor as a fire pole and each unit of work to run as a monkey, we can start to use my fun little analogy.

Everything you did was about queuing up the monkeys efficiently then sending them down the pole as quickly as possible for each individual Monkey thereby maximizing your monkey throughput. Great schemes for keeping the Monkeys in the right order, keeping the pole clean during non-monkey sliding moments (Garbage collection in Java-land) and other similar optimizations were the wave of the day.

Now the fire poll manufacturers (Intel and AMD) discovered that they just couldn't build a faster fire pole without violating the laws of physics (and it's not that they wouldn't break those laws, more that they couldn't figure out how) and therefore decided the best way to move forward was to build a multi-fire pole firehouse in the same amount of space. It was a great innovation... but the monkeys were still optimized for a single fire pole. So in many cases (not all, but many) the fire pole (CPU) utilizations actually was very low even though other areas were slammed. Monkey lines (memory) and even Monkey storage (I/O) became bottlenecks with a low 25% pole usage.

The time has come to build education on efficient fire pole usage. We need more effective ways to slide those little monkeys down. There are some basic practices that seem to make sense. Sun recently explained how they are building better threading into Java and things are rolling out from there. That's a great start... but that is just a start. To really use the power we are now putting into machines we need to figure out better ways to feed the cores.

Help me save the monkeys... do you have any good tips, observations, sites or do you have any bad monkey practices that you have seen that should be avoided? This practice is, of course, not limited to Java but C, C++, Ruby and others. I would like to pull these together to be shared with everyone. If you have a good monkey tail please drop a comment. I would love to hear from you.

Not dead yet...

Rumors of my death have been greatly exaggerated. Ok. So maybe there weren't really rumors of death. I did hear from enough of my regular readers though that I wanted to pop a quick post out to declare that I am still alive just HORRIBLY busy at the moment. I normally write my blogs at night and post when I get a chance but of late I have been working regular work during that time. Rude isn't it?

Rather than simply declare that I am alive (and whine that I am busy) I will share a bit of scalability learning as well. I was recently sent a link to a wonderful article on the stability journey of MySpace.com (if you don't know what MySpace is you probabl live under an internet rock) in Baseline Magazine. It's a fairly quick read and it discusses many of the standard things you learn with scale. (Partitioning, Virtualization, Simplicity, Segmentation etc.)