Making sense of nonstructured nonsense - Trend Spotting in the Over-Information Age

As we look out over the web and remember that all of this information at our finger tips didn't used to be there it's a little hard to believe. The current Generation Y has a reason to be asking as this information has always been there for them. They are used to being perpetually connected and living their lives online. Tools like Twitter and Facebook make connectivity via digital methods a default. Connectivity from access devices such as iPhones or something as strait-forward as SMS provide snippet updates and easy push of information. Even Microsoft is getting into the game with their recent Vine announcement. All the while "normal" information continues flowing out. Papers are published and go to the web, news releases feed directly out, blog posts go out every second. Lots and lots of unstructured data every single day.

This brings about the challenge of the next generation of systems. Simple links to information without context aren't necessarily useful any more. When the search engines first came out the ability to point you in the right direction was enough. It no longer is. If a search engine returns a particularly pithy 140 character tweet on my topic I probably won't be thrilled. If it returns a mesh of links based on that tweet connecting with others, blogs and research topics in a neat visualized package... well that would be cool. 

For example, I am obsessively (those that know me probably are nodding their heads right now... I would ask you to stop please) interested in ants, papers on ants and recent research about ants. I believe that how they work, organize and where collective intelligence emerges provides us with many great clues on building scalable systems out of unstable parts. But in order to keep up on this I need to keep running searches and queries against various engines, look to known research outlets, receive email pointers from friends and build my own views and filters to what I think is worthwhile.  

Another example, twitter and other outlets can give indicators of where traffic problems are. People get stuck in traffic and tweet to their friends they are stuck. What if that information were automatically gathered, from the problem areas themselves, with geocoded info and placed on a google map mashup. Talk about useful. (I will of course conveniently forget to mention the downsides of texting, tweeting, facebooking and driving. Don't do it.)

Visibility to all of this disorganized and free flow data is the next great mountain to climb. Terms such as Crowdsourcing, Trend Spotting, Data Visualization are the buzzwords of the day. For a quick and inspiring note on what you can do with this type of information check out Erik Hersman's Ted Talk.  Knowing what is getting buzzed on Twitter and where the collective consciousness is headed is good and in many cases ahead of "standard" news outlets. How to create tools to view this information in a meaningful way is the challenge we need designers and engineers to be working on. 

In the meantime if you see good ant stuff let me know.