Better Data not Bigger Data

Big Data is Big News. Big consultancies and senior managers are all using the term "Big Data" these days. The words are the "Information Superhighway" of this decade*. Over-used, over-hyped and mis-understood.

In practice I tend to refer as something as "Big Data" if I can't inset it & work with it in an Excel spreadsheet... even though we've has databases for half a century that can deal with more than 1,048,576 rows (which is actually the maximum rows you can get in an Excel spreadsheet).

However there can be a lot of insight that can be gleaned from much smaller data sources. You don't need to have access to every single customer record in a database to analyse most trends about your users. You only need to examine a smaller accurate and representative data set. It can't be incomplete, out of date or incorrect.

So shouldn't we really use the term "Better Data"?

 

*BTW: What is this decade actually called? Sure we had "the Eighties", "the nineties" and even the "noughties"... but are these "the teenies"?

Big Data and The Republican Loss

This is more of a question than an observation, which is why I've posted it on a Sunday (I'm not expecting an answer and I'm assuming most professional users are out or not reading blogs & Twitter ).

I've been reading a fair bit this weekend about Orca, Mitt Romney's voter assessment software, built by an anonymous supplier on a Microsoft platform.

According to various articles, Orca suffered stability and accuracy issues in the run-up to the election last week. Something many people are now claiming cost The Republicans the Presidency of The USA.

Orca was apparently built pretty hastily 7 months ago to rival the effect of a similar piece of kit used successful by The Democrats in 2008 and wheeled-out and improved upon again this year. Both systems apparently crunch 'big data' to identify likely voters and contributors for their respective parties. Only Orca didn't deliver.... It apparently crashed, locked out users and couldn't handle the data entry required of it.

So my question is simply this... If Mitt Romney is from such a prestigious financial management background (Bain), where spreadsheets have been the backbone of accurate decision making for over a decade ... then how come it was this crucial element that was so weak in The Republican campaign?

Does big data mean big problems?

The subject of Big Data is currently reaching the level of hype previously reserved for Social Media, Web2.0 and the dotcom boom & bust before it. Just like these previous concepts... big data is large enough, complex enough and technical enough to bewilder a lot of 'normal' people. And like its predecessors, it is sufficiently hard enough to describe... however I like the explanation:

"Big Data is anything that doesn't fit into an Excel spread sheet"

However it is obvious that some companies are making progress with big data... whilst others, especially the consultancies, are keen to make out they have been involved with big data since before it was big news.

Of interest to me was this video posting by McKinsey, which explains how to deliver big data and the necessary analytics tools into the hands of  managers who need it.
http://www.mckinsey.com/features/advanced_analytics

However I think the challenge with all of this talk about big data is not only of defining it , but also getting started in the subject. You can talk about big data all you like, but how do you actually start making progress with analysing it and when does it actually become database administration (which has been dealing with gigabytes of data since before the dotcom bubble).