Big Data Comes of Age
Many years ago, Walter Wriston, former Chairman and CEO of Citicorp, startled the world by claiming information about money has become almost as important as money itself. More recently, the World Economic Summit at Davos published a report implying it was an economic asset as valuable as currency or gold.[i]
The difference between ‘almost’ and ‘just’ might seem more semantic than systemic, but the reality is quite the opposite. The massive growth in technology in recent years is causing an explosive growth in information about ourselves, our lives and the world in which we live.
Thanks to the internet, not since the Great Library in Alexandria has so much information been stored in one place at one time and with far less chance of burning down. Of course it is not just in one ‘place’ but distributed across a mass of servers, nodes and networks accessible to all of us who hold an internet ‘library card’ (address) thanks to an open sourced software framework called Hadoop.
The implications of this massive explosion of data are huge. We call it ‘Big.’ And big it is, growing at 50% per annum according to IDC estimates and by even more according to others.[ii] But not just big in terms of growth, but also scale, because it covers our tweets, our wikis, our blogs, our photos, even our conversations and gestures thanks to dramatic progress in the way we engage with computers.
Big Data is different from traditional data in that it is more varied in format and structure, more volatile and often travels much more rapidly than traditional forms of information. It is also more valuable, because it provides granular information to marketers as to who we really are and offers new ways for corporations and financial institutions to engage with us with pinpoint accuracy.
In contrast’ traditional’ data is structured, machine readable (by traditional machines) and resides on data bases organized around planned queries – questions we thought we wanted answers to in some cases years ago. But in the last few years, we know the world has changed dramatically.
We have new questions and they are growing all the time. As a result, much of the way corporations are approaching data management and data warehousing in the past has to be revised.
Just a few years ago we believed free markets were self-correcting without knowing as much about them as we thought we did. Now we can manage financial markets in real time with the precision of a Swiss watch. We have the technology to achieve this, but not always the will to implement it.
For those of us who believe many of the world’s problems – war, diseases, poverty, hunger – are largely caused by ignorance, Big Data promises us a new Utopia.
The time has come to grasp it as quickly as we can.
[i] “Big Data, Big Impact: New Possibilities for International Development” World Economic Forum
[ii] “The Age of Big Data” Steve Lohr, New York Times, February 11th, 2012