The Economist on the Data Deluge

The Economist has a special report on the volumes of data that is now being regularly generated with a 2008 study by the International Data Corp (IDC) reporting that about 1,200 exabytes of digital data (equivalent to 10 million copies of The Economist) will be generated this year.

With the world becoming increasingly digital, the analysis of the data, and its aggregation, is expected to bring many benefits to different fields, from healthcare, to government, to the business processes and supply chains, to name just a few. Because of the tremendous amounts of data that are now becoming increasingly available through sensors, mobile technologies, cameras, computers, social networking sites, etc., business intelligence techniques, coupled with analytics, that can yield insights from statistical analyses are becoming very important. The deluge of data is so large that it is starting to overwhelm storage capacities of computers (not to mention the ability of humans to process).

Of course, as the report notes, the IT industry is diving deep into business intelligence and notes such companies as Accenture, IBM, and SAP.

I would have liked to have seen in this special report a greater emphasis given to optimization, although it did mention revenue and yield management. The writeup on the visualization of data was especially interesting to me and it highlighted the classic book, The Visual Display of Quantitative Information, by Edward Tuffe. Those of us who work in network analysis have long appreciated the power of networks to represent and depict numerous systems and phenomena.