Big data is more than just the latest IT trend, more than just a consultant buzz word, more than another technology improvement. Big data is a big deal. Although it involves more than simply a matter of size, the volumes of data being created, captured, stored and used to create value today are staggering. Let us reflect on the volumes and trends over the past generation with some numbers:
- 8 bits = 1 Byte, a single character
- 1024 B = 1 Kilobytes, a typewritten page
- 1024 KB = 1 Megabytes, a short novel or a photograph
- 1024 MB = 1 Gigabytes, a truck full of paper, or a symphony, or a movie
- 1024 GB = 1 Terabytes, one library
- 1024 TB = 1 Petabytes, thousands of libraries
The pattern keeps going up like that until you end up with numbers so large there aren’t really names for them outside of computer terminology. Ever heard of Exabytes or Zettabytes or Yottabytes (10 to the power of 24, presumably named after the Star Wars character Yoda)?
The practical applications of big data in the fields of science, healthcare, education, government services and business are impressive. But most of the data – estimated at 70% in organizations – remains unstructured (untagged) and difficult to use effectively.
If we look at how big data enables big science, the incredible large hadron collider (LHC) experiments represent about 150 million sensors delivering data 40 million times per second. There are nearly 600 million collisions per second. As a result, only working with less than 0.001% of the sensor stream data, the data flow from all four LHC experiments represents 25 petabytes annual rate before replication (as of 2012). This becomes nearly 200 petabytes after replication.
The dramatic increase in volume, velocity and variety of information is transforming the core capabilities of businesses as well. Data driven decision-making has swept into every industry and business function and are becoming as important a factor of production as labor and capital. This is the Information Age.
The increasing volume and detail of information captured by enterprises, the rise of multimedia, social media, and internet usage will fuel exponential growth in data for the foreseeable future.
Up to now, only very large corporations or government institutions have had access to big data value creation. The technology and skills required to pursue a big data strategy have been too time-consuming and costly for small to mid-sized firms to take advantage of the opportunities created by big data projects.
Big data is particularly a problem in traditional business analytics because standard tools and procedures are not designed to analyze these massive data sets. But the new ROI – Return on Information – for companies with less than $1B in annual revenue is changing.
Innovative service companies are popping up daily providing big data as a service (BDaaS). The trend is to offer firms which cannot afford or do not want to invest in pursuing big data strategies the possibility to take advantage of the opportunity.
Although better decision-making is key, the value proposition is more than business intelligence on steroids to enable better management decisions. Big data projects will collect more accurate and detailed performance information on everything from product inventories to sick days, and make information transparent and usable at much higher frequency.
Big data is already being used to improve the development of the next generation of products and services. For example, manufacturers are using data obtained from sensors embedded in products to create innovative after-sales service offerings such as proactive maintenance. And imagine the sensory data which will be captured and analyzed when nanobots are routinely introduced into human beings to monitor health functions.
In most industries, established competitors and new entrants alike will leverage data-driven strategies to innovate, compete, and capture value from deeper and real-time information. The computer and electronic products and information sectors, as well as finance and insurance, and government, healthcare and education are all poised to gain substantially from the use of big data. Other sectors will follow – and some lead – in unexpected ways leveraging big data frameworks.
Policies related to privacy, security, intellectual property, and even liability will need to be addressed differently in a big data world. We are already seeing the emerging trend appear in public policy making and international laws. as a lag indicator.
It is estimated that every day, we are currently creating 2.5 quintillion bytes of data. This data comes from everywhere: sensors used to gather climate information, posts to social media sites, digital pictures and videos, purchase transaction records, and cell phone GPS signals, to name a few sources. This data is big data. Some savants in the field estimate that as much as 90% of the data in the world today has been created in the last three years alone.