1.2 Data Sets Bigger Than the Largest Library
3
been termed “Big Data” and are creating an increasingly accurate digital picture of
our physical and social world, as well as the global economy.
“Big Data” will certainly change our world. The term was coined more than
15 years ago to describe data sets so big that they can no longer be analyzed using
standard computational methods. If we are to benefit from Big Data, we must learn
to “drill” and “refine” it into useful information and knowledge. This is a significant
challenge.
The tremendous increase in the volume of data is attributable to four important
technological innovations. First, the Internet enables global communication between
electronic devices. Second, the World Wide Web (WWW) has created a network
of globally accessible websites, which emerged as a result of the invention of the
Hypertext Transfer Protocol (HTTP). Third, the emergence of social media platforms
such as Facebook, Google+, WhatsApp and Twitter has created social communication
networks. Finally, a wide range of previously offline devices such as TV sets, fridges,
coffee machines, cameras as well as sensors, smart wearable devices (such as activity
trackers) and machines are now connected to the Internet, creating the “Internet of
Things” (IoT) or “Internet of Everything” (IoE). Meanwhile, the data sets collected
by companies such as eBay, Walmart or Facebook, must be measured in petabytes—
1 million billion bytes. This amounts to more than 100 times the information stored
in the US Library of Congress, which is the largest physical library in the world.
Mining Big Data offers the potential to create new ways to optimize processes,
identify interdependencies and make informed decisions. However, Big Data also
produces at least four major new challenges (the “four V’s”). First, the unprecedented
volume of data means that we need immense processing power and storage capacity
to deal with the huge amounts of data. Second, the velocity at which data must be
processed has increased: now, continuous data streams must often be analyzed in
real-time. Third, Big Data is mostly unstructured, and the resulting variety of data is
difficult to organize and analyze. Finally, the veracity of the data may be difficult to
handle because Big Data tends to contain errors and is usually neither representative
nor complete.
1.3 Will a Digital Revolution Solve Our Problems?
Let us see what an evidence-based approach building on the wealth of today’s data
can do for us. In the past, whenever a problem had to be solved, the best course
of action was to “ask the experts”. These experts would go to the library, collect
up-to-date knowledge, and supervise Ph.D. students who would help to fill gaps
in existing knowledge. But this was a slow process. Nowadays, whenever people
have a question, they ask Google or consult Wikipedia, for example. This might not
always give the definitive or best answer, but it delivers quick answers. On average,
decisions taken in this way may even be better than many decisions made in the past. It
is no wonder, therefore, that policymakers love the Big Data approach, which seems
3
been termed “Big Data” and are creating an increasingly accurate digital picture of
our physical and social world, as well as the global economy.
“Big Data” will certainly change our world. The term was coined more than
15 years ago to describe data sets so big that they can no longer be analyzed using
standard computational methods. If we are to benefit from Big Data, we must learn
to “drill” and “refine” it into useful information and knowledge. This is a significant
challenge.
The tremendous increase in the volume of data is attributable to four important
technological innovations. First, the Internet enables global communication between
electronic devices. Second, the World Wide Web (WWW) has created a network
of globally accessible websites, which emerged as a result of the invention of the
Hypertext Transfer Protocol (HTTP). Third, the emergence of social media platforms
such as Facebook, Google+, WhatsApp and Twitter has created social communication
networks. Finally, a wide range of previously offline devices such as TV sets, fridges,
coffee machines, cameras as well as sensors, smart wearable devices (such as activity
trackers) and machines are now connected to the Internet, creating the “Internet of
Things” (IoT) or “Internet of Everything” (IoE). Meanwhile, the data sets collected
by companies such as eBay, Walmart or Facebook, must be measured in petabytes—
1 million billion bytes. This amounts to more than 100 times the information stored
in the US Library of Congress, which is the largest physical library in the world.
Mining Big Data offers the potential to create new ways to optimize processes,
identify interdependencies and make informed decisions. However, Big Data also
produces at least four major new challenges (the “four V’s”). First, the unprecedented
volume of data means that we need immense processing power and storage capacity
to deal with the huge amounts of data. Second, the velocity at which data must be
processed has increased: now, continuous data streams must often be analyzed in
real-time. Third, Big Data is mostly unstructured, and the resulting variety of data is
difficult to organize and analyze. Finally, the veracity of the data may be difficult to
handle because Big Data tends to contain errors and is usually neither representative
nor complete.
1.3 Will a Digital Revolution Solve Our Problems?
Let us see what an evidence-based approach building on the wealth of today’s data
can do for us. In the past, whenever a problem had to be solved, the best course
of action was to “ask the experts”. These experts would go to the library, collect
up-to-date knowledge, and supervise Ph.D. students who would help to fill gaps
in existing knowledge. But this was a slow process. Nowadays, whenever people
have a question, they ask Google or consult Wikipedia, for example. This might not
always give the definitive or best answer, but it delivers quick answers. On average,
decisions taken in this way may even be better than many decisions made in the past. It
is no wonder, therefore, that policymakers love the Big Data approach, which seems