Learning Data Mining with Python

Chapter 12
In big data, we can't load our data into memory, In many ways, this is a good
definition for whether a problem is big data or not—if the data can fit in the memory
on your computer, you aren't dealing with a big data problem.
Application scenario and goals
There are many use cases for big data, in the public and private sectors.
The most common experience people have using a big-data-based system is
in Internet search, such as Google. To run these systems, a search needs to be
carried out over billions of websites in a fraction of a second. Doing a basic
text-based search would be inadequate to deal with such a problem. Simply
storing the text of all those websites is a large problem. In order to deal with
queries, new data structures and data mining methods need to be created and
implemented specifically for this application.
Big data is also used in many other scientific experiments such as the Large
Hadron Collider, part of which is pictured below, that stretches over 17 kilometers
and contains 150 million sensors monitoring hundreds of millions of particle
collisions per second. The data from this experiment is massive, with 25 petabytes
created daily, after a filtering process (if filtering was not used, there would be 150
million petabytes per year). Analysis on data this big has led to amazing insights
about our universe, but has been a significant engineering and analytics challenge.
[ 273 ]