Building Machine Learning Systems with Python - Richert, Coelho

Big(ger) Data
While computers keep getting faster and have more memory, the size of the data
has grown as well. In fact, data has grown faster than computational speed, and this
means that it has grown faster than our ability to process it.
It is not easy to say what is big data and what is not, so we will adopt an operational
definition: when data is so large that it becomes too cumbersome to work with, we
refer to it as big data. In some areas, this might mean petabytes of data or trillions of
transactions; data that will not fit into a single hard drive. In other cases, it may be
one hundred times smaller, but just difficult to work with.
We will first build upon some of the experience of the previous chapters and work
with what we can call the medium data setting (not quite big data, but not small
either). For this we will use a package called jug, which allows us to do the following:
• Break up your pipeline into tasks
• Cache (memoize) intermediate results
• Make use of multiple cores, including multiple computers on a grid
The next step is to move to true "big data", and we will see how to use the cloud
(in particular, the Amazon Web Services infrastructure). We will now use another
Python package, starcluster, to manage clusters.
Learning about big data
The expression "big data" does not mean a specific amount of data, neither in the
number of examples nor in the number of gigabytes, terabytes, or petabytes taken
up by the data. It means the following:
• We have had data growing faster than the processing power
• Some of the methods and techniques that worked well in the past now need
to be redone, as they do not scale well