Berry

CHAPTER
17
Preparing Data for Mining
As a translucent amber fluid, gasoline—the power behind the transportation
industry—barely resembles the gooey black ooze pumped up through oil
wells. The difference between the two liquids is the result of multiple steps of
refinement that distill useful products from the raw material.
Data preparation is a very similar process. The raw material comes from
operational systems that have often accumulated crud, in the form of eccentric
business rules and layers of system enhancements and fixes, over the course of
time. Fields in the data are used for multiple purposes. Values become obsolete.
Errors are fixed on an ongoing basis, so interpretations change over time.
The process of preparing data is like the process of refining oil. Valuable stuff
lurks inside the goo of operational data. Half the battle is refinement. The
other half is converting its energy to a useful form—the equivalent of running
an engine on gasoline.
The proliferation of data is a feature of modern business. Our challenge is to
make sense of the data, to refine the data so that the engines of data mining can
extract value. One of the challenges is the sheer volume of data. A customer
may call the call center several times a year, pay a bill once a month, turn the
phone on once a day, make and receive phone calls several times a day. Over
the course of time, hundreds of thousands or millions of customers are generating
hundreds of millions of records of their behavior. Even on today’s computers,
this is a lot of data processing. Fortunately, computer systems have
become powerful enough that the problem is really one of having an adequate
539