Extraction and Integration of MovieLens and IMDb Data - APMD

Extraction and Integration of MovieLens and IMDb Data – Technical Report
10
− Aggregate tasks build aggregations (P’ in Figure 13) for storing existing values of a specific attribute, i.e.
master tables (e.g. movie genres). Tasks are implemented as SQL operations of the form:
INSERT INTO P’ (
SELECT column1,... columnK
FROM P-ref
GROUP BY column1,... columnK;
The following sub-sections describe the specific tasks of each data set.
2.3.1. ETL process for the large data set
The large data set was quite consistent: no duplicates were found, no violations to referential constraints were
detected. However, we detected and solved some minor anomalies:
− Genres were represented as multi-valued attributes (pipe-separated lists) in source files. Normalization
routines were performed.
− ZIP codes contained letters and ZIP intervals. We ignored such errors and stored values as Strings.
− Master tables for ages and occupations were created from scratch in order to associate descriptions to age
ranges and occupation ids.
Figure 14 shows the extraction workflow. Note that not all the tasks presented in previous section were
necessary for extracting data from all text files. For ease of understanding, we omitted representing some data
flows and temporary tables; copy tasks were also omitted.
movies.dat
ratings.dat
users.dat
bulk copy
bulk copy
bulk copy
duplicate
chech
duplicate
check
duplicate
check
referential
check
genre
denormalisation
Figure 14 – ETL workflow for the large data set
referential
check
manual
insert
manual
insert
ML_MovieGenres
ML_Movies
ML_Ratings
ML_Users
ML_Ages
ML_Occupations
The semantics of the bulk copy, duplicate check and reference check tasks was described previously; the
semantics of specific validation and aggregation tasks is the following:
− Genre denormalisation task builds a normalized table expressing the relationship among movies and
genres. Each tuple, which has a list of genres, is decomposed in several tuples, one per genre. For
example, from the tuple (see Figure 3) we create
3 tuples , and . The task is implemented as a sequence of
SQL operations that split the genre list according to the token ‘|’.