Extraction and Integration of MovieLens and IMDb Data - APMD

Table Attributes Constraints
SML_Users
Demographic information
about users
SML_Movies
Information about movies
SML_Ratings
User ratings on movies
SML_Genres
Description of genres
SML_MovieGenres
Genres of movies
− UserId: Numeric(4)
− Age: Numeric(2)
− Gender: Char(1); value in {M,F}
− Occupation: String(15); finite set
− ZipCode: String(5)
− MovieId: Numeric(4)
− MovieTitle: String(100)
− ReleaseYear: Numeric(4)
− Imdb: String(150); represents the URL of an
IMDb page
− UserId: Numeric(4)
− MovieId: Numeric(4)
− Rating: Number(1), value in {1,2,3,4,5}
− Timestamp: Numeric(9); represents
milliseconds from an initial time
− Genre: String(12)
− GenreId: Numeric(2)
− MovieId: Numeric(4)
− GenreId: Numeric(2)
Table 2 – MovieLens schema (small data set)
2.3. MovieLens extraction, cleaning and transformation processes
Primary key: UserId
Primary key: MovieId
Unique: MovieTitle
Verónika Peralta
Primary key: UserId, MovieId
Foreign keys:
− UserId: ref. SML_Users
− MovieId: ref. SML_Movies
Primary key: GenreId
Primary key: MovieId, GenreId
Foreign keys:
− MovieId: ref. SML_Movies
− GenreId: ref. SML_Genres
Both data sets were quite consistent; however, we detected and solved some kinds of anomalies:
− Multi-valued attributes: Some source attributes are provided as comma-separated (or pipe-separated) lists
of values. As a solution, we normalise tables.
− Null or dummy values in mandatory attributes: Some mandatory attributes (especially key attributes)
contains Null or dummy values (e.g. MovieTitle = 'Unknown'). As a solution, such tuples are eliminated.
− Duplicates: Some tuples (or tuple keys) are duplicated. As a solution, we keep a unique tuple, reconciling
values of further attributes if necessary.
− Orphans: Some tuples do not satisfy referential constraints, i.e. their values are not included in a foreign
table. As a solution, we eliminate orphan tuples.
In order to solve such anomalies we implemented an ETL process for extracting, cleaning and transforming data
of each source file. ETL workflows consist of five major tasks as illustrated in Figure 12 (rectangles represent
P.txt
extraction
P-source
cleaning
P-cleaning
unduplication
P-group
referenciation
P-duplicates P-orphans
Figure 12 – High-level ETL process
referential
P-ref
exportation
P
P’
7