Extraction and Integration of MovieLens and IMDb Data - APMD

Verónika Peralta
− Movie type calculation task (cleaning for movies.txt) calculates movie types as derived attributes by
looking for some special characters in the movie title (see Table 9). The task is implemented as a
sequence of SQL operations (one for each movie type) that update the Type attribute according to Table
9, for example:
UPDATE movies-cleaning
SET Type=M
WHERE Mid(MovieTitle, 1, 2) = ’”’
AND Mid(MovieTitle, Len(MovieTitle)-5, 6) = ’(mini)’
Movie type Title is quoted Special ending word Generated attribute
Cinema film No none C
Series episode Yes (mini) S
Mini-series Yes none M
TV series No (TV) TV
Video No (V) V
Video game No (VG) VG
Table 9 – Movie type identification
− Null values elimination tasks (initial cleaning for biographies.txt and business.txt) eliminate tuples having
null values for all attributes (excepting keys). Tasks are implemented as SQL operations like:
INSERT INTO business-cleaning (MovieTitle, Budget, Revenue)
SELECT *
FROM business-source
WHERE Bugnet IS NOT NULL OR Revenue IS NOT NULL;
The implementation for biographies.txt is analogous.
− Splitting tasks (cleaning for several tasks) update several attributes from one concatenating several
concepts. For example, the ProductionCompany column of the productioncompanies.txt file embeds
company names and country codes. The tasks search for special characters (e.g. brackets, square brackets,
braces) and split attribute values according to such characters. Tasks are implemented as a sequence of
SQL operations, for example:
INSERT INTO productioncompanies-cleaning
(MovieTitle, ProductionCompany, CompanyInfo, Aux1)
SELECT MovieTitle, ProductionCompany, Comments,
InStr(1,ProductionCompany,’[‘)
FROM productioncompanies-source;
UPDATE productioncompanies-cleaning
SET CompanyName = Mid(ProductionCompany, 1, Aux1-2),
CountryCode = Mid(ProductionCompany,Aux1+1,Len(ProductionCompany)-Aux1-1)
WHERE Aux1 > 0;
UPDATE productioncompanies-cleaning
SET CompanyName = ProductionCompany
WHERE Aux1 = 0;
The first operation calculates the position of the special character and stores it in the Aux1 attribute. The
second operation performs the split. The third operation is necessary when some values are optional (in
the example, if country code can be omitted). It sets the first attribute and less the second as Null.
The implementation of splitting tasks for business.txt, distributors.txt, releasedates.txt and
runningtimes.txt is similar. Table 10 lists attributes to split and special characters for those files. Columns
of biographies.txt were not split because of their format diversity.
As a special case, the splitting of movielinks.txt is performed by table look-up, i.e. the first attribute
(movie type) is looked up in a table; the remaining of the text corresponds to the second attribute. The
look-up table contains the fifteen link types, namely: “alternate language version of”, “edited from”,
“edited into”, “featured in”, “features”, “followed by”, “follows”, “referenced in”, “references”, “remade
as”, “remake of”, “spin off”, “spoofed in”, “spoofs”, “version of”. The tasks is implemented as follows:
23