Extraction and Integration of MovieLens and IMDb Data - APMD

More documents

Recommendations

Info

Extraction and Integration of MovieLens and IMDb Data – Technical Report I_MovieRatings Global ratings on movies (not differentiated par user) I_MovieReleaseDates Dates of different releases I_MovieRunningTimes Running times of movies I_MovieSounds Sound mix of movies I_MovieTypes Types of movies I_MovieWriters Writers of movies I_MovieYears Years of movies I_Occupations User occupations I_Producers Master of producers I_Ratings Master of ratings I_Revenues Master of revenue intervals I_Sounds Master of sounds I_Types Types of movies I_UserRatings User ratings on movies 32 − MovieId: Numeric(4) − Distribution: String(10); unknown meaning − Votes: Numeric(8,2); quantity of votes (decimals are always zero) − Rating: Number(4,2), value between 0 and 10 − MovieId: Numeric(4) − Country: String(40) − ReleaseDate: String(20) − ReleaseYear: Numeric(4) − ReleaseInfo: String(90); further info (release city/festival) − MovieId: Numeric(4) − Country: String(40) − Duration: Numeric(5) − RunningInfo: String(75); further info (version, episodes, media, …) − MovieId: Numeric(4) − SoundMix: String(40) − MovieId: Numeric(4) − Type: String(2); {C=cinema, TV=television, V=video, VG=video game, S=serie, M=miniserie} − MovieId: Numeric(4) − Writer: String(45) − WriterInfo: String(110); further info (role, awards, …) − MovieId: Numeric(4) − Year: Numeric(4) − YearInfo: String(30); further info (shot years) − OccupationId: Numeric(2) − Occupation: String (25) − Producer: String(60) − MovieQuantity: Numeric(3); the number of produced movies − Rating: Number(4,2), value between 0 and 10 (internal use, for classifying ratings) − RevenueUSD: Numeric(15,2); start of an interval (internal use) Primary key: MovieId Foreign keys: − MovieId: ref. I_Movies Primary key: MovieId, Country, ReleaseDate Foreign keys: − MovieId: ref. I_Movies − ReleaseYear: ref. I_Years − Country: ref. I_Countries Primary key: MovieId, Country Foreign keys: − MovieId: ref. I_Movies − Country: ref. I_Countries Primary key: MovieId, SoundMix Foreign keys: − MovieId: ref. I_Movies − SoundMix: ref. I_Sounds Primary key: MovieId Foreign keys: − MovieId: ref. I_Movies − Type: ref. I_Types Primary key: MovieId, Writer Foreign keys: − MovieId: ref. I_Movies − Writer: ref.: I_Writers Primary key: MovieId, Year Foreign keys: − MovieId: ref. I_Movies − Year: ref. I_Years Primary key: OccupationId Primary key: Producer Primary key: Rating Primary key: RevenueUSD − SoundMix: String(40) Primary key: SoundMix − Type: String(2); {C=cinema, TV=television, V=video, VG=video game, S=serie, M=miniserie} − TypeDescription: String(10) − UserId: Numeric(4) − MovieId: Numeric(4) − Rating: Number(1), value in {1,2,3,4,5} − Timestamp: Numeric(9); represents milliseconds from an initial time Primary key: Type Primary key: UserId, MovieId Foreign keys: − UserId: ref. I_Users − MovieId: ref. I_Movies
I_Users Descriptive information about users I_Votes Master of votes I_Writers Master of writers I_Years Master of years I_Zones Master of geographical zones 4.3. Construction of the integrated database − UserId: Numeric(4) − Gender: Char(1); value in {M,F} − Age: Numeric(2) − Occupation: Numeric(2) − ZipCode: String(10) − Votes: Numeric(8,2); indicates the start of a voting interval (internal use, for classifying votes) − Writer: String(45) − MovieQuantity: Numeric(3); the number of written movies − Year: Numeric(4) − Decade: Numeric(4) − Zone: String(30); a country, region, sea, … − Country: String(40) Table 13 – Integrated schema Verónika Peralta Primary key: UserId Foreign keys: − Age: ref. I_Ages − Occupation: ref. I_Occupation Primary key: Votes Primary key: Writer Primary key: Year Primary key: Zone Foreign keys: − Country: ref. I_Countries Matching MovieLens and IMDb movie titles was the most difficult task in the construction of the integrating database. The remaining tasks basically consist in joining the I_Movies table (the master of movies resulting from the matching process) in order to keep movie features of movies included in the master (i.e. assuring referential integrity). This allowed the creation of the appropriate foreign keys. In addition, we created master tables for most movie features. This section describes these tasks. Figure 24 shows an overview of the process for building tables of the integrated database. In order to filter features of movies non-included in the master of movies, we use the orphan elimination task described in Subsection 2.3, which simply joins the IMDb feature table (IMDb_P in Figure 24) with the I_Movies table. Then, some master tables were aggregated using the aggregation task, also described in Sub-section 2.3, which group tuples according to a feature attribute. The aggregation task was not executed for some feature tables (I_MovieCostumeDesigners, I_MovieProductionDesigners) because they have too few repetitive values (for example, each entry of the I_MovieCustomeDesigners table describes a different costume designer). The master tables generated for each table can be deduced from Figure 23, because they coincide with referential constraints (excepting those referencing the I_Movies table). IMDb_P I_Movies orphan elimination aggregation I_MovieP I_P Figure 24 – Construction of tables of the integrated database Three special data reconciliation tasks were performed for years, currencies and countries. For years, we performed the union of years appearing in I_MovieYears and I_MovieReleaseDates tables and then, we calculated decade as a derived attribute, using typical date manipulation functions (similar to the year calculation task described in Sub-section 2.3.2). For currencies, we obtained budget and revenue currencies from the I_MovieBusiness table. We obtained dollar exchange (at October 2006) for such currencies from different Internet sites, e.g. [1] (a better exchange should be obtained by considering currencies at movie running time; we refer this to future work). For countries, we had country names in the I_MovieCountries and I_MovieReleaseDates tables, in addition to some zones of the I_MovieLocations table and we had country codes in the I_MovieDistributors and I_MovieProductionCompanies tables. It was a big mess! Country names were written in different manners (e.g. ‘UK’ and ‘United Kingdom’) and sometimes referenced non-sovereign territories (e.g. French Polynesia) or non-current countries (e.g. ‘URSS’), country codes were also ambiguous. 33
Page 1 and 2: Extraction and Integration of Movie
Page 3 and 4: Movies.dat 1::1193::5::978300760 1:
Page 5 and 6: u.genre Verónika Peralta This file
Page 7 and 8: Table Attributes Constraints SML_Us
Page 9 and 10: Verónika Peralta temporal table st
Page 11 and 12: Verónika Peralta − Manual insert
Page 13 and 14: Verónika Peralta corrections and a
Page 15 and 16: Verónika Peralta File Format Optio
Page 17 and 18: Table Attributes Constraints IMDb_M
Page 19 and 20: IMDb_ReleaseDates Dates of movie re
Page 21 and 22: Verónika Peralta In order to store
Page 23 and 24: Verónika Peralta − Movie type ca
Page 25 and 26: Verónika Peralta The implementatio
Page 27 and 28: ML_Movies SML_Movies IMDb_Movies ma
Page 29 and 30: 4.2. Integrated schema Verónika Pe
Page 31: I_MovieCountries Countries of movie
Page 35 and 36: Verónika Peralta integration proce

Extraction and Integration of MovieLens and IMDb Data - APMD

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?