Web Mining and Social Networking: Techniques and ... - tud.ttu.ee

148 7 Extracting and Analyzing Web Social NetworksR merge (c(t k−1 ),c(t k )) = N mg(c(t k−1 ))t k −t k−1. (7.4)The split rate, R split (c(t k−1 ), c(t k )), is the number of split URLs from c(t k−1 ) per unittime. When the split rate is low, c(t k ) is larger than other split communities. Otherwise, c(t k )is smaller than other split communities. The split rate is defined as follows.R split (c(t k−1 ),c(t k )) = N sp(c(t k−1 ))t k −t k−1. (7.5)By combining these metrics, some complex evolution patterns can be represented. Forexample, a community has stably grown when its growth rate is positive, and its disappearanceand split rates are low. Similar evolution patterns can be defined for shrinkage.Longer range metrics (more than one unit time) can be calculated for main lines. Forexample, the novelty metrics of a main line (c(t i ), c(t i+1) , ..., c(t j )) is calculated as follows.Other metrics can be calculated similarly.R novelty (c(t i ),c(t j )) = ∑ j k=i N ap(c(t k ))t j −t i. (7.6)7.1.3 Web Archives and GraphsFor experiments, M.Toyoda and M. Kitsuregawa used four Web archives of Japanese Webpages (in jp domain) crawled in 1999, 2000, 2001, and 2002 (See Table 1). The same Webcrawler was used in 1999 and 2000, and collected about 17 million pages in each year. In2001, the number of pages became more than twice of the 2000 archive through improvingthe crawling rate. The used crawlers collected pages in the breadth-first order.From each archive, a Web graph is built with URLs and links by extracting anchors fromall pages in the archive. The graph included not only URLs inside the archive, but also URLsoutside pointed to by inside URLs. As a result, the graph included URLs outside jp domain,such as com and edu. Table 1 also shows the number of links and the total URLs. For efficientlink analysis, each Web graph was stored in a main-memory database that provided out-linksand in-links of a given URL. Its implementation was similar to the connectivity server [30].The whole system was implemented on Sun Enterprise Server 6500 with 8 CPU and 4GBmemory. Building the connectivity database of 2002 took about one day.By comparing these graphs, the Web was extremely dynamic. More than half URLs disappearedor changed its location in one year. They first examined how many URLs in ourWeb graphs were changed over time, by counting the number of URLs shared between thesegraphs. About 60% of URLs disappeared from both 1999 and 2000 graphs. From 2001 to2002, about 30% of URLs disappeared in four months. The number of URLs surviving throughfour archives was only about 5 million. In [63], Cho reported that more than 70% of pages survivedmore than one month in their four month observation. This result is close to their resultsfrom 2001 to 2002 (30% disappearance in four months). Although it is not easy to estimatethe rate for one year from [63], they say that their results does not deviate so much.7.1.4 Evolution of Web Community ChartsThe global behavior of community evolution is described now. From the above four Webgraphs, four community charts are built using the technique described in Chapter 5(Section 5.4).