Roxana - Gabriela HORINCAR Refresh Strategies and Online ... - LIP6

frequency and in-place page update or page collection shadowing.
Third, some well studied factors that are widely considered in the design of a crawling
policy include:
• importance - of a piece of web content (e.g. web page, RSS feed) relative to other
pieces of content,
• relevance - of a piece of web content to the purpose served by the crawl, and
• content changes - or how the web content tends to evolve in time.
And last, we mention some other problems that must be considered in the crawler design,
such as:
• content avoidance - which is the appropriate way a crawler should detect and avoid
content that is redundant, malicious, wasteful or misleading?
• hidden or deep web - how should the web content available only by filling in HTML
forms be crawled?
• personalized content - in which manner a crawler should treat web sites that offer
personalized content to individual users?
• erased content - in the case a crawler runs out of storage space, how should it select
the pages to retire from its repository?
In the following sections we introduce selected references that are based on some of the
already discussed crawling considered factors and targeted objectives.
3.2.1 First-time Downloading Strategies
This group of techniques focuses on ordering the content for first-time download with the
goal of achieving wide coverage of the available web content. Roughly put, these strategies
download the web content in descending order of importance, where the importance
function reflects the purpose of the crawl.
When the goal is to cover high quality content of all types, an importance score extensively
used in literature is PageRank [PBMW99] and its variations. Three published
empirical studies evaluate different ordering policies on real web data: [CGMP98, NW01,
BYCMR05]. In [CGMP98], the authors use the indegree and the PageRank of web pages to
prioritize crawling. Indegree priority favors pages with higher numbers of incoming hyperlinks.
There is no consensus on prioritization by indegree: [CGMP98] finds that it works
fairly well (almost as well as prioritization by PageRank), whereas [BYCMR05] finds that
it performs very poorly. [NW01] uses breadth-first search, an appealing crawling strategy
due to its simplicity: pages are downloaded in the order in which they are discovered,
where discovery occurs via extracting links from each downloaded page. [BYCMR05] proposes
a crawl policy that gives priority to sites containing a large number of discovered but
36