Roxana - Gabriela HORINCAR Refresh Strategies and Online ... - LIP6

optimization method. In these works, a famous counterintuitive result was obtained: a
uniform crawl strategy (where all pages are crawled with the same frequency) is superior
to the proportional one (where a page is crawled proportional to its change frequency).
Furthermore, their optimal crawling strategy may never revisit pages that change too
often. The intuition is that the crawling resources (bandwidth) needed to keep weakly
synchronized a fast changing page can be better used to keep strongly synchronized the
pages that change slower for obtaining higher freshness scores. The same goal is also aimed
by [WSY + 02], with the difference that they no longer use a homogeneous Poisson page
change model, but a quasi-deterministic page change model in which page update events
are nonuniform in time and the change distribution is a priori known. The authors of
[EMT01] do not make any particular assumption about the page change evolution model,
but they rather use an adaptive approach that estimates and adapts the page change
frequency on the fly.
Continuous freshness measures typically classify a content element as being ”fresher” than
others. As a first example, [CGM03a] introduces the age of an element as the amount
of time the real and the cached copy of the element have been different. They propose
a refresh strategy that minimizes age. Content-based freshness measures are introduced
in [OP08], where the degree of change is reflected directly by the changes in the page
content. Fragment staleness (they use the reverse of the freshness) is estimated by the
Jaccard set distance on the component fragments of a page. In addition to characterizing
a page by its update frequency, they also introduce the longevity of a page, as the lifetime
of page fragments that appear and disappear in time. The principle of their best effort
refresh strategy is also based on the Lagrange multipliers [Ste91] optimization method,
but they use a variant that employs a utility function. This solution avoids computing the
differential equations system of the classical Lagrange method. This same method was
first presented in [OW02] in the context of cache synchronization with source cooperation
and push protocol and it also represents an inspiration source for our two steps refresh
strategy for RSS feeds introduced in chapter 4. For the case when the page revisitation
purpose is to maintain the index of a search engine, [PO05] proposed a user-centric method
that assigns weights to individual content changes, based on how this impacts the ranking
of the page.
If the crawl purpose is to capture as many individual content updates as possible, then the
refresh strategy goal is to maximize completeness. This is specific to applications such as
web archiving or temporal data mining analysis. In this sense, a first algorithm was proposed
in [PRC03]. In a subsequent work [PDO04], a more general algorithm is introduced
that supports a flexible combination of freshness and completeness optimization.
3.3 Crawling RSS Feeds
A refresh strategy specially conceived for crawling RSS feeds is proposed in [SCC07],
based on the Lagrange multipliers [Ste91] optimization method. The challenge for the
38