Roxana - Gabriela HORINCAR Refresh Strategies and Online ... - LIP6

fetch new or updated information or they receive information that suits their interests
from the content providers that push it to the aggregators.
One of the earliest search services that adopted a push model is the Harvest system
[BDH + 95], but this approach did not succeed on the long run. There are a number of
papers that discuss server initiated events (push), often in comparison with pull techniques.
Most of them focus on distributed publish-subscribe systems [JA06, SMPD05, RPS06,
RMP + 07, CY08, ZSA09], multi-casting with a single publisher [AFZ97, FZ98, TV00] and
AJAX applications [BMvD07]. In a more recent work, [STCR10] uses a hybrid push-pull
refresh strategy for materializing users views over event feeds.
An initiative to define a push based protocol for RSS feeds has been proposed by the
PubSubHubbub [pub] open protocol for distributed publish-subscribe communication on
the Internet. The main purpose is to provide near-instant notifications of change updates,
which would improve the typical situation where a client periodically polls the feed server
at some arbitrary interval. In essence PubSubHubbub provides pushed RSS/Atom update
notifications instead of requiring clients to poll whole feeds. However, the PubSubHubbub
adoption heavily depends on whether both RSS feeds publishers and clients support the
protocol extensions, which is rarely the case. For example, [URM + 11] estimates that only
2% of their 180, 000 feeds data set support the PubSubHubbub protocol.
Other researchers [OW02] have used the push protocol with a source cooperation approach,
where data sources actively notify the remote data cache servers about their changes. They
propose different types of divergence metrics and while lag is similar to our window/feed
divergence measures, these metrics are conceived for a different application context and
only partially reflect the particularities of RSS feeds.
Virtually all content providers and aggregators adopt the pull approach. Content providers
sometime exclude all or part of their content from being crawled and may provide hints
about their content, rate of change and importance. Such efforts to make web crawling
more efficient by improving the underlying protocol were made by introducing the Sitemaps
protocol [sit] that allows a site administrator to publish the list of available pages and their
last modification date. Tradeoffs between using the classical pull approach and Sitemaps
for content discovery were examined in [SS09]. Even though this protocol offers a crawler
useful information about new created pages and their changes, the crawler still has to use
the pull architecture in order to regularly get the information of its interest and thus, pull
based refresh strategies are still needed.
Both push and pull protocols have their advantages and disadvantages. When the new
content is pushed by the content providers, a big advantage is that the subscribed aggregators
can receive instant update notifications as soon as new content is published.
However, the problem is that the content providers must be aware of their subscribers list
and manage a trust relationship with them, which is difficult and highly unlikely in the
web’s reality. On the other hand, when the new available content is proactively pulled
from the content providers, the crawler obeys a refresh strategy that must predict the
update moments and sources and that decides when the crawler fetches the newly pub-
34