Roxana - Gabriela HORINCAR Refresh Strategies and Online ... - LIP6

and therefore, hard to manage. However, the pull protocol works without needing the
content providers to be aware of the subscribed aggregators. And finally, adopting a push
protocol requires the existence and maintenance of a trust relationship between content
providers and content aggregators, which is very difficult to manage and highly unlikely
in the Web’s reality. A more elaborate discussion together with related work references
on pull and push protocols is presented in Section 3.1.
When considering the underlying communication protocol, from the point of view of a
web server there is no distinction between RSS feeds and web pages or any other web
resources. All kinds of resources have to be refreshed by using the standard pull-based
HTTP protocol where changes can only be detected by explicitely contacting the content
provider server. Therefore aggregators as well as search engines face the same type of
challenge: deciding when is the optimum refresh time moment of each resource.
Our research work is situated in the context of the RoSeS project (Really Open Simple
and Efficient Syndication) [rosa], a content-based feed aggregation system. Its goal is to
define a set of web resource syndication services and tools for localizing, querying, generating,
composing and personalizing RSS feeds available on the Web. A RoSeS aggregator
represents an RSS feed sharing system in which users are allowed to register content-based
aggregation queries defined on a set of input RSS feeds, whose results are made available
to users as a newly created output RSS feed. For that, it has to periodically retrieve the
newly published content on the ever changing RSS feed sources.
Placed in the context of the content-based feed aggregation systems, the main goal of this
dissertation is to improve the aggregation quality by designing optimal refresh strategies
and online change estimation techniques adapted for crawling highly dynamic RSS feed
sources. Despite the apparent simplicity, feed aggregation systems have many inherent
challenges:
• Large scale. Not only that the Web is very large, but it is also permanently
evolving and so are the RSS feeds hosted on websites. Therefore, feed aggregators
must support an extremely high throughput by dealing with a large number of feed
content providers and consumers while seeking high aggregation quality.
• Highly dynamic content. As any web resource, RSS feeds evolve independently
of their clients and may suddenly change their publication activity. Even if the
aggregator supports large scale syndication, it could never keep up with all the
dynamic changes of all the feeds.
• Refresh decision. In order to assure the proper functioning of an aggregator, it
must use an intelligent refresh strategy that decides when to refresh each feed source
such as to assure aggregation quality maximization within a minimum cost.
• Limited resources. Not only that crawlers should respect politeness policies by
not imposing too much of a burden on the content providers, but they are also
constrained by their own internal resource limitations, such as bandwidth, storage,
memory or computing consumed resources, minimizing the overall cost.
3