Roxana - Gabriela HORINCAR Refresh Strategies and Online ... - LIP6

fraction of recently published feed items that have not been retrieved yet. A maximum
window freshness shows that the RSS feed result of an aggregation query is up to date
with all its RSS feed sources. The proposed quality metrics (feed completeness and window
freshness) considered simultaneously were designed to reflect the quality of newly
produced aggregation feeds within a content-based feed aggregation system.
When should the crawler of a feed aggregation system refresh the RSS feed
sources and what makes a refresh strategy optimal? We propose a best effort refresh
strategy based on the Lagrange multipliers [Ste91] optimization method that retrieves
new postings from different feed sources and maintains an optimal level of aggregation
quality, obtained with a minimum cost. Our strategy maximizes both feed completeness
and window freshness, fetching new items in time and avoiding item loss. The proposed
policy is supported by an extensive experimental evaluation, where its performance and
robustness are tested by comparing it to other refresh strategies.
How much new information do the RSS feeds publish every day? When exactly
do they publish it? Are there any patterns in their publication activity? In
order to better understand the publication behavior of real RSS feeds, we propose a general
characteristics analysis with a focus on the temporal dimension of real RSS feed sources,
using data collected over four weeks from more than 2500 RSS feeds. We start by looking
into the intensity of the feed publication activity and into their daily periodicity features.
We also classify the source feeds according to three different publication shapes: peaks,
uniform and waves.
How should the feed publication behavior should be modeled? What is an
appropriate estimation model? How can we keep the estimation models up to
date with the continually changing feed publication activities? Inspired by the
observations made on the real RSS feeds publication behavior, we propose and study two
different models that reflect different types of feed publication activities. Furthermore, we
propose two online estimation methods that correspond to the publication models that are
continually updated in order to reflect the changes in the real feed publication activities.
We provide an experimental evaluation of the online estimation methods in cohesion with
different refresh strategies and an analysis of their effectiveness on sources with different
publication behavior was made.
Organization of the Dissertation
This rest of the dissertation is structured as follows. Chapter 1 introduces some fundamental
notions of the content-based feed aggregation domain. We describe the architecture
of our aggregation system in order to better understand the proposed methods and also
introduce a formal content-based feed aggregation model. Chapter 2 defines our two aggregation
oriented quality measures: feed completeness and window freshness. We also
explain here the saturation problem specific to RSS feeds and give an item loss measure.
In Chapter 3 we discuss the general problem of web crawling, by introducing some key
5