Information Retrieval Techniques for non-textual media - Prof. A ...

INFORMATION RETRIEVAL
TECHNIQUES FOR
NON-TEXTUAL MEDIA
A.Bala subramanian

World of Information explosion
• Information produced globally are
enormous
• Print, film, magnetic, and optical
storage media produced about 5
exabytes (10^18 bytes) of new
information in 2002 itself (U of Calif-
B).

Media Storage
• Ninety-two percent of the new information was
stored on magnetic media, mostly in hard disks.
• If digitized, the nineteen million books and other
print collections in the Library of Congress would
contain about ten terabytes of information;
• Hard disks store most new information.
• Film represents 7% of the total, paper 0.01%,and
optical media 0.002%.

United States alone
• The produces about 40% of the world's new
stored information,
• including 33% of the world's new printed
information,
• 30% of the world's new film titles,
• 40% of the world's information stored on
optical media, and
• about 50% of the information stored on
magnetic media.

Population Reference Bureau
• the world population is 6.3 billion,
• thus almost 800 MB of recorded information
is produced per person each year.
• It would take about 30 feet of books to store
the equivalent of 800 MB of information on
paper.

Information Flows
• Information flows through electronic
channels -- telephone, radio, TV, and the
Internet – contained almost 18 exabytes
• three and a half times more than is recorded
in storage media.
• Ninety eight percent of this total is the
information sent and received in telephone
calls - including both voice and data on both
fixed lines and wireless.

The WWW
• contains about 170 terabytes of information
on its surface;
• Instant messaging generates five billion
messages a day (750GB), or 274 Terabytes a
year.
• Email generates about 400,000 terabytes of
new information each year worldwide.

P2P file exchange
• P2P file exchange on the Internet is growing
rapidly.
• Seven percent of users provide files for
sharing, while 93% of P2P users only
download files.
• The largest files exchanged are video files
larger than 100 MB, but the most frequently
exchanged files contain music (MP3 files).

Web content
• The median size of HTM/HTML pages was 8 KB, but the
mean was 605 KB.
• About 23% included images and
• 4% contained movies or animations, and about 20%
contained Javascript applications.
• There are about 2.9 million active weblogs ('blogs'),
containing about 81 GB of information.
• About 62 billion emails are sent daily, on the Internet
and elsewhere
• The average email is about 59 kilobytes in size, thus
the annual flow of emails worldwide is 667,585
terabytes.

Peer to Peer (P2P) File Sharing
• A significant new source of storing, creating
and exchanging media and data on the
Internet is through P2P file sharing networks.
• One of the most popular of these applications,
has recently reached over 230 million
downloads worldwide, with an average of 2
million more per week (source:
Download.com)

AVI & mp3 files
• In looking at file sizes, users frequently exchange
files larger than 100 MB
• The largest file types are .AVI Files.
• The most common files shared by P2P users are MP3
files, music files encoded using MP3 technology.
• Images (jpg, bmp) are also popular but take up much
less space.
• Sixty percent of the files on users' hard disks were
MP3 files, taking up about 30% of the space.

Use of non-textual digital
documents
• The use of non-textual digital documents is
increasing.
• A multimedia document may be seen as a
complex information object with
components of different kinds such as text,
images, video, animation and audio under
static displays or streaming applications.

Growth of multimedia content
• The exponential growth of multimedia content,
in on-line databases, broadcast and streaming of
media over the internet, demands effective
access and retrieval techniques of both textual
and non-textual resources.
• There is a need to have more sophisticated
technology for modeling of multimedia data
under networked environment.
• More user oriented methods for indexing,
compressing and warehousing of multimedia
information have been identified and used.

INFORMATION STORAGE AND
RETRIEVAL
• Information storage begins with the
collection of data and represent them
digitally.
• It may be a text or other media data.
• In order to be able to access and process the
data in ways that are applicable to the media
type, it is necessary to define structures and
data types for these binary strings.

Databases
• Collection of all organised data forms a
database.
• Databases are stored digitally in the form of
files with appropriate file extensions.
• The file extensions denote the type of digital
data contained in them.
• File type and file extensions are mostly used
by the user to fetch the links during a search.

Search/Retrieval/data model
• Underlying data model: hierarchic, network,
relational, object-oriented,
• Primary data-type: tabular, text, image, map/spatial,
audio, video,
• Content: document, record, multimedia,
bibliographic, statistical, geographic,
• Primary DB usage: administrative, library, geographic,
museum,
• Architecture: centralized, distributed, homogeneous,
heterogeneous, or
• Access type: real-time, interactive Web.

Metadata
• The word Metadata is used to define 'data about
data' or 'information about information'.
• In the other words, metadata is data that
describe information resources.
• Metadata captures the wide range of intrinsic or
extrinsic information about a variety of objects.
• These intrinsic or extrinsic characteristics and
features are described in the individually
structured data elements that facilitate object
use, identification and discovery.

Metadata
• is used to speed up and enrich searching for
resources.
• search queries using metadata can save users from
performing more complex filter operations manually.
• Metadata can be divided into 3 distinct categories as
• descriptive,
• administrative and
• structural.
• Search engines are used to search the desired
information from these databases using search key
words.

Types of Search Queries
• Most of the searches fall into the following 3
categories:
• Informational - seeking static information
about a topic
• Transactional - shopping at, downloading
from, or otherwise interacting with the result
• Navigational - send me to a specific URL.

Notable works
• Multimedia Information Retrieval Systems
• Applications of video-content analysis and
retrieval
• Intelligent Multimedia Information Retrieval
• Content-based multimedia information
retrieval:
• Content based image retrieval were some of
the works done on these areas.

THE MULTIMEDIA ANALYSIS AND
RETRIEVAL SYSTEM
• The Multimedia Analysis and Retrieval System (MARS)
was developed by Sharad Mehrotra of the
Department of Information and Computer Science,
University of California.
• The goals of the MARS project are to design and
develop an integrated multimedia information
retrieval and database management infrastructure,
entitled Multimedia Analysis and Retrieval System
(MARS), that supports multimedia information as firstclass
objects suited for storage and retrieval based on
their content.

Multimedia Analysis and Retrieval
System
• four sub-areas
• Multimedia Content Representation: extraction of multimedia content
and content-based representation of multimedia objects in databases.
• Multimedia Information Retrieval: content-based multimedia retrieval
techniques including multimedia retrieval models and interactive query
refinement techniques.
• Multimedia Feature Indexing: that overcomes the high-dimensionality
and non-Euclidean nature of feature data to efficiently support retrieval
based on feature similarity.
• Multimedia Database Management: techniques to effectively and
efficiently incorporate content-based retrieval of multimedia
information into structured database processing.

Information Retrieval Systems (IRS)
• Text-based Applications
• Text-based applications were among the first
computer applications, initiated by the users to
analyze large quantities of text documents.
• Other early applications included establishment
of information retrieval systems (IRS) to support
management, analysis and retrieval of
information from digitized medical journals and
legal documents, public access catalogs, on-line
catalogs (OPACs), and digital libraries, insurance
documents, and digitized literature.

Early text-based applications
• include,
• 1) literature analysis
• 2) language translation, and
• 3) compilation of programs.
• As access to network communications has gradually
increased and text-based applications have moved to
the public domain, with the most familiar and
commonly used services being e-mail, Web sites with
tagged and hyper-linked documents. From a data
management (storage and retrieval) point of view,
text-data is unstructured

Ways of locating text documents
• There are 2 ways of locating text documents as
• 1) Directly, using a set of query terms to be
matched to the terms used in the body of
documents to determine relevancy to the query or
• 2) Indirectly, using the metadata describing the
document, as done in most cataloging systems.
• In principle, there is no limit to the number of
search terms that can be included in a query.

Architecture of a Search Engine
• A search engine contains all data
managers, parsers, spiders, refresh
managers and Graphic applications.
• These are the channels of
communication for information
retrieval.

MIMD architectures
• Document /data /term partitioning-logical,
physical, inverted indexing, etc
• Models- Probabilistic, Bayesian networks,
ranking, Fuzzy logic, Semantic indexing,
extended Boolean , vector model
• Machine Learning-Information extraction,
sliding windows and boundary dtection,
supervised learning, Hidden Markov Models,

Situation Today
• Managing IRS is an expensive affair, we need to go for
alternative architectures and algorithms
• Parallel and Distributed Inf. Retrieval
• Parallel Computing is the simultaneous application of
multiple processors to solve a single problem
• Flynn’s Taxonomy:
Flynn’s Taxonomy:
• SISD- Single instruction, single data
• SIMD- Single instruction , multiple data
• MISD- multiple instruction, single data
• MIMD-multiple instruction, multiple data

Multimedia Mining Hierarchy
• Multimedia data – image, video, audio,
animation
• Data segmentation- object-oriented
representation, feature extraction, additional
information
• Pattern extraction- case(event) definition,
knowledge representation, information
modeling
• MMDB

Retrieval of Images
• Images were treated as document-
like –objects, considering the set of
image –specific elements.
• Images, movies, speeches and music
were characterized as document
like objects(the size, codec,
presenter, other options, security)

Image-based Applications
• due to computer capacity limitations, development of
image-based computer applications followed the path of
text-based applications.
• Perhaps the first image-based computer applications were
developed for weather forecasting, though other early
applications included
• Map generation and manipulation,
• Visual analysis of experiment data - in chemistry, physics
and statistics,
• Visualization of mathematical functions, and
• CAD/CAM (computer aided design and manufacture) for
ship building, quickly followed by architecture,
landscaping, interior decorating and fashion designing.

Raster and vector images
• CAD/CAM or geographic maps, business
graphics which are generated from database
values and interactive applications which
might have different content for each user
are considered to be no-document –like
objects.
• These sources do not contain images but
they generate images.

Images
• Images offer a number of technological and
descriptive challenges peculiar to themselves.
• Textual materials can be indexed and classified
even through automated methods.
• But the encoding schemes are critical for using
images.
• Rendering images require web graphic display
facilities with wide differences in display
properties.

Information necessary for
rendering
• are Type ( Bit – mapped , vector , video ),
• Format ( TIFF, GIF, JFIF, PICT, PCD , Photoshop,
CGM,TGA ),
• Compression schemes & ratios ( JPEG ,LZW,
QuickTime … ),
• Dimensions, Dynamic range and
• Color look up tables and related matrices
(CYMK, RGB ….).

Characteristics of original image
capture
• The (scanning) including
• light source (full spectrum or infrared),
• resolution,
• dynamic range,
• type of scanners,
• date of scan,
• journal/ audit trails and
• digital signatures for authentication.

Image retrieval elements
• All these are the critical elements of
metadata for a particular image or
collection.
• Several element names were used
to make the image description less
text -centric.

The elements used for image
representation are:
• Subject (keywords, controlled vocabulary
terms & formal classification designators),
• Description ( descriptive prose or content
description) and
• Rights management
Rights management field ( Null , No
restrictions on Reuse, URL or other pointers)

Image Retrieval
• Popular knowledge claims that an image is worth
1000 words. Unfortunately, these 1000 words may
differ from one individual to another depending on
their perspective and/or knowledge of the image
context.
• The problem is fundamentally one of communication
between an information/image seeker/user and the
image retrieval system.
• Since the user may have differing needs and
knowledge about the image collection, an image
retrieval system must support various forms for
query formulation.

Image retrieval queries
• can be classified as:
• Attribute-based queries, which use context and/
structural metadata values to retrieve images.
• Textual queries, which use a term-based
specification of the desired images that can be
matched to textual image descriptors.
• Visual queries, which give visual characteristics
(color, texture) or an image example, that can be
compared to visual descriptors.

Query types utilize different
image descriptors
image descriptors
• Image descriptors can be classified into:
• 1) Metadata descriptors, those that describe the image, as
recommended in the numerous metadata standards.
• These metadata can again be classified as:
• 1) Attribute-based context and structural metadata, such
as creator, dates, genre, (source) image type, size, file
name, ..., or
• 2) Text-based semantic metadata, such as a) title/caption,
subject/keyword lists, free-text descriptions and/or the
text surrounding embedded images, for example as used in
a html document.
• b) Visual descriptors that can be extracted from the image
during the storage process by an image retrieval system.

Text-based image retrieval :
• Often, the image requester is able to give a
verbal description/specification of the
content of the required images.
• These text-based queries can be formulated
as free-text or a term list that can be
compared to such text descriptors as
description, subjects, title and/or the text
surrounding an embedded image, , using the
text retrieval techniques.

Query by Image Retrieval
(Content Based IR)
• the term 'image content' refers to
the implementation content of the
image, i.e. to its pixel content.
image, i.e. to its pixel content.
• The objective of CBIR research and
development activities is to develop
automated routines that can analyze a digital
image or image stream and identify the
objects, actions and events that it portrays.

• Currently, CBIR routines are relatively adept at
identifying color and texture distributions, as
well as primitive shapes.
• This information is the basis for specification
of one or more image signatures that can act
as a surrogate for the image and that can be
indexed to provide rapid access to elements in
the image collection.

Using content descriptors:
• color and texture - An image is basically a
long string of pixels for which each pixel is
identified by its place in the image matrix, its
color and its intensity.
• An analysis of the pixel set can give
information about the distribution of
dominant colors, the image texture, and the
shapes formed by marked change in
neighbouring colors.

Basic color profile
• Though there are many techniques used for color
and texture extraction, they are variations
and/or refinements of a basic color profile
• An image query is given as an example or seed
image or sketch for which a feature vector is
calculated in the same way as that used for the
DB image set.
• The query signature is then compared to the DB
image signatures, using a distance measure.
•

Image analysis
• Identifying shapes - image objects:
• Shape identification and recognition is the most
difficult challenge to image analysis since it relies on:
• Isolation of the different objects/shapes within the
image, which may not necessarily be whole or
standardized, but are likely 'hidden' in the perspectives
of the image and then
• Normalizing the object's size and rotation
• Identification and possible connection of object parts.
• Semantic identification of the image components/
objects.

Automatic object recognition
• To date, automatic object recognition has only
been accomplished for well-defined domains
where objects of interest are well known and
well defined within the image, such as:
• Police images of faces and fingerprints,
• Medical images from domain x-rays, MRI and
scans, and
• Industrial surveillance of building structures,
such as bridges, tunnels or pipelines.

Visual query requires a query
language
• An object-based visual query requires a query
language that can accept a visual
object/image as an example.
• A shape thesaurus can be developed for
image collections from specific domains that
can be used in a way similar to that done for
the terms in text document collections.

Automatic Linguistic Indexing of
Pictures
• Categorized images are used to train a dictionary of hundreds
of statistical models each representing a concept.
• Images of any given concept are regarded as instances of a
stochastic process that characterizes the concept.
• To measure the extent of association between an image and
the textual description of a concept, the likelihood of the
occurrence of the image based on the characterizing
stochastic process is computed.
• A high likelihood indicates a strong association.
• Two-dimensional multi-resolution hidden Markov models
(2D MHMMs).

Video Information Retrieval (VIR)
• Recognition technologies
– Image
– Voice
– Text transcripts
• Document retrieval technologies
– Topic segmentation
– Topic matching
– Text summarization
• Presentation Technologies
– Combine Recognition and retrieval
technologies
• Result is an integrated application

Streamed media applications:
Source data + playback software
• Streaming media is multimedia that is constantly
received by, and normally presented to, an end-user
while it is being delivered by a streaming provider
(the term "presented" is used for denoting audio or
video playback).
• The name refers to the delivery method of the
medium rather than to the medium itself.
• Media delivery systems are either
• inherently streaming (e.g. radio, television) or
• inherently non-streaming (e.g. books, video cassettes,
audio CDs).

Streamed media/dynamic media
• is composed of a series of media objects that
have a time relationship that is necessary for
proper communication.
• Audio, film (a series of images) and video (a
combination of audio and film) data belong to
this class.

Streamed-data applications
include
• Speech analysis, Speech command of automated
artifacts: robots, equipment for the handicapped,
answering/ordering systems,
• Music archives, search and presentation,
Surveillance and film analysis,
• Animation for stories, games, simulations,
• Film archives, search and presentation and Video on
Demand.
• Webcasting, podcasting and on-line radio facilities
are coming under this information retrieval.

Streamed data representation
• as image data, have some metadata attributes:
• Context descriptors: artist name(s), publisher,
publication date, ...
• Content descriptors: title, genre, keywords,
description and for music: mood, musical theme,
melody, tempo, rhythm and instrumentation,
and
• Structural description, such as presentation
length, storage size, storage/presentation
format.

Search requests for streamed data
• from a media collection can be expressed in 3
ways as
• An exact-match query giving data values to be
compared to the regular metadata
attributes(Retrieve all films where the
artist_name='xxx ccccc ' )
• A text-based query using the content descriptive
metadata( Retrieve Hindi devotional songs) and
• An example-based query using a Video or audio
clip or a Whistled tune such as offered at
Musipedia.

Stand-alone Internet radio devices
• are offering listeners a "no-computer" option for
listening to audio streams.
• In general, multimedia content is large, so media
storage and transmission costs are still significant; to
offset this somewhat, media is generally compressed
for both storage and streaming.
• A media stream can be on demand or live.
• On demand streams are stored on a server for a long
period of time, and are available to be transmitted at a
user's request.
• Live streams are also available at any time , as in a
video stream of a live sporting event or temple pooja.

Web Server vs. Streaming Server:
CEC Model + ISRO Edusat Model
• There are two major methods of delivering
streaming audio and video content over the
Web.
• The first method uses a standard Web server
(E-content+ LOR) to deliver the audio and
video data to a media player. The second
method uses a separate streaming media
server (Edusat) specialized to the audio/video
streaming task.

Programmes are deployable
through a campus LAN

Watch Live
programmes

Streaming with a Web Server
• Deploying streaming media content with the Web
server approach is actually only a small
evolutionary step away from the download-andplay
model.
• Uncompressed audio and video is first compressed
into a single "media file" for delivery over a specific
network bandwidth such as a 28.8 kilobits per
second (Kbps) modem.

• This media file is then placed on a standard Web
server.
• Next, a Web page containing the media file's URL is
created and placed on the same Web server.
• This Web page, when activated, launches the clientside
player and downloads the media file.
• So far, the actions are identical to those in the
download-and-play case.
• The difference lies in how the client functions.

Streaming with a Streaming Media
Server
• In the streaming media server approach, the
initial steps are similar to the Web server
approach, except that the compressed media
file is produced and copied to a specialized
streaming media server (such as Microsoft
Windows Media Services) instead of a Web
server.
• Then a Web page with a reference to the media
file is placed on a Web server.
• Windows Media Services and the Web server
may run on the same computer.

Data Delivery
• The rest of the streaming media server delivery
process differs significantly from the Web server
approach.
• In contrast to the passive burst methodology
employed in Web server streaming, the data is
actively and intelligently sent to the client, meaning
that it delivers the content at the exact data rate
associated with the compressed audio and video
streams.
• The server and the client stay in close touch during
the delivery process, and the streaming media server
can respond to any feedback from the client.

Streaming bandwidth and storage
• Streaming media storage size (in the
common file system measurements
megabytes, gigabytes, terabytes, and so on)
is calculated from streaming bandwidth and
length of the media with the following
formula (for a single user and file):
• storage size (in megabytes) = length (in
seconds) * bit rate (in kbit/s) / (8 * 1024)

• For example,
• One hour of video encoded at 300 kbit/s will be:
(3,600 s * 300,000 bit/s) / (8*1024*1024) give
around 128 MB of storage.
• If the file is stored on a server for on-demand
streaming and this stream is viewed by 1,000
people at the same time using a Unicast protocol,
you would need: 300 kbit/s * 1,000 = 300,000 kbit/s
= 300 Mbit/s of bandwidth.
• This is equivalent to around 125 GiB per hour.

• Of course, using a Multicast protocol the
server sends out only a single stream that is
common to all users.
• Hence, such a stream would only use 300
kbit/s of serving bandwidth.

Protocol issues
• five major types of protocols used in multimedia
usage in internet.
• Datagram protocols, such as the User Datagram
Protocol (UDP), send the media stream as a
series of small packets. ( Edusat streaming
through VLC)
• This is simple and efficient
• It is up to the receiving application to detect loss
or corruption and recover data using error
correction techniques. If data is lost, the stream
may suffer a dropout.

• The Real-time Streaming Protocol (RTSP),
Real-time Transport Protocol (RTP) and the
Real-time Transport Control Protocol (RTCP)-
all were specifically designed to stream
media over networks.
• The latter two are built on top of UDP.
• Reliable protocols, such as the Transmission
Control Protocol (TCP), guarantee correct
delivery of each bit in the media stream.

Unicast protocols
• send a separate copy of the media stream from the
server to each recipient.
• Multicasting broadcasts the same copy of the
multimedia over the entire network to all clients.
• Multicast protocols were developed to reduce the
data replication (and consequent server/network
loads) that occur when many recipients receive
Unicast content streams independently.
• These protocols send a single stream from the
source to a group of recipients.

Continuous streaming of radio or
television
• Continuous streaming of radio or television
material usually precludes the recipient's
ability to control playback.
• IP Multicast provides a means to send a single
media stream to a group of recipients on a
computer network.
• Peer-to-peer (P2P) protocols arrange for prerecorded
streams to be sent between
computers.

Benefits of protocols
• UDP – provides the most efficient network
throughput and can have a very positive impact
on the user (player) experience.
• The only downside to UDP is that many network
administrators close their firewalls to UDP traffic,
limiting the potential audience of UDP-based
streams
• TCP –provides an adequate, though not
necessarily efficient, protocol for delivering
streaming media content from a server to a
client.

VLC – for open source streaming
• Direct draw
• Directshow
• Four protocols
• Multiple formats, devices, methods
• Single stream-output- suitable for
webcasting
• Videoconferencing, live stream recording

Vlc media player options

Vlc
• The network on which you setup the VideoLAN
solution can be as small as one ethernet 10/100Mb
switch or hub, and as big as the whole Internet! The
bandwidth needed is:
• 0.5 to 4 Mbit/s for an MPEG-4 stream,
• 3 to 4 Mbit/s for an MPEG-2 stream read from a
satellite card, a digital terrestial television card or
an MPEG-2 encoding card,
• 6 to 9 Mbit/s for a DVD.

Inputs
VLC features page
UDP Unicast /
Multicast
Yes Yes Yes unicast only Yes Yes
Output
RTP Unicast /
Multicast
Yes Yes Yes unicast only Yes Yes
File Yes Yes Yes Yes Yes Yes
HTTP Yes Yes Yes Yes Yes Yes
MMSH Yes Yes Yes Yes Yes Yes
Transcoding Yes Yes Yes Yes Yes Yes
Misc
Interfaces and
more
Send DVD
subtitles
Send SAP
announces
Partial Partial Partial Partial Partial No
Yes Yes Yes Untested Yes Untested
See the VLC features page

The VideoLAN streaming solution
includes:
• VLC media player (initially VideoLAN Client), which can be
used as a server to stream MPEG-1, MPEG-2 and MPEG-4 /
DivX files, DVDs and live videos on the network in unicast or
multicast; or used as a client to receive, decode and display
MPEG streams under multiple operating systems,
• VLS (VideoLAN Server), which can stream MPEG-1, MPEG-2
and MPEG-4 files, DVDs, digital satellite channels, digital
terrestial television channels and live videos on the network
in unicast or multicast. Most of the VLS functionality can
now be found in the much better VLC program. Usage of VLC
instead of VLS is advised.

Web 2.0 Ogg bitstream format
• Free, open std container format
• Designed to provide efficient streaming and
manipulation of high quality digital
multimedia
• It can multiplex a no of separate independent
free and open source codec for audio, video,
text, etc.
• Theora, vorbis,speex, flac, oggPCM,

Streaming Media Services
• Numerous subscription based streaming
media services began to appear on the web.
• These services provided a low cost, low
barrier to entry model that allowed small
businesses and internet marketers to place
streaming audio and streaming video on their
websites.
• On more addition was podcasting.

Internet television:
(ERNET Model)
• allows viewers to choose the show they want to
watch from a library of shows.
• The primary models for Internet television are
streaming Internet TV or selectable video on an
Internet location, typically a website.
• The video can also be broadcast with a peer-to-
peer network(P2PTV),which doesn't rely on
single website's streaming.
• It differs from IPTV in that IPTV offerings are
typically offered on discrete service provider
networks, requiring a special IPTV set-top-box.

• Internet TV is a quick-to-market and relatively
low investment service.
• Internet TV rides on existing infrastructure
including broadband, ADSL, Wi-Fi, cable and
satellite which makes it a valuable tool for a
wide variety of service providers and content
owners looking for new revenue streams.
• CEC-ERNET collaboration model shows-
Gyan-Vyas on internet.
• ( Shirdi Sai Baba Sansthan and other religious
websites)

P2PTV
• The term P2PTV refers to peer-to-peer (P2P)
software applications designed to redistribute
video streams in real time on a P2P network;
• the distributed video streams are typically TV
channels from all over the world but may also
come from other sources.
• The draw to these applications is significant
because they have the potential to make any
TV channel globally available.

Peercast
• A Peercast can be used to multicast streaming
audio (MP3, WMA) and/or video (Nullsoft
Streaming Video, or WMV), or any other
stream of data, over the internet.
• Peercast uses a distributed bandwidth
technique to lighten the load of the
broadcaster's upstream bandwidth where
each listener/viewer will relay the stream they
download to one or more additional listeners

• The benefits of using PeerCast is that it allows
any multicasters, particularly small or
independent ones, to distribute their streams
without need for much bandwidth, saving
them costs.
• It also allows, theoretically, an infinite number
of listeners as long as there are enough relays

Unreal Media Server
• It is a proprietary streaming server for Windows
platforms, designed to provide multimedia
delivery over LAN and Internet.
• It supports media files and live media streams.
• Supported file formats: AVI (DivX, XVid, VP6, any
other codecs), MPEG-1/2/4, WMV/WMA, MP3,
ASF, QuickTime.
• Playlist functionality allows automatically playing
all the files of server's virtual folder in a loop
mode.

• Supported live media sources include: digital cameras,
microphones, TV-tuners, analog video sources connected to
video card or to Video capture card that supports
DirectShow interface.
• Live audio/video is encoded with WMA/MP3/GSM 6.10 -
WMV/MPEG4 codecs in real time.
• Hardware encoder appliances are supported for streaming
hardware compressed content without software
transcoding.
• TCP, HTTP and MMS unicast, RTP multicast transport
protocols are supported.
• For HTTP delivery IIS web server is required on the server
computer.

Unreal Media Server includes 3 major
components
• Streaming video recording : Streaming video recorders can
record not only video streams but also many audio streams so sometimes they are
also called streaming media recorders. There are different approaches that are
used by the software to make the recording, depending on which stage of the
process one taps into. In order, they are:
• URL Snooping: Simplest is if the stream is served by simply requesting
it, just as web pages are, as in an HTTP GET request: this will directly copy the
encoded, streamed file. In this case, one simply needs to determine the URL, and
then download that, either by pasting it into one's web browser (location box or
"Open location..."), or via a specialized download manager.
• Encoded capture: Some streaming is not via a simple request to an
URL – in this case, to capture the stream requires some understanding and
implementation of the particular streaming protocol (the encoded media stream is
encapsulated within a network stream), either: passively / offline :- capturing the
actual traffic and extracting it (via deep packet capture, using a packet sniffer), or
actively / online :-implementing the streaming protocol / program enough to
request the encoded data.

Decoded capture
• An approach used to record the decoded information
at the end level- from the video and sound card of the
computer.
• This is essentially capturing what you are watching or
listening to directly from the screen, and could be
likened to recording off the air with a microphone.
• This solution makes it possible to record anything that
you are able to view or listen to, regardless of original
format or protection, though it suffers from a loss in
quality (digital generation loss) due to re-encoding.

On-going Research
• Multi-modal information retrieval from Broadcast Video
using OCR and Speech recognition.
• multi-modal information retrieval from broadcast video
where text can be read on the screen through OCR and
speech recognition can be performed on the audio track.
• OCR and speech recognition are compared on the 2001 TREC
Video Retrieval evaluation corpus.
• Results show that OCR is more important than speech
recognition for video retrieval.
• OCR retrieval can further improve through dictionary-based
post-processing.

• International Institute of Information
Technology, Hyderabad, India have worked on
“A System for Information Retrieval
Applications on Broadcast News Videos.
• This system is specifically designed for the
information retrieval applications on broadcast
news videos.
• The system is directly useful to an end user for
easy access to the news stories of interest.
• It also act as a platform for convenient
deployment and experimentation of various
video analysis and indexing techniques on real
data, and on a large scale.

AUDIO INFORMATION RETRIEVAL
• The problem of audio information retrieval is
familiar to anyone who has returned from
vacation to an answering machine which may
contain full of messages.
• recent advances in automatic speech
recognition, word spotting, speaker and music
identification, and audio similarity have been
explained.

Music Information Retrieval
• to determine the similarity between two given
melodies;
• there are several melodic similarity measures
• Audio Segment Retrieval Using a Synthesized
HMM was attempted
• The approach allows a user to query audio data
of any length by one or more example audio
segments and find similar segments.
• Audio identification using sinusoidal modeling
and application to jingle detection

Signal + Context = Better
Classification
• Typical signal-based approaches to extract
musical descriptions from audio only have limited
precision.
• architecture to train a large set of binary
classifiers simultaneously, for many different
musical metadata (genre, instrument, mood,
etc.), in such a way that correlation between
metadata is used to reinforce each individual
classifier.
• The system is iterative

Content-based music retrieval
using query integration
• For users whose preferences range in songs with
a wide variety of features.
• MIR method dynamically generates an optimal
set of query vectors from the sample set of songs
submitted by the user to express their
preferences, based on the similarity of the songs
in the sample set.
• Experiments conducted on a music collection
with subjective user ratings were very effective
to improve the accuracy of content based MIR.

Issues to be considered: Flash

Flash video + text + command
buttons

Total e-content in a package
format

E-content deployment
• Full content coverage of subjects, deployment
architecture, servers (at least 5, database,
webservice, message, backup, intrusion
detection, media server), tools and
customization at three levels( content,
localhost , webserver level)
• Distributed network architecture(with
EMMRcs as Hubs)- replication of database
servers

Other issues
• Gateway
• Bandwidth
• Load-balancing
• End user facilitation
• Application environment
• Integrated system

LOR: Front page- Blocked content

Allowing the blocked media

Radio Broadcast
TV Broadcast
Videoconference
Night Time
Loading
Asymmetric
Internet
Benefits
On-line
Education
Telephone
Voice Chat
on Internet
WebCAM
Talkback
Internet

Technological Convergence
• Computer professionals
• Information scientists
• Media Professionals
• User limitations
• Deliverer’s preferences
• Operational issues at local and global levels
• Co-ordinated approach

Thank you