Media Fingerprinting âPart II - SET
Media Fingerprinting âPart II - SET
Media Fingerprinting âPart II - SET
You also want an ePaper? Increase the reach of your titles
YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.
<strong>Media</strong> <strong>Fingerprinting</strong> – Part <strong>II</strong><br />
Applications in Broadcast Television<br />
Marco Lopez<br />
Sr. VP, Miranda Technologies
Agenda<br />
Next<br />
What is a media fingerprint<br />
How does it work<br />
Applications in broadcast TV<br />
Fingerprint standardization
What is a <strong>Media</strong> Fingerprint <br />
A form of identification for a<br />
piece of audio and video media<br />
Can be used to identify or<br />
recognize a specific piece of<br />
media later in time or<br />
downstream in a system
Key Attributes of a <strong>Media</strong> Fingerprint<br />
Does not alter the media itself<br />
Compact, ideally much, much smaller<br />
than the media it is applied to<br />
Robust and Resistant: able to survive<br />
normal processing of the media<br />
Efficient: economical to generate as<br />
well as compare or search in a<br />
database
Fundamental Difference Between<br />
Fingerprints and Watermarks<br />
Fingerprints extract properties<br />
from a source then store or<br />
transport it separately<br />
Watermarks ALTER sources by<br />
putting some form of<br />
mark or identifier in the media<br />
in a hard to remove way<br />
Send<br />
Store<br />
Actually the watermark is<br />
usually not visible/audible<br />
to the human eye or ear.
Agenda<br />
What is a media fingerprint<br />
Next<br />
How does it work<br />
Applications in broadcast TV<br />
Fingerprint standardization
Video Fingerprint Generation<br />
A finger Consider print a is series extracted of<br />
from consecutive each consecutive fields or field<br />
frames in or a frame video program<br />
Field or Frame X Field or Frame X+1<br />
Field or Frame N Field or Frame N+1<br />
t
Video Fingerprint Generation<br />
A change index is<br />
calculated between<br />
consecutive fields or frames<br />
Fingerprint<br />
Compare<br />
X<br />
change<br />
index<br />
= Fingerprint<br />
Compare<br />
= N<br />
change<br />
index<br />
Field or Frame X Field or Frame X+1<br />
Field or Frame N Field or Frame N+1<br />
t
Video<br />
Program A<br />
Matching Two Fingerprints<br />
Video<br />
System<br />
Video<br />
Program A’<br />
Delta Based Video<br />
Fingerprint<br />
Algorithm<br />
Delta Based Video<br />
Fingerprint<br />
Algorithm<br />
Streams of Change Index<br />
Fingerprint<br />
Comparison<br />
Video<br />
Matching<br />
Factor (%)<br />
Program to<br />
Program<br />
Delay<br />
(16.6 or 20<br />
mSec<br />
resolution)
Characteristics of this Fingerprint Scheme<br />
• Based on changes in image (levels and<br />
motion)<br />
Video<br />
Fingerprint<br />
Algorithm<br />
• Very simple to generate fingerprints<br />
• Very simple to compare two fingerprints<br />
• Fingerprint stream is very compact<br />
• Insensitive to common processing
Audio Fingerprint Generation<br />
One Channel of<br />
Digital Audio Program<br />
(48 KHz Sampling)<br />
Variation based audio<br />
fingerprint<br />
Mean +<br />
Envelope<br />
Extraction<br />
Fingerprint<br />
Extraction<br />
Stream of<br />
audio<br />
fingerprints
Audio<br />
Program<br />
Matching Two Audio Fingerprints<br />
Audio<br />
System<br />
Audio<br />
Program’<br />
Variation based<br />
Audio Fingerprint<br />
Algorithm<br />
Variation based<br />
Audio Fingerprint<br />
Algorithm<br />
audio fingerprint streams<br />
Fingerprint<br />
Comparison<br />
Content<br />
Match<br />
Factor (%)<br />
Program to<br />
Program<br />
Delay<br />
(1 ms<br />
resolution)
Characteristics of this Audio Fingerprint Scheme<br />
• Based on variations in audio mean level and envelope<br />
Audio Fingerprint<br />
Algorithm<br />
• Very simple to generate fingerprints<br />
• Very simple to compare two fingerprints<br />
• Fingerprint stream is very compact<br />
• Insensitive to common processing
Agenda<br />
What is a media fingerprint<br />
How does it work<br />
Next<br />
Applications in broadcast TV<br />
Fingerprint standardization
Audio and<br />
Video<br />
Programs<br />
<strong>Media</strong> <strong>Fingerprinting</strong> High Level Concept<br />
Audio+Video<br />
Fingerprint<br />
Algorithm<br />
Non intrusive<br />
Downstream,<br />
TV<br />
device Production analyses a<br />
second video+audio device<br />
and Delivery and<br />
generates generates a<br />
Systema low<br />
fingerprint bit rate of fingerprint the<br />
audio and video Fingerprint is<br />
time-stamped and<br />
streamed into a<br />
standard IT network<br />
Audio and<br />
Video<br />
Programs<br />
The two fingerprints are<br />
compared to establish if<br />
content is the same and<br />
Audio+Video what the relative delay is<br />
Fingerprint<br />
Algorithm<br />
Standard<br />
Network<br />
compare<br />
Content<br />
Same <br />
Audio to<br />
Video Delay<br />
Same
<strong>Media</strong> <strong>Fingerprinting</strong> High Level Concept<br />
Audio and<br />
Video<br />
Programs<br />
TV<br />
Production<br />
and Delivery<br />
System<br />
Audio and<br />
Video<br />
Programs<br />
Content<br />
In Server<br />
Audio+Video<br />
Off –Line<br />
Fingerprint Algorithm<br />
Fingerprint<br />
algorithm Fingerprints can also<br />
are<br />
be applied stored to on file<br />
a server<br />
based and anc content<br />
can later be<br />
sitting on searched a server<br />
for<br />
content match<br />
Fingerprint<br />
Database<br />
LAN /<br />
WAN<br />
Audio+Video<br />
Fingerprint<br />
Algorithm<br />
Search<br />
What<br />
Content is<br />
This
Two Key Applications For Finger Printing<br />
Comparisons in Playout<br />
Next<br />
Multi Point<br />
Electronic<br />
Signal<br />
Continuity<br />
LIP<br />
SYNC<br />
Multi Point,<br />
Multi Channel<br />
Lip Sync<br />
Detection
Operators typically monitor<br />
multiple points in playout<br />
chain<br />
Content is Valid and<br />
Same Across Entire<br />
Distribution Path<br />
rimary<br />
erver<br />
ackup<br />
erver<br />
R<br />
o<br />
u<br />
t<br />
e<br />
r<br />
Primary Path<br />
Branding<br />
CC/<br />
VBI<br />
Secondary Path<br />
Branding<br />
CC/<br />
VBI<br />
Neilsen<br />
Neilsen<br />
Audio<br />
Audio<br />
2x1<br />
Change<br />
-over<br />
Distribution<br />
Encoder<br />
Return<br />
IRD<br />
Up<br />
Link<br />
DTH<br />
return<br />
Cable<br />
return
Visual Content Comparison<br />
CC/<br />
VBI<br />
CC/<br />
VBI<br />
Primary Path<br />
Neilsen<br />
Audio<br />
Secondary Path<br />
Neilsen<br />
Audio<br />
2x1<br />
Change<br />
-over<br />
Distribution<br />
Encoder<br />
Return<br />
IRD<br />
DTH<br />
return<br />
Cable<br />
return<br />
TX<br />
Mismatch<br />
between Mismatch playout<br />
Between and satellite Playout<br />
and return Satellite<br />
Return<br />
Indicative of an<br />
Indicative uplink problem of an<br />
uplink problem
Traditional Playout Monitoring Scenario<br />
Channels are monitored by an operator looking at the monitor wall<br />
Operator watches & sometimes listens for operational errors such as wrong<br />
content, missing graphics, lip sync<br />
As channel count grows or channel complexity increases, so must operator<br />
count
Fingerprint Based Monitoring Scenario<br />
Compare All Points<br />
Finger Print Multi Comparison Viewer Engine<br />
Exception<br />
Notification<br />
Server Playback<br />
Primary and Backup<br />
Branded Outputs<br />
Primary, Backup & Final<br />
Returns<br />
Primary Path<br />
Primary<br />
Server<br />
Backup<br />
Server<br />
R<br />
o<br />
u<br />
t<br />
e<br />
r<br />
Branding<br />
Main<br />
Branding<br />
Backup<br />
CC/VBI<br />
Inserter<br />
Backup Path<br />
CC/VBI<br />
Inserter<br />
Neilsen<br />
Encoder<br />
Neilsen<br />
Encoder<br />
Audio<br />
Process<br />
Audio<br />
Process<br />
2x1<br />
Change<br />
-over<br />
Distribution<br />
Encoder<br />
Return<br />
IRD<br />
DTH<br />
return<br />
Up<br />
Link<br />
Cable<br />
return
Ideally Fingerprint Generation Built Into Devices<br />
Multi-Point Finger Print Correlation Engine<br />
Standard Ethernet Network<br />
Primary Path<br />
Primary<br />
Server<br />
Backup<br />
Server<br />
RFP<br />
o<br />
u<br />
t<br />
e<br />
FP r<br />
BrandingFP<br />
Main<br />
BrandingFP<br />
Backup<br />
CC/VBI<br />
Inserter<br />
Backup Path<br />
CC/VBI<br />
Inserter<br />
Neilsen<br />
Encoder<br />
Neilsen<br />
Encoder<br />
Audio<br />
Process<br />
Audio<br />
Process<br />
FP<br />
FP<br />
2x1<br />
Change<br />
-over<br />
FP<br />
Distribution<br />
Encoder<br />
FP<br />
FP<br />
Return<br />
IRD<br />
DTH<br />
return<br />
Up<br />
Link<br />
FP<br />
Cable<br />
return
Key Requirements for Playout Monitoring<br />
• Fingerprints must be robust to survive<br />
multiple conversion, encoding, decoding<br />
• Fingerprints must be “focusable” to detect<br />
specific problems such as missing<br />
graphics<br />
• Fingerprint generation must be simple so<br />
that it can be incorporated inside simple<br />
devices (e.g. routers, DA’s, IRDs)<br />
• Multi-vendor support so not all the<br />
devices have to be from the same vendor
Two Key Applications For Finger Printing<br />
Comparisons in Playout<br />
Multi Point<br />
Electronic<br />
Signal<br />
Continuity<br />
Next<br />
LIP<br />
SYNC<br />
Multi Point,<br />
Multi Channel<br />
Lip Sync<br />
Detection
Lip Sync Errors can Happen in Many Places Along<br />
the Broadcast Food Chain<br />
Special event venue<br />
Network Facility<br />
Local Station<br />
Mobile<br />
studio<br />
Encode<br />
+ Tx<br />
IRD<br />
Server<br />
Facility<br />
Infrastructure<br />
Branding<br />
+ Proc<br />
Encode<br />
+ Tx<br />
Return<br />
IRD<br />
Rx +<br />
Decode<br />
Local<br />
Process<br />
Encode<br />
+ Tx<br />
Off Air<br />
IRD<br />
Service Provider<br />
(Cable, DTH, IPTV)<br />
Rx+<br />
Decode<br />
Encode<br />
+ Tx
Fingerprint Comparison Provides Delay Measurement<br />
Program<br />
Video +<br />
N Ch Audio<br />
System<br />
Video<br />
Audio<br />
Fingerprint<br />
Ch …<br />
Generation<br />
1<br />
Audio<br />
Ch<br />
N<br />
Video<br />
Audio<br />
Fingerprint<br />
Ch …<br />
Generation<br />
1<br />
Audio<br />
Ch<br />
N<br />
Audio<br />
Fingerprint Comparison<br />
Video Ch<br />
Audio to Video Delay<br />
… Ch<br />
Calculation<br />
1<br />
Audio<br />
N<br />
Overall Program Delay<br />
Video to Audio Lip Sync Error (per channel)
Fingerprint Comparison Provides Delay Measurement<br />
If Video Delay ≠ Audio Delay<br />
we have lip sync errors<br />
Video Fingerprint Comparison<br />
Ch N Audio Fingerprint Comparison<br />
Ch 2 Audio Fingerprint Comparison<br />
Ch 1 Audio Fingerprint Comparison<br />
Can establish<br />
Lip Sync error as small<br />
as 1 mS<br />
Can establish<br />
Lip Sync error across<br />
any number of audio<br />
channels<br />
15<br />
PGM<br />
A<br />
0<br />
15<br />
PGM<br />
B<br />
0<br />
<br />
DLY<br />
PGM<br />
A<br />
Compare delays<br />
PGM<br />
B<br />
1<br />
0<br />
1<br />
0<br />
Ch1<br />
<br />
DLY
Primary<br />
Playout<br />
Backup<br />
Playout<br />
2x1<br />
Fingerprints for Distributed Monitoring<br />
Mismatch LipSync<br />
Error<br />
Error<br />
Playout Center<br />
Plant Sat Off Cable<br />
out Return Air Return<br />
Distribution<br />
Enoder<br />
probe<br />
probe<br />
M<br />
u<br />
x<br />
Return<br />
IRD<br />
Up<br />
Link<br />
STB Probe<br />
Rx<br />
Regional Station<br />
probe<br />
Router<br />
STL<br />
Tx<br />
STB Probe<br />
STB Probe<br />
Cable/IPTV Head End<br />
RX<br />
Prime<br />
time<br />
Sub<br />
Switch<br />
STL<br />
Rx<br />
TX Site<br />
ATSC<br />
TX<br />
Final<br />
Encode<br />
or<br />
Transcode<br />
probe<br />
STB Probe<br />
Normal Audio/Video Probe<br />
Set Top Box Probe
Distribution<br />
Probes Offer Proxies as Well<br />
Playout Center<br />
Mismatch Fingerprint<br />
Exception Correlation<br />
Monitoring LAN<br />
Proxy on demand<br />
Up-Link<br />
WAN<br />
Remote Monitoring Site<br />
Fp<br />
S<br />
Pr<br />
EdgeVision<br />
Cable/IPTV<br />
Head End<br />
Set Top Box<br />
Primary<br />
Playout<br />
Backup<br />
Playout<br />
2x1<br />
Fp<br />
S<br />
Pr<br />
Distribution<br />
Encoder<br />
Fp<br />
S<br />
Pr<br />
Return<br />
IRD<br />
M<br />
u<br />
x<br />
Up<br />
Link<br />
S<br />
receive<br />
Ad<br />
Insert<br />
Equipment and Signal Status<br />
Pr Audio / Video Proxy<br />
Final<br />
Encode<br />
or<br />
Transcode<br />
Fp Audio / Video Fingerprints
Fingerprints for Monitoring Ad Insertion Agreements<br />
Distribution<br />
Remote Monitoring Site<br />
Playout Center<br />
WAN<br />
EdgeVision Set Top Box<br />
Monitoring LAN<br />
Up-Link<br />
Cable/IPTV<br />
Head End<br />
Primary<br />
Playout<br />
Backup<br />
Playout<br />
2x1<br />
Distribution<br />
Enoder<br />
Return<br />
IRD<br />
M<br />
u<br />
x<br />
Up<br />
Link<br />
Receive<br />
Ad<br />
Insert<br />
Commerical<br />
Verification Post<br />
Cable Operator<br />
Final<br />
Encode<br />
or<br />
Transcode
Miranda End-to-End Lip Sync Monitoring<br />
FP<br />
Fingerprints<br />
Generated and<br />
Transmitted<br />
by various<br />
devices<br />
iCONTROL<br />
Collect & Compare<br />
Fingerprints +<br />
Detect and Measure<br />
Lip Sync Errors<br />
EdgeVision<br />
FP<br />
Frame Sync<br />
FP<br />
LAN/WAN<br />
FP<br />
FP<br />
FP<br />
2X1<br />
IRD<br />
Up Converters
iControl End-to-End Lip Sync Monitoring<br />
Server A Out<br />
MC Out<br />
TX Return
Agenda<br />
What is a media fingerprint<br />
How does it work<br />
Applications in broadcast TV<br />
Next<br />
Fingerprint standardization
The need for an<br />
interoperable fingerprint standard<br />
is similar to the need for an interoperable<br />
compression standard<br />
(ex. MPEG-2 or H.264)
Vendor A<br />
Lip Sync Application<br />
<strong>Fingerprinting</strong> Nirvana<br />
Special event venue<br />
Network Facility<br />
Local Station<br />
Mobile<br />
studio<br />
Vendor A<br />
Equipment<br />
Encode<br />
+ Tx<br />
IRD<br />
Vendor B<br />
Equipment<br />
Server<br />
Facility<br />
Infrastructure<br />
Branding<br />
+ Proc<br />
Vendor A<br />
Equipment<br />
Encode<br />
+ Tx<br />
Return<br />
IRD<br />
Rx +<br />
Decode<br />
Vendor Local C<br />
Equipment Process<br />
Encode<br />
+ Tx<br />
Off Air<br />
IRD<br />
Service Provider<br />
(Cable, DTH, IPTV)<br />
Vendor D<br />
Equipment<br />
Rx+<br />
Decode<br />
Encode<br />
+ Tx
<strong>Fingerprinting</strong> Nirvana<br />
Vendor A<br />
Lip Sync Application<br />
Vendor B<br />
Content Compare Application<br />
Special event venue<br />
Network Facility<br />
Local Station<br />
Mobile<br />
studio<br />
Vendor A<br />
Equipment<br />
Encode<br />
+ Tx<br />
IRD<br />
Vendor B<br />
Equipment<br />
Server<br />
Facility<br />
Infrastructure<br />
Branding<br />
+ Proc<br />
Vendor A<br />
Equipment<br />
Encode<br />
+ Tx<br />
Return<br />
IRD<br />
Rx +<br />
Decode<br />
Vendor Local C<br />
Equipment Process<br />
Encode<br />
+ Tx<br />
Off Air<br />
IRD<br />
Service Provider<br />
(Cable, DTH, IPTV)<br />
Vendor D<br />
Equipment<br />
Rx+<br />
Decode<br />
Encode<br />
+ Tx
SMPTE Standardization – Lip Sync<br />
• SMPTE standardisation work is ongoing in the Technology<br />
Committee 24TB (Television and Broadband <strong>Media</strong>) for Lip Sync,<br />
based on fingerprints<br />
• Documents are in early draft<br />
• Multiple vendors are collaborating<br />
• Contributions to the work is always welcomed (www.smpte.org)<br />
• Extensive testing is being conducted to ensure that the fingerprint<br />
algorithm can survive the signal processing that a signal could<br />
experience
Fingerprints<br />
Proven to be Robust<br />
In Real End to End<br />
Testing
HD/SD<br />
Playout<br />
Omneon 4102<br />
HD Server<br />
SV43 TPT 2<br />
Cross/<br />
Up Convert<br />
Miranda<br />
XVP1801UC<br />
AC-3<br />
Decode<br />
Evertz Decoder<br />
520ADA- DD- HD<br />
re-map SAP to 7/8<br />
Evertz Decoder<br />
520ADA- DD- HD<br />
pass 2.0 decode 5.1<br />
a1<br />
Routing<br />
33 .<br />
Utah 32 .<br />
400<br />
Series<br />
router<br />
HD<br />
SDI<br />
Master<br />
Control<br />
Utah<br />
MC400<br />
Master Ctr Sw.<br />
HD<br />
SDI<br />
Routing<br />
Utah<br />
8X1<br />
A/V Processing<br />
Video Processing<br />
Audio Processing<br />
Compression<br />
AFD Insertion<br />
Miranda<br />
MMX-1801<br />
a2<br />
Bug/EAS<br />
Evertz #1<br />
HD9625LGA<br />
SAP<br />
Silence Fix<br />
AJA HD10AM<br />
Titus TIT-3DRX<br />
a4<br />
Weather CG<br />
Ross<br />
MDK-111ALite<br />
HD<br />
SDI<br />
DA<br />
a5<br />
Audio<br />
Demux<br />
AJA HD10AM<br />
De-embedder<br />
AC-3<br />
Encoding<br />
Dolby DP569<br />
AC3 Encode<br />
Audio<br />
Frame Sync<br />
Dolby DP584<br />
Frame Sync<br />
Caption<br />
Translate<br />
Evertz<br />
7760<br />
Emission<br />
Encoder<br />
Harmonic<br />
MV500<br />
IP<br />
Emission<br />
Mux<br />
Harmonic<br />
Prostream<br />
1000<br />
ASI<br />
HD SDI<br />
Nielsen<br />
Insertion<br />
ASI<br />
Wegener<br />
NAVE<strong>II</strong>c<br />
ASI<br />
Convert<br />
ENEN<br />
ASI > 310m<br />
Fiber<br />
Link<br />
Force<br />
2753<br />
ATSC<br />
TX<br />
KTCA<br />
Ch 34<br />
Tx<br />
AES<br />
ATSC<br />
RX/Dec<br />
Sencore<br />
MRD8137A<br />
8VSB<br />
Utah<br />
400<br />
Series<br />
router<br />
b1<br />
MPEG Dec<br />
c1<br />
HD > SDc2<br />
MPEG Enc MPEG Dec c3<br />
Tandberg<br />
Tandberg<br />
Bitlink<br />
Miranda<br />
EN8030 IP RX1290<br />
ASI Dec<br />
1801DC<br />
IP SD Enc<br />
IP SD Dec<br />
SDI<br />
“SD Simultcast Distribution”<br />
SMPTE Lip Sync Testing<br />
Broadcaster Site<br />
2-10-2011
First Impressions & Comments<br />
• “…Gave me accurate audio to video delay measurements<br />
through-out the distribution chain…”<br />
• “…With the system, we were able to determine that we had a<br />
very slow drift once a week from one of our transmission<br />
encoders…”
Conclusion<br />
• <strong>Fingerprinting</strong> differs from watermarking<br />
• <strong>Fingerprinting</strong> has many applications<br />
• Standardization is essential for<br />
fingerprinting to reach its full potential
<strong>Media</strong> <strong>Fingerprinting</strong> – Part <strong>II</strong><br />
Applications in<br />
Broadcast Television<br />
Questions, Comments<br />
Observations <br />
Marco Lopez<br />
mlopez@miranda.com
Change language