[PDF] [PDF] - Ascom

TEMS SOLUTIONS 

for Testing Multimedia (Video, Audio) Applications 

Presented by: Dr. Irina Cotanis 

Testing Multimedia © Ascom Network Testing Inc. 

Global Applied Research 

1

OUTLINE 

• Aspects of Objective Multimedia Equality Evaluation on 3G Networks 

• Types of Multimedia Quality Evaluation Metrics 

• Usage of Various Metrics 

• Standardization Metrics and Development Status 

• TEMS Video/Multimedia Solution: MOS Scores and Beyond 

• Conclusions 

Testing Multimedia © Ascom Network Testing Inc. 2

ASPECTS OF OBJECTIVE MULTIMEDIA QUALITY EVALUATION 

ON 3G NETWORKS 

3G Challenges in a Nutshell 

Consequences 

Testing & Evaluation Aspects 


3

[ ASPECTS OF OBJECTIVE MULTIMEDIA QUALITY EVALUATION ON 3G NETWORKS ] 

3G CHALLENGES IN A NUTSHELL 

• Multimedia support for audio/speech, video 

(streaming, mobile/IPTV, video telephony, video sharing, gaming, etc.) 

Audio wideband/super-wide speech and 

music; from traditional speech to 

VoIP/VoLTE or VoLGA or 

VoIP over IMS (known as MMTel) 

Very low latencies, stable high throughput 

Variety of codecs (commercial/standardized) 

to ensure speech/audio for PSS / MMS / 

MBMS: 

multi-bandwidth with complex error 

concealment techniques 


Video delivery for a large variety of applications: 

mobile and IP TV, streaming, telephony, 

sharing, YouTube 

Variety of contents (news, sports, gaming, 

entertainment) 

Variety of devices/clients; video formats/sizes and 

displays 

Variable delay, jitter buffer (including adaptive 

buffering), packet loss 

Variety of codecs (commercial / standardized) 

with complex error concealment techniques and 

compression schemes 

4


CONSEQUENCES 

QoS and intrinsically QoE dependency on: 

• Performance of the network’s segments (RAN, Core-CS/PS or flat all IP, IMS) 

Heterogeneous networks differing in throughput, latency, and reliability 

Open interface difficulties (network challenges to interact with others in order to provide service continuity) 

Multiple players (operators, service providers) 

• Codec characteristics and performances 

Low/very-low to high video rates 

o Mobile (QCIF/QVGA): (>1.5)16/80kbpsec-320/800kbps; PC-CIF: 128kb/s-704kb/s; PC ITU-R Rec.601/HVGA: 320kb/s-2Mb/s 

Audio bandwidth and rates 

o Standardized: WB (300Hz-7kHz, 12.65kb/sec): AMR-WB (G.729.1); Audio Super-WB (300Hz-16kHz, 48-64kB/sec): E-, AAC+, AAC-LD 

o Commercial speech codecs: SILK (Skype), etc 

Variety of players with embedded codecs 

o Commercial: Windows MediaVideo9, RealVideo9, Flush, etc. 

o Standardized: H.264 (MPEG4-layer10; fixed and mobile MM)/AVC, H.263 (3GPP; video conferencing)) 

Variety of video sizes: QCIF(177x144), QVGA(320x240), CIF(352x288), HVGA 


5


TESTING & EVALUATION ASPECTS – 

TEST SCENARIOS 

The complexity of “simple” services implies a vast range of conditions to account for 

when determining subscriber expectations 

Multiple references x multiple field test conditions (not exhaustive) 

Multimedia: >400 reference scenarios for field test conditions 

Video: Codecs: Microsoft VC1, Real9, MPEG2/4, H.261/263, H.264/AVC+ 

Bitrates: 32..80…250…800kbs…2Mbs (HVGA) 

Frames per sec: 5, …8.33,12.5, 24, 25, 30fps 

Video sizes: QCIF, QVGA, CIF, HVGA 

o Audio: Codecs: AMR-WB+, AAC+, AAC-LD 

Bitrates: 96kb/s, 48kb/s, 32kb/s, 24kb/s, 18kb/s, 14kb/s, 12.2kb/s 


6



VIDEO SIGNAL ON IP STREAMS 

• Video frames ("pictures") 

• Transmitted at a rate of 15-60 frames per second 

(e.g., 10fps for 3G video telephony; 25/30fps mobile TV) 

• Compressed based on rather lossy than lossless techniques using temporal inter-frame and 

spatial intra-frame redundancy 

• Coding techniques: 

o each frame is divided into blocks (typically 16x16 pixels) 

o each block transformed using a Discrete Cosine Transform (DCT) 

o DCT coefficients quantized and compressed further 

• Group of Pictures (GoP). For example: 

I, B, B, P, B, B, P, B, B, P, B, B, P, B, B sent at 30 frames per second 

• I frame is independently compressed requiring a much larger number of IP packets than the 

P or B frames (which only encode changes from the previous frame) 


7




• Video stream processing: 

Compressed frames divided into some number of transport units for transmission over IP network 

Transport units first encapsulated in a transport packet (RTP or MPEG-2/4 Transport) , 

then in UDP or TCP, and then in IP 

• Video IP streams structure 

[IP header] [UDP or TCP header] [RTP header or MPEG-2 Transport ] [Video payload ] 





• Video IP streams behavior (error, lost) 

• Video systems with Forward Error Correction: redundancy to the packet stream, 

which allows some proportion of lost packets to be replaced at the receiving end 

• Usage of retransmission based protocol to replace lost packets 

(Reliable UDP, TCP, or multicast with unicast retransmission) 


9



MEASUREMENT CHALLENGES 

• Capturing procedures for video and audio 

• Embedded players don’t give access to 

capturing the received, impaired multimedia 

samples 

• Procedures required to measure audio and 

video misalignment (lip sync) for multimedia 

quality evaluation 

• Received samples must have audio and video 

re-aligned 

• The temporal and spatial misalignment 

measurement of the video sequences 


• Spatial: block translation within a frame 

• Analysis of inter-frame temporal and intra- 

frame spatial misalignment 

• Large variety of contents: 

sports, news, movies, games 

• No objectively defined “standard content” 

could cause low accuracy vs. range of content 

• Proven to strongly impact quality evaluation 

10

TYPES OF MULTIMEDIA QUALITY EVALUATION METRICS 


11

[ TYPES OF MULTIMEDIA QUALITY EVALUATION METRICS ] 

OBJECTIVE MEASUREMENTS 

E2E QoE monitoring 

Intrusive: 

Uses test original and degraded signals 

to provide quality score 

Advantages: 

• Direct estimator of subscriber’s opinion 

• Quality ensured by the entire network 

• Requires access only to the end point 

Disadvantages: 

• May push a network to capacity limits 

• Limited space-time granularity 

Algorithms: 

• Video (MM1): PEVQ, PVA (J.247) 

• Video HDTV: VQEG (2010) 

• Audio (MM2): VQEG on-going work (very early phase) 

• Third parties 


Perceptual (content-based) 

Troubleshooting/diagnosing in correlation 

with full reference perceptual metric 

Non-intrusive (non-reference, hybrid): 

Uses received impaired video content and 

some IP bit stream info (hybrid) 

Advantages: 

• Normal usage of the network 

• Troubleshooting the “problem generating” node 

• Flexible implementation 


• Low accuracy (high-order averaging is required and therefore 

possible problems could be smoothed out and not properly 

detected) 

Algorithms: 

• Video: ITU-T SG 16 report - 2008, J.246 (RR) 

• MM (video, audio): hybrid model (VQEG, on-going) 


12

[ TYPES OF MULTIMEDIA QUALITY EVALUATION METRICS ] 

OBJECTIVE MEASUREMENTS 

No Reference (parametric-based) 

Troubleshooting /diagnosing in correlation with full perception metric 

Non-intrusive: 

Uses IP/transport parameters or bit stream (payload) to predict quality 

Advantages: 

• Normal usage of the network 

• Troubleshooting the “problem generating” node (if access enabled) 

• High time and space granularity 

• Possibility for quick correlation with network behavior 


• Low accuracy (high-order averaging is required and therefore possible problems could be smoothed out) 

• Quality evaluation “one-dimensional,” taking into consideration metrics belonging to a single segment of entire network (generally IP) 

• Missing content information 

Algorithms (Video/MM): 

• P.NAMS (IP header based) – ITU-T on-going work (2011) 

• P.NBAMS (IP payload base; bit stream decoding) – ITU-T on-going work (early phase) (2011-2012) 



13

USAGE OF VARIOUS METRICS 

What do they measure? 

How do they measure? 


14

[ USAGE OF VARIOUS METRICS ] 

WHAT DO THEY MEASURE? Typical MM Degradations 

Application Main degradation sources 

(Network centric: QoS) 

Video telephony 

(CS) 

MM streaming 

Mobile/IP TV 

(PS) 

1. Terminals: 

• Camera lens and software 

• codecs 

2. Network: 

• Transmission delay and loss (BLER) 

1. Video creation: down-sampling, content 

2. Terminals: 

Transmitter (codec/player, server): 

• Encoding rate, Delay, codec 

Receiver (codec/player, client): 

• Decoding rate, error concealment 

scheme 

Delay, Jitter buffer 

3. Network (transport): 

• Transmission bit rate 

• Transmission delay and loss (packet 

loss) and their distribution 

Perceived video degradations and their main causes 

(Subscriber centric: QoE) 

1. Jerkiness 

• Motion does not look smooth 

• Causes: dropped frames 

2. Blockiness 

• Block structure 

• Causes: compression techniques, transmission 

errors 

3. Blurriness 

• Loss of fine detail and smearing of the edges 

• Causes: high frequency attenuation 

(during recording or encoding process) 

4. Noise: 

• Pixels whose content deviates from original video 

• Causes: noisy receiving equipment , compression, 

transmission error 

5. Colorfulness 

• Intensity of saturation of colors and their spread and 

distribution in the video 


[ USAGE OF VARIOUS METRICS ] 

HOW DO THEY MEASURE? Typical implementation 

Downsampling 

Encoding 

Video/multimedia streaming and 

mobile TV perceptual, parametric and hybrid measurements 

Streaming 

server 

Mobile TV 

server 


Cellular 

Voice & Data 

Network 

Video/MM 

or mobile TV 

FR (intrusive) measurement 

Hybrid measurement 

Video/MM 

or mobile TV 

Decoding 

Radio access and/or 

IP transport/payload 

parameters 

Non-intrusive (parametric) measurement 

16

STANDARDIZATION METRICS & DEVELOPMENT STATUS 


17

[ STANDARDIZATION METRICS & DEVELOPMENT STATUS] 

Model High level description - generalities Standardization 

P.NAMS Definition: parametric model that predicts the impact of observed IP network 

impairments on quality experienced by the end-user in multi-media mobile streaming 

and IPTV applications over transport formats such as: RTP (over UDP), MPEG2-TS 

(over UDP or RTP/UDP), 3GPP-PSS (over RTP) based on IP header information 

Characteristics: May be deployed in end-point and/or in mid-network monitoring points 

Primary applications: Monitoring of transmission quality for operations/maintenance 

P.NBAMS Definition: See P.NAMS + IP payload information 

Characteristics: See P.NAMS 

Primary applications: See P.NAMS 

PEVQ, PVA Definition: Perceptual full reference model that predicts the video only quality 

experienced by the end-user in multi-media mobile streaming, mobile/IPTV, video 

telephony applications delivered on different network conditions (e.g., bit error, packet 

loss) using different codecs (e.g., MPEG2, H.264, Real 9), in VGA, CIF, and QCIF 

formats 

Characteristics: Generally deployed at end-point 

Primary applications: Quality monitoring of deployed networks to ensure their 

operational readiness, lab testing for video systems 

NTT PSNR, 

VMon , 

Yonsei 

University 

Definition: Non-reference and reduced reference solution of J.247 and included in J.246 

Characteristics: Generally deployed at end-point, but mid-point also possible and 

actually recommended 


operational readiness 

ITU-T SG 12 /Q14 

Slow on-going work – 

estimated due date 2011 

ITU-T SG 12 /Q14 

Slow on-going work – 

estimated due date 2011 

VQEG study, ITU-T 

J.247 

(finished 

recommendation) 

VQEG study, ITU-T 

J.246 

(finished 

recommendation) 


[ STANDARDIZATION METRICS & DEVELOPMENT STATUS] 

Model High level description - generalities Standardization 

Hybrid Definition: See P.NBAMS + some video signal information 

(similar to non-reference J.246) 

Characteristics: Generally deployed at end-point, but mid-point also could possibly 

be recommended 


operational readiness 

MM-phase 2 

(mainly 

informative for 

now) 

Definition: Perceptual full reference audio, multimedia 

NOTE: To complete the MM-phase 1 model (J.247), which was video only 

Characteristics: Deployed at end-point 



HDTV Definition: Perceptual full reference model that predicts the video-only quality 

experienced by the end-user in HDTV (for IPTV) 

Characteristics: Deployed at end-point 

Input: Original and captured degraded video signal 

Output: Predicts Mean Opinion Scores (MOS) for video 



VQEG study – on going 

work; estimated due 

date 2011 

VQEG study, very early 

phase 

NOTE: A lot of 

subjective testing work is 

still needed 

Finalized VQEG study to 

be presented/submitted 

to ITU-T/SG9 

NOTE: Similar to J.247, 

but for HD IPTV 


TEMS VIDEO/MULTIMEDIA SOLUTION: MOS SCORES & 

BEYOND 


20

[ TEMS VIDEO/MULTIMEDIA SOLUTIONS: MOS SCORES & BEYOND] 

MOS SCORE SOLUTIONS 

Solution/ 

Product 

Solution: 

• PEVQ 

(perceptual FR) 

Products: 

• TEMS 

Investigation 

• TEMS Discovery 

• 2010-2011 

Solution: 

• MTQI 

(parametric IP 

header based) 

Products: 

• TEMS 

Investigation 

• TEMS Discovery 

• 2010-2011 

Applications Audio 

Codec 

• Mobile TV 

(MBMS) 

• IPTV 

• MM streaming 

• Mobile/wireline 

video telephony 

• Progressive 

download 

• Mobile TV 

(MBMS) 

• MM streaming 

• (Mobile video 

telephony)* 

• (Progressive 

download)* 

Video Codec / 

Format / fps 

N/A Video codec: 

Windows Media (VC-1), 

Real Video 9/10, H.261, 

H.263, MPEG4/H.264, 

MPEG2, Cinepak, DivX, 

Sorenson3, and Theora 

AMR- 

NB/WB+ 

MPEG4/ 

AAC 

AAC-LC 

(HE-AAC)* 

Format: VGA, CIF, QCIF 

fps: 

2.5, 5, 8, 12.5, 15, 20, 25, 30 

Video codec: 

MPEG4, H.264 

Format: 

QCIF, QVGA, (HVGA) 

fps: 

5, 8.33, 12.5, 15, 20, 25, 30 

Protocols Comments 

Transparent • High accuracy, to be used as a reference 

quality score 

• Covers a large range of applications and is 

protocol independent 

• Video only for now (audio is not expected 

earlier than 2012) 

• Test stimuli covering sports, news, 

entertainment 

3GPP 

MBMS, PSS 

or RTSP 

over 

RTP/UDP/IP 

MPEG2/ 

RTP/TCP/IP 

(HTTP/TCP 

/IP)* 

• To replace previous solutions (VSQI/VTQI) 

gradually 

• No backwards compatibility because they are 

parametric models designed with different 

scopes and it is not recommended either to 

compare or to map/convert to each other 

• Low bit rates focused (mobile apps) 

• Multiple video contents 

• Unique: 

Works also for H.263 

On-going tests to extend its application for 

flash player (applications such as Spotify 

(similar to Pandora); YouTube) 

* Later releases 



MOS SCORE SOLUTIONS 

Solution/ 

Product 

Solution: 

• VSQI 

Products: 

• TEMS 

Investigation 

• TEMS 

Automatic 

• TEMS 

Discovery 

Solution: 

• VTQI 

Products: 

• TEMS 

Investigation 

• TEMS 

Automatic 

• TEMS 

Discovery 

Applications Audio 

Codec 

MM streaming AMR- 

NB/WB+ 

Video telephony AMR- 

NB/WB+ 

Video Codec / 

Format / fps 

Video codec: 

Real Video, MPEG4/H.264, 

Format: QCIF 

fps: 15, 30 

Protocols Comments 

RTP/TCP/IP • Unique: 

MM score 

Buffering/re-buffering analysis 

(30-sec samples) 

1-sec MOS scores 

(high granularity specific to mobile 

environment) 

30-sec MOS scores 

(jitter buffer global effect) 

• Multiple video contents 

H.263 CS • Unique: 

MM score 

* Later releases 



BEYOND MOS SCORES 

Solution Outputs Scope / Usage Comments 

PEVQ • MOS-video only 

• Video metrics (e.g., blurriness, 

jerkiness, blockiness, delay, and 

delay variations) 

MTQI • MOS (video for now, audio ready 

also; multimedia later release) 

• IP parameters (jitter buffer size, bit 

rate, packet-loss, packet-discard, 

averaged burst packet loss length, 

burst packet loss variation range, 

mean packet size, delay, delay 

variation range) 

• Codec and bit rate 

VSQI • MOS (short and long term) 

• Buffering duration, delay 

• Overall end-user perception for a large set of 

applications 

• Reference video quality score 

• Video imagine diagnostic 

• Evaluation of the media path’s delay 

• Video, audio, and multimedia quality impacted by 

IP part of the network 

• Diagnostic: 

Video/audio path 

Bandwidth / admission control (congestion 

detection) optimization (per service type) 

MOS score correlated with bit rate / throughput 

and service type 

• For the video case, when correlated to PEVQ, 

scores could be used to identify whether RAN had 

an impact on a possible quality degradation 

• Unique metric to provide the most 

accurate subscriber's perception 

• To be used as reference metric when 

correlated with other metrics such as 

MTQI 

This solution could be used in correlation 

with PEVQ as well as with TEMS IP 

analysis in order to diagnose possible IP 

network problems and/or identify whether 

RAN might affect quality 

Similar to MTQI (MM streaming only) Similar to MTQI features, but for MM 

streaming only (no separate video-audio 

analysis) 



BEYOND MOS SCORES: VSQI & IP ANALYSIS 

• IP Analysis window: Highly versatile tool that 

reveals traffic over a large number of protocols 

allowing analysis and troubleshooting of any 

service in minute detail (per application / transport 

/ network / data link protocol) 

• Demo: Snapshot 

Streamed video clip replayed in the Video Monitor 

WCDMA Data Line Chart tracks the application 

throughput (blue curve) to evaluate the streaming 

quality in terms of the VSQI quality measure (green 

bars) 

IP Analysis window: Monitors the message traffic over 

the RTP, RTCP, and RTSP protocols that are used for 

streaming. 

Application layer: BOOTP, DNS, FTP, FTP-DATA, HTTP, NBDS, 

NBNS, RTCP, RTP, RTSP, SNMP 

Transport layer: TCP, UDP 

Network layer: ICMP, IPv4, IPv6 

Data link layer: PPP CC, PPP CHAP, PPP IPCP, PPP LCP, PPP PAP 


CONCLUSIONS 


25

CONCLUSIONS 

• Video / audio / multimedia itself (in addition to the network’s complexity) raises more complex 

and difficult challenges than speech ever did or would do (or is estimated to do) 

• Various types of metrics have different applications and usage limitations 

• Standardization status on the video/audio/multimedia topic 

• TEMS video/multimedia suite: MOS scores and beyond 


THANK YOU! 


27

Questions? 


28

[PDF] [PDF] - Ascom

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template ?