VoIP Performance Management - Telchemy
VoIP Performance Management - Telchemy
VoIP Performance Management - Telchemy
You also want an ePaper? Increase the reach of your titles
YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.
1<br />
<strong>VoIP</strong> <strong>Performance</strong> <strong>Management</strong><br />
Alan Clark<br />
CEO, <strong>Telchemy</strong><br />
Internet Telephony Fall 2005<br />
Internet Telephony - Fall 2005
2<br />
Outline<br />
• Problems affecting <strong>VoIP</strong> performance<br />
• Tools for Measuring and Diagnosing Problems<br />
• Protocols for Reporting QoS<br />
• <strong>VoIP</strong> <strong>Performance</strong> <strong>Management</strong> Architecture<br />
• Application to Enterprise and Service Provider<br />
Networks<br />
Internet Telephony Fall 2005
3<br />
Enterprise <strong>VoIP</strong> Application<br />
IP<br />
Phones<br />
Gateway<br />
IP VPN<br />
IP<br />
Phone<br />
Internet Telephony Fall 2005<br />
Branch Office<br />
Teleworker<br />
IP<br />
Phone
4<br />
Residential <strong>VoIP</strong> Application<br />
Residential<br />
Gateway<br />
Internet Trunking<br />
Gateway<br />
Internet Telephony Fall 2005<br />
PSTN
5<br />
Potential Issues<br />
IP<br />
Phones<br />
Gateway<br />
LAN congestion,<br />
Long Ethernet segments,<br />
Duplex mismatch<br />
Route flapping,<br />
Link failures,<br />
Delay<br />
Line<br />
Echo<br />
IP VPN<br />
Access Link<br />
Congestion<br />
Internet Telephony Fall 2005<br />
CODEC<br />
distortion<br />
Acoustic<br />
ECHO<br />
IP<br />
Phone
6<br />
Call Quality Problems<br />
• Packet Loss<br />
• Jitter (Packet Delay Variation)<br />
• Codecs and PLC<br />
• Delay (Latency)<br />
• Echo<br />
• Signal Level<br />
• Noise Level<br />
Internet Telephony Fall 2005
7<br />
Packet Loss and Jitter<br />
IP<br />
Network<br />
Packets lost<br />
in network<br />
Internet Telephony Fall 2005<br />
Jitter<br />
Buffer<br />
Packets discarded<br />
due to jitter<br />
Codec<br />
Distorted<br />
Speech
8<br />
Jitter measurements can be misleading!!!<br />
Delay (mS)<br />
150<br />
125<br />
100<br />
75<br />
50<br />
Average jitter level (PPDV) = 4.5mS<br />
Peak jitter level = 60mS<br />
0 0.5 1 1.5 2<br />
Time (Seconds)<br />
Internet Telephony Fall 2005
9<br />
Delay (mS) & RSSI<br />
WiFi can also cause jitter<br />
300<br />
250<br />
200<br />
150<br />
100<br />
50<br />
0<br />
Recvd Signal Strength<br />
Delay<br />
Internet Telephony Fall 2005<br />
0<br />
25<br />
50<br />
75<br />
100<br />
125<br />
150<br />
175<br />
200<br />
225<br />
250<br />
275<br />
300<br />
325<br />
350<br />
375<br />
400<br />
425<br />
450<br />
Time
10<br />
Effects of Jitter<br />
• Low levels of jitter absorbed by jitter buffer<br />
• High levels of jitter<br />
– lead to packets being discarded<br />
– cause adaptive jitter buffer to grow - increasing delay but<br />
reducing discards<br />
• If packets are discarded by the jitter buffer as they<br />
arrive too late they are regarded as “discarded”<br />
• Simple jitter metrics such as PPDV can be misleading<br />
Internet Telephony Fall 2005
11<br />
Packet Loss<br />
500mS Avge Packet Loss Rate<br />
50<br />
40<br />
30<br />
20<br />
10<br />
0<br />
Average packet loss rate = 2.1%<br />
Peak packet loss = 30%<br />
30 35 40 45 50 55 60 65 70<br />
Time (seconds)<br />
Internet Telephony Fall 2005
12<br />
200<br />
150<br />
100<br />
50<br />
0<br />
Consecutive loss<br />
20 percent burst density (sparse burst)<br />
Burst weight (packets) Example Packet Loss Distribution<br />
0 100 200 300 400 500<br />
Burst length (packets)<br />
Internet Telephony Fall 2005
13<br />
Loss and Discard<br />
• Loss is often associated with periods of high<br />
congestion<br />
• Jitter is due to congestion (usually) and leads<br />
to packet discard<br />
• Hence Loss and Discard often coincide<br />
• Other factors can apply - e.g. duplex<br />
mismatch, link failures etc.<br />
Internet Telephony Fall 2005
14<br />
Example Loss/Discard Distribution<br />
Burst weight (packets)<br />
200<br />
150<br />
100<br />
50<br />
0<br />
Consecutive loss<br />
25 percent burst density (sparse burst)<br />
0 100 200 300 400 500<br />
Burst length (packets)<br />
Internet Telephony Fall 2005
15<br />
Leads To Time Varying Call Quality<br />
Bandwidth (kbit/s)<br />
500<br />
400<br />
300<br />
200<br />
100<br />
5<br />
4<br />
3<br />
2<br />
MOS 0<br />
1<br />
High jitter/ loss/ discard<br />
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18<br />
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18<br />
Time<br />
Internet Telephony Fall 2005<br />
Voice<br />
Data
16<br />
Packet Loss Concealment<br />
Estimated by PLC algorithm<br />
• Mitigates impact of packet loss/ discard by replacing<br />
lost speech segments<br />
• Very effective for isolated lost packets, less effective for<br />
bursty loss/discard<br />
• But isn’t loss/discard bursty?<br />
– Need to be able to deal with 10-20-30% loss!!!<br />
Internet Telephony Fall 2005
17<br />
Effectiveness of PLC<br />
Codec<br />
distortion<br />
ACR MOS<br />
5<br />
4<br />
3<br />
2<br />
1<br />
0 5 10 15 20<br />
Packet Loss/Discard Rate<br />
Internet Telephony Fall 2005<br />
G.711 no PLC<br />
G.711 PLC<br />
Impact of<br />
loss/ discard<br />
and PLC<br />
Typical burst packet<br />
Loss/discard rate
18<br />
Call Quality Problems<br />
• Packet Loss<br />
• Jitter (Packet Delay Variation)<br />
• Codecs and PLC<br />
• Delay (Latency)<br />
• Echo<br />
• Signal Level<br />
• Noise Level<br />
Internet Telephony Fall 2005
19<br />
Effect of Delay on Conversational Quality<br />
MOS Score<br />
5<br />
4<br />
3<br />
2<br />
1<br />
Low echo level (55DB)<br />
Significant echo level (35dB)<br />
0 100 200 300 400 500 600<br />
Round trip delay (milliseconds)<br />
Internet Telephony Fall 2005
20<br />
Interaction of echo and delay<br />
• Echo with very low delay sounds like<br />
“sidetone”<br />
• Echo with some delay makes the line sound<br />
hollow<br />
• Echo with over 50mS delay sounds like…. Echo<br />
• Echo Return Loss<br />
– 55dB or above is good<br />
– 25dB or below is bad<br />
Internet Telephony Fall 2005
21<br />
Cause of Echo<br />
IP<br />
Gateway<br />
Echo<br />
Canceller<br />
Round trip delay - typically 50mS+<br />
Internet Telephony Fall 2005<br />
Line<br />
Echo<br />
Additional delay introduced by <strong>VoIP</strong><br />
makes existing echo problems more obvious<br />
Acoustic<br />
Echo
22<br />
Causes of Delay<br />
Network<br />
delay<br />
IP<br />
UDP<br />
TCP<br />
Accumulate and encode<br />
RTP<br />
IP<br />
UDP<br />
TCP<br />
RTP<br />
CODEC<br />
CODEC<br />
Echo<br />
Control<br />
Jitter buffer, decode and playout<br />
Echo<br />
Control<br />
Internet Telephony Fall 2005<br />
External delay
23<br />
Call Quality Problems<br />
• Packet Loss<br />
• Jitter (Packet Delay Variation)<br />
• Codecs and PLC<br />
• Delay (Latency)<br />
• Echo<br />
• Signal Level<br />
• Noise Level<br />
Internet Telephony Fall 2005
24<br />
Signal Level Problems<br />
0 dBm0<br />
-36 dBm0<br />
Amplitude Clipping occurs -- speech<br />
sounds loud and “buzzy”<br />
Temporal Clipping occurs with VAD or Echo<br />
Suppressors -- gaps in speech, start/end of words<br />
missing<br />
Internet Telephony Fall 2005
25<br />
Noise<br />
• Noise can be due to<br />
– Low signal level<br />
– Equipment/ encoding (e.g. quantization noise)<br />
– External local loops<br />
– Environmental (room) noise<br />
• From a service provider perspective - how to<br />
distinguish between<br />
– room noise (not my problem)<br />
– Network/equipment/circuit noise (is my problem)<br />
Internet Telephony Fall 2005
26<br />
Measuring <strong>VoIP</strong> performance<br />
Active Test<br />
- Measure test calls<br />
Passive Test<br />
- Measure live calls<br />
<strong>VoIP</strong> Specific<br />
VQmon<br />
ITU P.VTQ<br />
VQmon<br />
ITU P.VTQ<br />
Internet Telephony Fall 2005<br />
Analog signal<br />
based<br />
ITU P.862 (PESQ)<br />
ITU P.563
27<br />
“Gold Standard” - ACR Test<br />
• Speech material<br />
2<br />
3<br />
2<br />
– Phonetically balanced speech samples 8-10 seconds in length<br />
– Test designed to eliminate bias (e.g. presentation order different<br />
for each listener)<br />
– Known files included as anchors (e.g. MNRU)<br />
• Listening conditions<br />
– Panel of listeners<br />
– Controlled conditions (quiet environment with known level of<br />
background noise)<br />
4<br />
Internet Telephony Fall 2005
28<br />
Example ACR test results<br />
• Extract from an<br />
ITU subjective test<br />
• Mean Opinion<br />
Score (MOS) was<br />
2.4<br />
• 1=Unacceptable<br />
• 2=Poor<br />
• 3=Fair<br />
• 4=Good<br />
• 5=Excellent<br />
Votes<br />
50<br />
40<br />
30<br />
20<br />
10<br />
0<br />
Internet Telephony Fall 2005<br />
1 2 3 4 5<br />
Opinion Score
29<br />
Measuring <strong>VoIP</strong> <strong>Performance</strong><br />
• VQmon<br />
– Most widely used algorithm for <strong>VoIP</strong> performance monitoring. Fast efficient,<br />
supports narrow and wideband codecs, listening and conversational quality.<br />
Incorporates P.VTQ and G.107 as subsets, original model for RTCP XR.<br />
• ITU P.VTQ<br />
– In development - expected completion in mid-2006. Lightweight algorithm<br />
for narrowband use, currently only listening quality, may extend to<br />
conversational.<br />
• ITU G.107 E Model<br />
– Network planning tool, used as a basis for some monitoring applications.<br />
Inaccurate under conditions of bursty packet loss.<br />
• P.862<br />
– Intrusive speech quality algorithm. Slow - takes a PC to process one speech<br />
file in approx real time.<br />
• P.563<br />
– Non-intrusive algorithm that operates on analog speech data. Highly<br />
MIPS/Memory intensive and very inaccurate for individual calls.<br />
Internet Telephony Fall 2005
30<br />
Reminder - loss/jitter are time varying<br />
Delay (mS)<br />
150<br />
125<br />
100<br />
75<br />
50<br />
Average jitter level (PPDV) = 4.5mS<br />
Peak jitter level = 60mS<br />
0 0.5 1 1.5 2<br />
Time (Seconds)<br />
Internet Telephony Fall 2005
31<br />
VQmon algorithm<br />
Discarded<br />
Jitter<br />
buffer<br />
Arriving<br />
packets<br />
CODEC<br />
Loss/ Discard<br />
events<br />
Signal level<br />
Noise level<br />
Echo level<br />
Delay<br />
Metrics<br />
Calculation<br />
Internet Telephony Fall 2005<br />
4 State Markov Model<br />
Gather detailed<br />
packet loss info<br />
in real time<br />
Call Quality Scores<br />
Diagnostic Data
32<br />
Modeling transient effects<br />
Measured<br />
Call quality<br />
Ie(gap)<br />
Ie(burst)<br />
10 15 20 25 30 35<br />
Time (seconds)<br />
Internet Telephony Fall 2005<br />
User Reported<br />
Call quality<br />
Ie(VQmon)
33<br />
Computational model<br />
Burst loss<br />
rate<br />
Gap loss<br />
rate<br />
Signal level<br />
Noise level<br />
Echo<br />
Delay<br />
Ie mapping<br />
ETSI TS101 329-5<br />
Calculate<br />
Ro, Is<br />
Calculate<br />
Id<br />
Perceptual model<br />
Recency<br />
model<br />
Modified<br />
ITU-T G.107<br />
Internet Telephony Fall 2005<br />
Calculate<br />
R-LQ<br />
MOS-LQ<br />
Calculate<br />
R-CQ<br />
MOS-CQ
34<br />
Accuracy: Non-bursty conditions<br />
MOS Score<br />
5<br />
4.5<br />
4<br />
3.5<br />
3<br />
2.5<br />
2<br />
1.5<br />
Comparison of VQmon vs ACR MOS - ILBC 15.2k<br />
ACR MOS<br />
VQmon MOS-LQ<br />
1<br />
0 5 10<br />
Packet Loss Rate (%)<br />
15 20<br />
PESQ Score<br />
2.5<br />
1.5<br />
Internet Telephony Fall 2005<br />
4<br />
3.5<br />
3<br />
2<br />
Comparison of VQmon vs PESQ - ILBC 15.2k<br />
PESQ<br />
VQmon MOS-PQ<br />
1<br />
0 5 10 15 20 25 30<br />
Packet Loss Rate (%)
35<br />
Comparison of VQmon and E Model<br />
• VQmon<br />
– Extended G.107<br />
– Transient impairment model<br />
– Wide range of codec models<br />
– Narrow & Wideband<br />
– Jitter Buffer Emulator<br />
– Listening and Conversational<br />
Quality<br />
• G.107<br />
– Well established model for<br />
network planning<br />
– No way to represent jitter<br />
– Few codec models<br />
– Inaccurate for bursty loss<br />
– Conversational Quality only<br />
Estimated MOS<br />
4<br />
3.5<br />
3<br />
2.5<br />
2<br />
1.5<br />
1.5 2 2.5 3 3.5 4<br />
Internet Telephony Fall 2005<br />
VQmon<br />
ACR MOS<br />
Comparison of VQmon and E Model<br />
for severely time varying conditions<br />
E Model
36<br />
ITU P.563 - Passive monitoring<br />
• Analyses received speech<br />
file (single ended)<br />
• Produces a MOS score<br />
• Correlates well with MOS<br />
when averaged over many<br />
calls<br />
• Requires 100MIPS per call<br />
• NOT suitable for individual<br />
calls<br />
ACR MOS<br />
5.00<br />
4.00<br />
3.00<br />
2.00<br />
1.00<br />
Internet Telephony Fall 2005<br />
1 2 3 4 5<br />
P563 Score<br />
Comparison of P.563 estimated MOS scores with<br />
actual ACR test scores.<br />
Each point is average per file ACR MOS with 16<br />
listeners compared to P.563 score
37<br />
Active or Passive Testing?<br />
• Active testing<br />
– works for pre-deployment testing and on-demand<br />
troubleshooting<br />
• But!!!!<br />
– IP problems are transient<br />
• Passive monitoring<br />
– Monitors every call made - but needs a call to<br />
monitor<br />
– Captures information on transient problems<br />
– Provides data for post-analysis<br />
• Therefore - you need both<br />
Internet Telephony Fall 2005
38<br />
<strong>VoIP</strong> <strong>Performance</strong> <strong>Management</strong> Framework<br />
Call Server and<br />
CDR database<br />
<strong>VoIP</strong><br />
Endpoint<br />
Embedded<br />
Monitoring<br />
Signaling Based<br />
QoS Reporting<br />
VQ<br />
Network Probe,<br />
Analyzer or<br />
Router<br />
Media Path Reporting<br />
(RTCP XR)<br />
Internet Telephony Fall 2005<br />
VQ<br />
RTP stream (possibly encrypted)<br />
Network<br />
<strong>Management</strong><br />
System<br />
SNMP<br />
Reporting<br />
VQ<br />
Embedded<br />
Monitoring<br />
<strong>VoIP</strong><br />
Gateway
39<br />
RFC3611 - RTCP XR<br />
Loss Rate Discard Rate Burst Density Gap Density<br />
Burst Duration (mS) Gap Duration (mS)<br />
Round Trip Delay (mS) End System Delay (mS)<br />
Signal level RERL Noise Level Gmin<br />
R Factor Ext R MOS-LQ MOS-CQ<br />
Rx Config - Jitter Buffer Nominal<br />
Jitter Buffer Max Jitter Buffer Abs Max<br />
Internet Telephony Fall 2005
40<br />
RTCP XR Application<br />
Residential Subscriber<br />
“A”<br />
RTCP XR<br />
Quality of stream from B to A<br />
and acoustic echo at A (if known)<br />
RTCP XR<br />
Internet Telephony Fall 2005<br />
“B”<br />
Quality of stream from A to B<br />
and Echo level on trunk side<br />
Trunk<br />
side
41<br />
SIP Service Quality Reporting Event<br />
PUBLISH sip:collector@example.com SIP/2.0<br />
Via: SIP/2.0/UDP pc22.example.com;branch=z9hG4bK3343d7<br />
………<br />
Content-Type: application/rtcpxr<br />
Content-Length: ...<br />
VQSessionReport<br />
LocalMetrics:<br />
TimeStamps=START:10012004.18.23.43 STOP:10012004.18.26.02<br />
SessionDesc=PT:0 PD:G.711 SR:8000 FD:20 FPP:2 PLC:3 SSUP:on<br />
CallID=1890463548@alice.uac.chicago.com<br />
………<br />
Signal=SL:2 NL:10 RERL:14<br />
QualityEst=RLQ:90 RCQ:85 EXTR:90 MOSLQ:3.4 MOSCQ:3.3<br />
QoEEstAlg:VQMonv2.1<br />
DialogID:38419823470834;to-tag=8472761;from-tag=9123dh311<br />
Internet Telephony Fall 2005
42<br />
SIP QoS Reporting Application<br />
Residential Subscriber<br />
“A”<br />
Internet Telephony Fall 2005<br />
“B” Trunk<br />
side<br />
SIP QoS “Collector”<br />
SIP QoS report sent at end of call.<br />
Can report on both directions if RTCP XR is<br />
present in both endpoints, otherwise only<br />
on received direction.
43<br />
Enterprise Application<br />
VQ<br />
VQ<br />
VQ<br />
NMS<br />
VQ<br />
SNMP<br />
IP<br />
Phones<br />
VQ<br />
Gateway<br />
VQ<br />
<strong>VoIP</strong> SLA<br />
IP<br />
VPN<br />
RTCP XR<br />
IP<br />
Phone<br />
VQ<br />
SIP QoS Report<br />
Internet Telephony Fall 2005<br />
VQ<br />
VQ<br />
VQ<br />
Branch Office<br />
VQ<br />
Teleworker<br />
VQ<br />
IP<br />
Phone
44<br />
Actual (typical?) <strong>VoIP</strong> SLA<br />
Jitter < 20mS<br />
Loss < 0.1% per month<br />
Latency < 100mS<br />
Availability 99.9%<br />
What does this mean in practice?<br />
Internet Telephony Fall 2005
45<br />
A Better <strong>VoIP</strong> SLA<br />
99.9% of calls/intervals have<br />
MOS-LQ > 3.9<br />
MOS-CQ > 3.8<br />
Degraded Service Quality<br />
Events < 0.1/ hour<br />
[DSQ = ….]<br />
Latency < 100mS<br />
Availability 99.9%<br />
Internet Telephony Fall 2005<br />
Based on either reference<br />
or actual endpoint<br />
Transient quality problems<br />
Also reflected in MOS-CQ<br />
Availability of media AND<br />
Signaling path
46<br />
Enterprise Applications<br />
• Ensure network is <strong>VoIP</strong> ready before<br />
deployment!!<br />
• Use VQmon/RTCP XR/ SIP QoS in IP phones<br />
and gateways<br />
• Use passive monitoring on every call to catch<br />
transient problems for post analysis<br />
• Develop meaningful SLAs<br />
Internet Telephony Fall 2005
47<br />
Residential <strong>VoIP</strong> Application<br />
VQ<br />
VQ<br />
VQ<br />
VQ<br />
RTCP XR<br />
Residential<br />
Gateway<br />
Internet Trunking<br />
Gateway<br />
SIP QoS<br />
Internet Telephony Fall 2005<br />
VQ<br />
PSTN
48<br />
Residential <strong>VoIP</strong> Application<br />
VQ<br />
VQ<br />
VQ<br />
RTCP XR<br />
Residential<br />
Gateway<br />
Internet<br />
SIP<br />
QoS<br />
Internet Telephony Fall 2005<br />
VQ<br />
PSTN<br />
Trunking<br />
Gateway
49<br />
Residential <strong>VoIP</strong> <strong>Management</strong><br />
• Use RTCP XR between IP endpoints to provide<br />
more detailed call quality metrics and<br />
bidirectional reporting<br />
• Use SIP QoS reports to get data back to<br />
management systems<br />
• Insist that peer networks (either <strong>VoIP</strong> or<br />
PSTN) support RTCP XR and defined SLAs<br />
Internet Telephony Fall 2005
50<br />
Summary<br />
• Problems affecting <strong>VoIP</strong> performance<br />
• Tools for Measuring and Diagnosing Problems<br />
• <strong>Performance</strong> <strong>Management</strong> Architecture<br />
• Applications<br />
Internet Telephony Fall 2005
Change language