Commissioning and Using the 4 Gigabit Lightpath iNetTest ...

UDP Throughput London-100009000Stockholm8000Alcatel Metro Core ConnectMCC Flow control ONKernel 2.6.20-web100_pktd-plusMyricom 10G-PCIE-8A-R Fibrerx-usecs=25Coalescence ONMTU 9000 bytesMax throughput 4.05 Gbit/sNO Packet loss – flowcontrolSending host, 3 CPUs idleFor

UDP Jitter Stockholm-LondonAlcatel Metro Core ConnectMCC Flow control OFFKernel 2.6.20-web100_pktd-plusMyricom 10G-PCIE-8A-R Fibrerx-usecs=0Coalescence OFF8972 byte packets16.5 µs spacingPeak in packet spacing at16 µs as expectedOther peaks at 10 and 51 µsN(t)N(t)700000600000500000400000300000200000100000010000001000001000010001001018972B w=16.5 sto-man_rx-usecs0_5Apr080 20 40 60 80 100 120 140Latency us8972B w=16.5 sto-man_rx-usecs0_5Apr080 20 40 60 80 100 120 140Latency usconnect • communicate •collaborate

NORDUnet install TSSTest Stockholm – Manchester4 GigabitEthernetPath over TSSNORDUnet4 Gig TDM/SDHLighpathover GÉANT2+4 Gig on EthernetOptical TransmissionJANET & NNWconnect • communicate •collaborate

UDP Throughput Stockholm-Londonwith Alcatel TSS – 10GE – MCCKernel 2.6.20-web100_pktd-plusMyricom 10G-PCIE-8A-R Fibrerx-usecs=75Coalescence ONMTU 9000 bytesMax throughput 3.78 Gbit/s(Previously was 4.05 Gbit/s)Packet loss 10%at packet spacing for 4.096Gbit/swith a 8129 byte payload.Recv Wire rate Mbit/s% Packet loss70006000500040003000200010000100sto-gig6_rx75_TSS_21May080 5 10 15 20 25 30 35 40Spacing between frames us806040200sto-gig6_rx75_TSS_21May080 5 10 15 20 25 30 35 40Spacing between frames us1000 bytes1472 bytes2000 bytes3000 bytes4000 bytes5000 bytes6000 bytes7000 bytes8000 bytes8972 bytes1000 bytes1472 bytes2000 bytes3000 bytes4000 bytes5000 bytes6000 bytes7000 bytes8000 bytes8972 bytesManchester – StockholmNO packet loss at 4.096 Gbit/sconnect • communicate •collaborate

Packet loss Stockholm-Manchesterwith Alcatel TSS – 10GE – MCC12Send 1M packets from PC inStockholm8192 BytesAll cross the TSS cloud.All enter Alcatel MCCas 10 GE framesPacket loss was in MCC Card~10% at 4096 Mbit/s% Packet Loss10864200 1000 2000 3000 4000 5000Offered Rate Mbit/sClassic “bottleneck”performance.But WHY ??connect • communicate •collaborate

UDP Jitter Manchester-Stockholmwith Alcatel TSS MCC – 10GE – TSS – PCKernel 2.6.20-web100_pktd-plusMyricom 10G-PCIE-8A-R Fibrerx-usecs=0Coalescence OFF8192 Byte packetsAlcatel MCC is TDM 28 VC-4.Can only send packets to theTSSat 16 µs not at 10 Gbit/s.Peak packet spacing is~ 6 µs ie packets arriving at10 Gbit/sN(t)N(t)N(t)400000350000300000250000200000150000100000500000100000010000010000100010010700000600000500000400000300000200000100000016 µs8192B w=16 gig6-sto_rx0_19J un080 20 40 60 80 100 120 140Latency us8192B w=16 gig6-sto_rx0_19J un080 20 40 60 80 100 120 140Latency us8972B w=16.5 sto-man_rx-usecs0_5Apr08Sto-Man no TSS16 µs0 20 40 60 80 100 120 140Latency usSuggests packet bunching inthe TSS networkconnect • communicate •collaborate

UDP 1-way delayManchester-Stockholm withTSS Alcatel MCC – 10GE – TSS – PCKernel 2.6.20-web100_pktd-plusMyricom 10G-PCIE-8A-R Fibrerx-usecs=0Coalescence OFF8192 Byte packetsStructure similar to interruptcoalescence but this is OFF1-waydelayusJitter as function of recvtimesee delays of ~ 130 µsthen packets at ~6 µsThese packets at ~ 10Gbit/sjiterus190001895018900188501880018750187001865018600140120100806040200W16 gig6-sto_rx0_16J un087558000 7558500 7559000 7559500 7560000receive time us.W16 gig6-sto_rx0_16J un087558000 7558500 7559000 7559500 7560000receive time us.This confirms packet bunchingin the TSS networkconnect • communicate •collaborate

Why do we get packet lossin Copenhagen?The Alcatel TSS delivers the packets over the 10GE link to theAlcatel MCC in bunches at line-speed (with suitable pauses toget 4Gbit/s).The SDH circuit over GÉANT from the Alcatel MCC runs at 28VC-4 (~4.2Gbit/s).The Alcatel MCC ingress buffering is only about 57 k Byte.These factors together cause buffer overflow and the packetloss.connect • communicate •collaborate

NORDUnet & GÉANT changethe TSS – MCC linkA work-around !VLBI uses SDH to GÉANTAnd buffering in the TSSNORDUnetTSS Cloudconnect • communicate •collaborate

The Current 4 Gigabit link28 VC-4 SDHLinksTSS to MCC4 Gig onEthernetOver DWDMJANET & NNWconnect • communicate •collaborate

INETTESTconnect • communicate •collaborate

iNetTest: Simulink design10GE MAC core tenGbE0Inter-packet timeHistogram tenGbE0NetworkparametersCSRsTransmit & receiveevent time logInter-packet timeHistogram tenGbE110GE MAC core tenGbE1Jonathan Hargreaves JBOconnect • communicate •collaborate

iNetTest: DetailsiNetTest FPGA runs with a 200 MHz (5 ns) clocksTwo iNetTest units can be controlled over ethernet IP from a PCiNetTest Ethernet IP address and MAC preset in code– last digit selected by jumperTen GE IP address, Port and Gateway can be user configuredAutomatic or manual ARPing. ARP tables for each port can beexaminediNetTest responds to and generates PINGUDP packets can be sent by firmware between two iNetTestsor between iNetTest and PCsUser selects the number of packets to send, their length,and the time between themiBOBs count received and transmitted packets, and store up to2048 time stamped events per portiBOBs generate histograms of the arrival time distribution forreceived packetsconnect • communicate •collaborate

iNetTest & eMerlinIN:Architecture for Operation &Controlconnect • communicate •collaborate

iNetTest & eMerlinIN: GUIconnect • communicate •collaborate

iNetTest & eMerlinIN: GUIconnect • communicate •collaborate

Testing Long Term StabilityUDP Throughput Onsala-JodrellUse iNetTest to send trials of 100M 8192 Byte packets.Spacing 16 µs, Trial takes ~27 min.Measure:Throughput 4.094 Gbit/sPacket loss ZerojitteriBoB tenGbE0 locked up5000Ons-J bo_iNetTest_13May09805000Ons-J bo_iNetTest_14May097WireRateMbit/s45004000350030002500200015001000500706050403020%Packetlos10WireRateMbit/s45004000350030002500200015001000500654321%Packetlos00000 200 400 600 800 1000 12000 200 400 600 800 1000 1200Time during the transfer minTime during the transfer minconnect • communicate •collaborate

Testing Long Term StabilityUDP Jitter Onsala-JodrelliNetTestPlot the inter-packet arrival times for 3 trialsNo noticeable differencesSame distribution for tests over several weeksWireRateMbit/s5000450040003500300025002000150010005000Ons-J bo_iNetTest_14May09765432100 200 400 600 800 1000 1200Time during the transfer min%Packetlosconnect • communicate •collaborate

Testing Long Term StabilityUDP Jitter Onsala-JodrelliNetTestNO change in JitterTail ~ 10 -3 downPeak at 16 µs, tail extends to ~70 µsconnect • communicate •collaborate

What did we learn?The network is very stableiBOB state machine can lock the Ethernet transmit & receive(?)Added a reset function to the FPGA.Ethernet switches do age their MAC tables (as expected) SO:they loose the port of the destination iBOBflood packets at 4 Gbit/s to all ports in the VLANiNetTest & on2jbo_rx send “data ACK” packets back to thesource every n packet received.iNetTest and eMerlinIN operate as required.Signals from signal generator in Onsala have been sampledand transmitted to the Station board in the Jodrell correlator.connect • communicate •collaborate

State histograms from thee-Merlin Station Board88 MHz Signal, 1024 MHz clock, 2 bits, 1pol.Local eMerlinIN88 MHz Signal,1024 MHz clock,2 bits, 1 pol.eMerlinIN from Onsala-40 dBm -10 dBm -6 dBm-3 dBm 0 dBm 3 dBmconnect • communicate •collaborate

ANY QUESTIONS ?connect • communicate •collaborate