Network Traffic Characteristics of Data Centers in the Wild - Sigcomm

More documents

Recommendations

Info

theapplicationsrunineachdatacenterandthendrillingdownto theapplications’sendandreceivepatternsandtheirnetwork-level impact. Usingpackettraces,wefirstexaminethetypeofapplicationsrunningineachdatacenterandtheirrelativecontributiontonetworktraffic.Wethenexaminethefine-grainedsendingpatternsascapturedbydatatransmissionbehavioratthepacketand flowlevels.Weexaminethesepatternsbothinaggregateandata per-applicationlevel.Finally,weuseSNMPtracestoexaminethe network-levelimpactintermsoflinkutilization,congestion,and packetdrops,andthedependenceofthesepropertiesonthelocationofthelinksinthenetworktopologyandonthetimeofday. Ourkeyempiricalfindingsarethefollowing: • Weseeawidevarietyofapplicationsacrossthedatacenters, rangingfromcustomer-facingapplications,suchasWebservices,filestores,authenticationservices,Line-of-Businessapplications,andcustomenterpriseapplicationstodataintensiveapplications,suchasMapReduceandsearchindexing.Wefindthatapplicationplacementisnon-uniformacross racks. • Mostflowsinthedatacentersaresmallinsize(≤ 10KB), asignificantfractionofwhichlastunderafewhundredsof milliseconds,andthenumberofactiveflowspersecondis under10,000perrackacrossalldatacenters. • Despitethedifferencesinthesizeandusageofthedatacenters,trafficoriginatingfromarackinadatacenterisON/OFF innaturewithpropertiesthatfitheavy-taileddistributions. • Intheclouddatacenters,amajorityoftrafficoriginatedby servers(80%)stayswithintherack. Fortheuniversityand privateenterprisedatacenters,mostofthetraffic(40-90%) leavestherackandtraversesthenetwork’sinterconnect. • Irrespectiveofthetype,inmostdatacenters,linkutilizations areratherlowinalllayersbutthecore.Inthecore,wefind thatasubsetofthecorelinksoftenexperiencehighutilization.Furthermore,theexactnumberofhighlyutilizedcore linksvariesovertime,butneverexceeds25%ofthecore linksinanydatacenter. • Lossesoccurwithinthedatacenters;however,lossesarenot localizedtolinkswithpersistentlyhighutilization.Instead, lossesoccuratlinkswithlowaverageutilizationimplicatingmomentaryspikesastheprimarycauseoflosses. We observethatthemagnitudeoflossesisgreaterattheaggregationlayerthanattheedgeorthecorelayers. • Weobservethatlinkutilizationsaresubjecttotime-of-day andday-of-weekeffectsacrossalldatacenters.Howeverin manyoftheclouddatacenters,thevariationsarenearlyan orderofmagnitudemorepronouncedatcorelinksthanat edgeandaggregationlinks. Tohighlighttheimplicationsofourobservations,weconclude thepaperwithananalysisoftwodatacenternetworkdesignissues thathavereceivedalotofrecentattention,namely,networkbisectionbandwidthandtheuseofcentralizedmanagementtechniques. • BisectionBandwidth:Recentdatacenternetworkproposals havearguedthatdatacentersneedhighbisectionbandwidth tosupportdemandingapplications.Ourmeasurementsshow thatonlyafractionoftheexistingbisectioncapacityislikely tobeutilizedwithinagiventimeintervalinallthedatacenters,eveninthe“worstcase”whereapplicationinstancesare 268 DataCenter Typeof Typeof #ofDCs Study DataCenter Apps Measured Fat-tree[1] Cloud MapReduce 0 Hedera[2] Cloud MapReduce 0 Portland[22] Cloud MapReduce 0 BCube[13] Cloud MapReduce 0 DCell[16] Cloud MapReduce 0 VAL2[11] Cloud MapReduce 1 MicroTE[4] Cloud MapReduce 1 Flyways[18] Cloud MapReduce 1 Opticalswitching[29] Cloud MapReduce 1 ECMP.study1[19] Cloud MapReduce 1 ECMP.study2[3] Cloud MapReduce 19 WebServices ElasticTree[14] ANY WebServices 1 SPAIN[21] Any Any 0 Ourwork Cloud MapReduce 10 PrivateNet Webservices Universities DistributedF’S Table1: Comparisonofpriordatacenterstudies,including typeofdatacenterandapplication. spreadacrossracksratherthanconfinedwithinarack.This istrueevenforMapReducedatacentersthatseerelatively higherutilization.Fromthis,weconcludethatloadbalancingmechanismsforspreadingtrafficacrosstheexistinglinksinthenetwork’scorecanhelpmanageoccasionalcongestion,giventhecurrentapplicationsused. • CentralizationManagement: Afewrecentproposals[2, 14]havearguedforcentrallymanagingandschedulingnetworkwidetransmissionstomoreeffectivelyengineerdatacenter traffic.Ourmeasurementsshowthatcentralizedapproaches mustemployparallelismandfastroutecomputationheuristicstoscaletothesizeofdatacenterstodaywhilesupporting theapplicationtrafficpatternsweobserveinthedatacenters. Therestofthepaperisstructuredasfollows:wepresentrelated workinSection2andinSection3describethedatacentersstudied, theirhigh-leveldesign,andtypicaluses.InSection4,wedescribe theapplicationsrunninginthesedatacenters. InSection5,we zoomintothemicroscopicpropertiesofthevariousdatacenters. InSection6,weexaminetheflowoftrafficwithindatacentersand theutilizationoflinksacrossthevariouslayers.WediscusstheimplicationsofourempiricalinsightsinSection7,andwesummarize ourfindingsinSection8. 2. RELATEDWORK Thereistremendousinterestindesigningimprovednetworksfor datacenters [1,2,22,13,16,11,4,18,29,14,21];however,such workanditsevaluationisdrivenbyonlyafewstudiesofdatacentertraffic,andthosestudiesaresolelyofhuge(>10Kserver)data centers,primarilyrunningdatamining,MapReducejobs,orWeb services.Table1summarizesthepriorstudies.FromTable1,we observethatmanyofthedataarchitecturesareevaluatedwithout empiricaldatafromdatacenters. Forthearchitecturesevaluated withempiricaldata,wefindthattheseevaluationsareperformed withtracesfromclouddatacenters.Theseobservationsimplythat theactualperformanceofthesetechniquesundervarioustypesof realisticdatacentersfoundinthewild(suchasenterpriseanduniversitydatacenters)isunknownandthuswearemotivatedbythis toconductabroadstudyonthecharacteristicsofdatacenters.Such astudywillinformthedesignandevaluationofcurrentandfuture datacentertechniques.
Thispaperanalyzesthenetworktrafficofthebroadestsetof datacentersstudiedtodate,includingdatacentersrunningWeb servicesandMapReduceapplications,butalsoothercommonenterpriseandcampusdatacentersthatprovidefilestorage,authenticationservices,Line-of-Businessapplications,andothercustomwrittenservices.Thus,ourworkprovidestheinformationneeded toevaluatedatacenternetworkarchitectureproposalsunderthe broadrangeofdatacenterenvironmentsthatexist. Previousstudies[19,3]havefocusedontrafficpatternsatcoarse time-scales,reportingflowsizedistributions,numberofconcurrent connections,durationofcongestionperiods,anddiurnalpatterns. Weextendthesemeasuresbyconsideringadditionalissues,suchas theapplicationsemployedinthedifferentdatacenters,theirtransmissionpatternsatthepacketandflowlevels,theirimpactonlink andnetworkutilizations,andtheprevalenceofnetworkhot-spots. Thisadditionalinformationiscrucialtoevaluatingtrafficengineeringstrategiesanddatacenterplacement/schedulingproposals. Theclosestpriorworksare[19]and[3];theformerfocuses onasingleMapReducedatacenters,whilethelatterconsiders clouddatacentersthathostWebservicesaswellasthoserunning MapReduce.Neitherstudyconsidersnon-clouddatacenters,such asenterpriseandcampusdatacenters,andneitherprovidesascompleteapictureoftrafficpatternsasthisstudy.Thekeyobservations fromBenson’sstudy[3]arethatutilizationsarehighestinthecore butlossesarehighestattheedge.Inourwork,weaugmentthese findingsbyexaminingthevariationsinlinkutilizationsovertime, thelocalizationoflossestolink,andthemagnitudeoflossesover time.FromKandula’sstudy[19],welearnedthatwhilemosttraffic inthecloudisrestrictedtowithinarackandasignificantnumber ofhot-spotsexistinthenetwork.Ourworksupplementstheseresultsbyquantifyingtheexactfractionoftrafficthatstayswithin arackforawiderangeofdatacenters. Inaddition,wequantify thenumberofhot-spots,showthatlossesareduetotheunderlyingburstinessoftraffic,andexaminetheflowlevelpropertiesfor universityandprivateenterprise(bothareclassesofdatacenters ignoredinKandula’sstudy[19]). OurworkcomplementspriorworkonmeasuringInternettraffic[20,10,25,9,8,17]bypresentinganequivalentstudyonthe flowcharacteristicsofapplicationsandlinkutilizationswithindata centers.Wefindthatdatacentertrafficisstatisticallydifferentfrom wideareatraffic,andthatsuchbehaviorhasseriousimplications forthedesignandimplementationoftechniquesfordatacenter networks. 3. DATASETSANDOVERVIEWOFDATA CENTERS Inthispaper,weanalyzedata-setsfrom10datacenters,including5commercialclouddatacenters,2privateenterprisedatacenters,and3universitycampusdatacenters.Foreachofthesedatacenters,weexamineoneormoreofthefollowingdata-sets:networktopology,packettracesfromselectswitches,andSNMPpolls fromtheinterfacesofnetworkswitches. Table2summarizesthe datacollectedfromeachdatacenter,aswellassomekeyproperties. Table2showsthatthedatacentersvaryinsize,bothintermsof thenumberofdevicesandthenumberofservers.Unsurprisingly, thelargestdatacentersareusedforcommercialcomputingneeds (allownedbyasingleentity),withtheenterpriseanduniversity datacentersbeinganorderofmagnitudesmallerintermsofthe numberofdevices. Thedatacentersalsovaryintheirproximitytotheirusers.The enterpriseanduniversitydatacentersarelocatedinthewestern/mid- 269 DataCenterName NumberofLocations EDU1 1 EDU2 1 EDU3 1 PRV2 4 Table3:Thenumberofpackettracecollectionlocationsforthe datacentersinwhichwewereabletoinstallpacketsniffers. westernU.S.andarehostedonthepremisesoftheorganizations toservelocalusers. Incontrast,thecommercialdatacentersare distributedaroundtheworldintheU.S.,Europe,andSouthAmerica. Theirglobalplacementreflectsaninherentrequirementfor geo-diversity(reducinglatencytousers),geo-redundancy(avoidingstrikes,wars,orfibercutsinonepartoftheworld),andregulatoryconstraints(somedatacannotberemovedfromtheE.U.or U.S.). Inwhatfollows,wefirstdescribethedatawecollect.Wethen outlinesimilaritiesanddifferencesinkeyattributesofthedatacenters,includingtheirusageprofiles,andphysicaltopology. We foundthatunderstandingtheseaspectsisrequiredtoanalyzethe propertiesthatwewishtomeasureinsubsequentsections,suchas applicationbehavioranditsimpactonlink-levelandnetwork-wide utilizations. 3.1 DataCollection SNMPpolls: Forallofthedatacentersthatwestudied,we wereabletopolltheswitches’SNMPMIBsforbytes-inandbytesoutatgranularitiesrangingfrom1minuteto30minutes.Forthe 5commercialclouddatacentersandthe2privateenterprises,we wereabletopollforthenumberofpacketdiscardsaswell. Foreachdatacenter,wecollectedSNMPdataforatleast10 days. Insomecases(e.g.,EDU1,EDU2,EDU3,PRV1,PRV2, CLD1,CLD4),ourSNMPdataspansmultipleweeks. Thelong time-spanofourSNMPdataallowsustoobservetime-of-dayand day-of-weekdependenciesinnetworktraffic. NetworkTopology: Fortheprivateenterprisesanduniversity datacenters,weobtainedtopologyviatheCiscoCDPprotocol, whichgivesboththenetworktopologyaswellasthelinkcapacities.Whenthisdataisunavailable,aswiththe5clouddatacenters,weanalyzedeviceconfigurationtoderivepropertiesofthetopology,suchastherelativecapacitiesoflinksfacingendhostsversus network-internallinksversusWAN-facinglinks. Packettraces: Finally,wecollectedpackettracesfromafew oftheprivateenterpriseanduniversitydatacenters(Table2).Our packettracecollectionspans12hoursovermultipledays.Sinceit isdifficulttoinstrumentanentiredatacenter,weselectedahandfuloflocationsatrandomperdatacenterandinstalledsnifferson them.InTable3,wepresentthenumberofsniffersperdatacenter. Inthesmallerdatacenters(EDU1,EDU2,EDU3),weinstalled 1sniffer. Forthelargerdatacenter(PRV2),weinstalled4sniffers.AlltraceswerecapturedusingaCiscoportspan.Toaccount fordelayintroducedbythepacketduplicationmechanismandfor endhostclockskew,webinnedresultsfromthespansinto10microsecondbins. 3.2 High-levelUsageoftheDataCenters Inthissection,weoutlineimportanthigh-levelsimilaritiesand differencesamongthedatacenterswestudied. Universitydatacenters:Thesedatacentersservethestudents andadministrativestaffoftheuniversityinquestion. Theyprovideavarietyofservices,rangingfromsystemback-upstohosting distributedfilesystems,E-mailservers,Webservices(administra-
Page 1: Network Traffic Characteristics of
Page 5 and 6: Percent of Bytes Per App 100 80 60
Page 7 and 8: CDF (a) CDF (b) 1 0.8 0.6 0.4 EDU1
Page 9 and 10: Percent of Traffic 0 20 40 60 80 10
Page 11 and 12: CDF (a) CDF CDF (b) (c) 1 0.998 0.9
Page 13 and 14: fromtheexistingcapacity,ratherthana

Network Traffic Characteristics of Data Centers in the Wild - Sigcomm

Create successful ePaper yourself

Delete template?

Save as template?