Network Traffic Characteristics of Data Centers in the Wild - Sigcomm

More documents

Recommendations

Info

CDF CDF CDF (a) (b) (c) 1 0.8 0.6 0.4 0.2 0 0.01 0.1 1 10 100 1 0.8 0.6 0.4 0.2 Max-Trough for Edge Link EDU1 EDU3 PRV1 PRV2 CLD1 CLD2 CLD3 CLD4 CLD5 0 0.01 0.1 1 10 100 1 0.8 0.6 0.4 0.2 Max-Trough for Agg Link PRV2 CLD1 CLD2 CLD3 CLD4 CLD5 0 0.01 0.1 1 10 100 Max-Trough for Core Link EDU1 EDU3 PRV1 PRV2 CLD1 CLD2 CLD3 CLD4 CLD5 Figure15:Differencebetweenthepeakandtroughutilization. 7. IMPLICATIONSFORDATACENTER DESIGN 7.1 RoleofBisectionBandwidth Severalproposals[1,22,11,2]fornewdatacenternetworkarchitecturesattempttomaximizethenetworkbisectionbandwidth. Theseapproaches,whilewellsuitedfordatacenters,whichrun applicationsthatstressthenetwork’sfabricwithall-to-alltraffic, wouldbeunwarrantedindatacenterswherethebisectionbandwidthisnottaxedbytheapplications.Inthissection,were-evaluate theSNMPandtopologydatacapturedfromthe10datacentersand examinewhethertheprevalenttrafficpatternsarelikelytostressthe existingbisectionbandwidth. Wealsoexaminehowmuchofthe existingbisectionbandwidthisneededatanygiventimetosupport theprevalenttrafficpatterns. Beforeexplaininghowweaddressthesequestions,weprovide afewdefinitions. Wedefinethebisectionlinksforatiereddata centertobethesetoflinksatthetop-mosttierofthedatacenter’s treearchitecture;inotherwords,thecorelinksmakeupthebisectionlinks.Thebisectioncapacityistheaggregatecapacityoftheselinks.Thefullbisectioncapacityisthecapacitythatwouldberequiredtosupportserverscommunicatingatfulllinkspeedswith arbitrarytrafficmatricesandnooversubscription. Thefullbisec- 278 Precent of Bisection Utilized 0 10 20 30 CLD1 CLD2 CLD3 CLD4 CLD5 EDU1 Data Center EDU2 EDU3 Current Full Figure16:Thefirstbaristheratioofaggregateservertraffic overBisectionBWandthesecondbaristheratioofaggregate servertrafficoverfullbisectioncapacity. They-axisdisplays utilizationasapercentage. tioncapacitycanbecomputedassimplytheaggregatecapacityof theserverNICs. Returningtothequestionsposedearlierinthissection,weuse SNMPdatatocomputethefollowing:(1)theratioofthecurrent aggregateserver-generatedtraffictothecurrentbisectioncapacity and(2)theratioofthecurrenttraffictothefullbisectioncapacity. Indoingso,wemaketheassumptionthatthebisectionlinkscan betreatedasasinglepoolofcapacityfromwhichallofferedtraffic candraw. Whilethismaynotbetrueinallcurrentnetworks,it allowsustodeterminewhethermorecapacityisneededorrather betteruseofexistingcapacityisneeded(forexample,byimproving routing,topology,orthemigrationofapplicationserversinsidethe datacenter). InFigure16,wepresentthesetworatiosforeachofthedata centersstudied.Recall(fromTable2)thatalldatacentersareoversubscribed,meaningthatifallserverssentdataasfastastheycan andalltrafficlefttheracks,thenthebisectionlinkswouldbefully congested(wewouldexpecttofindutilizationratiosover100%). However,wefindinFigure16thattheprevalenttrafficpatternsare suchthat,evenintheworstcasewhereallserver-generatedtraffic isassumedtoleavetherackhostingtheserver,theaggregateoutput fromserversissmallerthanthenetwork’scurrentbisectioncapacity. Thismeanseveniftheapplicationsweremovedaroundand thetrafficmatrixchanged,thecurrentbisectionwouldstillbemore thansufficientandnomorethan25%ofitwouldbeutilizedacross alldatacenters,includingtheMapReducedatacenters.Finally,we notethattheaggregateoutputfromserversisanegligiblefraction oftheidealbisectioncapacityinallcases.Thisimpliesthatshould thesedatacentersbeequippedwithanetworkthatprovidesfullbisectionbandwidth,atleast95%ofthiscapacitywouldgounused andbewastedbytoday’strafficpatterns. Thus,theprevalenttrafficpatternsinthedatacenterscanbesupportedbytheexistingbisectioncapacity,evenifapplicationswere placedinsuchawaythattherewasmoreinter-racktrafficthan existstoday.Thisanalysisassumesthattheaggregatecapacityof thebisectionlinksformsasharedresourcepoolfromwhichall offeredtrafficcandraw.Ifthetopologypreventssomeofferedtrafficfromreachingsomelinks,thensomelinkscanexperiencehigh utilizationwhileothersseelowutilization.Eveninthissituation, however,theissueisoneofchangingthetopologyandselecting aroutingalgorithmthatallowsofferedtraffictodraweffectively PRV1 PRV2
fromtheexistingcapacity,ratherthanaquestionofaddingmore capacity. Centralizedrouting,discussednext,couldhelpinconstructingtherequisitenetworkpaths. 7.2 CentralizedControllersinDataCenters Thearchitecturesforseveralproposals[1,22,12,2,14,21,4, 18,29]relyinsomeformoranotheronacentralizedcontroller forconfiguringroutesorfordisseminatingroutinginformationto endhosts. Acentralizedcontrollerisonlypracticalifitisableto scaleuptomeetthedemandsofthetrafficcharacteristicswithinthe datacenters.Inthissection,weexaminethisissueinthecontextof theflowpropertiesthatweanalyzedinSection5. Inparticular,wefocusontheproposals(Hedera[2],MicroTE[4] andElasticTree[14])thatrelyonOpenFlowandNOX[15,23].In anOpenFlowarchitecture,thefirstpacketofaflow,whenencounteredataswitch,canbeforwardedtoacentralcontrollerthatdeterminestheroutethatthepacketshouldfollowinordertomeetsome network-wideobjective.Alternatively,toeliminatethesetupdelay, thecentralcontrollercanprecomputeasetofnetworkpathsthat meetnetwork-wideobjectivesandinstallthemintothenetworkat startuptime. OurempiricalobservationsinSection5,haveimportantimplicationsforsuchcentralizedapproaches.First,thefactthatthenumberofactiveflowsissmall(seeFigure4(a))impliesthatswitchesenabledwithOpenFlowcanmakedowithasmallflowtable,which isaconstrainedresourceonswitchestoday. Second,flowinter-arrivaltimeshaveimportantimplicationsfor thescalabilityofthecontroller.AsweobservedinSection5,asignificantnumberofnewflows(2–20%)canarriveatagivenswitchwithin10µsofeachother.Theswitchmustforwardthefirstpacketsoftheseflowstothecontrollerforprocessing.Evenifthedatacenterhasasfewasa100edgeswitches,intheworstcase,acontrollercansee10newflowsperµsor10millionflowspersecond.Dependingonthecomplexityoftheobjectiveimplementedat thecontroller,computingarouteforeachoftheseflowscouldbe expensive.Forexample,priorwork[5]showedacommoditymachinecomputingasimpleshortestpathforonly50Kflowarrivals persecond. Thus,toscalethethroughputofacentralizedcontrolframeworkwhilesupportingcomplexroutingobjectives,we mustemployparallelism(i.e.,usemultipleCPUspercontrollerand multiplecontrollers)and/orusefasterbutlessoptimalheuristicsto computeroutes. Priorwork[28]hasshown,throughparallelism, theabilityofacentralcontrollertoscaleto20millionflowsper second. Finally,theflowdurationandsizealsohaveimplicationsforthe centralizedcontroller.Thelengthsofflowsdeterminetherelative impactofthelatencyimposedbyacontrolleronanewflow.Recall thatwefoundthatmostflowslastlessthan100ms.Priorwork[5] showedthanittakesreactivecontrollers,whichmakedecisionsat flowstartuptime,approximately10mstoinstallflowentriesfor newflows.Givenourresults,thisimposesa10%delayoverhead onmostflows. Additionalprocessingdelaymaybeacceptable forsometraffic,butmightbeunacceptableforotherkinds. For theclassofworkloadsthatfindsuchadelayunacceptable,Open- Flowprovidesaproactivemechanismthatallowsthecontrollers, atswitchstartuptime,toinstallflowentriesintheswitches.This proactivemechanismeliminatesthe10msdelaybutlimitsthecontrollertoproactivealgorithms. Insummary,itappearsthenumberandinter-arrivaltimeofdata centerflowscanbehandledbyasufficientlyparallelizedimplementationofthecentralizedcontroller.However,theoverheadof reactivelycomputingflowplacementsisareasonablefractionof thelengthofthetypicalflow. 279 8. SUMMARY Inthispaper,weconductedanempiricalstudyofthenetwork trafficof10datacentersspanningthreeverydifferentcategories, namelyuniversitycampus,privateenterprisedatacenters,andcloud datacentersrunningWebservices,customer-facingapplications, andintensiveMap-Reducejobs.Tothebestofourknowledge,this isthebroadest-everlarge-scalemeasurementstudyofdatacenters. Westartedourstudybyexaminingtheapplicationsrunwithin thevariousdatacenters. Wefoundthatavarietyofapplications aredeployedandthattheyareplacednon-uniformlyacrossracks. Next,westudiedthetransmissionpropertiesoftheapplicationsin termsoftheflowandpacketarrivalprocessesattheedgeswitches. Wediscoveredthatthearrivalprocessattheedgeswitchesis ON/OFFinnaturewheretheON/OFFdurationscanbecharacterizedbyheavy-taileddistributions.Inanalyzingtheflowsthatconstitutethesearrivalprocess,weobservedthatflowswithinthedata centersstudiedaregenerallysmallinsizeandseveraloftheseflows lastonlyafewmilliseconds. Westudiedtheimplicationsofthedeployeddatacenterapplicationsandtheirtransmissionpropertiesonthedatacenternetwork anditslinks.Wefoundthatmostoftheservergeneratedtrafficin theclouddatacentersstayswithinarack,whiletheoppositeistrue forcampusdatacenters. Wefoundthatattheedgeandaggregationlayers,linkutilizationsarefairlylowandshowlittlevariation. Incontrast,linkutilizationsatthecorearehighwithsignificant variationsoverthecourseofaday. Insomedatacenters,asmall butsignificantfractionofcorelinksappeartobepersistentlycongested,butthereisenoughsparecapacityinthecoretoalleviate congestion. Weobservedlossesonthelinksthatarelightlyutilizedonaverageandarguedthattheselossescanbeattributedto theburstynatureoftheunderlyingapplicationsrunwithinthedata centers. Onthewhole,ourempiricalobservationscanhelpinformdata centertrafficengineeringandQoSapproaches,aswellasrecent techniquesformanagingotherresources,suchasdatacenternetworkenergyconsumption.Tofurtherhighlighttheimplicationsofourstudy,were-examinedrecentdatacenterproposalsandarchitecturesinlightofourresults.Inparticular,wedeterminedthatfullbisectionbandwidthisnotessentialforsupportingcurrentapplications.Wealsohighlightedpracticalissuesinsuccessfullyemployingcentralizedroutingmechanismsindatacenters.Ourempiricalstudyisbynomeansall-encompassing.Werecognizethattheremaybeotherdatacentersinthewildthatmayor maynotshareallthepropertiesthatwehaveobserved.Ourwork pointsoutthatitisworthcloselyexaminingthedifferentdesignand usagepatterns,asthereareimportantdifferencesandcommonalities. 9. ACKNOWLEDGMENTS Wewouldliketothanktheoperatorsatthevariousuniversities, onlineservicesproviders,andprivateenterprisesforboththetime anddatathattheyprovidedus. Wewouldalsoliketothankthe anonymousreviewersfortheirinsightfulfeedback. ThisworkissupportedinpartbyanNSFFINDgrant(CNS- 0626889),anNSFCAREERAward(CNS-0746531),anNSFNetSE grant(CNS-0905134),andbygrantsfromtheUniversityof Wisconsin-MadisonGraduateSchool.TheophilusBensonissupportedbyanIBMPhDFellowship. 10. REFERENCES [1] M.Al-Fares,A.Loukissas,andA.Vahdat.Ascalable, commoditydatacenternetworkarchitecture.InSIGCOMM, pages63–74,2008.
Page 1 and 2: Network Traffic Characteristics of
Page 3 and 4: Thispaperanalyzesthenetworktraffico
Page 5 and 6: Percent of Bytes Per App 100 80 60
Page 7 and 8: CDF (a) CDF (b) 1 0.8 0.6 0.4 EDU1
Page 9 and 10: Percent of Traffic 0 20 40 60 80 10
Page 11: CDF (a) CDF CDF (b) (c) 1 0.998 0.9

Network Traffic Characteristics of Data Centers in the Wild - Sigcomm

Create successful ePaper yourself

Delete template?

Save as template?