12.07.2015 Views

Benchmarking in the Cloud - Parallel and Distributed Systems

Benchmarking in the Cloud - Parallel and Distributed Systems

Benchmarking in the Cloud - Parallel and Distributed Systems

SHOW MORE
SHOW LESS

Create successful ePaper yourself

Turn your PDF publications into a flip-book with our unique Google optimized e-Paper software.

Someth<strong>in</strong>g as a Service (XaaS) term<strong>in</strong>ologies. Due to space limitations, here weonly mention Youseff et al. [12], who extend <strong>the</strong> three layer model with Hardwareas a Service, Data as a Service, <strong>and</strong> Communication as a Service concepts.Under <strong>Benchmark<strong>in</strong>g</strong> <strong>in</strong> <strong>the</strong> <strong>Cloud</strong>, we underst<strong>and</strong> <strong>the</strong> process of benchmark<strong>in</strong>gservices provided by <strong>the</strong> <strong>Cloud</strong>. A <strong>Cloud</strong> Benchmark <strong>the</strong>refore for us is abenchmark <strong>in</strong> which <strong>the</strong> SUT conta<strong>in</strong>s a <strong>Cloud</strong> service as component of <strong>in</strong>terest.A good summary of this topic was provided <strong>in</strong> [13].3.1 <strong>Cloud</strong> ActorsTo <strong>in</strong>troduce <strong>the</strong> different <strong>Cloud</strong> actors, we take a bus<strong>in</strong>ess orientated view of<strong>Cloud</strong> Comput<strong>in</strong>g go<strong>in</strong>g beyond <strong>the</strong> simple consumer/provider model. We claimthat each service layer br<strong>in</strong>gs its own actors who add value to <strong>the</strong> level below.Different types of actors might have different benchmark requirements for <strong>the</strong>same SUT. To have a strong target audience, a benchmark has to address <strong>the</strong>appropriate actors.Leimeister et al. [14] argue that <strong>the</strong> actors <strong>in</strong> <strong>the</strong> <strong>Cloud</strong> form a bus<strong>in</strong>essvalue network ra<strong>the</strong>r than a traditional bus<strong>in</strong>ess value cha<strong>in</strong>. We identify <strong>the</strong>follow<strong>in</strong>g actors <strong>in</strong> a <strong>Cloud</strong>-centric bus<strong>in</strong>ess value network (Figure 1): IT Vendorsdevelop <strong>in</strong>frastructure software <strong>and</strong> operate <strong>in</strong>frastructure services; ServiceProviders develop <strong>and</strong> operate services; Service Aggregators offer new services bycomb<strong>in</strong><strong>in</strong>g preexist<strong>in</strong>g services; Service Platform Providers offer an environmentfor develop<strong>in</strong>g <strong>Cloud</strong> applications; Consult<strong>in</strong>g supports customers with select<strong>in</strong>g<strong>and</strong> implement<strong>in</strong>g <strong>Cloud</strong> services; Customers are <strong>the</strong> end-users of <strong>Cloud</strong> services.Note that [14] uses <strong>the</strong> term Infrastructure Provider for what we call ITVendor. We deviate from [14] to stress <strong>the</strong> fact that vendors that offer softwarethat enables <strong>Cloud</strong> services should also be considered part of this actor group.We also use <strong>the</strong> term Customer where o<strong>the</strong>rs might use <strong>the</strong> term Consumer. Wedecided to adhere to [14] <strong>in</strong> this case because service aggregators <strong>and</strong> serviceplatform providers are consumers just as customers are.3.2 The System Under TestThe above def<strong>in</strong>ition of a cloud benchmark requires that at least one componentof <strong>in</strong>terest with<strong>in</strong> <strong>the</strong> benchmark SUT is a cloud service. This leads to severalimplications, which we now briefly discuss.SUT Disclosure. A common benchmark requirement is to lay open all propertiesof <strong>the</strong> <strong>in</strong>volved SUT components. This requirement is no longer realisticwhen <strong>the</strong> SUT conta<strong>in</strong>s public cloud services. We <strong>the</strong>refore propose to consider<strong>the</strong> follow<strong>in</strong>g relaxation: All properties of SUT components visible to <strong>the</strong>ir clientsshould be laid open. In terms of white-box vs. black-box benchmark<strong>in</strong>g this leadsto nearly white-box IaaS benchmarks, grey-box PaaS benchmarks <strong>and</strong> blackboxSaaS benchmarks. Overcom<strong>in</strong>g <strong>the</strong> SUT disclosure issue was also recentlydiscussed by <strong>the</strong> SPEC OSG <strong>Cloud</strong> Comput<strong>in</strong>g Workgroup. Their White Paper[15] provides fur<strong>the</strong>r <strong>in</strong>formation.


Fig. 1. <strong>Cloud</strong> Actors <strong>and</strong> <strong>the</strong>ir Value NetworkSUT Uniqueness. Traditional benchmarks run on SUT components copied<strong>in</strong>to an isolated test environment. In a cloud environment, components are modeledas services. These are s<strong>in</strong>gle <strong>in</strong>stance entities, cannot be copied, <strong>and</strong> a clearisolation is not possible. Dur<strong>in</strong>g a benchmark run <strong>the</strong> SUT services most likelywill be shared with third party clients, <strong>and</strong> avoid<strong>in</strong>g external clients is nei<strong>the</strong>rpossible nor desired.Carv<strong>in</strong>g out a SUT. One possible approach to deal with <strong>the</strong> disclosure <strong>and</strong>uniqueness issues discussed above is to carve out a dedicated set of servers with<strong>in</strong>a public cloud <strong>and</strong> have <strong>the</strong>se deliver <strong>the</strong> services for <strong>the</strong> SUT. We believe thatthis will not lead to relevant results.4 Use CasesIn this section we present a list of sample use cases. The list does not <strong>in</strong>tend to becomplete, but ra<strong>the</strong>r should help us illustrate <strong>the</strong> layers <strong>and</strong> actors def<strong>in</strong>ed above<strong>and</strong> motivate <strong>the</strong> discussion of different optimization questions. In Section 5we show how appropriate <strong>Cloud</strong> benchmarks help answer <strong>the</strong>se questions, <strong>and</strong><strong>in</strong> Section 6 we identify <strong>and</strong> discuss <strong>the</strong> most important challenges that <strong>the</strong>sebenchmarks should resolve.4.1 Simple IaaSBus<strong>in</strong>esses use <strong>Cloud</strong> IaaS for outsourc<strong>in</strong>g of non-critical processes like test<strong>in</strong>g,tra<strong>in</strong><strong>in</strong>g <strong>and</strong> reference l<strong>and</strong>scapes. They buy IaaS resources <strong>and</strong> <strong>in</strong>stall <strong>the</strong> desiredsystems, but do not use <strong>the</strong>m all <strong>the</strong> time. The expected load is moderateto low, be<strong>in</strong>g created ma<strong>in</strong>ly by non-critical offl<strong>in</strong>e processes. Our experiencewith such scenarios shows systems with 100GB disk space, 32GB memory, <strong>and</strong> 4


CPU cores. The average load of a runn<strong>in</strong>g system uses about 10% CPU resources<strong>and</strong> 25% of memory or disk space.The IaaS resources are not expected to scale with regard to <strong>the</strong> load persystem. Scalability is expected with regards to <strong>the</strong> number of systems that canbe started <strong>and</strong> shut down.4.2 Onl<strong>in</strong>e Gam<strong>in</strong>g PlatformGameX is produced by ACME Games.ACME Games runs its own server clusters which should host on averagehundreds of game <strong>and</strong> m<strong>in</strong>i-game sessions, each with vary<strong>in</strong>g number of users(say between 0 <strong>and</strong> 5,000 for <strong>the</strong> core game, <strong>and</strong> from 5 to 50 for <strong>the</strong> m<strong>in</strong>i- orsmaller game <strong>in</strong>stances [16]), <strong>and</strong> length rang<strong>in</strong>g of 20-30 m<strong>in</strong>s to several hours.When players overload <strong>the</strong> servers, no new player sessions can be established.Wait<strong>in</strong>g to log<strong>in</strong> is one of <strong>the</strong> worst breaches of <strong>the</strong> expected Service LevelAgreements for GameX, so ACME Games has to avoid <strong>the</strong>se situations at allcost. Because <strong>the</strong> resource consumption of <strong>in</strong>teraction operations between playersdoes not always scale l<strong>in</strong>early with <strong>the</strong> number of players, <strong>and</strong>, <strong>in</strong> fact, <strong>in</strong> somecases may grow cubically <strong>in</strong> <strong>the</strong> number of players, ACME Games has to beei<strong>the</strong>r overly-conservative <strong>in</strong> its server capacity estimates or risk severe overloads<strong>in</strong> some circumstances [17].Instead of over-provision<strong>in</strong>g, which as seen is highly undesirable, ACMEGames can use <strong>Cloud</strong> technology. They can lease mach<strong>in</strong>es on-dem<strong>and</strong> to alleviatesudden surges <strong>in</strong> <strong>the</strong>ir workload. This solution needs to fulfill strict ServiceLevel Agreements <strong>in</strong> terms of response time, which <strong>in</strong>cludes a non-trivial networklatency component, <strong>and</strong> comput<strong>in</strong>g workload distribution <strong>and</strong> <strong>in</strong>terference. Forits new games, ACME Games may also lease (reserve) <strong>in</strong>stances from a <strong>Cloud</strong>until <strong>the</strong> game is properly measured <strong>and</strong> future workloads can be predicted.4.3 High Workload Analytical PlatformACME Analytics is a start-up that wants to provide Big Data analytics services.Their target customers produce large amounts of data <strong>and</strong> want to analyze iton a daily or weekly basis. Potential customers may be smart grid or mobilephone providers. The company expects up to 500 TB data per client <strong>and</strong> asmuch as 1 PB of data shared among clients. An average task is expected to use50TB of <strong>the</strong> private <strong>and</strong> 1 TB of <strong>the</strong> shared data. Clearly, us<strong>in</strong>g a massivelyparalleldata process<strong>in</strong>g platform, such as MapReduce, as an off-<strong>the</strong>-shelf <strong>Cloud</strong>service is <strong>the</strong> most lucrative technical solution for ACME Analytics because of<strong>the</strong> low upfront <strong>in</strong>vestment <strong>and</strong> ma<strong>in</strong>tenance cost. The first technical decisionthat ACME Analytics has to make <strong>the</strong>refore is which PaaS to use.5 Build<strong>in</strong>g Benchmarks <strong>in</strong> <strong>the</strong> <strong>Cloud</strong>In this section we go through <strong>the</strong> previous use cases <strong>and</strong> try to envision howbenchmarks <strong>in</strong> <strong>the</strong> respective area could be build.


5.1 Simple IaaSFirst, we need to ref<strong>in</strong>e <strong>the</strong> optimization question for <strong>the</strong> use case. The consumerneeds many parallel <strong>in</strong>stances of some application for <strong>the</strong>ir test or tra<strong>in</strong><strong>in</strong>g systems.There is no necessity to scale up a s<strong>in</strong>gle application <strong>in</strong>stance. It is expectedto save cost by shutt<strong>in</strong>g down application <strong>in</strong>stances. We can <strong>the</strong>refore recap <strong>the</strong>optimization question posed by <strong>the</strong> use case as follows: Which IaaS does mosteffectively host a bunch of parallel mid-size application <strong>in</strong>stances? This <strong>in</strong>cludesreduc<strong>in</strong>g cost when fewer application <strong>in</strong>stances are active.Next, we discuss how <strong>the</strong> SUT <strong>and</strong> <strong>the</strong> workload should look like. We arenot <strong>in</strong>terested <strong>in</strong> a distributed or cluster scenario, so we can adapt a well knownworkload, say TPC-C, <strong>and</strong> run it <strong>in</strong>dependently on all <strong>in</strong>stances.To adapt TPC-C, we first pick a tpmC value representative for a mid-sizeload. In addition, we also set <strong>the</strong> maximum number of <strong>in</strong>dependent application<strong>in</strong>stances that will participate for a s<strong>in</strong>gle benchmark run. Let us putmaxInst=250.We will consider two workload variants: (1) runn<strong>in</strong>g all <strong>in</strong>stances with <strong>the</strong>same tpmC, <strong>and</strong> (2) runn<strong>in</strong>g a vary<strong>in</strong>g amount of <strong>in</strong>stances over <strong>the</strong> benchmarkruntime. Workload (1) is <strong>in</strong>tended to measure how many IaaS nodes are requiredto make maxInst <strong>in</strong>stances pass <strong>the</strong> TPC-C SLA requirements. Because differentproviders are expected to have different nodes, a f<strong>in</strong>al comparison of <strong>the</strong>respective services will only be possible by compar<strong>in</strong>g <strong>the</strong> price of <strong>the</strong> servicesconsumed. Workload (2) measures <strong>the</strong> elasticity of <strong>the</strong> underly<strong>in</strong>g IaaS by compar<strong>in</strong>g<strong>the</strong> price of <strong>the</strong> full workload with <strong>the</strong> price of <strong>the</strong> vary<strong>in</strong>g workload. For<strong>the</strong> second workload, we propose a schedul<strong>in</strong>g mechanism with number of active<strong>in</strong>stances def<strong>in</strong>ed by <strong>the</strong> formula:actInstances(timeElapsed) =∣maxInst2(× 1 − cos( ))∣ 3π × timeElapsed ∣∣∣totalRuntimeThese are only a few ideas about a parallel load IaaS benchmark. We listfur<strong>the</strong>r design questions for such a benchmark.1. As we are not benchmark<strong>in</strong>g <strong>the</strong> database we can fix <strong>the</strong> database systemto be used. But this would violate <strong>the</strong> fairness <strong>and</strong> portability requirement.2. Is it allowed to use different schemata of a database <strong>in</strong>stance for differentTPC-C <strong>in</strong>stances?3. Is it allowed to migrate TPC-C <strong>in</strong>stances between server nodes dur<strong>in</strong>g abenchmark run?4. Should we ra<strong>the</strong>r not design for possible multiple TPC-C per node <strong>and</strong> scaleby <strong>in</strong>creas<strong>in</strong>g <strong>the</strong> tpmC until a node is fully loaded?5. Where should <strong>the</strong> Remote Term<strong>in</strong>al Emulator (<strong>the</strong> driver) be located?We discuss <strong>the</strong>se questions <strong>in</strong> Section 6.5.2 Onl<strong>in</strong>e Gam<strong>in</strong>g PlatformThere currently exists no onl<strong>in</strong>e gam<strong>in</strong>g benchmark. Relevant prior work on <strong>the</strong>prerequisites of design<strong>in</strong>g onl<strong>in</strong>e gam<strong>in</strong>g benchmarks exists, ei<strong>the</strong>r <strong>in</strong> <strong>the</strong> form


of game workload characterizations or of benchmarks built for o<strong>the</strong>r types ofmedia. The volum<strong>in</strong>ous body of work on game workload characterization hasbeen recently surveyed [18]. For related benchmarks, we refer to ALPBench [19],MediaBench [20], <strong>and</strong> MiBench [21].We hold that <strong>the</strong> Gam<strong>in</strong>g use case ra<strong>the</strong>r gives rise to parallel than a distributedscenario. Still, it is much more complex than <strong>the</strong> Simple IaaS use case.In this case <strong>the</strong> various types of system operations necessary to fulfill variousrequests may lead to widely different response times. Ano<strong>the</strong>r fundamental differenceis that requests need to be fulfilled <strong>in</strong> different amount of times before<strong>the</strong> users consider <strong>the</strong>ir mental Service Level Agreement breached <strong>and</strong> decideto move to ano<strong>the</strong>r game. For example, role-play<strong>in</strong>g requests may be fulfilledwith<strong>in</strong> 1 second [22, 23]. For strategic decisions <strong>the</strong> response time needs to bearound 300 milliseconds [24], <strong>and</strong> requests for first-person activities need to befulfilled <strong>in</strong> under 100 milliseconds [23], [25].O<strong>the</strong>r important metrics for ACME Games are <strong>the</strong> 99th percentile of <strong>the</strong>wait time distribution <strong>and</strong> <strong>the</strong> <strong>the</strong> 99th percentile of <strong>the</strong> distribution of fractionof dropped requests. These high limits (vs <strong>the</strong> traditional 95th percentile) stemfrom <strong>the</strong> way players jo<strong>in</strong> <strong>and</strong> leave games as <strong>the</strong> result of positive <strong>and</strong> negativetrends, respectively. Strong community ties between players [26], [27] <strong>in</strong>dicatethat a percentage as low as 1% of unhappy players may trigger <strong>the</strong> departureof 10-20% <strong>in</strong> a matter of days [17], e.g. via social media rants, <strong>in</strong>-game negativeadverts, <strong>and</strong> pla<strong>in</strong> group-based discussions.5.3 High Workload Analytical PlatformIn this case, ACME Consult<strong>in</strong>g acts as service aggregator <strong>and</strong> has two options.They may ei<strong>the</strong>r bundle services of an exist<strong>in</strong>g analytical platform or deploy<strong>the</strong>ir own analytical tools on an exist<strong>in</strong>g <strong>in</strong>frastructure service. To compare <strong>the</strong>result<strong>in</strong>g service <strong>the</strong>y need a benchmark build around a set of representativeanalytical tasks. Most research <strong>in</strong> <strong>the</strong> area <strong>in</strong> done on actual MapReduce benchmarkslike MRBench [28] or design<strong>in</strong>g appropriate MapReduce workloads [29].Pavlo et al. [30] show how to have analytical workload run by both MapReduce<strong>and</strong> <strong>Distributed</strong> Databases <strong>and</strong> compare <strong>the</strong> results. These approaches makeconsiderable progress def<strong>in</strong><strong>in</strong>g representative workloads. They do not deal withservices <strong>and</strong> do not <strong>in</strong>tend to def<strong>in</strong>e <strong>Cloud</strong> benchmarks. Here are a few ideashow an analytical <strong>Cloud</strong> benchmark could look like:1. Start with a set of analytical tasks from <strong>the</strong> latest references.2. For each task f<strong>in</strong>d an appropriate runtime SLA def<strong>in</strong><strong>in</strong>g a 90% percentile.3. Run <strong>the</strong> tasks concurrently. Run <strong>the</strong> whole workload several times.4. Scale <strong>the</strong> tasks with <strong>the</strong> amount of data analyzed. Keep th<strong>in</strong>gs simple byscal<strong>in</strong>g up all task data sizes with <strong>the</strong> same l<strong>in</strong>ear factor anScale.5. We expect <strong>the</strong> SUT to acquire more server nodes as <strong>the</strong> amount of data tobe analyzed <strong>in</strong>creases. The primary metric is <strong>the</strong> maximal anScale, that canbe reached.


6. Devise a maximal scal<strong>in</strong>g factor maxAnScaleYYYY. The benchmark shouldnot scale <strong>the</strong> load fur<strong>the</strong>r than maxAnScaleYYYY.7. Also report <strong>the</strong> price of <strong>the</strong> services used so that services can be comparedif <strong>the</strong>y reach <strong>the</strong> maximal anScale.8. Periodically (for example once per year) revise <strong>and</strong> agree on a new (larger)maxAnScaleYYYY.9. Similarly to <strong>the</strong> Simple IaaS use case report elasticity by runn<strong>in</strong>g with avary<strong>in</strong>g load.In <strong>the</strong> next section we discusses <strong>the</strong> challenges for <strong>the</strong> hypo<strong>the</strong>tical benchmarkspresented above <strong>in</strong> more detail.6 The Art of Build<strong>in</strong>g <strong>and</strong> Runn<strong>in</strong>g a Good Benchmark<strong>in</strong> <strong>the</strong> <strong>Cloud</strong>We believe, that <strong>the</strong>re are four ma<strong>in</strong> steps for build<strong>in</strong>g <strong>and</strong> runn<strong>in</strong>g a goodbenchmark <strong>in</strong> <strong>the</strong> <strong>Cloud</strong>. These are Mean<strong>in</strong>gful Metric, Workload Design, WorkloadImplementation <strong>and</strong> Creat<strong>in</strong>g Trust. In this section we discuss <strong>the</strong> <strong>in</strong>herentchallenges <strong>in</strong> <strong>the</strong>se steps.6.1 Mean<strong>in</strong>gful MetricA metric is a function that transforms measured results <strong>in</strong>to a form that is easilyunderstood by <strong>the</strong> system analyst. The most simple metric is <strong>the</strong> runtime metric,which reports ei<strong>the</strong>r median, average, maximum or even m<strong>in</strong>imum runtimeamong transactions run by <strong>the</strong> SUT. When systems have to deal with concurrenttransactions, it makes more sense to consider a throughput metric report<strong>in</strong>g <strong>the</strong>maximal number of transactions adher<strong>in</strong>g to some runtime SLA. Some benchmarks(ma<strong>in</strong>ly those designed to support bus<strong>in</strong>ess decisions) also report <strong>the</strong> costof <strong>the</strong> underly<strong>in</strong>g system or <strong>the</strong> cost of runn<strong>in</strong>g a s<strong>in</strong>gle transaction at maximalload. There is an ongo<strong>in</strong>g debate if report<strong>in</strong>g cost makes sense [31]. Kossmannet al. have devoted a sequence of papers to this topic <strong>in</strong> its <strong>Cloud</strong> context [5, 2,32], argu<strong>in</strong>g that <strong>in</strong> this case cost should be <strong>the</strong> primary metric. The argumentis motivated by <strong>the</strong> fact that <strong>in</strong> <strong>the</strong>ory <strong>Cloud</strong> technology should provide <strong>in</strong>f<strong>in</strong>itescalability, which makes <strong>the</strong> classic throughput metric obsolete. However, <strong>in</strong> [32]<strong>the</strong> authors observe that (to no surprise) <strong>in</strong>f<strong>in</strong>ite scalability is an illusion <strong>and</strong>for most providers breaks down sooner or later due to bottlenecks <strong>in</strong> <strong>the</strong> SUTarchitecture. In that case it makes perfect sense to report how far <strong>the</strong> load canbe scaled up.Challenge 1: Price vs. Performance Metric. For <strong>the</strong> parallel load of <strong>the</strong>Simple IaaS use case we had decided to report <strong>the</strong> number of nodes required torun <strong>the</strong> 250 TPC-C <strong>in</strong>stances. As nodes of different vendors will most probablyhave different characteristics, <strong>the</strong> number of nodes is <strong>in</strong> fact not a suitable metric.Money is <strong>the</strong> only possible yardstick <strong>and</strong> <strong>the</strong>refore we have to report <strong>the</strong> price


for <strong>the</strong>se nodes. We propose to use <strong>the</strong> Price Metric as primary metric whendeal<strong>in</strong>g with parallel load: Report <strong>the</strong> m<strong>in</strong>imal cost to run maxInst of a givenapplication <strong>and</strong> a given workload on <strong>the</strong> SUT.For a distributed load we propose to use a mixed Price/Performance Metricas suggested by <strong>the</strong> High Analytical Workload Platform use case: Scale up <strong>the</strong>load along a well def<strong>in</strong>ed dimension, but do not pass a very ambitious limit ofmaxScaleYYYY which should be reconsidered annually. Also report <strong>the</strong> price of<strong>the</strong> SUT. This enables comparison <strong>in</strong> case systems reach maxScaleYYYY.Challenge 2: Elasticity Metric. We propose to use a Elasticity Metric assecondary metric. How can elasticity of a service under test be measured? Previouswork has <strong>in</strong>troduced for this purpose concepts such as over- <strong>and</strong> underprovision<strong>in</strong>gof resources [17, 33], or counted <strong>the</strong> number of SLA breaches [17]dur<strong>in</strong>g periods of elastic system activity. However, measur<strong>in</strong>g <strong>and</strong> characteriz<strong>in</strong>gelasticity rema<strong>in</strong> activities without <strong>the</strong>oretical support. Even underst<strong>and</strong><strong>in</strong>gwhich periods are elastic, that is, dist<strong>in</strong>guish<strong>in</strong>g between normal fluctuations<strong>and</strong> significant elastic changes <strong>in</strong> <strong>the</strong> system behavior, requires advances <strong>in</strong> <strong>the</strong>current body of knowledge. Moreover, <strong>the</strong> proposal <strong>in</strong> [17, 33] relies on <strong>the</strong> benchmarkbe<strong>in</strong>g able to collect CPU consumption <strong>in</strong>formation of <strong>the</strong> provider system.In some cases <strong>the</strong>se numbers might not be freely available to <strong>the</strong> consumer. Consequentlya consumer benchmark should not rely on <strong>the</strong>se numbers. B<strong>in</strong>nig etal. [2] def<strong>in</strong>e a Scalability Metric, which does not take <strong>in</strong>to account CPU <strong>in</strong>fo<strong>and</strong> solely relies on <strong>the</strong> successful <strong>in</strong>teractions under an <strong>in</strong>creas<strong>in</strong>g load. Thismethodology could be extended to also capture elasticity. Never<strong>the</strong>less, it is oflittle <strong>in</strong>terest to <strong>the</strong> consumer if <strong>the</strong> provider can deal effectively with a vary<strong>in</strong>gload. What <strong>the</strong> consumer needs to know is whe<strong>the</strong>r a varied <strong>and</strong> reduced loadleads to a reduced bill. We <strong>the</strong>refore propose to measure elasticity by runn<strong>in</strong>ga vary<strong>in</strong>g workload <strong>and</strong> compare <strong>the</strong> result<strong>in</strong>g price with <strong>the</strong> price for <strong>the</strong> fullload. Some details can be found <strong>in</strong> <strong>the</strong> discussion of <strong>the</strong> Simple IaaS benchmark.Challenge 3: Percentile. Which percentiles are appropriate for <strong>the</strong> responsetime SLAs of <strong>Cloud</strong> like <strong>in</strong>teractions? One might argue, that <strong>the</strong> <strong>Cloud</strong> calls forhigher percentiles. Disappo<strong>in</strong>ted users might just move to <strong>the</strong> next service offered.We hold that percentiles depend on <strong>the</strong> respective scenario. For <strong>the</strong> SimpleIaaS scenario <strong>the</strong> 90% percentiles of TPC-C are f<strong>in</strong>e. For <strong>the</strong> Gam<strong>in</strong>g scenario<strong>the</strong>y are not. We propose not to run <strong>the</strong> same benchmark (scenario) with differentpercentiles but to have different benchmarks model<strong>in</strong>g different scenarios<strong>and</strong> respective percentiles.Discuss<strong>in</strong>g <strong>the</strong> metric topic <strong>in</strong> <strong>the</strong> <strong>Cloud</strong> community we were asked to alsoconsider several o<strong>the</strong>r metrics like consistency, security, user experience <strong>and</strong>trust. In our op<strong>in</strong>ion each scenario has its own consistency requirement. Theconsistency level of a service has to fulfill this consistency requirement. If a servicepromises a higher degree of consistency than required, this should be stated


<strong>in</strong> <strong>the</strong> benchmark report. The same more or less holds for security. User experience<strong>and</strong> trust are hard to capture. We <strong>in</strong>tend to create trust by do<strong>in</strong>g <strong>the</strong>benchmark<strong>in</strong>g right.6.2 Workload DesignThe workload of a benchmark must be designed towards <strong>the</strong> metric to be used.It has to model real world workloads on a given application scenario. Workloaddesign is a traditional challenge <strong>in</strong> <strong>the</strong> design of benchmarks.Challenge 4: Limit <strong>the</strong> Resources Acquired. S<strong>in</strong>ce many <strong>Cloud</strong>s try toprovide <strong>the</strong> illusion of <strong>in</strong>f<strong>in</strong>ite scale, benchmarks cannot merely load <strong>the</strong> systemuntil it breaks down. This applies <strong>in</strong> particular for parallel load, where we expectto be able to crash <strong>the</strong> complete service. This is not realistic <strong>and</strong> might violate <strong>the</strong>benchmark requirement to stay with<strong>in</strong> economical boundaries. We have to devisea maximal scal<strong>in</strong>g factor limit<strong>in</strong>g <strong>the</strong> load. However, hav<strong>in</strong>g this accepted by <strong>the</strong>majority of stakeholders poses a serious challenge. We propose to annually renew<strong>the</strong> maximal scal<strong>in</strong>g factor. We expect a discussion about hav<strong>in</strong>g a maximalscal<strong>in</strong>g factor or not. Those aga<strong>in</strong>st could argue, that providers have to takecare not to have <strong>the</strong>ir services crashed. A benchmark would check implicitly, ifthis k<strong>in</strong>d of check is <strong>in</strong> place.Challenge 5: Variable Load. As user-triggered elasticity <strong>and</strong> automatic adaptivityare expected features of <strong>Cloud</strong> services, benchmarks can no longer focussolely on steady-state workloads. For <strong>the</strong> Simple IaaS use case we proposed to usea harmonic variability of <strong>the</strong> load. In general, captur<strong>in</strong>g <strong>and</strong> us<strong>in</strong>g characteristicsof real <strong>Cloud</strong> workloads might lead to better accepted benchmarks.Challenge 6: Scalability. The benchmark requirements listed <strong>in</strong> Section 2.3ask for Scalability. This means that <strong>the</strong> benchmark has to be enabled to have<strong>the</strong> load aga<strong>in</strong>st <strong>the</strong> SUT <strong>in</strong>creased along a well def<strong>in</strong>ed scale. What is <strong>the</strong> bestc<strong>and</strong>idate for a ’well def<strong>in</strong>ed scale’ for our use case? The answer to this questionis not obvious.Let us return to <strong>the</strong> Simple IaaS use case <strong>and</strong> recap <strong>the</strong> design of its benchmark.The general question is: how well can a service host multiple <strong>in</strong>stances ofa given application? To answer this question <strong>the</strong> benchmark <strong>in</strong>creases <strong>the</strong> load<strong>and</strong> reports how far it can get. We have several options to <strong>in</strong>crease <strong>the</strong> load:1. Increase <strong>the</strong> number of users per application <strong>in</strong>stance.2. Increase <strong>the</strong> size of <strong>the</strong> application <strong>in</strong>stance.3. Increase <strong>the</strong> number of application <strong>in</strong>stances.4. Increase <strong>the</strong> number of application <strong>in</strong>stance per node.As we do not model for high load per application <strong>in</strong>stance <strong>the</strong> first twooptions are not relevant. The third option leaves open <strong>the</strong> question where to set


<strong>the</strong> limit discussed <strong>in</strong> <strong>the</strong> previous challenge. We choose <strong>the</strong> last option because itaddresses resource shar<strong>in</strong>g. Once we deal with distributed load we face differentoptions. In <strong>the</strong> case of MapReduce scenarios we choose <strong>the</strong> size of <strong>the</strong> data tobe analysed as ’well def<strong>in</strong>ed scale’.6.3 Workload ImplementationChallenge 7: Workload Generation. The <strong>in</strong>creased complexity of <strong>the</strong> workloadsalso imposes challenges for <strong>the</strong> implementation of workload generator programs.First, efficient scalability is a hard requirement because of <strong>the</strong> expectedworkload sizes. Second, <strong>the</strong> workload generators should be able to implement allrelevant characteristics of <strong>the</strong> designed application scenario.As an example, consider <strong>the</strong> analytical platform use-case from Section 5.3. Arelevant benchmark should use test data with similar order of magnitude, so anatural problem that becomes evident here is how to ship benchmark data of thatmagnitude to <strong>the</strong> SUT. Clearly, import<strong>in</strong>g data from remote locations or us<strong>in</strong>gsequential data generators are not feasible options due to <strong>the</strong> unrealistic amountof required time (accord<strong>in</strong>g to our estimates generat<strong>in</strong>g 1PB of TPC-H data for<strong>in</strong>stance will take at least a month). The most promis<strong>in</strong>g alternative to solve thisproblem is to utilize <strong>the</strong> computational resources <strong>in</strong> <strong>the</strong> <strong>Cloud</strong> environment <strong>and</strong>generate <strong>the</strong> data <strong>and</strong> <strong>the</strong> workloads <strong>in</strong> a highly parallel manner. Recent work<strong>in</strong> <strong>the</strong> area of scalable data generation [34, 35] demonstrates how a special classof pseudo-r<strong>and</strong>om number generators (PRNGs) can be exploited for efficientgeneration of complex, update-aware data <strong>and</strong> workloads <strong>in</strong> a highly parallelmanner. In general, partitionable PRNGs can be used to ensure repeatability<strong>and</strong> protect aga<strong>in</strong>st unwanted correlations for all sorts of parallel simulations.Challenge 8: Fairness <strong>and</strong> Portability. Like B<strong>in</strong>nig et al. [2] we propose tohave <strong>Cloud</strong> benchmarks model <strong>the</strong> complete application stack. In case we do notbenchmark SaaS services, some party has to provide an implementation of <strong>the</strong>respective application to be deployed on <strong>the</strong> service under test. What k<strong>in</strong>d ofrules <strong>and</strong> restrictions should apply for <strong>the</strong> implementation of this application?We have to take care not to violate <strong>the</strong> Fairness <strong>and</strong> Portability requirements.In <strong>the</strong> Simple IaaS use case we discussed <strong>the</strong> option to fix a database system for<strong>the</strong> benchmark. In our op<strong>in</strong>ion, this would violate <strong>the</strong> Fairness <strong>and</strong> Portabilityrequirements. Allow<strong>in</strong>g different database systems <strong>in</strong>creases <strong>the</strong> degrees of freedom.This <strong>in</strong> turn makes it hard to compare <strong>the</strong> underly<strong>in</strong>g IaaS SUT. How canthis challenge be overcome? Here is our solution:1. Set application specs without touch<strong>in</strong>g implementation details.2. Service providers are free to provide <strong>the</strong>ir own optimal implementation of<strong>the</strong> respective application.3. Service providers can require submissions to be based on <strong>the</strong>ir own implementation.


With this solution services allow<strong>in</strong>g for good implementations will have an advantage,but this is only fair.PaaS providers offer a zoo of different languages, data stores <strong>and</strong> applicationservers. Some of <strong>the</strong>se follow st<strong>and</strong>ards, some do not <strong>and</strong> o<strong>the</strong>rs are head<strong>in</strong>g tobecome <strong>the</strong> de facto st<strong>and</strong>ard. PaaS benchmarks have <strong>the</strong> choice ei<strong>the</strong>r to begeneric enough to cover all or most of <strong>the</strong>se offers or to restrict <strong>the</strong>mselves to awell def<strong>in</strong>ed st<strong>and</strong>ard, which is implemented by a considerable group of serviceproviders. We propose to h<strong>and</strong>le <strong>the</strong> first option like <strong>the</strong> IaaS case discussedabove. This <strong>the</strong>n leads to a s<strong>in</strong>gle benchmark, which can be used for IaaS <strong>and</strong>PaaS services. This is <strong>the</strong> approach of Kossmann et al. [32], who implement <strong>the</strong>TPC-W benchmark for each of <strong>the</strong> services under test. The second PaaS optionmight lead to a reuse of exist<strong>in</strong>g benchmarks like SPECjEnterprise <strong>in</strong> <strong>the</strong> <strong>Cloud</strong>.We conclude <strong>the</strong> discussion with a short remark about SaaS benchmarks. Wecannot expect to have one benchmark for all SaaS offer<strong>in</strong>gs. Still, benchmarksfor an important application doma<strong>in</strong> like Human Capital Management(HCM)are worth consideration. Their biggest challenge would be to def<strong>in</strong>e a set of representativetransactions, which most HCM providers offer. The more specialized<strong>the</strong> services under test are, <strong>the</strong> more difficult it is to f<strong>in</strong>d a common denom<strong>in</strong>ator.6.4 Creat<strong>in</strong>g TrustTrust, which is a major concern for any <strong>Cloud</strong> operation [36], is particularly importantfor benchmark<strong>in</strong>g <strong>Cloud</strong> services. Choos<strong>in</strong>g a representative benchmarkscenario <strong>and</strong> properly deal<strong>in</strong>g with above design <strong>and</strong> implementation challengessupports creat<strong>in</strong>g trust. We hold that we must also consider benchmark operations<strong>and</strong> list three operational challenges.Challenge 9: Location. <strong>Benchmark<strong>in</strong>g</strong> <strong>in</strong> <strong>the</strong> <strong>Cloud</strong> raises a non-trivial challenge<strong>in</strong> decid<strong>in</strong>g where each component of <strong>the</strong> benchmark is located. Should<strong>the</strong> <strong>Cloud</strong> provider host <strong>the</strong> driver? Should <strong>the</strong> request queues of <strong>the</strong> user [37]be located close to <strong>the</strong> <strong>Cloud</strong> or even <strong>in</strong> different time zones? Does this decisiondepend on <strong>the</strong> workload or not? If yes, is <strong>the</strong>re a general rule of thumb that canhelp us decide where to place <strong>the</strong> driver?Challenge 10: Ownership. Which actor, from <strong>the</strong> group <strong>in</strong>troduced <strong>in</strong> Section3.1, should run <strong>the</strong> benchmark? How to prevent that <strong>the</strong> <strong>Cloud</strong> service provider“games” <strong>the</strong> results, for example by putt<strong>in</strong>g more resources <strong>in</strong>to <strong>the</strong> benchmarkedservice? Should <strong>the</strong> <strong>Cloud</strong> service provider be <strong>in</strong>formed about when <strong>the</strong>benchmark is be<strong>in</strong>g run <strong>in</strong> <strong>the</strong>ir system?Challenge 11: Repeatability. We expect a variability <strong>in</strong> <strong>the</strong> results reportedby a benchmark <strong>and</strong> list three possible reasons. a) The performance variability


of production IaaS <strong>Cloud</strong>s has been <strong>in</strong>vestigated [38] <strong>and</strong> found to exhibit pronouncedtime patterns at various time scales. The ma<strong>in</strong> reason is that <strong>Cloud</strong>services time-share <strong>and</strong> may over-commit <strong>the</strong>ir resources. b) Ever-chang<strong>in</strong>g customerload may affect <strong>the</strong> performance of <strong>Cloud</strong> <strong>in</strong>frastructures. c) Moreover,providers are liable to change prices, which directly affects <strong>the</strong> proposed PricePerformancemetrics. Thus, benchmark<strong>in</strong>g results may vary significantly overtime. In contrast to traditional benchmark<strong>in</strong>g of large-scale comput<strong>in</strong>g systems,what is <strong>the</strong> value of numbers measured at any particular po<strong>in</strong>t <strong>in</strong> time? How toensure <strong>the</strong> repeatability of results? Should benchmarks be simply re-run periodically,<strong>the</strong>refore lower<strong>in</strong>g <strong>the</strong> economical requirement, or should <strong>the</strong> repeatabilityrequirement be lowered or even not imposed?Our first draft of a solution to <strong>the</strong>se operational challenges is to set up an<strong>in</strong>dependent consortium. The ma<strong>in</strong> tasks of this consortium would be to:1. Provide a geo-diverse driver <strong>Cloud</strong>.2. Run benchmarks without fur<strong>the</strong>r notice to <strong>the</strong> provider.3. Rerun benchmarks periodically.4. Charge benchmark runs to <strong>the</strong> provider.5. Offer different levels of trust by hav<strong>in</strong>g runs repeated more or less frequently.6. Store benchmark applications implemented (see Challenge 8) by <strong>the</strong> provideror third party.7 Conclusion<strong>Benchmark<strong>in</strong>g</strong> plays an important role <strong>in</strong> <strong>the</strong> wide-spread adoption of cloudcomput<strong>in</strong>g technologies. General expectations of ubiquitous, un<strong>in</strong>terrupted, ondem<strong>and</strong>,<strong>and</strong> elastic cloud services must be met through <strong>in</strong>novative yet universallyaccepted benchmark<strong>in</strong>g practices. In this work we described our underst<strong>and</strong><strong>in</strong>gwhat benchmark<strong>in</strong>g should, can, <strong>and</strong> cannot be. This underst<strong>and</strong><strong>in</strong>gis governed by general benchmark requirements listed <strong>in</strong> Section 2.3 . It is alsobased on a sequence of papers [5], [2], [32] by Kossmann et al. <strong>and</strong> <strong>the</strong> respectiveexperiments performed.We first def<strong>in</strong>ed <strong>the</strong> actors <strong>in</strong>volved <strong>in</strong> cloud benchmark<strong>in</strong>g, <strong>in</strong>clud<strong>in</strong>g <strong>the</strong>irvalue network, <strong>and</strong> <strong>the</strong> system under test (SUT). Unlike traditional benchmark<strong>in</strong>g,<strong>the</strong> SUT <strong>in</strong>cludes numerous components that are ei<strong>the</strong>r black boxes or <strong>in</strong>herentlyunstable. Next, we analyzed several use cases where benchmark<strong>in</strong>g can playa significant role, <strong>and</strong> discussed <strong>the</strong> ma<strong>in</strong> challenges <strong>in</strong> build<strong>in</strong>g scenario-specificbenchmarks. Last, we collected <strong>the</strong> challenges of scenario-specific benchmarks<strong>and</strong> proposed <strong>in</strong>itial steps towards <strong>the</strong>ir solution. Besides propos<strong>in</strong>g solutions fortechnical challenges we propose found<strong>in</strong>g a consortium, which is able to tackle<strong>the</strong> operational challenges. We hope to be able to discuss our solutions with<strong>the</strong> TPC audience <strong>and</strong> are strongly committed to use our current presence <strong>in</strong><strong>the</strong> related SPEC work<strong>in</strong>g groups to foster <strong>the</strong> adoption of <strong>the</strong>se benchmark<strong>in</strong>gtechnologies.


References1. Huppler, K.: The art of build<strong>in</strong>g a good benchmark. In Nambiar, R.O., Poess,M., eds.: TPCTC. Volume 5895 of Lecture Notes <strong>in</strong> Computer Science., Spr<strong>in</strong>ger(2009) 18–302. B<strong>in</strong>nig, C., Kossmann, D., Kraska, T., Loes<strong>in</strong>g, S.: How is <strong>the</strong> wea<strong>the</strong>r tomorrow?:towards a benchmark for <strong>the</strong> cloud. In: DBTest, ACM (2009)3. SPEC: The SPEC CPU2006 Benchmark URL: http://www.spec.org/cpu2006/.4. TPC: The TPC-C Benchmark URL: http://www.tpc.org/tpcc/.5. Florescu, D., Kossmann, D.: Reth<strong>in</strong>k<strong>in</strong>g cost <strong>and</strong> performance of database systems.SIGMOD Record 38(1) (2009) 43–486. Gray, J., ed.: The Benchmark H<strong>and</strong>book for Database <strong>and</strong> Transaction <strong>Systems</strong>(2nd Edition). Morgan Kaufmann (1993)7. Kounev, S.: Performance Eng<strong>in</strong>eer<strong>in</strong>g of <strong>Distributed</strong> Component-Based <strong>Systems</strong>- <strong>Benchmark<strong>in</strong>g</strong>, Model<strong>in</strong>g <strong>and</strong> Performance Prediction. PhD <strong>the</strong>sis, TechnischeUniversität Darmstadt (2005)8. Sachs, K., Kounev, S., Bacon, J., Buchmann, A.: Performance evaluation ofmessage-oriented middleware us<strong>in</strong>g <strong>the</strong> SPECjms2007 benchmark. PerformanceEvaluation 66(8) (Aug 2009) 410–4349. Sachs, K.: Performance Model<strong>in</strong>g <strong>and</strong> <strong>Benchmark<strong>in</strong>g</strong> of Event-Based <strong>Systems</strong>.PhD <strong>the</strong>sis, TU Darmstadt (2011)10. Madeira, H., Vieira, M., Sachs, K., Kounev, S. Dagstuhl Sem<strong>in</strong>ar 10292. In: Resilience<strong>Benchmark<strong>in</strong>g</strong>. Spr<strong>in</strong>ger Verlag (2011)11. NIST: The NIST Def<strong>in</strong>ition of <strong>Cloud</strong> Comput<strong>in</strong>g (2011)http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf.12. Youseff, L., Butrico, M., Silva, D.D.: Towards a unified ontology of cloud comput<strong>in</strong>g.In: Proc.of <strong>the</strong> Grid Comput<strong>in</strong>g Environments Workshop(GCE08). (2008)13. Huppler, K.: <strong>Benchmark<strong>in</strong>g</strong> with your head <strong>in</strong> <strong>the</strong> cloud. TPCTC (2011)14. Leimeister, S., Böhm, M., Riedl, C., Krcmar, H.: The bus<strong>in</strong>ess perspective of cloudcomput<strong>in</strong>g: Actors, roles <strong>and</strong> value networks. In Alex<strong>and</strong>er, P.M., Turp<strong>in</strong>, M., vanDeventer, J.P., eds.: ECIS. (2010)15. SPEC Open <strong>Systems</strong> Group: Report on cloud comput<strong>in</strong>g to <strong>the</strong> OSG Steer<strong>in</strong>gCommittee. Technical Report OSG-wg-f<strong>in</strong>al-20120214 (Feb 2012)16. Shen, S., Visser, O., Iosup, A.: Rtsenv: An experimental environment for real-timestrategy games. In Shirmohammadi, S., Griwodz, C., eds.: NETGAMES, IEEE(2011) 1–617. Nae, V., Iosup, A., Prodan, R.: Dynamic resource provision<strong>in</strong>g <strong>in</strong> massively multiplayeronl<strong>in</strong>e games. IEEE Trans. <strong>Parallel</strong> Distrib. Syst. 22(3) (2011) 380–39518. Ratti, S., Hariri, B., Shirmohammadi, S.: A survey of first-person shooter gam<strong>in</strong>gtraffic on <strong>the</strong> <strong>in</strong>ternet. IEEE Internet Comput<strong>in</strong>g 14(5) (2010) 60–6919. Li, M., Sasanka, R., Adve, S., Chen, Y., Debes, E.: The ALPBench benchmark suitefor complex multimedia applications. In: Workload Characterization Symposium,2005. Proceed<strong>in</strong>gs of <strong>the</strong> IEEE International. (2005) 34–4520. Lee, C., Potkonjak, M., Mangione-Smith, W.H.: Mediabench: A tool for evaluat<strong>in</strong>g<strong>and</strong> syn<strong>the</strong>siz<strong>in</strong>g multimedia <strong>and</strong> communicatons systems. In: MICRO. (1997)330–33521. Guthaus, M.R., R<strong>in</strong>genberg, J.S., Ernst, D., Aust<strong>in</strong>, T.M., Mudge, T., Brown,R.B.: MiBench: A free, commercially representative embedded benchmark suite.In: Proceed<strong>in</strong>gs of <strong>the</strong> Fourth Annual IEEE International Workshop on WorkloadCharacterization. WWC-4 (Cat. No.01EX538), IEEE (2001) 3–14


22. Fritsch, T., Ritter, H., Schiller, J.H.: The effect of latency <strong>and</strong> network limitationson mmorpgs: a field study of everquest2. In: NETGAMES, ACM (2005) 1–923. Chen, K.T., Huang, P., Lei, C.L.: How sensitive are onl<strong>in</strong>e gamers to networkquality? Commun. ACM 49(11) (2006) 34–3824. Claypool, M.: The effect of latency on user performance <strong>in</strong> real-time strategygames. Computer Networks 49(1) (2005) 52–7025. Beigbeder, T., Coughlan, R., Lusher, C., Plunkett, J., Agu, E., Claypool, M.: Theeffects of loss <strong>and</strong> latency on user performance <strong>in</strong> unreal tournament 2003. Inchang Feng, W., ed.: NETGAMES, ACM (2004) 144–15126. Bal<strong>in</strong>t, M., Posea, V., Dimitriu, A., Iosup, A.: User behavior, social network<strong>in</strong>g,<strong>and</strong> play<strong>in</strong>g style <strong>in</strong> onl<strong>in</strong>e <strong>and</strong> face to face bridge communities. In: NETGAMES,IEEE (2010) 1–227. Iosup, A., Lascateu, A.: <strong>Cloud</strong>s <strong>and</strong> cont<strong>in</strong>uous analytics enabl<strong>in</strong>g social networksfor massively multiplayer onl<strong>in</strong>e games. In Bessis, N., Xhafa, F., eds.: Next GenerationData Technologies for Collective Computational Intelligence. Volume 352of Studies <strong>in</strong> Computational Intelligence. Spr<strong>in</strong>ger (2011) 303–32828. Kim, K., Jeon, K., Han, H., Kim, S.g., Jung, H., Yeom, H.Y.: Mrbench: A benchmarkfor mapreduce framework. In: Proceed<strong>in</strong>gs of <strong>the</strong> 2008 14th IEEE InternationalConference on <strong>Parallel</strong> <strong>and</strong> <strong>Distributed</strong> <strong>Systems</strong>. ICPADS ’08, Wash<strong>in</strong>gton,DC, USA, IEEE Computer Society (2008) 11–1829. Chen, Y., Ganapathi, A., Griffith, R., Katz, R.H.: The case for evaluat<strong>in</strong>g mapreduceperformance us<strong>in</strong>g workload suites. In: MASCOTS, IEEE (2011) 390–39930. Pavlo, A., Paulson, E., Ras<strong>in</strong>, A., Abadi, D.J., DeWitt, D.J., Madden, S., Stonebraker,M.: A comparison of approaches to large-scale data analysis. In Çet<strong>in</strong>temel,U., Zdonik, S.B., Kossmann, D., Tatbul, N., eds.: SIGMOD Conference,ACM (2009) 165–17831. Huppler, K.: Price <strong>and</strong> <strong>the</strong> tpc. In Nambiar, R.O., Poess, M., eds.: TPCTC.Volume 6417 of Lecture Notes <strong>in</strong> Computer Science., Spr<strong>in</strong>ger (2010) 73–8432. Kossmann, D., Kraska, T., Loes<strong>in</strong>g, S.: An evaluation of alternative architecturesfor transaction process<strong>in</strong>g <strong>in</strong> <strong>the</strong> cloud. In: Proceed<strong>in</strong>gs of <strong>the</strong> 2010 ACM SIGMODInternational Conference on Management of data. SIGMOD ’10, New York, NY,USA, ACM (2010) 579–59033. Islam, S., Lee, K., Fekete, A., Liu, A.: How a consumer can measure elasticity forcloud platforms. [39] 85–9634. Rabl, T., Poess, M.: <strong>Parallel</strong> data generation for performance analysis of large,complex rdbms. In Graefe, G., Salem, K., eds.: DBTest, ACM (2011) 535. Frank, M., Poess, M., Rabl, T.: Efficient update data generation for dbms benchmarks.[39] 169–18036. Armbrust, M., Fox, A., Griffith, R., Joseph, A.D., Katz, R.H., Konw<strong>in</strong>ski, A., Lee,G., Patterson, D.A., Rabk<strong>in</strong>, A., Stoica, I., Zaharia, M.: A view of cloud comput<strong>in</strong>g.Commun. ACM 53(4) (2010) 50–5837. Villegas, D., Antoniou, A., Sadjadi, S.M., Iosup, A.: An analysis of provision<strong>in</strong>g<strong>and</strong> allocation policies for <strong>in</strong>frastructure-as-a-service clouds. In: CCGRID. (2012)38. Iosup, A., Yigitbasi, N., Epema, D.H.J.: On <strong>the</strong> performance variability of productioncloud services. In: CCGRID, IEEE (2011) 104–11339. Kaeli, D.R., Rolia, J., John, L.K., Krishnamurthy, D., eds.: Third Jo<strong>in</strong>tWOSP/SIPEW International Conference on Performance Eng<strong>in</strong>eer<strong>in</strong>g, ICPE’12,Boston, MA, USA - April 22 - 25, 2012. In Kaeli, D.R., Rolia, J., John, L.K.,Krishnamurthy, D., eds.: ICPE’12 ICPE, ACM (2012)

Hooray! Your file is uploaded and ready to be published.

Saved successfully!

Ooh no, something went wrong!