Agile Performance Testing - Testing Experience

More documents

Recommendations

Info

to create high, spiking, random loads. We would sustain these loads for up to 48 hours. If none of the systems crashed, none of the network bridges lost connectivity to one or both networks, no data were lost, and if no applications terminated abnormally, then that was a passing test result. Figure 5: Maximum network configuration for the operating system Lesson 3: Test the Tests - and the Models Okay, so now it’s clear that realism of the environment and the load are important. I mentioned earlier that performance, load, and reliability testing are complex endeavors, and that means that it’s easy to screw them up. These two areas of realism are especially easy to screw up. So, if realism of environment and load is important, how can we be sure we got it right, before we go live or ship? The key here is to build a model or simulation of your system’s performance and use the model or simulation both to evaluate the performance of the system during the design phase and to evaluate the tests during test execution. In turn, since the model and simulation are equally subject to error, the tests also can evaluate the models. Furthermore, a model or simulation which is validated through testing is useful for predicting future performance and for doing various kinds of performance “what-iffing” without having to resort to expensive performance tests as frequently. There are a couple of key tips to keep in mind regarding modeling and simulating. First, spreadsheets are good for initial performance, load, and reliability modeling and design. Second, once these spreadsheets are in place, you can use them as design documents both for the actual system and for a dynamic simulation. The dynamic simulations allow for more “what-if” scenarios and are useful during design, implementation, and testing. Here’s a case study of using models and simulations properly during the system test execution period. On an Internet appliance project, we needed to test server-side performance and reliability at projected loads of 25,000, 40,000, and 50,000 devices in the field. In terms of realism, we used a production environment and a realistic load generator, which we built using the development team’s unit testing harnesses. The test environment, including the load generators and functional and beta test devices, are shown in Figure 6. Allowing some amount of functional and beta testing of the devices during certain carefully selected periods of the performance, load, and reliability tests gave the test team and the beta users a realistic sense of user experience under load. Figure 6: Internet appliance test environment So far, this is all pretty standard performance, load, and reliability testing procedure. However, here’s where it gets interesting. Because a dynamic simulation had been built for design and implementation work, we were able to compare our test results against the simulation results, at both coarse-grained and finegrained levels of detail. Let’s look first at the course-grained level of detail, using an extract from our performance, load, and reliability test results presentation. We had simulation data for the mail servers, which were named IMAP1 and IMAP2 in Figure 6. The simulation predicted 55% server CPU idle at 25,000 devices. Table 1 shows the CPU idle statistics under worst-case load profiles, with 25,000 devices. We can see that the test results support the simulation results at 25,000 devices. Server CPU Idle IMAP1 59% IMAP2 55% Table 1: Server CPU idle figures during performance tests Table 2 shows our more fine-grained analysis for the various transactions the IMAP servers had to handle, their frequency per connection between the device and the IMAP server, the average time the simulation predicted for the transaction to complete, and the average time that was observed during testing. The times are given in milliseconds. Transaction Frequency Simulation <strong>Testing</strong> Connect 1 per connect 0.74 0.77 Banner 1 per connect 2.57 13.19 Authorize 1 per connect 19.68 19.08 Select 1 per connect 139.87 163.91 Fetch 1 per connect 125.44 5,885.04 Table 2: Server transactions and times, simulated and observed during testing 70 The Magazine for Professional Testers www.testingexperience.com
© iStockphoto.com/arlindo71 © iStockphoto.com/ScantyNebula Testen für Entwickler 28.06.10-29.06.10 Berlin 23.08.10-24.08.10 Berlin 18.10.10-19.10.10 Berlin 29.11.10-30.11.10 Berlin Während die Ausbildung der Tester in den letzten Jahren große Fortschritte machte – es gibt mehr als 13.000 zertifizierte Tester alleine in Deutschland – wird die Rolle des Entwicklers beim Softwaretest meist unterschätzt. Dabei ist er beim Komponententest oftmals die treibende Kraft. Aus diesem Grunde ist es wichtig, dass auch der Entwickler Grundkenntnisse im Kernbereichen des Softwaretestens erlangt. http://training.diazhilterscheid.com IREB - Certified Professional for Requirements Engineering - Foundation Level 25.08.10-27.08.10 Berlin 26.10.10-28.10.10 Berlin 01.12.10-03.12.10 Berlin www.testingexperience.com Die Disziplin des Requirements Engineering (Erheben, Dokumentieren, Prüfen und Verwalten) stellt die Basis jeglicher Software-Entwicklung dar. Studien bestätigen, dass rund die Hälfte aller Projekte scheitern aufgrund fehlerhafter oder unvollständiger Anforderungsdefinitionen. http://training.diazhilterscheid.com The Magazine for Professional Testers 71
Page 1 and 2:
10 ISSN 1866-5705 www.testingexperi
Page 3 and 4:
© Hans Schaefer www.testingexperie
Page 5 and 6:
Five Lessons in Performance, Load,
Page 7 and 8:
Bug type Bug description String con
Page 9 and 10:
Biography Anuj Magazine is a softwa
Page 11 and 12:
folder). Thus, every time an applic
Page 13 and 14:
Feierabend! Seine Last- und Perform
Page 15 and 16:
vices/resolutions. 3. Repeat the te
Page 17 and 18:
Figure 2: DBAP Test Data Usage Cube
Page 19 and 20: Database Constraints restrict which
Page 21 and 22: XLT Xceptance LoadTest Functional &
Page 23 and 24: Biography Ronny Vogel is technical
Page 26 and 27: In this article we describe how Hud
Page 28 and 29: Foto: © swoodie - Fotolia.com Why
Page 30 and 31: init_server The init server job hel
Page 32 and 33: Load and Performance Testing for So
Page 34 and 35: © iStockphoto.com/cogal limited pl
Page 36 and 37: Altogether the criteria are grouped
Page 38: Testing is carried out primarily fo
Page 41 and 42: Berlin, Germany IT Law Contract Law
Page 43 and 44: © iStockphoto.com/naphtalina Agile
Page 45 and 46: October 6 Conference Day 2 Track 1
Page 47 and 48: availability of empirical data to n
Page 49 and 50: ous causal analysis in each case. T
Page 51 and 52: www.testingexperience.com An approa
Page 53 and 54: Figure 3 - Script line analysis 2.1
Page 55 and 56: Testing & Finance 2010 - The Confer
Page 57 and 58: tially desired test script shown in
Page 59 and 60: Daddy’s performance test Saturday
Page 61 and 62: i.e.people that know the product. H
Page 63 and 64: © iStockphoto.com/Petrovich9 Knowl
Page 65 and 66: ting is good. Tests often don’t u
Page 67 and 68: and software engineering is not yet
Page 69: Subscribe for the printed issue! Pl
Page 73 and 74: As you can see, we had good matches
Page 75 and 76: Lesson 5: Start Early and Stay on C
Page 77 and 78: © Pitopia / Klaus-Peter Adler, 200
Page 79 and 80: This is even when the transaction r
Page 81 and 82: the experience of educational speci
Page 83 and 84: Time to change a winning team? Duri
Page 85 and 86: lity Center, TestManager and VSTS E
Page 87 and 88: Masthead EDITOR Díaz & Hilterschei
show all

Agile Performance Testing - Testing Experience

Create successful ePaper yourself

Delete template?

Save as template?