3 Examples of Reliability Testing - The Workshop On Performance ...

# 1 – Component Failover Testing• Application:– Donor Management system for a national non-profit– W2K, SQL Server, vbscript• Reliability issue:– Major load-related outage experienced in prior peak season where donationtransactions were lost; suspected component failure (web? Db?)• Solution:– Add redundant firewall, web/app server, hot-standby DB server• Testing focus:– Test fail-over of components– Measure recovery time and quantify failed transactions, especially lost donations• Result:– Showed that redundant components took up current workload quickly, minimizingfailed transactions, lost donations

Bandwidth throughput and error rates under load asprimary firewall & load balancer are failedLoad BalancerTurned Off tosimulateProduction DownLoad Balancer FailOver: Throughputnot affectedFirewall FailOver: Bandwidthdrops andrecoversThroughputErrorsRunningUsersColor Graph Graph's Min. Graph's Ave. Graph's Max.Throughput 0.0 2343400 4879640Running Vusers 0.0 146 375Errors per Second 00 0.0 0233 0.233 2.078

Key Informant CategoryPayers/HospitalAdministrators[n=2]ProductDevelopers/IndustryRepresentatives[n=2-3]Potential Questions1. What policies do payers put in place to influence use of imaging for treatmentevaluation of metastatic breast cancer?2. How are decisions to purchase imaging equipment used to evaluate treatment formetastatic breast cancer made at your institution?3. What are the major trends in imaging to evaluate treatment for the metastaticbreast cancer population? In general, is imaging use for metastatic breast cancerincreasing, decreasing, or staying the same?4. What are some research questions about imaging for treatment evaluation formetastatic breast cancer that you would like answered?5. What types of imaging is most commonly reimbursed for treatment evaluation ofmetastatic breast cancer? What are the advantages and disadvantages of thesetypes of imaging?6. Are you considering new policies to improve the use of imaging in thispopulation? If so, please describe them.1. What are the major trends in imaging to evaluate treatment for the metastaticbreast cancer population? In general, is imaging use for metastatic breast cancerincreasing, decreasing, or staying the same?2. What issues (e.g., cost, accuracy, patient discomfort) with imaging for treatmentevaluation for metastatic breast cancer do you see as most important toaddress?3. What are some imaging technologies that are not commonly used now but havethe potential to be widely used in the U.S. in the next 5 years or so for treatmentevaluation of metastatic breast cancer? Do you think these new technologies willreplace or be added onto existing technologies?4. What types of imaging technologies are you developing for the assessment oftreatments for metastatic breast cancer?5. What do you believe will be the role of biomarkers for the assessment oftreatments for metastatic breast cancer?6. What topics regarding imaging for treatment evaluation for metastatic breastcancer do you see as the most important areas for further research?B. Grey Literature SearchWe will conduct a search of the grey literature, which will be identified using advice fromour Key Informants as well as Internet searches (e.g., clinicaltrials.gov, NIH Reporter, GoogleScholar, ProQuest Dissertations and Theses, and/or PQDT Open). We will search forprofessional society consensus statements, conference abstracts and proceedings, and otherpreliminary, unpublished study findings. Professional societies may include American Societyfor Clinical Oncology (ASCO), the American College of Radiology (ACR), the AmericanCancer Society (ACS), the National Comprehensive Cancer Network (NCCN), and others.C. Published Literature SearchInclusion and exclusion criteria for our search of published literature are shown in Table 2.Source: www.effectivehealthcare.ahrq.govPublished online: December 17, 20136

# 3 – JVM Heap Utilization Analysis• Application:– National B2B retailer-supplier messaging system– Complex Java, Oracle, WebSphere MQ; Windows, AIX, mainframe– Re-architecture and redevelopment of middleware tier using JFS IDE• Reliability issue:– Severe user performance degradation under increasing load• Testing focus:– Monitor jvm heap utilization as is load increased (using Jconsole)• Resolution:– Uncovered and reduced JFS (Java Server Faces) default “Viewstatequeue depth” to eliminate heap space depletion which was triggeringcpu-grabbing garbage collection “sweeps”

JVM heap space profile showing steady-state heap usagebetween 200 and 300 MB on all app servers after fixapplied