"Experiences in Applying Testing Automation Research in an ...
"Experiences in Applying Testing Automation Research in an ...
"Experiences in Applying Testing Automation Research in an ...
- No tags were found...
You also want an ePaper? Increase the reach of your titles
YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.
"<strong>Experiences</strong> <strong>in</strong> Apply<strong>in</strong>g Test<strong>in</strong>g<strong>Automation</strong> <strong>Research</strong> <strong>in</strong> <strong>an</strong>Industrial Context."Marlon Vieira, PhDProgram M<strong>an</strong>ager - Software Development TechnologiesSoftware Eng<strong>in</strong>eer<strong>in</strong>g DepartmentSiemens Corporate <strong>Research</strong>Pr<strong>in</strong>ceton, NJEmail: marlon.vieira@siemens.comCopyright © Siemens AG 2008. All rights reserved.
Outl<strong>in</strong>e1. Background2. Some aspects from <strong>Research</strong> toTechnology Tr<strong>an</strong>sfer3. A Broader View: Software Eng<strong>in</strong>eer<strong>in</strong>g,Test<strong>in</strong>g <strong>an</strong>d <strong>Automation</strong>4. <strong>Experiences</strong> from apply<strong>in</strong>g our MBT <strong>Research</strong> <strong>in</strong>several projects5. Typical Aspects Used to Evaluate Introduction ofnew Technologies <strong>an</strong>d F<strong>in</strong>al RemarksPage 2Siemens Corporate <strong>Research</strong>
Siemens has a long tradition oftechnological <strong>in</strong>novations.Page 3Siemens Corporate <strong>Research</strong>
Siemens is one of the world’s largestsoftware comp<strong>an</strong>ies• Siemens has more th<strong>an</strong> 41,000 software developers.• 60% of Siemens’ bus<strong>in</strong>ess is based on software.• Siemens spends more th<strong>an</strong> 3 billion Euros per year onsoftware development.Page 4Siemens Corporate <strong>Research</strong>
But, Siemens is not recognized as a softwarecomp<strong>an</strong>y, s<strong>in</strong>ce most of our software isembedded.Examples:• <strong>Automation</strong> devices• Industrial control systems• Medical devices• Communication systems• Rail systems• Automotive componentsPage 5Siemens Corporate <strong>Research</strong>
Corporate Technology:About 2,300 <strong>Research</strong>ers <strong>an</strong>d Developers WorldwideRomsey (RMR)Berl<strong>in</strong>St. PetersburgMoscowPr<strong>in</strong>cetonSiemensCorporate TechnologyA Global NetworkBeij<strong>in</strong>gBerkeleySh<strong>an</strong>ghaiTokyoErl<strong>an</strong>genMunichB<strong>an</strong>galorePage 6Siemens Corporate <strong>Research</strong>
Global Technology FieldSoftware Development TechnologiesPage 7Siemens Corporate <strong>Research</strong>
From <strong>Research</strong> to Practice“Genius is one percent <strong>in</strong>spiration <strong>an</strong>dn<strong>in</strong>ety-n<strong>in</strong>e percent perspiration.”- Thomas EdisonFrom <strong>Research</strong> to Prototypes & Solutions Tr<strong>an</strong>sferPre-development• <strong>Research</strong> requirements• Technology evaluation• Academic partners• Feasibility study• PrototypePrototype/Work-<strong>in</strong>-Progress• Prototypeoptimization• Pilots• Feedback• ValidationTr<strong>an</strong>sfer• Integration withproducts <strong>an</strong>d/orprojects• More CompleteDocumentation <strong>an</strong>dknowledge tr<strong>an</strong>sferProductization &Ma<strong>in</strong>ten<strong>an</strong>ce• Ma<strong>in</strong>ten<strong>an</strong>ce• UpgradesPage 8Siemens Corporate <strong>Research</strong>
Typical Characteristics <strong>in</strong> <strong>Research</strong> that weapply <strong>in</strong> practices• Scalability• Usability• Early results• Support Evolution <strong>an</strong>d,• Different levels of abstractionPage 9Siemens Corporate <strong>Research</strong>
Broader View“Software Challenges at Industrial Context”• Functionality previously realized <strong>in</strong>electrical or electro-mech<strong>an</strong>icalsystems is now be<strong>in</strong>g realized <strong>in</strong>software• Meet<strong>in</strong>g functional <strong>an</strong>d non-functionalrequirements is import<strong>an</strong>t to bus<strong>in</strong>esssuccess• Multisite development projects.• High quality (i.e., thoroughly tested,reliable) software is import<strong>an</strong>t tobus<strong>in</strong>ess success.Page 10Siemens Corporate <strong>Research</strong>
Software Test<strong>in</strong>g: Some challenges• Pressure to release software to the market(with quality <strong>an</strong>d on time)• Interoperability Test<strong>in</strong>g• need to <strong>in</strong>teract with other constituents(Siemens products <strong>an</strong>d/or from othercomp<strong>an</strong>ies)• Specification is a Mov<strong>in</strong>g Target• new customer requirements•Test<strong>in</strong>g is Expensive• laboratory test<strong>in</strong>g vs. field test<strong>in</strong>gPage 11Siemens Corporate <strong>Research</strong>
An Essential challengeEffective ways to design <strong>an</strong>d automate library of teststhat conta<strong>in</strong>s the right subset of possible test caseshav<strong>in</strong>g the highest probability of detect<strong>in</strong>g errorsPage 12Siemens Corporate <strong>Research</strong>
Typical test<strong>in</strong>g processTest Pl<strong>an</strong>n<strong>in</strong>g Test Design Test Execution Test Report<strong>in</strong>gTest Pl<strong>an</strong>n<strong>in</strong>g Test Design Test Execution Test Report<strong>in</strong>gPage 13Siemens Corporate <strong>Research</strong>
<strong>Automation</strong> of Test<strong>in</strong>g:Observations (Diverse Siemens Projects)The test process <strong>an</strong>d its automation <strong>in</strong> practice (Tools)Test M<strong>an</strong>agement ToolsCapture-ReplaySimulators/emulatorsCustom FrameworksResult Analysis ToolsDefect ClassificationMetrics AnalysisUnitComponent Integration ValidationHigh * Medium * Medium * Low *Page 14* <strong>Automation</strong> level (usage)Siemens Corporate <strong>Research</strong>
Common FrameworkExistence of different frameworks for different steps of the test processtest scriptsGUISUTTest resultsobjectTest M<strong>an</strong>agementdatabaseToolPage 15Siemens Corporate <strong>Research</strong>
<strong>Automation</strong> <strong>an</strong>d Test Design• Test Design is usually the weakest l<strong>in</strong>k• Use a test m<strong>an</strong>agement tool for what belongs <strong>in</strong> it• Test Design should be leveraged <strong>in</strong>to <strong>Automation</strong>Page 1616Siemens Corporate <strong>Research</strong>
Model-Based Test<strong>in</strong>g: Our View“ MBT is a technology <strong>an</strong>d not a solution by itself ”• Automatic generation of test cases us<strong>in</strong>g models createddur<strong>in</strong>g development process or extracted (created) fromartifacts of that process.• Average testers are already us<strong>in</strong>g models.• Test<strong>in</strong>g creation activities are based (at least) on mentalmodelPage 18Siemens Corporate <strong>Research</strong>
MBT ProcessModel<strong>in</strong>g Generat<strong>in</strong>g Execut<strong>in</strong>g Analyz<strong>in</strong>g ResultsMentalModelGenerateTest CasesTest SuiteTest SuiteRequirementor DesignDocumentationDecide whether to• Generate more tests• Modify the model• Stop test<strong>in</strong>gTest passor FailOraclesVerifyResults<strong>an</strong>d/orGenerateExecutableCreateExecutableExecutableExecutableScriptsScriptsRunScriptsExecuteM<strong>an</strong>uallyApplication Under TestPage 19Siemens Corporate <strong>Research</strong>
MBT <strong>Research</strong> at Siemens Corporate <strong>Research</strong>• Workbench for develop<strong>in</strong>g efficienttechniques for test case generation• Ma<strong>in</strong> goal - Support <strong>an</strong> <strong>in</strong>dustrial <strong>an</strong>d“affordable“ approach• Increased use of model-based development (UML) bySiemens Units• Leverage test<strong>in</strong>g by reus<strong>in</strong>g documents/models• Opportunity for automatic test case generationPage 20Siemens Corporate <strong>Research</strong>
TDE/UML UC-Activ : Ma<strong>in</strong> Concepts• Support a Functional (Sub) System (black box) test<strong>in</strong>g• Integration to different tools to support the Test Methodology• Methodology comb<strong>in</strong>es:Model<strong>in</strong>g system behavior− UML Model<strong>in</strong>g tools / SCR Eclipse UML Editor− UML UC <strong>an</strong>d Activity Diagrams− Extension (via Stereotypes) to better describe testablespecificationsGenerat<strong>in</strong>g test cases− Diverse control/data coverage criterion− Test M<strong>an</strong>agement ToolsExecut<strong>in</strong>g test cases− Approach to create executable test cases− Test Execution ToolsPage 21Siemens Corporate <strong>Research</strong>
MBT with TDE/UMLModel<strong>in</strong>g• Focus on UML Use CaseDiagrams ref<strong>in</strong>ed by UMLActivity DiagramsTest Case Generation• Comb<strong>in</strong>es Graph Coverage <strong>an</strong>dData Coverage to form a TestSuiteCode Generation• Tr<strong>an</strong>sforms a Test Suite <strong>in</strong>toother formats, such asexecutable code, reports, xml…Page 22Siemens Corporate <strong>Research</strong>
Model<strong>in</strong>g for TDE/UMLAll Use Cases must have <strong>an</strong> Activity Diagram that describes itUse Cases have a network of <strong>in</strong>cluded <strong>an</strong>d extended use casesUse Cases c<strong>an</strong> be <strong>in</strong>cluded <strong>an</strong>d extended by m<strong>an</strong>y use casesThis promotes reusePage 23Siemens Corporate <strong>Research</strong>
Model<strong>in</strong>g for TDE/UML• Categories <strong>an</strong>d Choices• Not part of the UML• Refers to the Category-Partition method developedwith<strong>in</strong> SCR - Thomas J. Ostr<strong>an</strong>d <strong>an</strong>d Marc J.Balcer[CACM ,1988].• Categories:• The ma<strong>in</strong> idea is to partition the <strong>in</strong>put doma<strong>in</strong> offunction be<strong>in</strong>g tested, <strong>an</strong>d then select test data for eachclass of the partition• A category is def<strong>in</strong>ed by specify<strong>in</strong>g all possible datachoices that it c<strong>an</strong> represent• Choices:• Data values of a category• With<strong>in</strong> a category, choices are required to be of thesame typePage 24Siemens Corporate <strong>Research</strong>
Test Case GenerationData Coverage Criterion• Sampl<strong>in</strong>g• Choice-per-suite• Choice-per-variable• Choice-per-path• Exhaustive2 Tests, 48 ProceduresPage 25Siemens Corporate <strong>Research</strong>
TDE/UML - Some Publication• “<strong>Automation</strong> of GUI Test<strong>in</strong>g Us<strong>in</strong>g a Model-driven Approach”, M. Vieira, J. Leduc, B. Hasl<strong>in</strong>g, R.Subram<strong>an</strong>y<strong>an</strong>, J. Kazmeier Workshop on <strong>Automation</strong> of Software Test (AST'06). At 28th InternationalConference on Software Eng<strong>in</strong>eer<strong>in</strong>g (ICSE'06), Sh<strong>an</strong>ghai, Ch<strong>in</strong>a.• “An approach for generat<strong>in</strong>g <strong>an</strong>d execut<strong>in</strong>g tests based on the UML test<strong>in</strong>g profile”, M. Vieira. InProceed<strong>in</strong>gs of the IASTED International Conference on Software Eng<strong>in</strong>eer<strong>in</strong>g, February, 2006 Innsbruck,Austria• "A UML-based approach to system test<strong>in</strong>g". Published at Innovations <strong>in</strong> Systems <strong>an</strong>d SoftwareEng<strong>in</strong>eer<strong>in</strong>g (A NASA Journal), Hartm<strong>an</strong>n J, Vieira M., <strong>an</strong>d Ruder A.. Issue: Volume 1, Number 1 , ISSN:1614-5046 Publisher: Spr<strong>in</strong>ger-Verlag London Ltd DOI: 10.1007/s11334-005-0006-0, Pages: 12 - 24 , April2005• "TDE/UML: A UML-based Test Generator to Support System Test<strong>in</strong>g", Hartm<strong>an</strong>n J, Vieira M., <strong>an</strong>dRuder A.. In Proceed<strong>in</strong>gs of the QAI's 5th International Software Test<strong>in</strong>g Conference, February, 2005,Hyderabad, India• The Category-Partition Method for Specify<strong>in</strong>g <strong>an</strong>d Generat<strong>in</strong>g Functional Tests, T. J. Ostr<strong>an</strong>d, Marc J.Balcer, CACM 31(6): 676-686 (1988)• “Apply<strong>in</strong>g Model-Based Test<strong>in</strong>g to Healthcare Products: Prelim<strong>in</strong>ary <strong>Experiences</strong>” Marlon Vieira,Xip<strong>in</strong>g Song, Gilberto Matos, Steph<strong>an</strong> Storck, Raj<strong>an</strong>ik<strong>an</strong>th T<strong>an</strong>ikella, Bill Hasl<strong>in</strong>g --ICSE 2008– HealthcareSessionPage 26Siemens Corporate <strong>Research</strong>
Practical ApplicationRequirements+horizontal workflowsvertical, screen-based workflowsPage 27Siemens Corporate <strong>Research</strong>
Process OverviewPage 28Siemens Corporate <strong>Research</strong>
Some Results• Recognition that Incorporation of MBT reducecomplexity <strong>an</strong>d <strong>in</strong>crease productivity• “It took about 5 m<strong>in</strong>utes to update the model for the newrequirement. 3 m<strong>in</strong>utes to generate the tests <strong>an</strong>d 2 m<strong>in</strong>utes toexport the tests to Quality Center. What a time saver!” –TestAnalyst• Time to address bigger test issues• Decreased the number of escaped defects• Improved tester job satisfactionPage 29Siemens Corporate <strong>Research</strong>
Some Challenges <strong>in</strong> Apply<strong>in</strong>g MBT (Supportedby TDE/UML) <strong>in</strong> <strong>an</strong> Industrial Project• Comfort factor• Difficult to conv<strong>in</strong>ce of benefits <strong>in</strong>itially• Process shift• Up front <strong>in</strong>vestment <strong>in</strong> test• Personnel shift• Test designers with stronger abstraction skills• <strong>Automation</strong> of test execution• Not easy effortPage 30Siemens Corporate <strong>Research</strong>
TDE/UML From <strong>Research</strong> to PracticeFrom <strong>Research</strong> to Prototypes & Solutions Tr<strong>an</strong>sferPre-development• <strong>Research</strong> requirements• Technology evaluation• Academic partners• Feasibility study• PrototypePrototype/Work-<strong>in</strong>-Progress• Prototypeoptimization• Pilots• Feedback• ValidationTr<strong>an</strong>sfer• Integration withprojects (process)• More CompleteDocumentation <strong>an</strong>dknowledge tr<strong>an</strong>sferProductization &Ma<strong>in</strong>ten<strong>an</strong>ce• Ma<strong>in</strong>ten<strong>an</strong>ce• UpgradesPage 31Siemens Corporate <strong>Research</strong>
Typical Aspects Used to EvaluateIntroduction of new TechnologiesDef<strong>in</strong>e goals & process; then consider costs:1. Analysis goals, requirements - level of dissatisfaction withthe status-quo (compell<strong>in</strong>g event)2. Selection: Initial (<strong>in</strong>ternal) knowledge acquisition orpurchase or partnership3. Pilot4. Installation & Rollout5. Tra<strong>in</strong><strong>in</strong>g6. Ma<strong>in</strong>ten<strong>an</strong>ce7. Ev<strong>an</strong>geliz<strong>in</strong>g, <strong>in</strong>ternal sell<strong>in</strong>g8. Org<strong>an</strong>izational process ch<strong>an</strong>gesWeigh the cost (each of the steps) aga<strong>in</strong>st the benefit thatimplement<strong>in</strong>g this new technology will br<strong>in</strong>g, is it worth it?Page 32Siemens Corporate <strong>Research</strong>
F<strong>in</strong>al Remarks (1)"<strong>Experiences</strong> <strong>in</strong> Apply<strong>in</strong>g Test<strong>in</strong>g <strong>Automation</strong> <strong>Research</strong><strong>in</strong> <strong>an</strong> Industrial Context." Introduc<strong>in</strong>g new Technologies is not easy.• Ch<strong>an</strong>ge must be positive <strong>an</strong>d well m<strong>an</strong>aged, with am<strong>in</strong>imum of disruption <strong>an</strong>d culture shock. Determ<strong>in</strong>e read<strong>in</strong>ess for ch<strong>an</strong>ge• level of dissatisfaction with the status-quo (compell<strong>in</strong>gevent)versus• cost of ch<strong>an</strong>ge (resist<strong>an</strong>ce)Page 33Siemens Corporate <strong>Research</strong>
F<strong>in</strong>al Remarks (2)"<strong>Experiences</strong> <strong>in</strong> Apply<strong>in</strong>g Test<strong>in</strong>g <strong>Automation</strong> <strong>Research</strong><strong>in</strong> <strong>an</strong> Industrial Context." Justify the improvements; def<strong>in</strong>e the benefits• Sell the ch<strong>an</strong>ge; ga<strong>in</strong> consensus– From senior m<strong>an</strong>agement to practitioner– From practitioner to senior m<strong>an</strong>agementShow value added to m<strong>an</strong>agement, developers <strong>an</strong>drest of org<strong>an</strong>ization• Measure the effectivenessPage 34Siemens Corporate <strong>Research</strong>
Th<strong>an</strong>k you for your attention!Marlon Vieiramarlon.vieira@siemens.comPhone +1 609. 2332300Page 35For <strong>in</strong>ternal use only / Copyright © Siemens AG 2007. All rights reserved.Siemens Corporate <strong>Research</strong>