YAVE ITE Integrated Test Environment White Paper - FTI Group

Test CaseV&V MatrixRequirementsDocumentsAuthoritiesRequirementsDocumentsV&V MatrixRequirementsQualificationTest ReportRequirementsEvaluationTestDefinitionTest CoverageReportReportRequirements-BasedTest EvaluationTestProcedureTestEvaluationTest ExecutionYAVE ITEIntegrated Test EnvironmentWhite Paper

IntroductionThe Integrated Test Environment (ITE), developed by FTI EngineeringNetwork GmbH and Razorcat Development GmbH, is designedto overcome complex test management problems. ITE has everyfunction necessary for systematic testing. These range from thedefinition of requirements to the creation and the execution of testcases, including the subsequent review and reporting of results.V&V MatrixRequirementsDocumentsAuthoritiesRequirementsDocumentsV&V MatrixRequirementsQualificationTest ReportRequirementsEvaluationTestDefinitionTest CoverageReportTest CaseReportRequirements-BasedTest EvaluationTestProcedureTestEvaluationTest ExecutionSince YAVE ITE is modular test management software, it can be implementedin all environments where testing, planning and verificationtake place. Test engineers can use ITE to manage test cases andrequirements, to link them with each other, to review and recordthe results and to automatically compile appropriate reports. Oncecompiled, these reports also contain relevant information for thecreation of a special test or a complete test campaign.In a nutshell, ITE is an all-in-one software solution for the problemsencountered by test engineers and test teams. In particular, ITE canbe used for development testing of hardware and software. Beyondsupporting requirements management, test definition, planning,evaluation and reporting, it is logically independent of the test executionlevel. The client-server architecture of ITE allows multipleusers to simultaneously work on the same project in different locationswithout disrupting each other.2

ITEState of the Art System TestingSystem testing is an essential step in the development and productionof a product. Since test processes must combine several requirements,they commonly demand different tools:• a tool for requirements management,• a tool for test creation and planning,• a tool for recording the test results and• a tool for report writing/compiling.There are specific commercial software solutions on the market foreach of these individual tasks (Requirements Management Tools,DTP, Spreadsheet Tools). However, as the number of componentsand requirements increases, coordinating their use can becomea task that is time-consuming, hard to manage and susceptible toerrors.To avoid such problems, all of the information test engineers needmust be available in a single Integrated Test Environment (ITE).This consolidation into the ITE enables the test engineer to easilysee which test procedures will be influenced by a particular actionas well as those that may need to be adjusted. Results are also consolidated,which makes reporting easy and, if desired, automatic.ITE can also generate reports that are immediately ready for productqualification, without the need for manual reconditioning.ITE not only combines all of the software tools needed to run testsin one environment, it also works in collaboration with existing infrastructure.ITE’s modular structure and highly flexible architectureallow for the retention of existing solutions (for example, forrequirements management or test execution). At the same time,it glues together the whole process and supports test engineers intheir daily tasks. All aspects from test definition to reporting are automated,and ITE allows the test engineers to focus on their mainpriorities: to run tests and verify products instead of managing data.Managing the EntireVerification Processin One ToolThe key strength of ITE is that the entire verification process iscovered. All necessary functions for organized testing and verificationare included. All of the information test engineers need atany stage of the verification process is available in a single environment.From requirements to test equipment, everything in the testprocess can be managed in ITE.3

State of the Art System TestingMulti-User FunctionalityTraceabilityVersioningImpact AnalysisAutomatic Creation ofTest ReportsSupport of Manualand AutomatedProcedure LanguagesAs a result of its client-server architecture, ITE has a multi-user feature,which enables multiple users to work on the same project atthe same time. ITE uses Oracle and PostgreSQL databases as well asJ2EE EJB technology with Eclipse RCP client applications.ITE allows the whole test chain of a system under test to be tracked.As a result, test engineers have an overview of the requirements,the created test cases and the test runs already executed, includingtheir results. The test equipment, test creator, executor and evaluatorare also recorded so that administrative efforts are kept to aminimum and the testing process is accelerated. Beyond this, a wellstructuredand field-proven user interface as well as a powerful andintelligent search function help the user to keep a clear overviewregardless of the volume of data managed in ITE.ITE automatically records a history of requirements and test caseswhen users transfer their version of a requirement or a test case tothe ITE server. If necessary, the history view also allows easy examinationof any version of a requirement or test case.ITE can analyze the impact of modifications on test data, for example,if test cases become invalid since they no longer match the underlyingrequirements. Changes can also be tracked at all processlevels, making it easy to keep the entire test process up to date aswell as to locate problems resulting from changes.ITE offers different types of automatically created test reports inPDF, RTF, ODF and other formats. Detailed configurable reports forone or more test cases, for a whole test program or a specific testcampaign can be generated. The meta data of a report can be savedas a template and used for further test reports. Reports generatedby ITE are generally accepted by certification authorities.In addition to supporting standard test definition languages like Cor Python, ITE also comes with a much more versatile and powerfulsolution: Check Case Definition Language (CCDL). CCDL makes automatedsystem testing easier by providing a high-level script languageto allow test procedures to be defined in a more readable form. As aresult, the test engineer does not need any special programmingskills to create CCDL test procedures. The CCDL code is run in a testexecution environment and used for automated testing of a system.4

ITEState of the Art System TestingConnection to TestExecution EnginesImport/Export ofRequirementsITE can send test runs with a CCDL script directly to any test runmanagement server – even if the real-time engine is not capable ofrunning CCDL in native mode. After the execution of the test run onthe test bench, the results are imported and evaluated in ITE automatically.Existing requirements in CSV and XML format can be imported. Thisworks well with DOORS and other requirements management systems.Requirements and validation results may also be exportedinto XML and ReqIF format.RequirementsDocumentsRequirementsManagementTraceability byV&V MatrixTestDefinition &ProcedureTestExecution &ReportUnit &System TestsRequirementsEvaluationTestEvaluationReports5

The Testing Process in YAVE ITEUser InterfaceITE’s user interface contains view modes for different tasks. Each ofthese view modes, in turn, is divided into information fields. Thesecan be arranged and configured to create a customized workspace.Requirements ManagementSince it helps to better capture the overall structure, ITE uses thehierarchical tree view mode for managing requirements. New requirementscan be created in ITE by using the integrated requirementseditor, while existing requirements can be imported fromdatabases like DOORS.Once created or imported, each requirement is assigned a uniqueID and listed as Version 1.0. This version label is then automaticallyincremented whenever a requirement is modified. At the same time,all previous versions of requirements are retained to be viewed orrestored at any time. It is also possible to mark requirements aspriority and to add pictures.Requirements and test means can then be easily linked by using the“V&V matrix”, which conveniently displays all available test meansalong with a full list of requirements. Once linked, test means appearin the “Test Means” field of the requirements editor.6

ITEThe Testing Process in YAVE ITETest Case Management/Test ExecutionThe test case editor allows users to create test cases which containall necessary data for test engineers to execute test runs. The hierarchicaltree view mode is used for this task as well since it offers anorganized division of test cases into documents, chapters and subchapters. Within each test case, there are sections for test definitionand test procedure. The test definition section is further divided intofields for description, initial conditions and linked requirements.In order to link test means to test definitions and to add initial conditiondefault sets, users simply drag and drop the desired portionsinto the test case editor. Requirements can then be linked to testcases in the “RQMT Test Definition dependencies” matrix.Though users are free to select among procedure languages (e.g.CCDL, C, etc.), CCDL is recommended since it enables users to takefull advantage of the powerful features of ITE (see CCDL White Paper).One of these ITE features is the built-in test case procedureeditor with syntax highlighting.Another helpful feature is the labeling of test cases as “suspicious”.Here, ITE alerts users when original requirements or test definitionshave been changed and requests that the test case be examinedcarefully. With the help of the “suspicious information view”, ITEthen guides users to the passages that require adjustment.7

The Testing Process in YAVE ITEExecution of a TestOnce test case setup is complete, a test run can be created. Theexecution of this test run depends on the procedure language. Fora CCDL procedure, the test run is sent to a test run managementserver and executed automatically. If, on the other hand, manualtest execution is preferred, ITE guides the user through each step ofthe process with the help of procedure forms.ITE also allows test cases to be organized into test campaigns andthen managed in the “Campaign Explorer” view.Logbook DatabaseManaging test campaigns for complex products with extensive requirementsinvolves high-capacity test benches with numerousparts. As a result, implemented hardware needs to be continuouslyupgraded and reconfigured in order to maintain efficient test operationsand avoid delays.The ITE Logbook Database supports the efficiency of configurationmanagement. Bench data sets contain parts allocations andcan be configured simply by drag and drop. Parts are also stored inparts catalogues with a unique identifier and a permanently loggedoperational history. Beyond this, a completed configuration is linkedto dedicated test runs, which means that any part used in any testexecution can be traced.8

ITEThe Testing Process in YAVE ITEEvaluation of Test ResultsOnce a test run has been executed, the results are available inITE and ready for evaluation. In the evaluation mode, ITE not onlyshows a summary of the test run (with, for example, test definition,test execution/evaluation, test procedure, etc.) but also evaluatesthe requirements. Users can then evaluate the requirements on thebasis of the test results.ReportingFollowing evaluation, the test run can be closed and a test run informationreport may be compiled. Users can either specify the metadata of the report in the report configuration or choose a templateand create the output file (e.g. PDF). If desired, users can also selectthe ITE approval process option, which requires that every report beconfirmed by internal authorities (“Final TI-Confirmation”).ITE reporting tools also allow users to create a single report foran entire test campaign, which summarizes test cases, test runs,requirements and evaluation results in one document.9

The Testing Process in YAVE ITEAdvanced SearchThe Advanced Search feature makes it easy to find specific informationsince it allows users to search for almost all elements, includinglinked/dependent elements and other attributes available in ITE.Pre-defined queries can also be saved on the ITE server as eitherprivate or public search templates.ITE also simplifies the process of creating a query by only showingthe user the applicable options in a context menu.10

ITEIntegrated Test Environment to Reduce Test EffortsTime to market remains one of the most critical aspects in modernproduct development. However, reducing test efforts in the validationstage can help to speed up the process.The use of ITE puts the test and validation process on a new level.Working in an integrated test environment keeps all data in oneplace. Beyond that, efforts to log and manage data as well as to createreports and keep everything up to date are reduced to a minimum.Benefits at a Glance• By reducing manual tasks to a minimum, the verificationprocess becomes scalable, transparent and less susceptibleto errors.• Using CCDL, the high-level test definition language, testprograms can be run on almost any test bench, eliminatingthe need for native code programming such as C or Python.• Staff training is only needed for one environment and oneeasy-to-learn test definition language.• A common test definition language means higher flexibilityand better utilization of test rigs and staff.• Automated evaluation and reporting save a vast amountof time. The reporting features in ITE easily create commonlyaccepted reports for authorities.Recent FTI customer projects reveal that ITE can bring time savingsof over 50% to the validation process. At the same time, it reduceseffort and improves the utilization of test equipment.11

ContactFTI GroupSchmiedestr. 215745 Wildauphone +49 (0)3375 5235 0fax +49 (0) 3375 5235 150mailwebinfo@ftigroup.netwww.ftigroup.netin co-operation withLegal NoticeThe contents of this white paper are intended to provide informationonly and to show possible examples. We reserve the right tochange technical data and design at any time without prior notice.The technical specifications of customer systems and of our currentproducts have to be clarified when ordering. All trademarks andregistered trademarks mentioned in this document are the propertyof their respective owners. Date 05/2012Copyright ©FTI Group / FTI_DB_000012 / 002