A Successful Strategy for Satellite Development and Testing - Inpe

More documents

Recommendations

Info

The fundamental premise of software reliabilityengineering is that the rate at whichsoftware defects are found and removed canbe described mathematically and thereforepredicted. These discovery and removal ratescan be constant or variable, depending onthe models used. If the testing environmentsimulates the operational environment, thenfailure rates observed at any point in the testwould be similar to the operational failurerates, and the model would enable a predictionof the future failure rate as the testingprogram proceeded. They would thereforeprovide an ability to predict the software’sfuture reliability.Software reliability engineering originatedin the 1970s and has been the subjectof extensive research since that time. Toolshave been developed to fit various models totest data to enable determination of the bestfit and subsequent extrapolation to enableprediction. Software reliability engineeringprovides a cost-effective method to determinewhen to stop testing. Cost typicallyranges from 0.1 to 3.0 percent of project developmentcosts.To help improve the accuracy and valueof these prediction models, Aerospace hasbeen working to develop a database schemafor software reliability data. The project,Space Systems Mission Assurance via SoftwareReliability Monitoring, will correlatesoftware life-cycle engineering practices(including test) with the reliability measuredfrom deployed space-systems software. Aneventual goal is to provide a risk-assessmenttool for program managers that will allowthem to compare key software life-cyclemetrics and test practices from their programto historical data from other programs. Thedatabase is being designed to support threetypes of analyses: exploratory, quantitative,and qualitative. Exploratory analysis wouldallow users to investigate relationships thatcould be used to predict software and systemreliability based on project, structural, andtest program attributes. Quantitative analysiswould allow users to extract event data topredict software reliability. Qualitative analysiswould allow users to address questionssuch as what are the major failure causes, effects,or developmental problems.Safety-Critical SoftwareAlthough software reliability engineeringcan benefit many types of software, specialconsiderations must be made for safetycriticalsoftware—the failure of which canlead to death, major injury, or extensiveActivitiesSoftware Test PlanningSoftware TestDescriptionSoftware Test Executionand ReportingKey software test issues.Key Issuesproperty damage. A good example is thesoftware supporting the Global PositioningSystem (GPS). An undetected failure inthe navigation signal from any of the GPSsatellites might result in an aircraft receivingmisleading information on its position oraltitude, thereby exposing its occupants to ahigh risk of a crash landing. Thus, the softwarecomponents involved in integrity monitoring,which would detect and announce anavigation signal failure, must receive specialscrutiny.Test organization, including personnel, responsibilities, discrepancy reportingrequirements, and release processes.Budget and schedule requirements, test schedule estimates, milestones, anddeliverables.Plans for maintaining and updating test plans, test cases, test environment, andautomated tools through the life cycle.Strategy for changes in the requirements and software items (in particular, regressiontesting).Testing of commercial or nondevelopmental item software.Particular equipment, procedures, methods, or data necessary to address the requirementsof the specific program for which the plans are developed.Completion criteria.Software requirements addressed by the test.Test driver environment (interfacing hardware, software, communications, etc.).Automated testing tools (record/playback tools, coverage analyzers, test tracking,etc.).Test completion criteria.Means of evaluating correctness of results (test method).Test tracking, logging, and archiving processes.Test setup steps.Metrics for the reporting of results.Retest criteria.Test input data requirements.Evaluation of test case data to assess success or failure.Procedures to undertake when an anomaly occurs to capture the circumstancessurrounding the failure.Overall assessment of software tested.Identification of deficiencies.Problem reports filed.Test environment version, constraints, etc.Recommendations for improvement.Deviations from procedures for each test case.Details for analysis required to document the pass/fail conclusion.Who executed the tests.Who witnessed the tests.Where the test results are archived.When were the test cases executed.Aerospace is supporting the GPS programoffice in producing high-integrity softwarefor the next-generation GPS constellation.For safety-critical software, testing is partof a process of analysis, documentation,and traceability that starts at the beginningof the project and continues throughout thesystem lifetime. For example, when requirementsare being formulated, a preliminaryor functional hazard analysis is performed toidentify major hazards and develop mitigationstrategies. At the design phase, two more34 • Crosslink Fall 2005
system-safety analyses are performed todetermine the safety impact of the softwarecomponents in their normal and failed states.For critical software components, verification,testing, and documentation must be performedintensively. For example, in aviationapplications, the RTCA DO 178B standardprovides for testing of all combinations ofconditions in branches in such software.Even intensive testing has the same limitationdiscussed earlier: it can only prove thepresence of defects in software, not their absence.Thus, Aerospace and other organizationsare researching methods that use mathematicaltechniques to prove the correctnessof the specification, the verification test suite,and the automatic code generators that createthe software. The goal is to use formalmethods and testing together to significantlydecrease development time while producingdependable software.ConclusionWith the addition of progressively more softwarefunctionality in both space and groundsegments, program managers will facetougher challenges in ensuring software reliability.Software testing efforts will requirebetter analytical methods and oversight approachesto meet the greater demand withoutadversely affecting budgets and schedules.By participating in software test planningand data analysis, reviewing softwaredevelopment standards and practices, and byperforming research on software reliability,Aerospace is helping to make the softwaretesting process more efficient and effective.The results of this research should augmentsoftware-intensive system acquisition practiceswith tools to help program managersensure mission success.Further ReadingAerospace Report No. TOR-2004(3909)-3537,“Software Development Standard for Space Systems.”(The Aerospace Corporation, El Segundo,CA, 2004)AIAA/ANSI R-013-1992, Recommended Practice:Software Reliability, American Institute ofAeronautics and Astronautics (Reston, VA).P. Cheng, “Ground Software Errors Can CauseSatellites to Fail too—Lessons Learned,”Ground Systems Architecture Workshop (ManhattanBeach, CA, March 4, 2003); availablefrom http://sunset.usc.edu/gsaw/gsaw2003/agenda03.html (last visited April 29, 2005).G. Durrieu, C. Seguin, V. Wiels, and O. Laurent,“Test Case Generation Guided by a CoverageCriterion on Formal Specification,” IEEE InternationalSymposium on Software ReliabilityEngineering (ISSRE, Nov. 2004).120100Total Failures8060402000 5 10 15 20 25 30 35 40 45 50Test Interval25020015010050NHPP Mean Value Function=m(t)=a(l -e-bt)a=Expected cumulative number of errorsb=Error detection rateRaw DataNHPP (intervals)Schneidewind: all0 0 5001000Test Interval NumberJ. T. Harding, “Using Inspection Data to ForecastTest Defects,” Crosstalk (May 1998);available at http://www.stsc.hill.af.mil/crosstalk/frames.asp?uri=1998/05/inspection.asp(lastvisited January 19, 2005).K. Hayhurst, et al., “A Practical Tutorial onModified Condition/Decision Coverage,” NASATM-2001-210876 (NASA Langley ResearchCenter, May 2001); available at http://techreports.larc.nasa.gov/ltrs/PDF/2001/tm/NASA-2001-tm210876.pdf (last visited May 10, 2005).M. Hecht and H. Hecht, “Digital System SoftwareRequirements Guidelines,” NUREG/CR-6734, Vol. I, Office of the Chief InformationOfficer, U.S. Nuclear Regulatory Commission(Washington, DC, 2001).C. Kaner, “An introduction to Scenario-BasedTesting,” available at http://www.testingeducation.org/articles/scenario_intro_ver4.pdf(lastvisited, January 22, 2005).D. Leffingwell and D. Widrig, Managing SoftwareRequirements (Addison Wesley, Longman,Reading, MA, 1999).b=0.8b=0.4b=0.2b=0.1This graph shows thebenefit of error-detectioneffectiveness under the assumptionthat the defectdetection can be modeledas a nonhomogenousPoisson process (NHPP). Asthe proportion of defectsremoved per test case orinterval moves from 0.2 to0.8, the number of test intervalsneeded to remove80 percent of the defectsgoes from 8.03 down to2.01.This figure shows the outputof a software reliabilitymodeling tool called CASRE(Computer Aided SoftwareReliability Estimation) developedat CalTech/JPL.Two of the models, thenonhomogenous Poissonprocess (NHPP) model andthe Schneidewind model,closely fit the cumulativedefect history curve fromsystem testing for a flightsoftware project. The bluepart of the curve displaysthe end of data bar andthe failure prediction resultstwo weeks into thefuture.S. McConnell, “Gauging Software Readinesswith Defect Tracking,” IEEE Software, Volume14, Issue 3, p. 135 (May-June 1997).J. Musa, Software Reliability Engineering (Mc-Graw Hill, New York, 1998).D. R. Wallace, “Is Software Reliability Modelinga Practical Technique?” 2002 SoftwareTechnology Conference, available at http://www.stc-online.org/stc2002proceedings/SpkrPDFS/ThrTracs/p411.pdf (last visited January 19,2005).M. C. K. Yang, A. Chao, “Reliability Estimationand Stopping Rules for Software Testing,Based on Repeated Appearances of Bugs,” IEEETransactions On Reliability, Vol. 44, No. 2, p.315 (June 1995).U.S. Department of Defense, Military Standard,Software Development and Documentation,December 1994, available from http://diamond.spawar.navy.mil/498/mil-498.html; also availablein a commercial variation as EIA/IEEE J-STD-016 from http://standards.ieee.org.Crosslink Fall 2005 • 35
Page 2 and 3: and environment for testing the att
Page 4 and 5: prime contractors by the government
Page 6: Environmental Testingfor Launch and
Page 9 and 10: Measured accelerationtime histories
Page 11 and 12: Software Testingin Space ProgramsAs
Page 13: Method and Description Objective Te

A Successful Strategy for Satellite Development and Testing - Inpe

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?