A Successful Strategy for Satellite Development and Testing - Inpe

and environment for testing the attitude control system, providingsolar illumination in vacuum for fully extended solar arrays, andfitting large systems into a relatively confined vacuum chamber).But even in light of these acknowledged limitations, Aerospacefound an increasing trend away from applying flight-like testingmethodologies where they were previously considered routine,such as in unit and subsystem performance testing and softwarecompatibility with hardware in the loop. Similarly, intersegmenttesting between the ground and space segments was often eliminatedentirely or greatly reduced in scope.The decision to omit or scale back these tests must be accompaniedby a clear assessment of the attendant risks. Where thereis significant risk exposure by not being able to test appropriately,mitigation strategies can be developed early in the life cycle.Aerospace found these risks were not always well understood, andconsequently, mitigation strategies were not effectively applied.Aerospace found that best practices for flight-like testing hadnot been codified in the industry. There was a general lack ofpractical guidance for determining how well or poorly the testingwas conducted. This was particularly true for “day in the life” operationaltesting.Traditionally, issues and problems uncovered during satellitedevelopment and testing would result in design and processchanges, which would in turn be scrutinized for insights thatvery often improve the development and verification process. Asa third-party observer, Aerospace could look across contractorboundaries and identify key lessons and practices, which couldthen be used to help prioritize the reintroduction of industry-widespecifications and standards. With the cancellation of these standardsin the mid-1990s, contractors were left on their own to accommodatetechnological changes and lessons learned into theirown processes—with variable success.A Leap in ComplexityWhile verification rigor had dropped, overall satellite complexityrose, often exponentially, as a result of advances in electronicstechnology and software. Not only were these systems usingmore parts, but the parts themselves were often far more complex,requiring much more stringent design verification and qualificationpractices. The greater use of field-programmable gate arrays(FPGAs) and application-specific integrated circuits (ASICs),with millions of embedded transistors on a single device, poses aneven greater testing challenge.Not only does increasing complexity pose a challenge tothe verification process, but it also implies an increase in thelikelihood of latent design and workmanship defects. Given theincreases in complexity, the corresponding pressures on the verificationprocesses, and the increased failure potential, the industryand government had embarked on a path of conflicted logic thatresulted in numerous problems that were often not detected untillate in development cycle, or even on orbit.Under acquisition reform, the government did not always specifyrequirements for qualifying the parts used in space systems.The manufacturers assumed responsibility for piece-part qualification,based on the application and the performance requirementsat the system level. This led to problems for several reasons.Acquiring qualified parts had become more difficult as suppliersfocused on commercial markets at the expense of the militaryspace market (which, although relatively small, typically requiresCumulative success rate (percent)Environmental test thoroughness1009080706050TraditionalacquisitionpracticesHigher-riskacquisitionpractices401980 1985 1990 1995 2000Year of launchOut of a sample of more than 450 vehicles manufactured in the United States,those developed using traditional acquisition practices show a consistentlyhigher success rate in the first year of operations. In contrast, vehicles developedusing higher-risk acquisition approaches show markedly lower successrates in the first year of operations.U.S. space asset loss (millions of dollars)100908070605030002500200015001000500Pre - 1995NRO programsPost - 1995Landsat 6NOAA 13UFO-1MarsObserverSMC programsAcquisition reform in the national security space arena reduced verificationrigor, as illustrated in the drop in environmental test thoroughness.3 Titan IV’sMilstar/NRO/DSP2 Delta III’sMars Climate Orbiter090 91 92 93 94 95 96 97 98 99YearThis chart shows the value in dollars of U.S. space assets lost during the 1990s.Recent independent studies have shown that reducing technical verificationrigor and diminishing the role of independent technical oversight in the developmentof government and commercial space systems results in greater problems,as evidenced by higher failure rates and cost and schedule overruns.

A Return to StandardsThe Air Force Space and Missile Systems Center (SMC) and the National ReconnaissanceOffice (NRO) have established policies that embrace the use of government, industry, andprofessional society specifications and standards to define program technical baselines.The effort includes the processes for the evaluation, selection, and preparation of documentsand also the processes and ground rules for implementation as compliance documentsin requests for proposals and contracts.Aerospace plays an integral role in the review of existing technical standards, the developmentand publication of new standards in several engineering disciplines, and the implementationof standards in the acquisition process for new systems. Aerospace, NRO, andSMC compiled a list of the key standards and have kept the list updated and publishedas an Aerospace technical report. Eventually, appropriate documents will be revised andreissued as military, industrial, or international standards. For example, five Aerospacestandards were recently issued as AIAA standards. In the meantime, Aerospace technicalreports will be used as compliance documents.—Valerie Lang, Joe Meltzer, and Jacqueline Unitisstricter parameter control, higher reliability,wider temperature ranges, higher dynamicresponse, radiation hardness, and similartraits). In addition, as suppliers switchedfrom a product qualification model to aprocess qualification a model, both primarycontractors and government lost insightand traceability into parts because supplierswere not required to provide technicaldata for qualification and traceability. Thegovernment had even less insight, with fewerpeople to track problems and less oversightinto manufacturing details.Cost and schedule assumed a greater rolein determining which tests and analysesshould be used to demonstrate that a devicewas acceptable and could meet system requirements.Because of inadequate resourcesand shifting priorities, only new or problematicsuppliers were evaluated or closelymonitored. Verification of compliance wasless disciplined for subtier contractors, andthe prime contractor’s role changed from“right of approval” to “right of rejection.”Flight software complexity had increasedeven more, and it is now statistically impossibleto find all possible defects in large softwaresystems. Despite continuing advances,debugging code remains time-consuming:up to 50 percent of a programmer’s timecan be spent debugging code. Furthermore,testing requires a test plan, detailed testprocedures, and scripts for providing inputto an automated testing tool—an effort thatcan be just as prone to error as the code itpurports to test. Altogether, complex softwareentails meticulous verification planningand software development, a challenge thatis not addressed in development and budgetallocations. This underscores the need for aReemerging Part SpecificationsBased on a number of recent problems, experience clearly indicates that a more stringentand consistent approach to parts, materials, and processes—including qualification—mustbe followed. One major objective is to establish a revised standard that defines the necessarycharacterization, qualification, and screening tests for microelectronics and otherpiece-part commodities (e.g., hybrids, capacitors, resistors, relays, and connectors) thatwould clear them for use in space applications. This includes government participation toensure that risks are not solely quantified on cost and schedule, but life performance aswell. A rewrite of technical requirements for space parts, materials, and processes is underway based on previously existing military standards.rigorous independent assessment of interrelatedsoftware and hardware requirementsdevelopment early in the process.Today’s satellite systems involve multipleuser nodes. The increasing number and complexityof interfaces led to a rise in interfaceproblems during system-level and end-toendtesting among ground, user, and spacesegments. These complex interfaces presenta challenge to simulation tools and limit theaccuracy of design-margin predictions andverification by use of models and simulations.A Breakdown inSystems EngineeringIn addition to finding problems with verificationand testing, the Aerospace study identifiednumerous problems with systems engineeringpractices, including source selection,requirements definition and flowdown,system design, engineering requirementverification, manufacturing and integrationsupport, and scheduling.Data analyzed pointed to a number of systemsengineering deficiencies that resulted innumerous late-build-cycle problems, highlightedby the large increase in design flaws(detected in system-level testing) since 1995.Specific deficiencies include marginalizingthe peer design review process and relateddocumentation, descoping preliminary andcritical design processes, and marginalizingthe risk management process. In general,Aerospace found that systems engineeringprocesses were fragmented.Several additional systems engineeringchallenges were also discovered—mostnotably, personnel shortfalls, flawed assumptionsregarding the insertion of commercialproducts in a given design, less emphasison achieving flight-like testing, and greateremphasis on cost and schedule versus performanceand reliability.Spacecraft are extremely complex, andprogram managers have always felt pressureto reduce costs and head count. Coupledwith the aging demographics of the spaceindustry workforce, the pressure to minimizestaffing levels had decimated governmentand contractor systems engineering teams—sometimes depleting teams from five or sixdeep to one individual who may not haveenough technical breadth to understand thepotential impact of design issues and themany problems that occur during production.This increased the chances that design errorswould go unidentified (and uncorrected) untilthey caused a failure. The lack of personnelalso led to a reduction in oversight of the8 • Crosslink Fall 2005

prime contractors by the government and ofthe subcontractors by the prime contractors.This increased the likelihood that problemscaused by streamlined design and verificationprocess changes at one level would notbe communicated to another.Another common shortfall in systemsengineering and verification planning involvedoverly optimistic assumptions aboutthe use of commercial off-the-shelf (COTS)or heritage components. In many cases, thedeveloper assumed that a commercial orheritage product was suitable for a new applicationwithout giving sufficient scrutiny tothe intended design use conditions. In reality,commercial or heritage products almostalways require more modifications thanexpected, and this adversely affects programschedule. Sometimes, problems with theseproducts were overlooked until they causedcostly failures in ground testing or even onorbit because assumptions regarding thesuitability of the original design to the newapplication’s actual design environment andoperational scenarios did not pan out.A Get-Well Road MapThe Aerospace study concluded with a seriesof specific recommendations for the nationalsecurity space community. In particular, acquisitionmanagers must:• strictly adhere to proven conservativedevelopment practices embodied in bestof-classspecifications and standards;• apply rigorous systems engineering, includingdisciplined peer design reviewsand clearly traceable verification processes;• emphasize requirements verification andtesting of all hardware and software,focusing on the early development phaseand lower-level unit design;• apply updated and consistent softwaredevelopment and verification processes,including meaningful metrics;• instill effective closed-loop design andcommunication processes, with specialattention to new technology insertion,application of COTS components, anddetailed assessment of operational dataand lessons learned;• strengthen the qualification and verificationof parts, materials, and processes;• develop a pyramidal and flight-like requirementsverification policy and assessthe risk of deviations from this policy;The Testing HandbookAerospace is developing a comprehensive test and evaluation handbook that will codifybest practices for planning a successful qualification and acceptance test strategy. Thishandbook will deal with the up-front planning and production phase as well as orbitalcheckout. The list of guidelines includes lessons learned from the study, such as:• Base schedules on realistic and executable models that account for system productionmaturity, reasonable levels of integration returns, and realistic problem resolution.• Plan a test program that implements a pyramidal requirements verification approach.• Test all high-power electronic units, including RF hardware, in thermal vacuum priorto system-level testing.• Ensure a conservative retest philosophy on all anomalous hardware that accounts forfatigue life from prior test exposures.• Develop an EMI/EMC control plan that ensures pertinent EMI/EMC testing at unitand appropriate levels of assembly and always conduct a system-level EMI/EMC testprior to the thermal vacuum test.• Perform early interface and harness compatibility checks on all hardware, preferablyduring development.• Plan for intersegment testing at the spacecraft level of assembly and include all flighthardware.• Plan for a rigorous verification and checkout of ground support equipment well inadvance of qualification testing.• Plan for a disciplined anomaly tracking and resolution process that determines rootcause on all anomalies and includes all factory, subcontractor, launch-base, andoperational anomaly data.• develop a set of engineering handbookswritten from the perspective of the systemprogram office;• manage the product life-cycle data withinthe system program office and across theenterprise and learn from it.When these practices are applied togetherthroughout development, they have historicallyresulted in successful program acquisitionsand mission success. Recommendationsfrom major government review panelsare largely consistent with Aerospace conclusionsregarding the proper application ofindustry best practices and lessons learned.Moreover, the Aerospace study providesdetailed evidence as to why national security,long-life, space acquisition—and morepointedly, the verification process—requiresa different approach than that of a purelycommercial space program. As a result, acquisitionleaders are once again emphasizinga more traditional, proven, and disciplinedapproach to engineering space systems.One critical part of such an approachis to ensure that appropriate specificationsand standards are applied on a given contract.Specifications and standards arisefrom an often painful and costly evolutionaryprocess, and in a sense, they form theembodiment of decades of lessons learnedand best practices. These specifications,standards, and guidelines therefore form thecornerstone of traditional best practices thathelp ensure successful execution of a satelliteprogram. Realizing this, Aerospace hasalready helped introduce revised and newnational security space standards for spacesystems development, which draw upon thepreviously canceled military standard withenhancements to bring them up to date withcurrent best practices.Additional best practices related to asuccessful qualification and acceptance teststrategy will be defined in a comprehensivetest and evaluation handbook under developmentat Aerospace. In addition, Aerospace isdeveloping and publishing handbooks thatCrosslink Fall 2005 • 9

General Design and Verification Former CurrentTest Requirements for Launch, Upper Stage, and Space Vehicles MIL-STD-1540C SMC-TR-04-17 1Test Method Standard for Environmental Engineering Considerations and Laboratory Tests MIL-STD-810F MIL-STD-810FSoftware Development Standard for Space Systems MIL-STD-498 Aero. TOR-2004(3909)-3537, Rev. ATest Requirements for Ground Equipment and Associated Computer Software Supporting SpaceVehiclesMIL-STD-1833MIL-STD-1833Verification Requirements for Space Vehicles — Aero. TOR-2005(3901)-4243 2Domain-Specific Design and Verification Former CurrentElectromagnetic Compatibility Requirements for Space Systems MIL-STD-1541A Aero. TOR-2005(8583)-1Electromagnetic Emissions and Susceptibility, Requirements for the Control of EMI MIL-STD-461D Aero. TOR-2005(8583)-1Wiring Harness, Space Vehicle, Design and Testing DoD-W-8357A DoD-W-8357ASpace Battery Standard — Aero. TOR-2004(8583)-5, Rev. 1Qualification and Quality Requirements for Space-Qualified Solar Panels — AIAA S-112-2005 1Electrical Power Systems, Direct Current, Space Vehicle Design Requirements MIL-STD-1539A Aero. TOR-2005(8583)-2Moving Mechanical Assemblies for Space and Launch Vehicles MIL-A-83577C AIAA S-114-2005 1Space Systems - Structures Design and Test Requirements — AIAA S-110-2005 1Space Systems - Flight Pressurized Systems MIL-STD-1522A Aero. TOR-2003(8583)-2896Space Systems - Metallic Pressure Vessels, Pressurized Structures, and Pressure Components MIL-STD-1522A AIAA S-080-1998 1Composite Overwrapped Pressure Vessels MIL-STD-1522A AIAA S-081-2000 1Solid Motor Case Design and Test Requirements — Aero. TOR-2003(8583)-2895, Rev. 1Criteria for Explosive Systems and Devices Used on Space Vehicles DoD-E-83578A AIAA S-113-2005 1Mass Properties Control for Satellites, Missiles, and Launch Vehicles MIL-STD-1811 Aero. TOR-2004(8583)-3970Part-Level Design and Verification Former CurrentQualification and Quality Standards for Space-Qualified Solar Cells — AIAA S-111-2005 1Electronic Parts, Materials, and Processes Control Program Used in Space Programs MIL-STD-1547B Aero. TOR-2004(3909)-3316, Rev. A1. These specifications have been converted to AIAA specifications based on Aerospace technical domain expertise submitted to AIAA technical committees via Aerospace TORs.2. Not included on SMC compliance list. Included with NRO standards.This table lists the specifications and standards that are being introduced orreintroduced into the national security space acquisition process that relate specificallyto design and verification requirements. General design and verificationspecifications and standards typically apply to multiple levels of assembly andinclude some discussion of requirements oriented toward the integrated systemlevelarchitecture. Domain-specific standards are oriented toward environmental,functional, or hardware type testing and include requirements for batteries, solarpanels, mechanisms, and structures. Part-level standards typically focus on spacerelateditems. For details, see Aero. TOR-2003(8583)-2, Rev. 4, “Systems EngineeringRevitalization Specifications and Standards Implementation Plan and Status.”provide technical rationale, methodology,and tailoring guidance for mission assuranceand space vehicle systems engineering.SummaryThe findings of the Aerospace study arehelping spur national security space initiativesto establish more disciplined systemsengineering, verification, and missionassurancestrategies. The assessment of developmentpractice changes, together with ananalysis of on-orbit and factory test failures,provided a greater degree of insight into theeffectiveness of the integration and testingprocesses, the critical role of the systemsengineering process, and the sensitivity ofdesign and verification processes to the consequenceof acquisition policy change. Thestudy also shed new light on the relationshipsamong test parameters, levels of assemblytested, test effectiveness, test-related fatigue,and the resulting influence on cost, schedule,and mission success.Successful space systems in the past adheredto a rigorous requirements flowdownprocess that was tied to a comprehensive anddisciplined verification process that ensuredeach requirement was properly verified andtraceable to a specific test, analysis, or inspectiondocument. By reemphasizing verificationand testing at the lowest level andtesting under flight-like conditions, the governmentis underscoring the importance ofapplying technical rigor in areas where conflictingand often marginally successful verificationmethods were being applied becauseof the lack of paradigmatic specifications andstandards. Systems engineering and missionassurance revitalization initiatives are wellattuned to the urgency to correct the lapsesin the acquisition strategy and have consolidatedefforts to accelerate development ofa common and technically relevant set ofspecifications, standards, and best practicesfor all national security space programs.10 • Crosslink Fall 2005

Environmental Testingfor Launch and Space VehiclesSpace systems must endure a physically stressful journey from thelaunchpad to their final destinations. Adequate testing can helpensure they survive the trip.Erwin Perl, Thinh Do, Alan Peterson, and John WelchThe structural design of space systems is dictated bythe rigors of the liftoff and ascent environments duringlaunch as well as the extreme thermal conditionsand operational requirements of spacecraft equipment andpayloads on orbit. At liftoff and for the next several seconds,the intense sound generated by the propulsion system exertssignificant acoustic pressure on the entire vehicle. This pressureinduces vibration, externally and internally, in the space vehiclestructures. In addition, the vehicle experiences intense vibrationsgenerated by engine ignitions, steady-state operation,and engine shutdowns as well as sudden transients or “shocks”generated by solid rocket motor jettison, separation of stagesand fairings, and on-orbit deployments of solar arrays and payloads.Space vehicles will also experience wide fluctuations intemperature from the time they leave the launchpad to the timethey settle into orbit. Both individually and in combination, themechanical environments of pressure, vibration, shock, andthermal gradients impose design requirements on many structuralcomponents. Ensuring the survivability of the delicatehardware poses challenges that can be met only by extensivepreflight tests encompassing acoustic, shock, vibration, andthermal environments.Environmental testing is performed at varying magnitudesand durations to verify the design of space systems and toscreen flight hardware for quality of workmanship. The firststep in this process is the definition of the maximum expectedenvironments during launch and on-orbit operation. Data fromprevious flights and ground tests are analyzed to generate predictionsfor a specific mission. These environments are thenflowed down from the space vehicle level to the various subsystemsand components for use as design requirements and, later,as test requirements.Aerospace performs a crucial role for the government inensuring that these environments are properly defined and thedesign qualification tests and the hardware acceptance tests areproperly planned and carried out. By reviewing test requirementsand analysis methodologies, for example, Aerospacehelps verify that the results will be accurate and meaningful.Reviewing the maximum predicted environments ensures thatspace systems are designed to withstand the rigors of flight.Reviewing test plans helps develop perceptive test procedures.Observing the tests builds confidence that they were conductedaccording to specification. Reviewing the test data provides anindependent validation of the results. Archiving and catalogingtest data helps test planners ensure that test methods reflect thecurrent state of the art. And of course, by observing test anomalies,Aerospace retains relevant lessons for future programs in acontinuous cycle driving toward improved reliability of spacesystems.Acoustic TestingA principal source of dynamic loading of space vehicles occursduring liftoff and during atmospheric flight at maximumdynamic pressure. It is caused by the intense acoustic pressuregenerated by turbulent mixing of exhaust gases from the mainengines and rocket motors with the ambient atmosphere.This acoustic excitation starts when the main engine is ignitedand lasts approximately 3 to 6 seconds. Ignition producesan exhaust plume that exerts acoustic pressure on the launchpadand reflects back to the space vehicle to induce vibration.The magnitude of the exhaust plume and the amount of pressureit exerts depends on factors such as engine thrust, exit velocity,engine nozzle diameter, location of structures, and ductconfiguration. As the speed of the launch vehicle increases, therelative velocity between the vehicle and the ambient atmospheregenerates fluctuating pressures in a turbulent boundarylayer between the exterior surface and the atmosphere. As thevehicle traverses the speed of sound, the so-called region oftransonic flight, and shortly thereafter, the region of maximumdynamic pressure, the airflow together with aerodynamicshock waves that attach, oscillate, and reattach cause acousticexcitations comparable to liftoff, but with different frequencycharacteristics. The sound pressure and its induced vibrationare random in character. The spectra used to assess damage

sometimes used to derive test levels. The predictedacoustic environment is adjusted usingstatistical methods to derive a maximum predictedflight environment. Margin is addedto ensure that the hardware is sufficientlyrobust and to account for analytical uncertaintiesin the derivation of the environmentand design of the hardware. A typical qualificationmargin is 6 decibels, or four timesthe energy of the maximum predicted environment.The test lasts at least 1 minute toestablish a duration margin of four times theexposure in flight. Additional test time maybe accumulated depending on the programrequirements. Hardware that is susceptible tothe acoustic-pressure loading are items withlarge surfaces and low mass density such ascomposite material solar arrays and antennareflectors. These composite structures mayhave design or workmanship deficiencies,which result in bond or material failures.Vibration TestingAs the launch vehicle lifts off from the standand throughout powered flight, the vibrationcaused by the operating engines excitesthe vehicle and spacecraft structure. Additionalvibration is caused by the fluctuatingacoustic pressure experienced during liftoff,transonic flight, and the maximum-dynamicpressurephase of flight.Vibration testing helps demonstrate thathardware can withstand these conditions.Random vibration tests are conducted onan electrodynamic vibration machine or“shaker,” which consists of a mounting tablefor the test item rigidly attached to a drivecoilarmature. A control system energizesthe shaker to the desired vibration level.Feedback for the control system is providedby a series of accelerometers, which aremounted at the base of the test item at locationsthat correspond to where the launchvehicle adapter would be attached. Twocontrol approaches can be used to provide realisticstructural responses. Most spacecraftvibration tests use response-limiting majorappendageaccelerations to reduce input atdiscrete frequencies so as not to cause unrealisticfailures. For test structures that exhibitdistinct, lightly damped resonances on ashaker, force limiting is used in conjunctionwith input vibration to control the shaker.In the force-limiting approach, transducersthat measure the input force are mounted betweenthe test item and the shaker. The goalis to reduce the response of the test item at itsresonant frequencies on the shaker to replicatethe response at the combined system atSound pressure levels(decibels)140130120110100908010 100 1000 10,0001/3 Octave frequency (hertz)Maximum predicted acoustic level for liftoffMaximum predicted acoustic level for transonic/max QMaximum workmanship requirement for acoustic testTypical acoustic test level is a smooth envelope plus 6 dB marginthe resonant frequencies that would exist inthe flight-mounting configuration.As in the case of acoustic testing, heritageflight and test data are used to predict vibrationtest levels, and analytical methods aresometimes used to develop transfer functionsto scale heritage data to new hardware configurations.In most cases, the predicted environmentsare verified later with system-levelacoustic tests and rocket engine static firetests. As with acoustic testing, a 6-decibelmargin is typically added to the maximumpredicted environment. Structural failures ofpiece parts, unit assemblies, and secondaryand primary space vehicle structures can anddo occur from vibration-induced stress andmaterial fatigue. Failures of inadequately designedor poorly manufactured or assembledstructural interfaces are commonly revealed.Aerospace personnel, using predictivesoftware, provide analysis confirmation foroptimal instrumentation for vibration testing.Aerospace confirms hardware test perceptivenessand effectiveness with analysis,Power spectral density1.000000.100000.010000.001000.00010Typical acoustic test levelused to simulate thelaunch vehicle environment.The spectrum isdivided into 1/3-octavebands, and the soundpressure level is specifiedfor each band in decibels.The frequency range istypically from 30 to 10,000hertz.testing experience, and consideration ofinterface constraints.Shock TestingStage, fairing, and vehicle separations areoften accomplished by means of pyrotechnicdevices such as explosive bolts, separationnuts, bolt cutters, expanding-tube separationsystems, clamp bands, ordnance thrusters,and pressurized bellows. When activated,these devices produce powerful shocks thatcan damage equipment and structures. Thecharacteristics of these shocks depend onthe particular separation mechanism, butthe energy spectrum is usually concentratedat or above 500 hertz and is measured in afrequency range of 100 to 10,000 hertz. Atypical shock response spectrum plot is usedto gauge the damage potential of a givenseparation event.Separations or deployments generatebrief impulsive loads even if no pyrotechnicdevices are used. Nonexplosive initiatorsmay produce significant shock levels simply0.0000110 100 1000Frequency (hertz)10,000Maximum predicted vibration level for liftoffMaximum predicted vibration level for transonic/max QMaximum workmanship requirement for vibration testTypical vibration test level is a smooth envelope plus 6 dB marginTypical vibration testlevel used to simulatethe launch vehicle environment.A 6-decibelqualification margin istypically added to themaximum predictedenvironment to ensurethat the hardware is sufficientlyrobust.Crosslink Fall 2005 • 13

Measured accelerationtime histories are usedto derive shock test requirements.Shock levels are specifiedas shock responsespectra defined over afrequency range. Theshock response spectrauses the responseof single-degree-offreedomoscillators,computed in 1/6 octavebands to convertthe time history to thefrequency domain.300200100–100–200through the release of structural strain. Experiencehas shown that shock can induce ahard or intermittent failure or exacerbate alatent defect. Commonly encountered hardwarefailures include relay transfer, crackingof parts, dislodging of contaminants, andcracking of solder at circuit-board interfaces.Unit-level shock tests are accomplishedusing one of several methods, which generallyentail securing the component to afixture that is then subjected to impact. This“ringing plate” approach has provided thebest practicable simulation of unit exposureto shock. In addition, vibration shakers areused in some applications to impart a transientshock. Shock testing is typically notTemperatureFFHS, FFCS, FF0–300552.50 552.55 552.60 552.65 552.70 552.75 552.8010 310 210 1 10 310 2 10 4AFAFAFCS, FFperformed as a unit workmanship screen,but is deferred to the system level for greaterdetection of functional defects. System-levelshock tests usually activate the separationor deployment systems, providing a directsimulation of the mission event. Thus, theydo not include any amplitude margin. Testfixtures are used to support hardware thathas been deployed or separated to preventsubsequent contact or damage. System-levelshock tests provide an excellent opportunityto measure shocks incident on componentsthroughout the space vehicle.Accurate prediction of high-frequencyshock levels, such as those associated withexplosive ordnance, remains an elusive goal.HS, FFTimeFFA typical thermal cyclingor thermal vacuumtest profile. Theprofile shows temperaturehistory and alwaysstarts and ends at roomtemperature. Hot starts(HS), cold starts (CS),full functional tests (FF),and abbreviated functionaltests (AF) are performedat temperatureplateaus.Therefore, it is important that the shock environmentbe assessed during the developmentphase of the program through both analysisand test simulations. Shock analysis includesconsideration of the source amplitudes,durations, transmission paths, path materials,and path discontinuities. Developmenttests employ an accurate replica of the flightstructure with all significant constituentssimulated. Deployed hardware is forced tophysically separate at least a small amountto provide realistic shock transmission paths.When practical, a shock-producing event isrepeated several times to permit meaningfulstatistical evaluation of the resultingdata. Qualification margins at the unit levelare typically 6 decibels on amplitude andtwice the number of flight activations. At thesystem level, it is generally impractical toimpose an amplitude qualification margin;however, a margin of two or three activationsis imposed. Aerospace provides expertise forthe prediction of test levels and the configurationof the hardware interfaces to achievean effective test.Thermal TestingLaunch vehicles and spacecraft must endurea wide range of temperatures associated withliftoff and ascent through the atmosphere,direct impingement of solar radiation, andtravel through the extreme temperatures ofspace. The thermal environment is generallyconsidered the most stressful operating environmentfor hardware in terms of fatigue,and it has a direct bearing on unit reliability.For example, the use of materials with differingcoefficients of thermal expansionhas resulted in unsuccessful deploymentsof mechanical assemblies and payloads.Outgassing increases significantly withtemperature, and the resulting contaminantswill more readily adhere and chemicallybond to colder surfaces. Electronic parts areespecially sensitive to the thermal conditionsand are subject to problems such as cracks,delamination, bond defects, discoloration,performance drift, coating damage, andsolder-joint failure.Thermal testing is used to screen outcomponents with physical flaws and demonstratethat a device can activate and operatein extreme and changing temperatures. Thefour most common thermal tests are thermalcycling, thermal vacuum testing, thermalbalance testing, and burn-in testing. Thermalcycling subjects the test article to a numberof cycles at hot and cold temperatures in anambient-air or gaseous-nitrogen environ-14 • Crosslink Fall 2005

ment; convection enables relatively rapidcycling between hot and cold levels. Thermalvacuum testing does the same thing, but in avacuum chamber; cycles are slower, but themethod provides the most realistic simulationof flight conditions. In thermal balancetesting, also conducted in vacuum, dedicatedtest phases that simulate flight conditions areused to obtain steady-state temperature datathat are then compared to model predictions.This allows verification of the thermal controlsubsystem and gathering of data for correlationwith thermal analytic models. Burnintests are typically part of thermal cycletests; additional test time is allotted, and theitem is made to operate while the temperatureis cycled or held at an elevated level.For electronic units, the test temperaturerange and the number of test cycles have thegreatest impact on test effectiveness. Otherimportant parameters include dwell time atextreme temperatures, whether the unit isoperational, and the rate of change betweenhot and cold plateaus. For mechanical assemblies,these same parameters are important,along with simulation of thermal spatialgradients and transient thermal conditions.Thermal test specifications are based primarilyon test objectives. At the unit level,the emphasis is on part screening, whichis best achieved through thermal cycle andburn-in testing. Temperature ranges aremore severe than would be encountered inflight, which allows problems to be isolatedquickly. Also, individual components areeasier to fix than finished assemblies.At the payload, subsystem, and spacevehicle levels, the emphasis shifts towardperformance verification. At higher levels ofShock response spectra10,00010001001010.1100 1000Frequency (hertz)10,000Maximum predicted shock level for fairing separationMaximum predicted shock level for stage separationMaximum predicted shock level for spacecraft separationTypical shock test level is a smooth envelope plus 6 dB marginassembly in flight-like conditions, end-toendperformance capabilities can be demonstrated,subsystems and their interfacescan be verified, and flightworthiness requirementscan be met. On the other hand, at thehigher levels of assembly, it is difficult (if notimpossible) to achieve wide test temperatureranges, so part screening is less effective.At the unit, subsystem, and vehicle levels,Aerospace thermal engineers work with thecontractor in developing test plans that provethe design, workmanship, and flightworthinessof the test article. Temperature rangesare selected that will adequately screen or accuratelysimulate mission conditions, and theproper number of hot and cold test plateausare specified to adequately cycle the testequipment. Aerospace will provide expertiseduring the test to protect the space hardwarein the test environment, resolve test issuesand concerns, and investigate test articlediscrepancies. The reason, of course, is thatTypical test level usedto simulate the shockenvironment. Qualificationmargins at theunit level are typically6 decibels.identifying and correcting problems in thermaltesting significantly increases confidencein mission success.ConclusionSince the first satellite launch in 1957, morethan 600 space vehicles have been launchedthrough severe and sometimes unknown environments.Even with extensive experienceand a wealth of historical data to consult,mission planners face a difficult task in ensuringthat critical hardware reaches spacesafely. Every new component, new process,and new technology introduces uncertaintiesthat can only be resolved through rigorousand methodical testing. As an independentobserver of the testing process, Aerospacehelps instill confidence that environmentalrequirements have been adequately definedand the corresponding tests have been properlyplanned and executed to generate usefuland reliable results.Left: A spacecraft is placed in the acoustic chamber and is ready for testing. Air horns at the corners of thechamber generate a prescribed sound pressure into the confined space and onto the spacecraft. Microphoneslocated around the spacecraft are used to monitor and control the pressure levels. Middle: The sudden separationof the payload fairing is used to expose spacecraft components to the shock environment expected inflight. Right: Space instrument placed on an electrodynamically controlled slip table for vibration testing. Thecontrol accelerometers are mounted at the base of the test fixture at a location that represents the interface tothe launch vehicle adapter. Accelerometers mounted on the test specimen measure the dynamic responses.Crosslink Fall 2005 • 15

Software Testingin Space ProgramsAs space-system software grows in size and complexity, adequate testingbecomes more difficult—and more critical.Myron Hecht and Douglas BuettnerPhoto courtesy of European Space Agency; Ada code from http://www-aix.gsi.de/~giese/swr/ariane5.html...declarevertical_veloc_sensor: float;horizontal_veloc_sensor: float;vertical_veloc_bias: integer;horizontal_veloc_bias: integer;...begindeclarepragma suppress(numeric_error, horizontal_veloc_bias);beginsensor_get(vertical_veloc_sensor);sensor_get(horizontal_veloc_sensor);vertical_veloc_bias := integer(vertical_veloc_sensor);horizontal_veloc_bias := integer(horizontal_veloc_sensor);...exceptionwhen numeric_error => calculate_vertical_veloc();when others => use_irs1();end;end irs2;The Ariane 5 launch vehicle failed on its maiden flight in June1996. About 40 seconds after liftoff, a software bug in theflight controller made the rocket veer off course, leading to itsdestruction via ground command. Ariane 5 reused softwarefrom Ariane 4 without proper testing. Contributing to themishap, run-time range checking had been turned off becauseof processor limitations. Also, the backup channel had failedmilliseconds earlier because of the same coding defect.Failures attributed to software defects are becoming increasinglyvisible in space systems. Recent newsworthy examples include thefailure of the Mars rover Spirit to execute any task that requestedmemory from the flight computer; the unanticipated descent of the MarsClimate Orbiter into the Martian atmosphere, ultimately traced to a unitconversion defect in a navigation system; and the crash of the Mars PolarLander onto the Martian surface after a premature shutdown of its descentengines. In 1996, the first launch of the Ariane 5 booster ended with aspectacular crash off the coast of French Guiana. The cause was traced toa variable overflow that affected software running in both channels of itsdual redundant inertial reference system. Earlier this year, the EuropeanSpace Agency’s Huygens probe successfully beamed back only half ofits image data. The other half was lost because of a single missing line ofcode.In the period from 1998 to 2000, nearly half of all observed spacecraftanomalies were related to software. Anomalies, less severe than failures,have been occurring with increasing frequency on U.S. national securityspace vehicles. One reason is that space-system software has been growingmore complex to meet greater functional demands. Another reason isthat software quality is inherently difficult to determine. The challenge indeveloping the next generation of national security space vehicles will beto ensure reliability despite increasing software size and complexity. Softwaretesting is an important factor in meeting this challenge.Types of Software TestingSoftware testing methods generally fall into two categories: “black box”and “white box” (while some authors also identify a third category, the“ticking box,” which involves not doing any testing).Black-box methods disregard the software’s internal structure andimplementation. The test data, completion criteria, and procedures aredeveloped solely to test whether the system meets requirements, withoutconsideration of how the software is coded. Black-box testing is used at alllevels of testing and is particularly applicable at higher levels of integration,where the underlying components are no longer visible.White-box testing, on the other hand, does account for the internalsoftware structure in the formulation of test cases and completion criteria.The most common types of white-box testing include branch testing,

which runs through every instruction in eachconditional statement in a program, and pathtesting, which runs through every set of conditionalstatements or branches. White-boxtesting is typically conducted at the unit level(i.e., the smallest testable component of software)and at the unit integration level.Both methods would typically includesome sort of nominal testing, in which testcases are designed to mimic normal operation,and negative testing, in which test casesare selected to try and “break” the program.For example, the software might be runusing input values of the correct type andwithin the expected range to verify conformancewith nominal requirements. It mightalso be run using input values and data ratesbeyond expected ranges to check failsafe andrecovery capabilities.The Testing ProgramWhite-box and black-box testing is performedwithin the context of an overallsoftware test program that starts during therequirements phase and continues throughproduct release and maintenance. Softwaredevelopment standards provide a basis fordefining the activities of the overall test program.Although the use of such standardsdeclined in the 1990s, they are now increasinglyrecognized as an important way tohelp ensure software quality despite risingcomplexity.For example, the National ReconnaissanceOffice (NRO) and the Air Force Spaceand Missile Systems Center (SMC) recentlyasked Aerospace to recommend a set ofsoftware development standards to be usedas compliance documents on NRO and SMCcontracts. Aerospace assisted with a detailedsurvey of existing life-cycle standards andrecommended the use of MIL-STD-498 orits commercial equivalent, J-STD-016-1995.However, MIL-STD-498 was canceled inthe mid-1990s, and J-STD-016 is no longerbeing maintained by the technical organizationsthat produced it. Therefore, SMCand NRO felt that a new software standardshould be developed.Aerospace helped analyze MIL-STD-498 in greater detail and identified waysto modernize J-STD-016. Based on thiseffort, Aerospace prepared a new standard,published as Aerospace Report No. TOR-2004(3909)-3537, “Software DevelopmentStandard for Space Systems.” It uses MIL-STD-498 as a foundation, but incorporatesadditional requirements from J-STD-016.It also adds exit criteria for various levelsOn-orbit anomalies per vehicle8765432102vehiclesYear Ranges1981 - 19951986 - 19901991 - 19951996 - 20002001 - 6/0310vehicles12vehicles1985 1990 1995 2000 >20005-Year Incrementsof software testing and requirements thatbring the standard up to date with modernterminology and best practices in softwaredevelopment.Many software development standards,including MIL-STD-498 and the Aerospacerevision, set forth requirements for threemajor activities of software testing: planning,definition, and execution.Software test planning addresses all levelsof coding and integration, from the highestlevelsoftware package down to the lowestlevelsoftware units. The results are documentedin a software test plan. Lower-leveltest plans are independently created if thesoftware’s size and complexity warrants it.The software test plan enables the programmanager to assess the adequacy of test planningfor each of the software items and forthe software system qualification testing. Inaddition, the software test plan lists the issuesthat should be considered in the developmentof the software test definition.In the test definition stage, the test preparations,test cases, and test procedures areall described and documented. This mayinvolve a significant design and developmenteffort—in some cases, equal to or exceedingthat of the software itself. This is particularlytrue for software item qualification testing,in which individual software componentsare accepted for integration into the system.Software item qualification testing is criticallydependent on the accuracy of the softwaretest definition.Once the test definition has been completed,it is possible to actually run the testsand record the results in the software testreport. As part of this process, the test organizationshould emphasize findings and observationsof anomalies. The software test reportcan also include suggestions for furthertesting based on the limitations of the testequipment or limitations arising from budgetor time constraints. The software test report5vehicles3 vehiclesAnomaly trend attributedto software in five-yearincrements from the firstthree years of the spacecraft’soperation usingavailable failure data froma wide range of satellitecategories.documents the test results and includes accumulatedtest analyses, results, summaries,deviations from dry runs, and metrics.Limitations of Software TestingDespite its obvious importance, softwaretesting is only a partial solution to creatingreliable software. In a sense, the purpose oftesting is to show that a program has bugs.Thus, while it can provide a means to findand fix defects, it cannot by itself provide anassurance of failure-free operations. Softwaretesting must be pursued in conjunctionwith other appropriate practices in systemsengineering, requirements definition, andsoftware development (such as inspection,the use of automated development aids, staticsource code and design analysis, and peerreview).A significant limitation is that softwaretesting cannot occur until after the code iswritten—about halfway or more throughproject development. The cost of fixingerrors rises dramatically as the project progressesbecause more deliverables are affected.For example, requirements errors cost10 times more to fix in the code phase than inthe requirements phase. Methods of softwareverification other than testing (under thebroad categories of inspection, analysis, ordemonstration) must be used to catch errorsin the earlier phases of design.A related limitation is that the effectivenessof a testing program is no better than therequirements on which it is based. Aerospaceanalysis has shown that the generation ofsoftware requirements is a major source oferrors in system development. Specific challengesinclude poorly stated requirements,changing or “creeping” requirements, andnonfunctional requirements. A study ofrequirements-originated software failuresshowed that roughly half resulted frompoorly written, ambiguous, unclear, andincorrect requirements. The rest came from32 • Crosslink Fall 2005

Method and Description Objective Test Type Applicable LevelScenario-based (also called thread) testing: Testing using data basedon usage scenarios, e.g., simulation of the mission.Requirements-based testing: Testing to assess the conformance of thesoftware with requirements.Nominal testing: Testing using input values within the expected rangeand of the correct type.Stress testing (a type of negative testing): Testing with simulated levelsbeyond normal workloads, or starving the software of the computationalresources needed for the workload; also called workload testing(usually run concurrently with endurance tests).Robustness testing (a type of negative testing): Testing with values,data rates, operator inputs, and workloads outside expected ranges.Boundary-value testing (a type of negative testing): Testing the softwarewith data at and immediately outside expected value ranges.Extreme-value testing (a type of negative testing): Testing for largevalues, small values, and the value zero.Random testing: Testing the software using input data randomly selectedfrom the operational profile probability distribution.Fault-injection testing: Testing on the nominal baseline source codeand randomly altered versions of the source (white box) or object code(black box).Branch testing: Testing using test cases selected to test each branch atleast once.Path testing: Testing using test cases selected to test each path (i.e.,feasible set of branches) at least once. Also called flow-graph testing.Modified-condition decision coverage: Every point of entry and exitin the program has been invoked at least once, every condition in adecision in the program has taken all possible outcomes at least once,every decision in the program has taken all possible outcomes at leastonce, and each condition in a decision has been shown to independentlyaffect that decision’s outcome.Typical black-box and white-box test methods.Assess overall conformance anddependability in nominal usage.Determine whether the softwaremeets specific requirements.Verify conformance with nominalrequirements.Measure capacity and throughput;evaluate system behavior underheavy loads and anomalous conditionsto determine workload levelsat which system degrades or fails.Challenge or “break” the systemwith the objective of testing failsafe and recovery capabilities.Test error detection and exceptionhandling behavior of software withanticipated exception conditions.Same as boundary-value testing.Assess overall stability, reliability,and conformance with requirements.Assess failure behavior, ensure thatsystem properly responds to componentfailures.Test correctness of code to thelevel of branches.Test correctness of code to thelevel of paths.Test for safety-critical softwarewhere a failure would probably oralmost inevitably result in a lossof life.Black box.Black box.Black box.Black box.Black andwhite box.Black andwhite box.Black andwhite box.Black box.Black andwhite box.White box.White box.White box.Integrated software andsystem.All levels at which requirementsare defined.All.Integrated software andsystem.All.Unit, software subsystem.Unit, software subsystem.Integrated system.Integrated software.Software unit.Software unit.Software unit (assemblycode created by compilerunder some circumstances).requirements that were completely omitted.Most problems introduced into software canbe traced directly to requirements flaws.An additional limitation is the difficulty—and hence the time, cost, and effort—ofsoftware testing. Ideally, a software systemcould be exhaustively tested and therebyproven correct. However, this is impossiblefor all but the simplest systems. Manyspace-system software applications are socomplex, and run in such an interdependentenvironment, that complete testing can neverbe achieved. Instead, program managersmust prioritize their testing objectives andoptimize their testing procedures to ensurethat the most important tests are completed.Skill in risk analysis is therefore essentialfor establishing an appropriate test coverageobjective—usually stated as a proportion ofthe requirements, input data, instructions, orprogram paths tested (e.g., testing is completewhen the tests addressing 100 percentfunctional coverage of the system have allexecuted successfully).Proper selection of input data can increasethe testing efficiency by either increasingthe error-detection effectiveness or reducingthe number of test cases needed to achieve agiven test coverage objective. For example,tests can be partitioned to exercise the samecode using only one representative case. Thenumber of test cases for each class of failurebehavior can be limited. If software inspectionis used in the development process, thedistribution of defects (by category) detectedby inspection can be used to drive the distributionof test data. The amount of coupling(intermodule referencing of variablesor subroutines) can be used to focus testcases—particularly if a significant amount ofsoftware changes have been made. Test casescan also be concentrated on areas exhibitingan abnormally high number of failures. Testcase input data can also be selected using a“design of experiments” approach.How Much Testing is Enough?Considering that complete test coverage isgenerally not possible, project managersface a difficult question in deciding when tostop testing. In practice, this decision is oftenbased not on specific and quantifiable goalsbut on deadlines, budgets, or completion ofan arbitrary number of test runs.For national security space systems, a bettercriterion would be the point at which thesoftware reaches an acceptable level of reliability,as measured in time between failures.This method, often referred to as softwarereliability engineering, is a recommendedpractice by the American Institute of Aeronauticsand Astronautics (AIAA).Crosslink Fall 2005 • 33

The fundamental premise of software reliabilityengineering is that the rate at whichsoftware defects are found and removed canbe described mathematically and thereforepredicted. These discovery and removal ratescan be constant or variable, depending onthe models used. If the testing environmentsimulates the operational environment, thenfailure rates observed at any point in the testwould be similar to the operational failurerates, and the model would enable a predictionof the future failure rate as the testingprogram proceeded. They would thereforeprovide an ability to predict the software’sfuture reliability.Software reliability engineering originatedin the 1970s and has been the subjectof extensive research since that time. Toolshave been developed to fit various models totest data to enable determination of the bestfit and subsequent extrapolation to enableprediction. Software reliability engineeringprovides a cost-effective method to determinewhen to stop testing. Cost typicallyranges from 0.1 to 3.0 percent of project developmentcosts.To help improve the accuracy and valueof these prediction models, Aerospace hasbeen working to develop a database schemafor software reliability data. The project,Space Systems Mission Assurance via SoftwareReliability Monitoring, will correlatesoftware life-cycle engineering practices(including test) with the reliability measuredfrom deployed space-systems software. Aneventual goal is to provide a risk-assessmenttool for program managers that will allowthem to compare key software life-cyclemetrics and test practices from their programto historical data from other programs. Thedatabase is being designed to support threetypes of analyses: exploratory, quantitative,and qualitative. Exploratory analysis wouldallow users to investigate relationships thatcould be used to predict software and systemreliability based on project, structural, andtest program attributes. Quantitative analysiswould allow users to extract event data topredict software reliability. Qualitative analysiswould allow users to address questionssuch as what are the major failure causes, effects,or developmental problems.Safety-Critical SoftwareAlthough software reliability engineeringcan benefit many types of software, specialconsiderations must be made for safetycriticalsoftware—the failure of which canlead to death, major injury, or extensiveActivitiesSoftware Test PlanningSoftware TestDescriptionSoftware Test Executionand ReportingKey software test issues.Key Issuesproperty damage. A good example is thesoftware supporting the Global PositioningSystem (GPS). An undetected failure inthe navigation signal from any of the GPSsatellites might result in an aircraft receivingmisleading information on its position oraltitude, thereby exposing its occupants to ahigh risk of a crash landing. Thus, the softwarecomponents involved in integrity monitoring,which would detect and announce anavigation signal failure, must receive specialscrutiny.Test organization, including personnel, responsibilities, discrepancy reportingrequirements, and release processes.Budget and schedule requirements, test schedule estimates, milestones, anddeliverables.Plans for maintaining and updating test plans, test cases, test environment, andautomated tools through the life cycle.Strategy for changes in the requirements and software items (in particular, regressiontesting).Testing of commercial or nondevelopmental item software.Particular equipment, procedures, methods, or data necessary to address the requirementsof the specific program for which the plans are developed.Completion criteria.Software requirements addressed by the test.Test driver environment (interfacing hardware, software, communications, etc.).Automated testing tools (record/playback tools, coverage analyzers, test tracking,etc.).Test completion criteria.Means of evaluating correctness of results (test method).Test tracking, logging, and archiving processes.Test setup steps.Metrics for the reporting of results.Retest criteria.Test input data requirements.Evaluation of test case data to assess success or failure.Procedures to undertake when an anomaly occurs to capture the circumstancessurrounding the failure.Overall assessment of software tested.Identification of deficiencies.Problem reports filed.Test environment version, constraints, etc.Recommendations for improvement.Deviations from procedures for each test case.Details for analysis required to document the pass/fail conclusion.Who executed the tests.Who witnessed the tests.Where the test results are archived.When were the test cases executed.Aerospace is supporting the GPS programoffice in producing high-integrity softwarefor the next-generation GPS constellation.For safety-critical software, testing is partof a process of analysis, documentation,and traceability that starts at the beginningof the project and continues throughout thesystem lifetime. For example, when requirementsare being formulated, a preliminaryor functional hazard analysis is performed toidentify major hazards and develop mitigationstrategies. At the design phase, two more34 • Crosslink Fall 2005

system-safety analyses are performed todetermine the safety impact of the softwarecomponents in their normal and failed states.For critical software components, verification,testing, and documentation must be performedintensively. For example, in aviationapplications, the RTCA DO 178B standardprovides for testing of all combinations ofconditions in branches in such software.Even intensive testing has the same limitationdiscussed earlier: it can only prove thepresence of defects in software, not their absence.Thus, Aerospace and other organizationsare researching methods that use mathematicaltechniques to prove the correctnessof the specification, the verification test suite,and the automatic code generators that createthe software. The goal is to use formalmethods and testing together to significantlydecrease development time while producingdependable software.ConclusionWith the addition of progressively more softwarefunctionality in both space and groundsegments, program managers will facetougher challenges in ensuring software reliability.Software testing efforts will requirebetter analytical methods and oversight approachesto meet the greater demand withoutadversely affecting budgets and schedules.By participating in software test planningand data analysis, reviewing softwaredevelopment standards and practices, and byperforming research on software reliability,Aerospace is helping to make the softwaretesting process more efficient and effective.The results of this research should augmentsoftware-intensive system acquisition practiceswith tools to help program managersensure mission success.Further ReadingAerospace Report No. TOR-2004(3909)-3537,“Software Development Standard for Space Systems.”(The Aerospace Corporation, El Segundo,CA, 2004)AIAA/ANSI R-013-1992, Recommended Practice:Software Reliability, American Institute ofAeronautics and Astronautics (Reston, VA).P. Cheng, “Ground Software Errors Can CauseSatellites to Fail too—Lessons Learned,”Ground Systems Architecture Workshop (ManhattanBeach, CA, March 4, 2003); availablefrom http://sunset.usc.edu/gsaw/gsaw2003/agenda03.html (last visited April 29, 2005).G. Durrieu, C. Seguin, V. Wiels, and O. Laurent,“Test Case Generation Guided by a CoverageCriterion on Formal Specification,” IEEE InternationalSymposium on Software ReliabilityEngineering (ISSRE, Nov. 2004).120100Total Failures8060402000 5 10 15 20 25 30 35 40 45 50Test Interval25020015010050NHPP Mean Value Function=m(t)=a(l -e-bt)a=Expected cumulative number of errorsb=Error detection rateRaw DataNHPP (intervals)Schneidewind: all0 0 5001000Test Interval NumberJ. T. Harding, “Using Inspection Data to ForecastTest Defects,” Crosstalk (May 1998);available at http://www.stsc.hill.af.mil/crosstalk/frames.asp?uri=1998/05/inspection.asp(lastvisited January 19, 2005).K. Hayhurst, et al., “A Practical Tutorial onModified Condition/Decision Coverage,” NASATM-2001-210876 (NASA Langley ResearchCenter, May 2001); available at http://techreports.larc.nasa.gov/ltrs/PDF/2001/tm/NASA-2001-tm210876.pdf (last visited May 10, 2005).M. Hecht and H. Hecht, “Digital System SoftwareRequirements Guidelines,” NUREG/CR-6734, Vol. I, Office of the Chief InformationOfficer, U.S. Nuclear Regulatory Commission(Washington, DC, 2001).C. Kaner, “An introduction to Scenario-BasedTesting,” available at http://www.testingeducation.org/articles/scenario_intro_ver4.pdf(lastvisited, January 22, 2005).D. Leffingwell and D. Widrig, Managing SoftwareRequirements (Addison Wesley, Longman,Reading, MA, 1999).b=0.8b=0.4b=0.2b=0.1This graph shows thebenefit of error-detectioneffectiveness under the assumptionthat the defectdetection can be modeledas a nonhomogenousPoisson process (NHPP). Asthe proportion of defectsremoved per test case orinterval moves from 0.2 to0.8, the number of test intervalsneeded to remove80 percent of the defectsgoes from 8.03 down to2.01.This figure shows the outputof a software reliabilitymodeling tool called CASRE(Computer Aided SoftwareReliability Estimation) developedat CalTech/JPL.Two of the models, thenonhomogenous Poissonprocess (NHPP) model andthe Schneidewind model,closely fit the cumulativedefect history curve fromsystem testing for a flightsoftware project. The bluepart of the curve displaysthe end of data bar andthe failure prediction resultstwo weeks into thefuture.S. McConnell, “Gauging Software Readinesswith Defect Tracking,” IEEE Software, Volume14, Issue 3, p. 135 (May-June 1997).J. Musa, Software Reliability Engineering (Mc-Graw Hill, New York, 1998).D. R. Wallace, “Is Software Reliability Modelinga Practical Technique?” 2002 SoftwareTechnology Conference, available at http://www.stc-online.org/stc2002proceedings/SpkrPDFS/ThrTracs/p411.pdf (last visited January 19,2005).M. C. K. Yang, A. Chao, “Reliability Estimationand Stopping Rules for Software Testing,Based on Repeated Appearances of Bugs,” IEEETransactions On Reliability, Vol. 44, No. 2, p.315 (June 1995).U.S. Department of Defense, Military Standard,Software Development and Documentation,December 1994, available from http://diamond.spawar.navy.mil/498/mil-498.html; also availablein a commercial variation as EIA/IEEE J-STD-016 from http://standards.ieee.org.Crosslink Fall 2005 • 35