2014DOTEAnnualReport

FY14 DOT&E Activity and oversightFor programs with upcomingOT events in the next threeyears, I found that slightlymore than one-third (15 of42) of the programs currentlydo not exhibit performanceproblems significant enoughto jeopardize successfulperformance in OT. Table 9shows these results by type ofproblem and phase of testing.Upcoming pre-IOT&E testevents are far more likely tobe delayed to correct problemscompared to both upcomingIOT&E and FOT&E. Infact, Table 9 shows that forthe programs covered in thisAnnual Report, there wereno upcoming FOT&E eventsthat were delayed to correctPrograms with upcoming OT eventsCategoryNo problems forupcoming OTProblems havedelayed upcoming OTProblems have notdelayed upcoming OTOther problemsthreaten upcoming OTTable 9. programs Commencing OT within the Next Three YearsNumber of Number and Type ofPhase of Testing (program count)Programs 1 Problems (program count) 2Pre-IOT&E IOT&E FOT&E15 4 5 6719Effectiveness 11 3 0 0Suitability 9 4 2 0Cybersecurity 3 5 0 0 0Total 5 2 0Effectiveness 2 7 4Suitability 1 5 4Cybersecurity 3 1 1 1Total 4 8 64 2 2 01. Forty-two programs will undergo an OT in the next three years. The number of programs adds up to more than 42 becausesome programs have problems that delayed their upcoming OT as well as problems that did not delay OT.2. The number of programs summed across type of problems adds up to more than the number of programs because someprograms have multiple problems or more than one type of problem..problems. Of the programs for which potential problems exist for upcoming OT events, 5 of 7 of the pre-IOT&E events weredelayed to address at least one issue, 2 of 10 of the IOT&E events were delayed, and 0 of 6 of the FOT&E (or post-IOT&E)events were delayed.For programs that have not delayed their upcoming OT and have known problems, the distribution between effectivenessand suitability problems is about the same (13 compared to 10), with three cybersecurity problems. Table 10 expands furtherCategoryTable 10. Actions taken to Address Problems for Upcoming OT Eventsfor which OT has not (yet) been DelayedHow has the problem been addressed to date (program count)?pre-IOT&E IOT&E FOT&EType of ProblemFix NotIdentifiedFixDocumentedFiximplemented(tested)Fix NotIdentifiedupon programs thathave not delayedtheir upcomingOT and shows thatFOT&E events areconsiderably morelikely to have a fiximplemented (and inmany cases tested)going into the OT,regardless of type ofproblem, comparedto both pre-IOT&Eand IOT&E. Thedata are currently insufficient to determine whether the differences between the rate of implementing and testing fixes prior toFOT&E compared to IOT&E is a trend or simply random.Specific Programs that have upcoming OT in the next three yearsNo problems for upcoming OTFifteen of the 42 programs with upcoming OT events have not yet exhibited problems considered to significantly jeopardizeperformance in upcoming OT events. Such programs could be examples where the program development process, includingDT and OT, appears to be moving along well. Alternatively, the testing to date might not have been sufficiently stressing tosurface any problems. These programs are listed below in Table 11.FixDocumentedFiximplemented(tested)Fix NotIdentifiedFixDocumentedFiximplemented(tested)upcoming OT 1 Cybersecurity 1 0 0 0 0 1 (0) 0 0 1 (1)Problems haveEffectiveness 2 0 1 (0) 4 4 2 (1) 2 0 2 (2)not delayed Suitability 0 1 0 4 1 1 (0) 0 1 3 (2)1. Numbers in parentheses are the number of problems that have already tested fixes prior to the upcoming OT.24 Problem Discovery Affecting OT&E