A Process Research Framework - Software Engineering Institute ...

7available to “the rest of us,” and sometimes it adds a level of complexitythat we as professional systems developers cannot yet cope with (think oftrying to de-skill nuclear power plant operations).7.4.1 Characterizing the Current State of the PracticeFor system contexts in which incomplete knowledge is (or will be) prevalent,today’s component- and system-build approaches are a poor fit, becausethey force systems to drive to certainty. We often push for certaintybefore understanding an adequate set of contexts—for use, for test beds,or for configurations. We do not know what data is safe to ignore or simplifyin different configurations or system states. We have trouble characterizinghow much data is enough to collect in relation to making processand other choices. Processes that help to assess the probability that whatis “known” is actually “true” are not in widespread use.In addition, when faced with a set of data or an analysis result, techniqueswe currently use are insufficient for characterizing the completenessof our knowledge about the data. Many of the standard analysistechniques (e.g., FMEA, the failure modes and effects analysis) do nottranslate well into an environment of incomplete system knowledge(i.e., where you do not know if the component at the end of a fault treeis actually working or not working). Quantitative assurance with an explicitprobabilistic focus is not an accepted mode in the test and evaluationcommunity, even though it’s a necessary mode in systems of highuncertainty, especially where there is minimum tolerance for failure.Therefore, release processes based on quantitative assurance are neithersufficient nor adequately used.7.4.2 Characterizing the Desired State of the PracticeWhen we have systems that are characterized by continuing, incompleteknowledge, we would use system building approaches that effectivelydeal with low probability but high impact risks. We would consistentlyuse processes that identify clearly the boundaries of complete and incompleteknowledge about operational scenarios, system state, and systemconfiguration. Approaches would be well understood and used forcollecting the “right” data (that which cannot be ignored or simplified)in different operational contexts. Quantitative assurance with an explicitprobabilistic focus would be an accepted mode in the test and evaluationcommunity. When making release decisions, we would use releaseprocesses based on quantitative assurance results. We would also haveprocesses for managing configurations that effectively deal with unboundedand incomplete contexts.Overall, we would have processes that drive to certainty only when thetime is right, and the knowledge needed is available. And we would78 Process Effects of Emerging Technology Trends | IPRC Framework