Project: Error, Fault, and Failure Data
Collection and Analysis

PROJECT LEADER: Dolores R. Wallace, Software Diagnostics and Conformance Testing Division; 301-975-3340 dwallace@nist.gov

GOALS: To help industry assess software system quality by collecting, analyzing, and providing error, fault, and failure data and providing statistical methods and tools for analysis on software systems.

BRIEF DESCRIPTION: We are acquiring error, fault and failure data to develop profiles for industry use and for statistical analysis methods; developing handbook of faults with detection and prevention methods; developing software tools for industry to collect & analyze data; will provide public data base and analysis tools on WEB. Summary of current data

IMPETUS/HOW PROJECT BEGAN: Industry and researchers have been asking for reference data on software failures for some time. There are some small collections of data for individual companies and agencies, but no comprehensive collection is available. Requests from industry and researchers for reference data of this kind from NIST indicated the need for a database.

OBJECTIVES: To help industry protect against releasing software systems with faults and to help assess software system quality by providing statistical methods and tools for analysis of software systems. To develop techniques for measuring effectiveness of software methods. To provide profiles against which developers can test their software.

1^st Qtr FY99 - Prototype public data base capability
1^st Qtr FY99 - Prototype Fault Handbook
4^th Qtr FY99 - Enhancements to data base capability
4^th Qtr FY99 - Additional chapters handbook
2^nd Qtr FY00 - Analytic capabilities
3^rd Qtr FY00 - Additional chapters handbook
all Qtrs           - Collect, analyze data

STAFF:
Dolores Wallace, Neva Carlson, Dr Larry Reeker, Tamer Ahmed, Michael Koo and Roger Gima

OUTSIDE INVOLVEMENT: SoHaR, Inc under CRADA; RST Corporation under Advanced Technology Project(FY97-98); pending Memorandum of Agreement with NASA IV&V Facility; contracts with University of Louisville, RST Corporation, Software Engineering Institute; non-disclosure agreements with six companies, others pending; continuing advice from researchers, industry

WHAT WORK HAS ITL DONE: We are acquiring error, fault and failure data to develop profiles for industry use and for statistical analysis methods; developing software tools for industry to collect & analyze data; will provide public data base and analysis tools on WEB.

WHAT WORK HAS BEEN DONE BY COLLABORATORS: Assistance in specifying requirements for the data base and associated tools. Data provided by industry and government

• Wallace, Dolores R and D. Richard Kuhn, Converting System Failure Histories into Future Win Situations,
• Wallace, Dolores R. and D. Richard Kuhn, Lessons Learned from 342 Failures of Medical Devices,
• Wallace, Dolores R., Enhancing Competitiveness via a Public Fault & Failure Data Repository [postscript], The 3rd IEEE High-Assurance Systems Engineering Symposium, IEEE Computer Society, November 13-14, 1998, Washington, DC
• Wallace, Dolores R. and Mark LaRoy Zimmerman, "The EFFective Manager Tool," NISTIR 6137, Gaithersburg, MD April 1998.
• Hecht, Herbert and Dolores R. Wallace, "Toward More Effective Testing for High Assurance Systems", High Assurance Systems Engineering Conference, Washington, DC, 8/1997.
• Wallace, Dolores R., Laura M. Ippolito, Herbert Hecht, "Error, Fault, and Failure Data Collection and Analysis," Quality Week 97, San Francisco, May 27-30, 1997.

Tools

• EFF Fault Tool, WWW-based Data Collection and Analysis Tool For Tutuorial, Download, and Demo
• EFF public facility (public data collection, fault handbook, the data collection tool, links to other data)

Presentations

• Project fault/failure data for risk assessment, Workshop on Risk Assessment, NASA IV&V Facility, October 26-27, 1998
• Improving Software Process with Fault Analysis,Philadelphia Chapter Software Process Improvement Network, October 8, 1998
• 14th International Test Conference, 1997 Tyson's Corners,VA
• "Error, Fault, and Failure Data Collection and Analysis," Quality Week ‘97, San Francisco, May 27-30, 1997.
• COMPASS 97, Gaithersburg, MD
• 16th International Testing Conference, "What Can We Do to Improve Testing?" Orlando, FL, November 14, 1996

Industry: baseline of development practices and industry benchmarks; better use of test resources; better selection of tools to build; reference methods. Researchers: access to data, & knowledge of relevant problems.

ACKNOWLEDGMENTS OF ITL'S EFFORTS: In addition to the collaborators listed above, several companies have agreed to provide data for the database because they view the results as valuable for the quality improvement efforts. As of 12/31/1998, commitments have produced data from over 10 projects. More are expected in early 1999.

RELATED ACTIVITIES: We are developing sets of software with known errors that can be used to evaluate the effectiveness of testing tools and methods. Also, we will be providing urls for related work and for tools for analyzing fault data.