next up previous contents
Next: References Up: No Title Previous: Using Testability to

Summary

This SBIR Phase II effort provided NIST with a methodology and a tool for predicting where faults will hide during testing and where assertions should be placed. Our deliverable tools work for the C++ and Java languages. To our knowledge, these tools are the only of their kind, and are based on nearly 10 years of basic research into why faults hide during testing. The key US Government organizations that have funded this research are: National Institute of Standards and Technology, NASA, National Science Foundation, and DARPA.

This methodology improves the testability for a fixed test suite. It is true that bigger test suites could be used as an alternative way to boost testability while ignoring assertions altogether. This issue has cost trade-offs that need further exploration to determine whether smaller test suites with assertions are better at revealing faults than larger test suites without assertions.

Generating additional test cases and embedding assertions are not the only verification and validation tricks that could be employed once it is known where testing is unlikely to be capable of detecting faults. Manual inspections or formal proofs that certain semantic constraints are true could also be applied to ensure that defects are not hiding.

Our conclusion that assertions are beneficial to software testing parallels the comments by Osterweil and Clarke concerning the value of assertions to testing; in their 1992 IEEE Software article, they classified assertions as ``among the most significant ideas by testing and analysis researchers'' [3]. From our previous work studying why faults hide during testing, we believe that we have provided insights into why assertions work well and how their placement can be made more systematic and practical.

We conjectured that assertions that are based on deficits in some testing suite D may still be valuable tools for improving the defect observability rates of another testing suite, . If generally true, this suggests that current research into which testing approach is better may be wasted effort, i.e., possibly any testing technique can be massaged into an excellent fault detector after assertions are instrumented to test untestable regions. Most research in software testing is geared toward finding some method K that will produce a test suite D for which . This suggests that when high fault detection is the goal, instead of looking for the ultimate K, derive assertions and generate additional tests, , such that . In summary, assertions and additional test cases (that ensure reachability) can transform your code from being untestable to testable. Let them.



next up previous contents
Next: References Up: No Title Previous: Using Testability to



Roger Gima
Tue Jan 20 16:43:17 EST 1998