This SBIR Phase II effort provided NIST with a methodology and a tool for predicting where faults will hide during testing and where assertions should be placed. Our deliverable tools work for the C++ and Java languages. To our knowledge, these tools are the only of their kind, and are based on nearly 10 years of basic research into why faults hide during testing. The key US Government organizations that have funded this research are: National Institute of Standards and Technology, NASA, National Science Foundation, and DARPA.
This methodology improves the testability for a fixed test suite. It is true that bigger test suites could be used as an alternative way to boost testability while ignoring assertions altogether. This issue has cost trade-offs that need further exploration to determine whether smaller test suites with assertions are better at revealing faults than larger test suites without assertions.
Generating additional test cases and embedding assertions are not the only verification and validation tricks that could be employed once it is known where testing is unlikely to be capable of detecting faults. Manual inspections or formal proofs that certain semantic constraints are true could also be applied to ensure that defects are not hiding.
Our conclusion that assertions are beneficial to software testing parallels the comments by Osterweil and Clarke concerning the value of assertions to testing; in their 1992 IEEE Software article, they classified assertions as ``among the most significant ideas by testing and analysis researchers'' [3]. From our previous work studying why faults hide during testing, we believe that we have provided insights into why assertions work well and how their placement can be made more systematic and practical.
We conjectured that assertions that are based on deficits in some
testing suite D may still be valuable tools for improving the defect
observability rates of another testing suite,
. If generally
true, this suggests that current research into which testing approach
is better may be wasted effort, i.e., possibly any testing technique
can be massaged into an excellent fault detector after assertions are
instrumented to test untestable regions. Most research in software
testing is geared toward finding some method K that will produce a
test suite D for which
. This suggests that
when high fault detection is the goal, instead of looking for the
ultimate K, derive assertions and generate additional tests,
, such that
. In
summary, assertions and additional test cases (that ensure
reachability) can transform your code from being untestable to
testable. Let them.