Introduction

Software testing is generally performed for one of two reasons: (1) to detect the existence of defects (faults), and (2) to estimate the reliability of the code. Although there are other uses for software testing, most applications of it can be grouped into one of these two categories. Errors are mental mistakes made by programmers, and faults are the manifestation of those errors in the code.

Testing is considered effective when it uncovers defects. Testing will often be considered ineffective when no failures occur, because few people today consider defect-free code to be an achievable goal. Residual software defects not discovered during testing can have significant and dangerous consequences after the software is released. Since debugging is usually invoked only after software failure is observed, test schemes that provide a greater ability to cause software failure if defects are present are very desirable. Using software assertions in conjuction with testing is one way to make software faults more visible.

Software testability is a characteristic that either suggests how easy software is to test, how well the tests are able to interact with the code to reveal defects, or some combination of the two [8]. Describing how easy software will be to test is valuable information for project schedules and project cost estimation, but this information provides little insight into how well your test case generator is at creating ``defect-detecting'' test cases. Because of this deficiency, it is useful to define software testability as a measure of how well test cases make defects detectable. This will be the perspective taken throughout this report.

Using this definition of software testability, when software is assessed as having higher testability, it means that incorrect output will likely occur if a defect exists. To understand why faults hide during testing, it is necessary to know the sequence of events that must occur in order to observe incorrect output:

1. An input must cause a defect to be executed (or what is sometimes referred to as ``reached'').
2. Once the defect is executed, the succeeding data state must become corrupted. This data state is hence referred to as containing a data state error.
3. After a data state error is created, the data state error must propagate to an output state, meaning that incorrect output exits the software.

This report argues that by using heuristics that identify regions in which faults cannot be detected (by testing alone, i.e., without assertions), we can identify where additional validation efforts (which may include more testing or non-testing approaches) should be performed. We provide a methodology that is complementary to traditional testing and that reduces the possibility of defect-hiding in those code regions. Our approach can be thought of as a strategic assertion placement heuristic. Using this technique improves the likelihood of defect detection and hence improves the software's reliability.

Over the years, we have observed that although testers would like to use assertions to improve the quality of their testing process, they find that they do not know the code well enough to inject correct assertions. For this reason, assertions are primarily used by developers. However when developers are responsible for creating assertions, this increases the likelihood that if the code is wrong, the assertions that are derived will be wrong. Further, neither group really knows where to place assertions that are the most cost-effective. All of these problems should not be disheartening, because as you will see, a handful of correct and strategically-placed assertions can dramatically improve the quality of a finished software product.

Unfortunately, this report does not offer foolproof solutions to these problems, but it does provide additional reasons for why assertions must be used to improve the deficiencies of software testing (even if that requires deriving assertions from formal specifications). Interest in assertions has in fact become so great that several recent languages support assertion placement, including Anna [5] and Eiffel [7]. This report suggests how developers and testers can form a partnership that, if successful, can begin to alleviate several of the aforementioned problems.