The creative idea of this project was to use testability scores (which are derived from a fault injection technique called sensitivity analysis) to suggest where assertions can find faults that a test suite cannot. Testability scores are composed of three factors: execution estimates, infection estimates, and propagation estimates.
After much research, and given our results from Phase I [4], the propagation estimates were determined to be the most crucial information for assertion placement. When you consider that this effort is focused on enhancing the testing process of object-oriented software, this makes intuitive sense. After all, tiny objects are very easily tested in isolation. It is not until they are combined with other tiny objects that we realize that if faults hide, they will hide by virtue of the fact that the corrupted states do not propagate properly. Hence propagation scores provide the most useful information to guide assertion placement.
Propagation scores measure the likelihood that an erroneous data value will result in erroneous program output. The higher the propagation score, the more likely it is that bad output will be produced, and hence it is more likely that regular testing will succeed in finding those types of faults. It is not until low testability scores are observed that concern should occur, since the lower scores indicate that testing is unlikely to find faults at those places. To compute the propagation score for a given program, variables are perturbed and the programs output is checked to see if the output of the perturbed program differs from the output of the unperturbed program.
Testers and developers can usually detect when a program produces corrupted output (if they cannot, the usefulness of testing is seriously brought into question). When programs do ``wear their faults on their sleeves'' in this manner, the need for additional assertions is reduced. The need for assertions occurs when programs do not have this property, and low propagation scores are observed. Low propagation scores suggest a much greater likelihood that testing will fail to detect existing faults, and thus by adding probes (assertions) into the software before the software is tested, we get a much clearer picture as to whether faults are present.
Examining propagation scores can aid a programmer in the placement of
ASSERT statements. The CBrowse utility that we have built as a part
of this effort can read propagation results and display them to the
user. CBrowse is capable of loading multiple files, and has been
modified to help the user place assertion statements. RST has also
created an Assertion Placement Wizard that aids the user in stepping
through propagation results loaded in CBrowse. The Wizard asks the user
to choose an upper bound for testability scores (the threshold). It
then allows the user to move between statements that have scores below
this threshold by clicking on either the Next or Previous buttons (See
Figure 
). The Wizard displays the score for the current
statement, and the assertion (if one exists). When all of the
relevant scores in the current file have been visited, the tool will
automatically load another file from the program and locate the first
testability score that is within the specified threshold. The Wizard
provides an easy means of navigating through source-code.
Figure: Assert Wizard for Placement.
The Assertion Edit Window (AEW) can be used in conjunction with the
Assertion Wizard to simplify the placing of ASSERT statements. The
AEW provides a GUI that allows the user to place
complex assertions with less effort (See
Figure ).
The user brings up the AEW utility by clicking on the Edit button of the Assertion Placement Wizard. The AEW provides lists of symbols, expressions, and relevant variables that allow the user to build ASSERT statements with minimal typing. When building an assertion statement, the user can use any item in these lists by simply clicking on it. The AEW also provides guidelines to help build syntactically correct statements. After building the assertion statement, the user has the option of placing the ASSERT statement either before or after the statement that is currently being examined. When the user inserts an assertion statement it appears in the Assertions window.
When the user places an ASSERT statement using the AEW, it is not immediately placed in the program's source code. Assertions built in this manner are first stored in memory, and then placed in the source code of a program when the user choses the FILE - SAVE ASSERTIONS menu option. Once the user has placed his or her ASSERT statements into the source code by choosing the FILE - SAVE ASSERTIONS menu option, the user must re-instrument for testability (using RST's testability tool) in order to attain valid testability results.
