Reference Information for the Software Verification and Validation Process

Computing systems may be employed in the health care environment in efforts to increase reliability of care and reduce costs. Software verification and validation (V&V) is an aid in determining that the software requirements are implemented correctly and completely and are traceable to system requirements. It helps to ensure that those system functions controlled by software are secure, reliable, and maintainable. Software V&V is conducted throughout the planning, development and maintenance of software systems, including knowledge-based systems, and may assist in assuring appropriate reuse of software.

Health care; independent verification and validation; knowledge-based systems; software reuse; software development; software diagnostic tools; software verification and validation.

This report was funded by the Advanced Technology Program (ATP) of the National Institute of Standards and Technology (NIST) under Solicitation 94-04, Information Infrastructure for Health Care.

Like many other industries in the United States, the health care industry is turning to computing systems to reduce administrative overhead, control escalating costs, and improve accuracy of stored information. New technology is affecting the form and usage of patient information, diagnostic tools, and the tools which provide treatment. In particular, the application of information technology is a promising enabler for transferring gains in medical science research to patient benefit, for ensuring appropriate availability of patient information, and for managing the billing processes.

Computing systems may be employed in the health care environment in efforts to increase reliability of care and reduce costs. Software verification and validation (V&V) is an aid in determining that the software requirements are implemented correctly and completely and are traceable to system requirements. (Software V&V does not verify the correctness of the system requirements, only that the software requirements can be traced to the system requirements.) It helps to ensure that those system functions controlled by software are secure, reliable, and maintainable. It uses a structured approach to analyze and test the software. It evaluates software against its requirements for quality attributes such as performance. Software V&V is conducted throughout the planning, development, and maintenance of software systems.

The major objective of the software V&V process is to determine that the software performs its intended functions correctly, ensure that it performs no unintended functions, and provide information about its quality and reliability. Software V&V evaluates how well the software is meeting its technical requirements and its safety, security and reliability objectives relative to the system. It also helps to ensure that software requirements are not in conflict with any standards or requirements applicable to other system components. Software V&V tasks analyze, review, demonstrate or test all software development outputs.

The guidelines in this report address V&V issues related to the recognition that different health care systems may:

• -- execute in real time (e.g., implantable medical devices and treatment devices);
• -- rely heavily on existing software;
• -- contain many units contributing to overall system complexity;
• -- incorporate knowledge-based systems (KBS) (e.g., diagnostic systems).

The software V&V process is tightly integrated with the software development process. For each activity in software development there is a corresponding software V&V activity to verify or validate the products of those activities. This report explains these relationships, the software V&V tasks supporting each activity, and the types of techniques that may be used to accomplish specific software V&V tasks.

Software V&V has long been employed on new development projects. Today, more and more systems are built using commercial off-the-shelf (COTS) software products, software components from sources external to the developer, and software from a previous version of a similar product built by the same organization. Some of the issues concerning software V&V for systems reusing any of these software types are addressed in this document.

The health care industry has been interested in, and made use of, artificial intelligence (AI) techniques by developing KBSs to understand the complex medical and patient data used for diagnosis. This interest has grown as the scale of the problem of managing data and knowledge has grown in the health care industry. While there are techniques available for V&V of the KBS which employ AI techniques, the V&V and AI communities still need to do more research especially in the areas of making knowledge maintenance easier and more reliable. These guidelines provide an overview of the issues in using V&V and KBS techniques.

AI             Artificial Intelligence
ATP            Advanced Technology Program
CASE           Computer-Aided Software Engineering
CBR            Case-Based Reasoning
COTS           Commercial Off-The-Shelf
FSM            Finite State Machines
IA             Intelligent Agents
IV&V           Independent Verification and Validation
KBS            Knowledge-Based System
KADS           KBS Analysis and Design Support
LOC            Lines Of Code
NIST           National Institute of Standards and Technology
SFMECA         Software Failure Mode, Effects, and Criticality Analysis
SPC            Statistical Process Control
SQA            Software Quality Assurance
SVVP           Software Verification and Validation Plan
SVVR           Software Verification and Validation Report
V&V            Verification and Validation

TABLE OF CONTENTS

Like many other industries in the United States, the health care industry is turning to computing systems to control escalating costs and improve the quality of service. New technology is affecting the form and usage of patient information, diagnostic tools, and the tools which provide treatment. In particular, the application of information technology is a promising enabler for transferring gains in medical science research to patient benefit, for ensuring appropriate availability of patient information, and for managing the billing processes.

In response to the increasing dependence of the health care industry on information technology, the Advanced Technology Program (ATP) at the National Institute of Standards and Technology (NIST) issued Solicitation 94-04, Information Infrastructure for Health Care. The recognition by the ATP that the computer-based systems used in health care must be of high integrity ⁽¹⁾ resulted in one element of that solicitation being technology for verification and validation (V&V). This report is the result of an effort funded by the ATP to produce guidance for the software V&V of computer-based health care systems. Software V&V helps to ensure and assess the quality of software-based systems.

Computing systems may be employed in the health care environment in efforts to increase reliability of care and reduce costs. To achieve these benefits, those functions controlled by software in health care systems must be secure, reliable, and maintainable. Software V&V will help to provide all these assurances. Software verification and validation (V&V) is an aid in determining that the software requirements are implemented correctly and completely and are traceable to system requirements. (Software V&V does not verify the correctness of the system requirements, only that the software requirements can be traced to the system requirements.) It uses a structured approach to analyze and test the software. It measures software against its requirements for quality attributes such as performance, safety ⁽²⁾, and computer security. Software V&V includes activities ⁽³⁾ to determine that the software system performs its intended functions correctly, to ensure that it performs no unintended functions, and to provide information about its quality and reliability.

The guidelines in this report address V&V issues related to the recognition that different health care systems may:

• -- execute in real time (e.g., implantable medical devices and treatment devices);
• -- rely heavily on existing software;
• -- contain many units contributing to overall system complexity;
• -- incorporate knowledge-based systems (KBS) (e.g., diagnostic systems).

Software V&V has long been employed on new development projects. Today, more and more systems are built using commercial off-the-shelf (COTS) software products, software components from sources external to the developer, and software from a previous version of a similar product built by the same organization. Some of the issues concerning software V&V for systems reusing any of these software types are addressed in this document. This particular aspect of software V&V for reused software requires additional research from the reuse and V&V communities.

The health care industry has been interested in and made use of artificial intelligence (AI) techniques by developing KBSs to understand the complex medical and patient data used for diagnosis. This interest has grown as the scale of the problem of managing data and knowledge has grown in the health care industry. While there are techniques available for V&V of the KBS which employ AI techniques, the V&V and AI communities still need to do more research especially in the areas of making knowledge maintenance easier and more reliable. These guidelines provide an overview of the issues in using KBS techniques on systems requiring high reliability and on some of the techniques for V&V of KBS especially KBS which employ expert systems.

The guidance in this report is generally applicable to most software systems and is compatible with the following existing NIST guidance documents:

• -- "A Study on Hazard Analysis in High Integrity Software Standards and Guidelines " [NIST5589]
• -- "A Framework for the Development and Assurance of High Integrity Software" [NIST223]
• -- "Quality Characteristics and Metrics for Reusable Software" [NIST5459]
• -- "Software Error Analysis" [NIST209]
• -- "Software Quality Assurance: Documentation and Reviews" [NIST4909]
• -- "Software Verification and Validation: Its Role in Computer Assurance and Its Relationship with Software Project Management Standards" [NIST165]
• -- "Guideline for Software Verification and Validation Plans" [FIPS132]

The overview of software V&V in section 2 of this report describes considerations for determining who performs software V&V and provides details on the management of software V&V. Section 2 also discusses the scope, objectives, and tasks of software V&V. Section 3 explains the categories of techniques supporting V&V. It also presents short descriptions of the more common techniques, the problems they help to uncover, and the other tasks they may support. Sections 4 and 5 address issues regarding reused software and KBS. In both cases, more research is needed to provide a comprehensive approach for software V&V. Appendix A addresses software metrics, statistical processes, and reliability estimation models that may be applied to the collective findings of software V&V.

Software verification and validation (V&V) is an aid in determining that the software requirements are implemented correctly and completely and are traceable to system requirements. (Software V&V does not verify the correctness of the system requirements, only that the software requirements can be traced to the system requirements.) The major objective of the software V&V process is to comprehensively analyze and test the software during development to determine that the software performs its intended functions correctly, ensure that it performs no unintended functions, and provide information about its quality and reliability [NIST165]. Software V&V evaluates how well the software is meeting its technical requirements and its safety, security, and reliability objectives relative to the system. It also ensures that software requirements are not in conflict with any standards or requirements applicable to other system components. Software V&V tasks analyze, review, demonstrate or test all software development outputs.

Software verification examines the products of each development activity (or increment of the activity) to determine if the software development outputs meet the requirements established at the beginning of the activity. The scope of each software development activity is defined by software program management. A software design may consist of many small increments for each iteration of the total system. Hence, V&V tasks can be performed on small outputs. Validation that the software is a correct implementation of the system requirements for which the software is responsible is conducted concurrently with, and at the end of, all software development activities.

The software V&V process produces a software verification and validation plan (SVVP), individual plans and reports for tasks, summary reports, anomaly reports, and a final software verification and validation report (SVVR). Software V&V planning is conducted against system requirements at the highest level of planning, and then on the software requirements, which should be traceable to the system requirements. Many software V&V tasks, such as planning for software system test, are actually performed in early development activities. The software system test plan is developed concurrently with the software requirements activity. The plan is updated with additions or changes in details as the project progresses. While different management and technical staff may be responsible for different types of test, staff who perform verification of the software requirements may be staff who prepare preliminary plans for software system tests. The development of the test plans and designs may lead to discovery of software requirements errors because of the analysis needed to plan tests.

One issue that often arises in planning a project and its software V&V effort is how to ensure the objectivity of the staff performing software V&V tasks. Independent V&V (IV&V) for software grew out of this concern. Software IV&V is the performance of software V&V tasks by a team that is separate from the software development group. IV&V is described in section 2.1.

This guideline is intended for use with any software development methodology. The software V&V process comprises the software V&V management activity and software V&V technical activities. Each activity consists of several tasks, shown in Table 2-1. These tasks are defined in [FIPS132] and expanded in [WALLACE94]. Software V&V management is described in section 2.2. It ensures that task selection is appropriate for achieving the software V&V objectives; ensures the performance and quality of the V&V effort; selects appropriate metrics and techniques applied to the V&V results; and, conveys results of the V&V tasks to appropriate places.

The software V&V technical activities each have several tasks. Each task is accomplished by applying one or more techniques. A specific technique, such as control flow analysis, focuses on finding a specific type of problem, for example, a logic error. An aggregate of techniques is usually necessary to achieve the objectives of a task. Section 2.3 discusses the tasks for each activity (sec. 3 describes techniques and the problem areas related to those techniques).

Some software V&V activities may be performed by two different groups. The use of a different organization (other than the software development group) for software V&V is called independent verification and validation (IV&V). The following is summarized from the chapter on IV&V in [WILEY].

Technical independence requires that members of the IV&V team (organization or group) may not be personnel involved in the development of the software. This team must have some knowledge about the system design or have related experience and engineering background enabling them to understand the system. The IV&V team must not be influenced by the development team when the IV&V team is learning about the system requirements, proposed solutions for building the system, and problems encountered. Technical independence is crucial in the team's ability to detect the subtle software requirements, software design, and coding errors that escape detection by development testing and SQA reviews.

The technical IV&V team may need to share tools from the computer support environment (e.g., compilers, assemblers, utilities) but should execute qualification tests on these tools to ensure that the common tools themselves do not mask errors in the software being analyzed and tested. The IV&V team uses or develops its own set of test and analysis tools separate from the developer's tools whenever possible.

Managerial independence means the responsibility for IV&V belongs to an organization outside the contractor and program organizations that develop the software. While assurance objectives may be decided by regulations and project requirements, the IV&V team independently decides the areas of the software/system to analyze and test, techniques to conduct the IV&V, schedule of tasks (within the framework of the system schedules), and technical issues to act upon. The IV&V team provides its findings in a timely fashion simultaneously to both the development team and the systems management who acts upon the reported discrepancy and findings.

Financial independence means that control of the IV&V budget is retained in an organization outside the contractor and program organization that develop the software. This independence protects against diversion of funds or adverse financial pressures or influences that may cause delay or stopping of IV&V analysis and test tasks and timely reporting of results.

The extent that each of these parameters is vested in the IV&V team's responsibilities defines the degree of independence achieved. Based on the definitions of IV&V and how much IV&V a specific project requires, some software V&V activities may be conducted by both the developer and another organization. For example, unit test by one organization may focus on demonstrating that specific objectives (e.g., safety objectives relative to the system), which may differ from the objectives of the developer (e.g., logic structure, test coverage), have been met [IEEEP1059].

The process of software V&V needs to be managed and performed comprehensively over the entire software development process. Management tasks, spanning all of the software development activities, are to:

• -- Plan and maintain the software V&V process.
  
  
• -- Coordinate and interpret performance and quality of the software V&V effort.
  
  
• -- Report discrepancies promptly to the user or development group.
  
  
• -- Identify early problem trends and focus software V&V tasks on them.
  
  
• -- Provide a technical evaluation of the software performance and quality attributes at each major software program review (so a determination can be made of whether the software product has satisfied its set of software requirements well enough to proceed to the next activity).
  
  
• -- Assess the full impact of proposed software changes.

An SVVP contains the information necessary to manage and perform software V&V. Major steps in developing an SVVP are to:

• -- Define (or confirm, if already provided) the quality and performance objectives (e.g., verify conformance to specifications, verify compliance with safety and computer security objectives relative to the system, assess efficiency and quality of software, and assess performance across the full operating environment).
  
  
• -- Characterize the types of problems anticipated in the system and define how they would be manifested in the software.
  
  
• -- Select the software V&V analysis and testing techniques to effectively detect the system and software problems.
  
  
• -- Select the metrics and techniques applied to V&V results to measure and predict the quality of the software.

The SVVP may include details for acquiring tools and for training personnel. The SVVP is revised as knowledge accumulates about the characteristics of the system, the software, and the problem areas in the software and in software V&V activities.

The software V&V process could be tailored to specific applications; however, the risk of the software failing and the subsequent consequences must be considered when selecting software V&V activities.

One software V&V management task is to monitor the software V&V technical progress and quality of results. During each software V&V activity, planned software V&V tasks are reviewed and new ones are added to focus on the critical performance/quality functions of the software and its system. The monitoring task includes formal reviews of software V&V discrepancy reports and technical evaluations to provide a check of their correctness and accuracy. Internal monitoring of the quality and accuracy of software V&V results is essential because the development group must make the necessary software changes as indicated by the software V&V results. If the software V&V results are erroneous, or of poor quality, the development group wastes its time and resources in attempting the changes, and more importantly, loses confidence in the effectiveness and helpfulness of the software V&V results. Software V&V studies [RADATZ] have shown that responding to discrepancy reports and software V&V evaluation reports consumes the largest portion of a development group's interface time with the software V&V group.

Boehm and Papaccio [BOEHM] report that the Pareto effect, that is, 20% of the problems cause 80% of the rework costs, applies to software. They recommend that software V&V "focus on identifying and eliminating the specific high-risk problems to be encountered by a software project." This does not mean that software V&V should examine only 20% of the software. Rather, software V&V needs to examine all the software. This includes: identifying potential hazards or threats to the safety and security of the system, prioritizing the software functions by criticality, and allocating software V&V analysis resources to those areas of the software which contain critical ⁽⁵⁾ functions and high-risk problems (i.e., more error-prone). Identifying and focusing on critical and high-risk areas of the software can be addressed by these software V&V methods:

• -- examination of early program deliveries to software V&V staff;
• -- use of software hazard (or threat) analysis; and
• -- conduct of a "criticality analysis" to identify the most critical functions of the software.

Various approaches in development can provide early product information to software V&V. These include: prototypes, incremental software development, and handing over each unit or subfunction following development unit testing. Incremental software development is an effective method of providing early product information to software V&V. The early deliveries reinforce the systematic analysis and test approach used by software V&V to examine the software in smaller pieces while progressively evaluating larger software pieces as each new piece is integrated. High-risk software areas are easier to identify by using the incremental approach because the software V&V can:

• -- Provide an early lead time to evaluate each engineering solution and allow time to suggest alternative solutions which can be incorporated in subsequent incremental deliveries without adversely impacting the schedule.
  
  
• -- Isolate each new set of requirements and evaluate their impact on the system performance.
  
  
• -- Provide early indications of system performance so that adjustments can be made to refine the desired performance.
  
  
• -- Develop trend information about software anomalies and risk issues to allow time to adjust the development and software V&V resources and planning to accommodate evolving software risk issues.

In incremental development, a software build (or early product) represents a basic program skeleton including draft documentation containing portions of the full software capabilities. Each successive build integrates additional functions into the skeleton. Based on discrepancy or progress reports from software V&V, software program management can make the technical and management decisions to refocus the software V&V and development team onto the program's specific problem areas of the software.

Two related analyses, criticality and hazard, can help focus the V&V effort on those parts of the program whose consequence of failure are most severe. A hazard is an (unsafe) "condition that may lead to an unintended event that causes an undesirable outcome" [WALLACE91]. For example, a driver of a car ignores warning lights at a railroad crossing and drives the car onto the tracks. The hazard is the presence of the car and train on the track at the same time. The unintended event (mishap) is the train colliding with the car. The undesirable outcome is the probable loss of life and damage to the car and train. The term hazard generally is used to refer to safety problems; the term threat generally is used to refer to security problems. In this document, the methods and issues related to hazard analysis are also applicable to security issues; the terms threat and security could be used in place of hazard and safety respectively.

Criticality analysis locates and reduces high-risk problems and is performed at the beginning of a project. It identifies the functions and units which are required to implement critical program functions or quality requirements (e.g., safety, computer security). The steps of the analysis are:

• -- Develop a block diagram or control-flow diagram of the system and its software. Each block or control-flow box represents a system or software function (unit).
  
  
• -- Trace each critical function or quality requirement through the block or control flow diagram.
  
  
• -- Classify all traced software functions (units) as critical to either the proper execution of critical software functions or the quality requirements.
  
  
• -- Focus additional analysis on these traced software functions (units).
  
  
• -- Repeat criticality analysis for each activity to observe whether the implementation details shift the criticality emphasis to other or additional functions (units).

System hazard analysis is used to identify potential events and circumstances that might lead to problems of varying degrees of severity, from critical failures resulting in loss of life or national security problems, to less serious malfunctions in the system. Software hazard (or threat) analysis focuses on the role of software relative to the hazards, or threats. Specific techniques that can be used for hazard analysis are included in section 6 with the V&V techniques; these include event tree analysis, software fault tree analysis, Petri nets, and software failure mode, effects, and criticality analysis. (Hewlett-Packard's Medical Systems Unit has developed a software hazard avoidance process utilizing some aspects of these techniques [CONNOLLY].)

When identification of high risk areas from early deliveries, criticality analysis, and hazard (or threat) analysis are used together, the software V&V approach can focus on the most critical areas of the early software products. Software V&V results, obtained early enough in the software development process, can have significant impact on the quality and performance of the system under development.

Software V&V should begin when the project begins. Usually the first software V&V tasks are conducted during the software requirements V&V activity. One V&V task is to examine the early project documentation, often called concept documents, to verify that the system to be built is not only feasible but will use the rules, conventions, algorithms, and practices appropriate to the application domain of the system. Software requirements V&V is performed to ensure that the specified software requirements are correct, complete, consistent, accurate, readable, and testable, and will satisfy the system requirements. Poorly specified software requirements (e.g., incorrect, incomplete, ambiguous, or not testable) contribute to software cost overruns and problems with reliability. Even when software fully meets its requirements upon delivery, there may be problems in the maintenance activity because general requirements (e.g., maintainability, quality, and reusability) were not accounted for during the original development. Identifying software requirements is difficult because the complexity of the problems being solved causes uncertainty in developing the intended system performance requirements. The occurrence of changes in requirements (e.g., to incorporate new technologies, new missions, new knowledge, new interfacing systems, new people coming on the scene) throughout the software development process adds significantly more chance for error. Software requirements V&V is intended to prevent these problems from occurring.

Design errors can be introduced by misrepresentation of the functional requirements and by implementation constraints relating to timing, data structures, memory space, and accuracy. Software design V&V provides assurance that software requirements are not misrepresented or incompletely implemented; that extraneous software requirements are not designed into the solution by oversight; that software requirements are not left out of the software design; and that other constraints are managed correctly.

Clerical and syntactical errors have been greatly reduced through use of structured programming, reuse of code, adoption of programming standards and style guides, availability of more robust computer languages, better compiler diagnostics and automated support, and, finally, more knowledgeable programmers. Nevertheless, problems still occur in translating design into code and code V&V continues to be an important software V&V activity.

Test management is an important part of the software V&V activity in which all testing needed for the software system is considered and planned. Software V&V test planning begins with software requirements and spans almost the full range of activities. Test planning tasks encompass different types of testing--unit test, software integration test, and software system test. The planning activities result in documentation for each test type consisting of test plan, test design, test case, and test procedure documents.

Unit test verifies the design and implementation of software units. Software integration test verifies functional requirements as the software units are integrated together. Special attention is focused on software, hardware, and operator interfaces. Software system test validates the entire software program against system requirements and software performance objectives. Software system tests validate that the software executes correctly within its stated operating environment. The software's ability to deal properly with anomalies and stress conditions is emphasized. Software V&V tests are not intended to duplicate or replace the user and development group's test responsibilities, but instead test behavior not normally checked by the user or development group.

Software installation test validates that the software operates correctly with the operational hardware system and with other software, as specified in the interface specifications. It verifies that the installation procedures are correct and adequate, that the software is the same as the executable code delivered for installation, and that all supporting software products are the proper versions. Software installation test verifies that the software has been accurately tailored for site-dependent parameters and that the configuration of the delivered product is correct.

In software operation and maintenance V&V, when a software change is made, all software V&V activities are considered and possibly repeated to ensure that nothing is overlooked. Software V&V activities include examining the impact of the change throughout the system to understand what software V&V activities are needed. Software V&V activities are added or deleted to address the type of software change made. In many cases, an examination of the proposed software change shows that software V&V needs to repeat its activities on only a small portion of the software. Also, some software V&V activities, such as verifying the original concepts, require little or no effort to verify a small change. Small changes can have subtle but significant side-effects in a software program; for this reason, change analysis (a software operation and maintenance V&V task) is significant in preventing unintended functions and problems from making their way into new versions of the system.

The software requirements V&V activity checks that the allocation of system requirements to software is appropriate and correct, and how well the software requirements have been specified (e.g., correct, complete, nonambiguous, testable). It should be structured to ensure that the software objectives have been met. Verification of the software requirements should include an examination of documentation produced earlier in system development (e.g., initial feasibility studies, concepts on which the system has been designed) if this examination has not already been performed. If the assumptions, algorithms, and physical rules imposed on the software requirements previously have not been verified to be appropriate for this project, then software V&V should perform those checks. Inputs to the software requirements V&V activity may be documents written in natural or formal mathematical languages and may include graphics and charts. When formal mathematical languages are used, other forms of representations may be provided to different users of the specifications. Software requirements verification must ensure fidelity among the forms of representation. [NIST223]

Concurrently with software requirements V&V, software system test planning in initiated. Software V&V examines all the proposed testing for the system to ensure that comprehensive testing and appropriate resources are planned. Each type of testing (unit, software integration, software system) is discussed more fully in this report. When the system requirements and the software requirements have been specified and reuse of software is identified, reuse issues identified in section 4 must be checked to ensure the software is suitable for the application domain and the operating environment.

The remainder of this section elaborates on software requirements V&V for general V&V tasks, for V&V tasks specifically designed for reused software, and those for knowledge-based systems (KBS).

• -- Conduct a concept documentation evaluation.

  - Evaluate the defined concept to determine whether it satisfies user needs and project objectives in terms of system performance requirements, feasibility (e.g., compatibility of hardware capabilities), completeness, and accuracy.

  - Identify major constraints of interfacing systems and constraints/limitations of the proposed approach and assesses the allocation of system functions to hardware and software, where appropriate.

  - Assess the criticality of each software item defined in the concept.
  
  

• -- Begin test planning.
  
  
• -- Conduct a software traceability analysis - Trace software requirements to system requirements (and vice versa) and check the relationships for accuracy, completeness, consistency, and correctness; check that allocation is appropriate and complete.
  
  
• -- Conduct a software requirements evaluation.
  
  

  - Evaluate the software requirements for accuracy, completeness, consistency, correctness, testability, and understandability.
  
  

  -- Measure completeness by verifying existence and correctness of defining properties: initiator of action, action, object of action, conditions, constraints, source, destination, mechanism, and reason.

  --Verify correctness and appropriateness of software requirements and assertions (executable statements that may be required in the software as fault tolerance protections for the system safety and computer security objectives (e.g., checking algorithms, states and integrity of system and the responses to unfavorable results of the assertions) ). Verify the operation of the assertions will not adversely impact system performance.

  --Verify correctness and appropriateness of fault tolerance requirements. Verify that the operation of the assertions will not adversely impact system performance.
  
  

  - Assess how well the software requirements accomplish the system and software objectives.
  
  
  - Identify critical areas of software by assessing criticality of software requirements.
  -Evaluate software requirements for compliance to software requirements standards and software engineering practices.
  
  
• -- Conduct a software interface analysis - Evaluate software requirements with hardware, user, operator and software interface requirements for accuracy, completeness, consistency, correctness, and understandability.

• -- Evaluate the reused software for conformance to its performance goals, to identify constraints of interfacing systems, to allocation functions to hardware and software, and to assess criticality of each software item.
  
  
• -- Conduct software interface analysis to evaluate reused software to new requirements for accuracy, completeness, consistency, correctness, and understandability, relative to the operating environment of both the reused and the new software and to the application domain. When COTS is considered for use in a new system, this task is especially significant for ensuring that the COTS will match the system interfaces in the operating environment.
  
  
• -- Compare the new software system objectives to the content of the reused documentation and the reused code to ensure the:
  
  

  - availability of all necessary files;

  - adequacy of user manual (compare to the requirements for the user manual in software development); and,

  - compatibility of the software, hardware, and system environment (e.g., was the old system designed for a 16 bit machine and will now be on a 32 bit machine?).
  
  

• -- If the reused software is COTS, consider whether any functions of the software are to be blocked out from usage; if the consequences of any functions are unknown; and, the operational history of the COTS relative to failure.

• -- Verify the scope and complexity of the proposed domain for the KBS.
  
  
• -- Verify the correctness and appropriateness of the requirements on accuracy and completeness of the expected results (e.g., is the system supposed to perform like a student or an expert?).
  
  
• -- Verify that the selected tools can implement a domain model of the expected scope and complexity.
  
  
• -- Determine how accuracy of the system will be evaluated and against what standard it will be evaluated.
  
  
• -- Determine the volatility of the domain model and strategy for updating the knowledge base.

The software design V&V activity occurs after the software requirements have undergone the software V&V process and the software design, or an increment of the software design, has been completed. ⁽⁷⁾ The software V&V tasks of traceability, evaluation, and interface analysis provide assurance that software requirements are not misrepresented, incompletely implemented, or incorrectly implemented. By verifying that the software design meets its software requirements, the software design V&V activity also supports validation that the software design meets system requirements. There may be several instantiations of the software requirements and software design verification before the entire system is verified. [NIST223]

When the software system is designed, decisions may be made to incorporate existing software. Again, the issues identified in section 3 must be considered by software V&V to ensure that the reused software is appropriate, and that the software design takes into account any changes that must be made to the reused software to accommodate the operating environment and the application domain. The tasks and techniques are the same as for the software being developed, but the objectives and issues are specific for reuse.

The remainder of this section elaborates on software design V&V for general V&V tasks, for V&V tasks specifically designed for reused software, and those for KBSs.

• -- Conduct a software design traceability analysis - Trace software design to software requirements, and vice versa. Check the relationships for accuracy, completeness, consistency, and correctness.
  
  
• -- Conduct a software design evaluation.
  

  - Evaluate the software design for accuracy, completeness, consistency, correctness, and testability.

  - Evaluate software design for compliance with software design standards; language standards if appropriate; and software engineering practices.

  -Assess software design against assigned quality attributes.

  
  
• -- Conduct a software design interface analysis - Evaluate software design for accuracy, completeness, consistency, and correctness of hardware, operator and software interface requirements.
  
  
• -- Verify that the software requirements for assertions, responses to assertions and other required system algorithm and integrity checks or fault tolerance protections have been designed into the software. Check that the software design is complete and accurate and will not adversely affect system performance.
  
  
• -- Coordinate with software integration test planning.

• -- Conduct an evaluation of the original software design documentation for compliance to software design requirements of the new system. Verify interface requirements. Generate any needed software design information or justify the use of the software without the required information. This determination should be based on recognized risk (safety, cost of modifications, impact of various degrees of uncertainty on the project) and coordinated with the user.
  
  
• -- If any modifications are needed, evaluate whether or not the software and documentation are adequate to support the modification (e.g., for change analysis, testability). If not, the needed information should be obtained or developed. If this is not prudent, modifications should not be made when they cannot be supported by adequate software design information.

• -- Verify that the domain model:
  

  - is complete and consistent; and,

  - represents the domain knowledge.

  
  
• -- Verify that the domain model addresses, at the required level of accuracy and completeness, the range of expected problems.
  
  
• -- Verify that the domain model operates in the specified scope.
  
  

The code verification activity verifies correct implementation of software design into code. Often this activity requires tedious checking of details within the code; automation provides protection against human error in gathering the code information for analysis and also can speed the process. Code verification is the last opportunity to find and remove errors that could cause unnecessary costs and delays from advancing poor code into any of the test activities. Code validation is accomplished through unit test which is described in section 2.3.4. [NIST223]

At this point in the software development process, the reuse issues should have been examined and the decision made to reuse or not to reuse the software. In the case that changes are to be made to the code, or if there is a possibility changes will be needed in a future version of the software system under development, some software V&V tasks may be needed.

The knowledge base should be internally consistent and reflect the domain model. In its simplest form, maintaining knowledge base consistency (or integrity) means not allowing a fact and its negation to both be part of the knowledge base. More extensive consistency checks can disallow rules that would, potentially, infer both a fact and its negation. Knowledge consistency is a key issue. A consistent domain model and a consistent representation of that model is critical. This is especially true for domains representing physical structures or controlled equipment. The model of the equipment and the physics controlling the behavior of the equipment must be consistent for computer controllers to function properly. In other domains, expert disagreement over the interpretation of a set of facts may be normal and expected. For example, legal disputes frequently involve the interpretation of the facts themselves. Probabilities can be one way to handle conflicting knowledge.

The remainder of this section elaborates on code verification for general tasks, for verification tasks specifically designed for reused software, and those for KBSs.

• -- Conduct a source code traceability analysis - Trace source code to software design, and vice versa. Check the relationships for accuracy, completeness, consistency, and correctness.
  
  
• -- Conduct a source code evaluation.
  

  - Evaluate the source code for accuracy, completeness, consistency, correctness, and testability.

  - Evaluate source code for compliance with code standards, language standards if appropriate, and software engineering practices.

  - Assess source code against assigned quality attributes.

  
  
• -- Conduct a source code interface analysis - Evaluate the source code for accuracy, completeness, consistency, and correctness with respect to the hardware, operator, and software interfaces.
  
  
• -- Evaluate draft code-related documents (e.g., user manual, commentary within the code) with source code for completeness, consistency, and correctness.
  
  
• -- Coordinate with unit test ⁽⁸⁾.

• -- If the source code is available, compare it to known design specifications. Evaluate for correctness, consistency, completeness, and accuracy. Assess the interfaces for consistency with the system in which the reused code will be placed. Assess source code quality. (This task may be needed in instances where the history of the code is not well-known.)
  
  
• -- Evaluate source code for testability. Evaluate code-related documentation received from the source for suitability for any future modifications.

• -- Conduct a logical verification of the structure of the knowledge and rules in the knowledge base for consistency, completeness, etc.
  
  
• -- Verify that the knowledge base implements the domain model accurately.

Unit test is the test of the software elements at the lowest level of development. Units may be aggregates of software elements. Planning for unit test should occur concurrently with the software design activity. Reused software will probably not undergo unit test; unless changes were made to the units. Then, appropriate testing is performed as in regression testing.

The remainder of this section elaborates on unit test for general V&V tasks, for V&V tasks specifically designed for reused software, and those for KBSs.

• -- Test planning - Establish the objectives of the unit test, the strategies to be employed, the coverage requirements, reporting and analysis, and close-out of anomalies.
  
  
• -- Generate, monitor, and update the unit test plan to accomplish objectives.
  
  
• -- Trace test design, cases, procedures, and execution results to the unit designs.
  
  
• -- Confirm that anomalies during test are software anomalies, and not problems detected for other reasons.
  
  
• -- Generate test cases and procedures - Develop test cases and procedures for unit test and continue tracing as required by software test plans.
  
  
• -- Perform unit test - Check individual software units for typographical, syntactic, and logic errors to ensure that each correctly implements the software design and satisfies the software requirements; execute the test cases; analyze results to verify anomalies; recommend changes to software design or code; and conduct re-testing as necessary.
  
  
• -- Document test activities and results.

• -- Evaluate existing test cases and reports for suitability for intended use.
  
  
• -- Prepare test cases and test procedures if any modifications are made to the reused software.
  
  
• -- Follow the criteria for unit test.

• -- Evaluate the knowledge and rules in the knowledge base against the domain knowledge.
  
  
• -- Establish objective for testing portions of domain knowledge.
  
  
• -- Plan tests for accuracy and completeness of domain model.
  
  
• -- Define test procedures to test for expected performance level of the system.

The software integration test activity is performed to examine how units interface and interact with each other with the assumption that the units and the objects (e.g., data) they manipulate have all passed unit tests [BEIZER]. Software integration tests check the interaction with other software (e.g., libraries) and hardware. The software integration test schedule depends upon the development and integration schedules for software units, hardware, and other components. For large systems, software integration test planning may require close coordination among all system personnel to ensure that the overall test objectives are achieved by the selected test strategy. For each major integration that has successfully undergone interface and interaction testing, functional tests may be developed and executed [BEIZER]. When all system components have been integrated and have successfully undergone software integration tests, then the system moves into software system test. During software integration test, reused software units are integrated into the system. It is critical to test that the interfaces are correct, and that the resulting software meets operating requirements.

The remainder of this section elaborates on software integration test for general V&V tasks, for V&V tasks specifically designed for reused software, and those for KBSs.

• -- Test planning - Establish the objectives of the software integration test, the strategies to be employed, the coverage requirements, reporting and analysis, and close-out of anomalies. Ensure that interface testing of reused software to other system software is planned.
  
  
• -- Generate, monitor, and update a software integration test plan to accomplish identified objectives.
  
  
• -- Trace test design, cases, procedures, and execution results to software requirements.
  
  
• -- Generate test cases and procedures - Develop test cases and procedures for unit test and continue tracing as required by software test plans.
  
  
• -- Perform software integration test.

  - Check the inter-unit communication links and test aggregate functions formed by groups of units.

  - Confirm that anomalies during test are software anomalies, and not problems detected for other reasons.

  - Ensure any changes to software requirements, software design, or code are made. Conduct retesting as necessary.

  - Conduct functional, structural, performance, statistical, and coverage testing of successfully integrated units after each iteration of software integration and successful testing of interfaces and interactions.

  
  
• -- Document test activities and results.

• -- Perform software integration test in accordance with test procedures.
  
  
• -- Analyze results to determine if the software implements the intended use requirements and known design and that the software units function correctly together.
  
  
• -- Conduct interface tests of reused units with other system components.
  
  
• -- Conduct tests of reused units with other system components to validate performance requirements.
  
  
• -- Evaluate existing test cases and reports for suitability for intended use.
  
  
• -- Prepare test cases and test procedures if any modifications are made to the reused software.
  
  
• -- Follow the criteria for software integration test.

• -- Evaluate knowledge base for completeness and consistency.
  
  
• -- Verify that the knowledge base represents the full scope of the domain model.

Software system test, in the context of software V&V, involves the conduct of tests to execute the completely integrated system. Software system test is the validation that the software meets its requirements. Validation of the complete system may involve many tests involving all system components. The software system tests exercise those system functions that invoke software to determine whether the software behaves as intended relative to complete system performance. These tests must be conducted in such a manner as to stress the system based on software responses to system inputs (e.g., from sensors, operators, databases). Tests and data collected from the tests are designed to provide an operational profile of the system which support a statistical analysis of the system reliability [MUSA87, MUSA89, BUTLER]. This section of the report addresses only the tests that validate that the software implements the system requirements; other tests for other components and perspectives are necessary for complete system validation.

While software system tests are conducted after the system has been built, it is imperative that planning for these tests is conducted concurrently with the software requirements activity because:

• -- Analyzing the software requirements for test requirements may result in finding software requirements errors and/or discovery of untestable requirements.
  
  
• -- Establishing test facilities (e.g., model of operational environment) and Computer-Aided Software Engineering (CASE) tools (e.g., test case generators, test data base) may require as much time as development of the system.

For reused software, software system test is performed to assure that the software is correct, consistent with prior documentation, complete for use and/or modification, and accurate. At the system level, reused software should be considered part of the system. Tests are in accordance with test procedures. Results are documented and traced as required by the software system test plan.

The remainder of this section elaborates on software system test for general V&V tasks, for V&V tasks specifically designed for reused software, and those for KBSs.

• -- Test planning - Establish the objectives of the software system test, the strategies to be employed, the coverage requirements, reporting and analysis, and close-out of anomalies.
  
  
• -- Generate, monitor, and update a software system test plan to accomplish objectives.
  
  
• -- Trace system and software requirements to test software design, cases, procedures, and execution results.
  
  
• -- Generate test cases and procedures - Develop test cases and procedures for unit test and continue tracing as required by software system test plans.
  
  
• -- Test the operation of the software as an entity (sometimes a simulated environment may be used); confirm that anomalies during test are software anomalies, not problems detected for other reasons; ensure any changes to software (software requirements, software design, code, or test cases) have been made; and conduct retesting as necessary.
  
  
• -- Document test activities and results.

• -- Evaluate existing test cases and reports for suitability for intended use.
  
  
• -- Prepare test cases and test procedures if any modifications have been made to the reused software.
  
  
• -- Follow the criteria for software system test within the boundaries of the known and documented software design.

• -- Define procedures for testing the system according to the expected knowledge of the end user.

The software installation test activity is the final step before launching full customer acceptance testing. The purpose of installation test is to demonstrate that the correct software has been delivered and that the software interfaces are correct relative to any interfaces at the installation site. Acceptance testing, which involves the user/customer, is outside the scope of this document.

The remainder of this section elaborates on installation test for general V&V tasks, for V&V tasks specifically designed for reused software, and those for KBSs.

• -- Conduct an installation configuration audit.
  

  - Determine that all software outputs needed to operate the system are present.

  - Check that the software installed in the system is the software that underwent software V&V.

  
  
• -- Develop and execute tests that will examine and stress site-unique parameters (e.g., printer interface, operating system interface, monitor interfaces).
  
  
• -- Generate applicable documentation.
  
  
• -- Generate an SVVR (or generate it at the end of the software V&V process).

• -- Conduct an installation configuration audit to verify that any reused software that has not been modified is the current version.

• -- Ensure that data and updates to the knowledge base which are supplied from external sources are in an acceptable form.

The software operation V&V activity requires periodic checks that the integrity of the system has been maintained, that any changes to the system which affect its operation have been documented, and operators have received training in new or changed procedures. The software maintenance V&V activity requires planning for software V&V based on the extent of the maintenance (e.g., adaptive, corrective, perfective [FIPS106]), and hence a revisit of all the software development activities to identify to what extent each software V&V activity must be performed.

If software V&V has not been performed during software development, then the V&V during software operations and maintenance must consider performing a selected set of tasks from the software V&V activities related to earlier development activities. Some activities may include generating software requirements or software design information from source code, an activity known as reverse engineering. While costly and time consuming, it is necessary when the need exists for a rigorous software V&V effort.

The remainder of this section elaborates on software operation and maintenance V&V for general V&V tasks, for V&V tasks specifically designed for reused software, and those for KBSs.

• -- Conduct an anomaly evaluation - Evaluate the severity of anomalies during software operation and their effect on the system.
  
  
• -- Conduct a proposed change assessment - Assess proposed changes to the software and their effect on the system to determine software V&V activities from earlier development to be repeated. Conduct them.
  
  
• -- Develop an SVVP.

Follow the guidance for reuse in section 4.

• -- Plan for update of knowledge base including domain model.
  
  
• -- Determine mechanisms used for updating knowledge base.

The conduct of software V&V tasks to fulfill the requirements of the V&V activities generally involves techniques selected from three major classes: static, dynamic, and formal analysis. Static analysis techniques are those which directly analyze the form and structure of a product without executing the product [FIPS101]. Reviews, inspections, audits and data flow analysis are examples of static analysis techniques. Static analysis techniques are traditionally applied to software requirements, software design and source code. They may also be applied to test documentation, especially test cases, to verify their traceability to the software requirements, their adequacy to fulfill test requirements, and their accuracy.

Dynamic analysis techniques involve execution, or simulation, of a development activity product to detect errors by analyzing the response of a product to sets of input data [FIPS101]. For these techniques, the output values, or ranges of values, must be known. Testing is the most frequent dynamic analysis technique. Prototyping, especially during the software requirements V&V activity, can be considered a dynamic analysis technique; in this case the exact output is not always known but enough knowledge exists to determine if the system response to the input stimuli meets system requirements.

Formal analysis is the use of rigorous mathematical techniques to analyze the algorithms of a solution [FIPS101]. Sometimes the software requirements may be written in a formal specification language (e.g., VDM, Z) which can be verified using a formal analysis technique like proof-of-correctness. The term formal often is used to mean a formalized process, that is, a process that is planned, managed, documented, and is repeatable. In this sense, all software V&V techniques are formal, but do not necessarily meet the definition of the mathematical techniques involving special notations and languages.

Table 3-1, at the end of this section, lists the software V&V techniques addressed in this report and indicates under which V&V activities these techniques can be applied. This report does not necessarily address all software V&V techniques.

Some software V&V techniques used during software requirements V&V tasks are control flow analysis, data flow analysis, algorithm analysis, and simulation. Control and data flow analysis are most applicable for real time and data driven systems. These flow analyses transform logic and data requirements text into graphic flows which are easier to analyze than the text. PERT, state transition, and transaction diagrams are examples of control flow diagrams. Algorithm analysis involves re-derivation of equations or evaluation of the suitability of specific numerical techniques. Simulation is used to evaluate the interactions of large, complex systems with many hardware, user, and other interfacing software units.

Some software V&V techniques used during software design V&V tasks include algorithm analysis, database analysis, sizing and timing analysis, and simulation. Algorithm analysis examines the correctness of the equations or numerical techniques as in the software requirements activity, but also examines truncation and round-off effects, numerical precision of word storage and variables (e.g., single- vs. extended-precision arithmetic), and data typing influences. Database analysis is particularly useful for programs that store program logic in data parameters. A logic analysis of these data values is required to determine the effect these parameters have on program control. Sizing and timing analysis is useful for real-time programs having response time requirements and constrained memory execution space requirements.

Some software V&V techniques used during code V&V tasks are control flow analysis, database analysis, regression analysis, and sizing and timing analysis. For large code developments, control flow diagrams showing the hierarchy of main routines and their subfunctions are useful in understanding the flow of program control. Database analysis is performed on programs with significant data storage to ensure common data and variable regions are used consistently between all call routines. Data integrity is enforced and no data or variable can be accidentally overwritten by overflowing data tables. Data typing and use are consistent throughout all program elements. Regression analysis is used to reevaluate software requirements and software design issues whenever any significant code change is made. This technique ensures project awareness of the original system requirements. Sizing and timing analysis is done during incremental code development and compared against predicted values. Significant deviations between actual and predicted values is a possible indication of problems or the need for additional examination.

Another area of concern to software V&V is the ability of compilers to generate object code that is functionally equivalent to the source code, that is, reliance on the correctness of the language compiler to make data dependent decisions about abstract programmer coded information. For critical applications, this problem is solved by validating the compiler or by validating that the object code produced by the compiler is functionally equivalent to the source.

Code reading is another technique that may be used for source code verification. An expert reads through another programmer's code to detect errors. In an experiment conducted at the National Aeronautics and Space Administration Goddard Space Flight Center, code reading was found to be more effective than either functional testing or structural testing [BASILI]. The reason was attributed to the expertise of the readers who, as they read the code, were simulating its execution and were able to detect many kinds of errors.

Other techniques commonly used are walkthroughs, inspections and reviews. These tasks occur in interactive meetings attended by a team which usually includes at least one member from the development group. Other members may belong to the development group or to other groups involved in software development. The duration of these meetings is usually no more than a few hours in which code is examined on a line-by-line basis. In these dynamic sessions, it may be difficult to examine the code thoroughly for control logic, data flow, database errors, sizing, timing and other features which may require considerable manual or automated effort. Advance preparation for these activities may be necessary and includes code analysis techniques. The results of these techniques provide appropriate engineering information for discussion at meetings where code is evaluated. Regardless of who conducts or participates in walkthroughs and inspections, software V&V analyses may be used to support these meetings.

A comprehensive test management approach to testing recognizes the differences in strategies and in objectives for unit, software integration, and software system test. Unit test verifies the design and implementation of software units. Software integration test verifies functional requirements as the software units are integrated. Special attention is focused on software, hardware, and operator interfaces. Software system test validates the entire software program against system requirements and software performance objectives. Software system tests validate that the software executes correctly within its stated operating environment. The software's ability to deal properly with anomalies and stress conditions is emphasized. These tests are not intended to duplicate or replace the user and development group's test responsibilities, but instead supplement the development testing to test behavior not normally tested by the user or development group.

Effective testing requires a comprehensive understanding of the system. Such understanding develops from systematically analyzing the software's concept, requirements, design, and code. By knowing internal software details, software V&V testing is effective at probing for errors and weaknesses that reveal hidden faults. This is considered structural, or white-box, testing. It often finds errors for which some functional, or black-box, test cases can produce the correct output despite internal errors.

Functional test cases execute part or all of the system to validate that the user requirement is satisfied; these test cases cannot always detect internal errors that will occur under special circumstances. Another software V&V test technique is to develop test cases that violate software requirements. This approach is effective at uncovering basic design assumption errors and unusual operational use errors. The process of planning functional test cases requires a thorough examination of the functional requirements. An analyst who carefully develops those test cases is likely to detect errors and omissions in the software requirements. In this sense test planning can be effective in detecting errors and can contribute to uncovering some errors before test execution.

The planning process for testing must take into account the specific objectives of the software V&V for the software and the impact of different test strategies in satisfying these objectives. Frequently, the most effective strategy may be to combine two or more strategies. More information and references on software testing may be found in [WILEY].

Criticality analysis may be used to identify software V&V techniques to address high-risk concerns. The selection of V&V techniques for use on each critical area of the program is a method of tailoring the intensity of the software V&V against the type of risk present in each area of the software. For example, software V&V would apply algorithm analysis to critical numerical software functions, and techniques such as sizing and timing analysis, data and control flow analysis and interface analysis to real-time executive functions.

The following are summary descriptions of techniques taken from [BAHILL], [BEN], [EWICS3], [KIRANI], [NBS93], [NGUYEN], [NIST209], [NIST5589], [NUREG6316], [OKEEFE], [OLEARY], [TURING], [VOAS91,92,95], [WALLACE94], and [WILEY]. Issues (in italics at the end of each description) include the types of errors the technique may find, the tasks the technique supports, and other related techniques (to or from which supporting information is provided).