NISTIR 5954
RISQ: A Web-Based Tool for Referencing Information
on Software Quality
Charles B. Weinstock
Software Engineering Institute
Carnegie Mellon University
Pittsburgh, PA 15213
Dolores R. Wallace
U.S.DEPARTMENTOF COMMERCE
National Institute of Standards and Technology
Computer Systems Laboratory
Gaithersburg, MD 20899
World Wide Web (Web) technology has provided access to enormous amounts of information on-line. As such it has a tremendous potential for use as an aid to technology transfer. However, locating relevant information is often difficult partially because of the volume of information available. This report describes a new search engine which is designed to allow the user to do efficient searches for information within a specific domain. The initial application of the engine is in the domain of high integrity software systems and is called RISQ which stands for "Reference Information for Software Quality." The search engine allows searches by taxonomy based keywords, other keywords, and artifact type. Artifacts can range from simple abstracts, documents and software to video, audio and on-line interactive demonstrations of software tools. This report discusses the philosophy of the system, the acceptance criteria for artifacts, and provides instructions for using the RISQ system.
high integrity software, search, search engine, quality, software,
taxonomy, World Wide Web
This document describes the Reference Information for Software Quality (RISQ) system developed by members of the National Institute of Standards and Technology (NIST) and Software Engineering Institute (SEI) technical staff.
The RISQ system is an important part of CHISSA, the Center for High Integrity Software Systems Assurance. CHISSA was organized by NIST's Computer Systems Laboratory in 1994 as a collaborative approach for government, industry, and academia to make available the technology which is necessary for assuring high integrity software in an ever growing number of applications.
Software is a major factor in corporate operation, consumer goods, military systems, environmental and energy services, communications, health care, and government operations. High integrity software, that is, software that can and must be trusted to work dependably, is a necessity for the ability of United States industries and government to function. Good dependability techniques developed in laboratories have not made it into use by development organizations. Conversely, very real problems faced by development organizations are not being addressed by the research community. A major goal of CHISSA is to make it easier for researchers to collaborate with developers to
1. see how research results perform in practice,
2. improve the dependability of the resulting applications, and
3. direct the researchers' efforts more towards helping the developers with their real problems.
The RISQ facility helps achieve these goals by making available a wide variety of artifacts related to software quality in a highly organized manner. The overarching goal is to help to transition technologies so that the best of them become standard practices when developing software systems with a requirement for high integrity.
The current RISQ facility is located on the World Wide Web ( Web) at hissa.ncsl.nist.gov/risq/. By locating the facility on the Web, we achieve the goal of making the material available to a large and geographically diverse set of potential users.
When CHISSA was first established, a Call for White Papers was issued to collect ideas on how to shape the organization to best serve the high integrity software systems community. There were over 100 responses to the Call, and it soon became apparent that an important role for CHISSA would be to provide a central location for industry, government, and academia to exchange information about tools and techniques available for achieving high integrity software systems.
Originally a CHISSA demonstration facility, to be hosted at NIST, was proposed. This facility would act as a repository for demonstrations of high integrity software systems technologies, and would behave much like a library. Patrons would visit the facility to view the demonstrations and leave with relevant literature.
It soon became apparent that this approach was not practical. Not only would it require a physical plant on the NIST campus, but it would also require NIST personnel to conduct the demonstrations. This was clearly not feasible from a budgetary point of view. Worse, it would require the patron to journey to the NIST campus, something that would limit the effectiveness of the facility in terms of its ability to serve a large number of people.
The second approach to a CHISSA demonstration facility was prototyped on the Web in 1995. This facility would consist of canned and live demonstrations, all hosted on a NIST Web site. The canned demonstration would be a series of Web pages with clickable images to simulate the actual operation of the system being demonstrated. A live demonstration would take advantage of a combination of the Web and windowing systems to actually allow the user to experiment with the real tool, or at least a special modified version of the real tool.
The prototype of this demonstration facility worked well. However a useful facility would consist of more than tool demonstrations. It became clear that a facility of any size would need to be distributed over many sites. This would allow, for instance, home organizations to install and maintain their own demonstrations on differing types of computers and greatly ease the work load on NIST personnel. As a result of this thinking the current RISQ system was prototyped in early 1996 by Tony Cincotta of NIST and the current version was implemented later that same year.
The designers of the RISQ system took the point of view that generality is not always the best way to proceed. Since the domain of interest is relatively well defined, the system has been organized to make searching within that specific domain more efficient for the user.
Another key part of the philosophy behind RISQ is that the best way to populate it, given the limited resources available to CHISSA, is to allow the community to submit suggested artifacts for inclusion. Although the RISQ administrators will have the final say as to what gets added to the database, allowing users to nominate artifacts provides a tremendous amount of leverage.
A third part of the philosophy is that users should have confidence in artifacts located through the RISQ system. This is embodied in the published acceptance criteria.
Objects maintained by the RISQ database (i.e., RISQ artifacts) are classified according to three axes:
Ordinary search engines are either free form or keyword based.
The user is left with the quandary of how to carefully choose
keywords to elicit the appropriate artifacts. The RISQ system
has taken a different approach. It presents the user with a taxonomy
of high integrity terms. Every artifact will fit into one or more
of the categories in the taxonomy. The result is that the user
does not have to guess at what the appropriate search terms might
be. Figure 1 shows the current RISQ taxonomy.
The user may select as many terms as desired to restrict the number of items located. As with any search engine, selecting incompatible terms will result in no items being returned. Selected terms are combined according to the following rules:
Any artifact can be located by selecting an appropriate set of terms from the taxonomy described above. As the database of artifacts gets more fully populated, restricting the search to manageable sets of artifacts will become increasingly difficult. To help with this problem, and to allow searching for keywords that are not in the taxonomy, the RISQ system also allows searches based on user defined keywords.
Searching using user defined keywords is somewhat problematic in that the user must guess at what keywords will elicit the appropriate artifacts. If the person entering the artifact into the database has a different world view than the person doing the search, important artifacts might not be retrieved. In the future we plan to develop the capability to automatically search for synonyms of the user defined keywords.
The third way in which searches may be restricted is by selecting specific artifact types. Currently defined artifact types include:
Abstract A text artifact that is an extended abstract describing some object that is not in the RISQ database (e.g., a paper published in a journal.)
Audio Artifacts that are aural in nature. This could be a .WAV file with relevant information, or a live lecture on a high integrity topic.
Code Source program artifacts that implement some high integrity function.
Data Artifacts consisting of real-world data which can be studied to glean additional information about how systems behave.
Design A design document related to high integrity systems.
Document A complete document related to high integrity systems. This could be an HTML document, a text file, a PDF file suitable for viewing with Acrobat, or any other format that can be read on-line.
Image A picture or figure related to high integrity systems.
PostScript A document in PostScript form. These documents typically must be printed to be read.
Requirement A requirements document.
Resource A resource or on-line service that cannot otherwise be categorized. This often will be the location on the web of additional high integrity artifacts.
Taxonomy Alternative taxonomies relevant to high integrity systems.
Test Testing materials.
Tool A tool. This may be just a demonstration (live or canned), or it may be the actual tool itself.
Tutorial A document artifact that is a tutorial covering some aspect of high integrity software systems.
Video An on-line video relevant to high integrity systems.
If the user does not specify an artifact type, the RISQ system defaults to type document.
All of these search mechanisms can be combined. For example the taxonomy search can pick "Hazard Analysis" and if the artifact type is "tool", any tools in the RISQ database that address hazard analysis will be retrieved.
One of the key philosophies underlying the RISQ System is that, for the repository to grow, we will need the assistance of the user community to populate it. Using an almost identical facility to what the RISQ administrators use to add artifacts, any user of the RISQ System is able to nominate an artifact for inclusion in the database. The administrators are notified of this nomination and have an opportunity to review the artifact to see if it meets the appropriate acceptance criteria (as discussed below). The administrator may reject the submission, accept it as is, or edit it.
In order to ensure that the information provided by the RISQ system is of maximum usefulness, the methods, tools and techniques described in RISQ artifacts must represent accepted practice or new technology, and meet certain criteria reflecting professional standards for publication.
All artifacts must be related to high integrity software. Any of the requirements discussed below may be waived at the discretion of the RISQ administrator.
There are two overarching requirements to consider when looking at any submission to the RISQ system:
The RISQ administrators will enforce acceptance criteria for each artifact type. A preliminary set of criteria are identified for document (including those in abstract or postscript form) and tool artifacts. Acceptance criteria will eventually be developed for all artifact types. The RISQ/CHISSA project manager will review recommendations from the RISQ administrator and has final authority for accepting or rejecting submissions for this facility.
An article is an artifact of the type abstract, document or PostScript. An article submitted to RISQ must have had a technical peer review (e.g., a review process for a published journal article; a refereed conference paper) or in the case of an abstract, the item referenced by the abstract must meet this criterion. It must be associated with the RISQ taxonomy (i.e., it must address one or more topics identified in the taxonomy).
The article may describe a software development or assurance method, a software tool associated with the method or an experiment with the method. It may contain case studies of using the method, or may provide a tutorial for using the method. Articles may also be surveys of a specific subject area (e.g., software safety, testing).
A software tool may implement a software development or assurance (verification, testing, any diagnostic) method or a new approach or algorithm to solve a specific problem. It must be associated with the taxonomy of the RISQ (i.e., it must address one or more topics identified in the taxonomy.) Current acceptance criteria are:
The RISQ system is on the World Wide at hissa.ncsl.nist.gov/risq/. The initial page is presented as shown in Figure 2.
The page is divided into three logical sections. Section one gives access to the RISQ system's search facility. To enter it, push the button marked Access Resource Center. Section two gives access to the user submission facility. To submit an artifact push the Submit an Artifact button. Finally, the third section is a source of help, and general information about CHISSA.
When the user pushes the Access Resource Center button, a screen similar to that shown in Figure 3 appears. There are three ways to limit a RISQ system search, and thus there are three major portions of interest in the RISQ Query Selection Criteria screen; a section for selecting artifact types, a section for user defined keywords, and a section for taxonomy based searches.
The search can be restricted to any of a number of artifact types. To select a particular artifact type, click on the box next to that artifact type. A check mark will appear next to those artifact types selected. To deselect an artifact type, click on it to remove the check mark. Checking the "all" artifact type is equivalent to individually checking each of the other artifact types.
To provide a user defined keyword phrase, type it in the box above the button labeled "Keyword". Then push the button. As keywords are defined they will appear, with a check box, under the button. To undefine a user keyword, uncheck the box next to it. Note that case is not important in user defined keywords. For purposes of searching, "Fault" is exactly the same as "fault".
To select taxonomy terms, click on them in the tree. Initially the tree is shown as "High Integrity" plus a set of five buttons-one for each of the areas covered by the taxonomy. Clicking on "High Integrity" will select all terms in the taxonomy. (Indeed, clicking on "High Integrity" and selecting "All" for the artifact type will result in the search engine returning all artifacts in the database.) Clicking on one of the five buttons will result in a taxonomy tree being displayed for that particular area.
Within a tree, clicking on a term will select that term. Clicking on a child of an already selected term will deselect the original term and select the child instead. Likewise, clicking on a parent will replace any previously selected children with the parent. So in the example shown in Figure 4 if "Hazard Analysis" had been previously selected, clicking on "FTA" would replace "Hazard Analysis" with "FTA". Then clicking on "Safety" would replace "FTA" with "Safety".
In the example the artifact types chosen are Abstract, Document, PostScript, and Tool. The user defined keyword is software hazard analysis, and the taxonomy based search will cover the whole Safety taxonomy.
Once the selection criteria are set, clicking on the button labeled "Find Matching Artifacts" will result in a screen similar to that of Figure 5. This screen gives a capsule summary of the section criteria used, and shows the retrieved artifacts, if any. If you are unhappy with what was retrieved, the selection criteria may be modified by clicking on the "Change Selection Criteria" button.
Otherwise the retrieved artifacts may be inspected. Clicking on the underlined text will produce a RISQ abstract similar to that shown in Figure 6. Clicking on a button like the one under the underlined title (or the one shown on the abstract page), will take you to the artifact itself. In this case clicking on the button will take you to the document shown in Figure 7.
The RISQ system provides a means for the user to submit artifacts for possible inclusion in the database. To submit an artifact the submitter provides information about the artifact, including its taxonomy classifications, any user defined keywords, and the artifact types involved. This information is stored for later retrieval by the RISQ administrators who will make the ultimate decision as to whether the artifact will be added to the database.
The process begins when the user pushes the "Submit an Artifact"
button on the RISQ home page (Figure 2). This causes a page very
similar to the search selection page of Figure 3 to be shown.
In fact, except for the title, the button labels, and the instructions,
the page is identical. In particular the taxonomy, user defined
keyword, and artifact type sections behave identically.
Figure 8 shows the screen as it looks after pushing the "Submit an Artifact" button on the RISQ home page. The submitter uses this page to classify the artifact as precisely as possible. Note that it is permissible to have one description apply to multiple artifacts (e.g., a document and a tool may share the same description.) In this case only one entry will appear in the database, but it will be displayed with multiple buttons-one for each of the artifact types.
Once the user has set the search terms for the artifacts, he pushes the "Submit Artifact" button and a page similar to Figure 9 appears. This page includes a capsule summary of the terms selected, along with additional fields to be filled in. The first field is the e-mail address of the submitter. This is important as it is the only way that the RISQ administrators can get in touch with the submitter. The next field is for the title of the artifact. Following that field, there will be a locator field for each of the artifact types selected. In this case there is only the one locator-for the document artifact type. This field will typically be filled in with a
URL that points to a location where the artifact may be found. Finally, there is a field for a short abstract that describes the artifact. Once the user is satisfied with his submission, a click on the "Submit" button saves it away and notifies the RISQ administrator that the submission is ready for them to look at.RISQ administrator that the submission is ready for them to look at.
For the RISQ administrators, maintaining the RISQ database is done almost the same as browsing the RISQ system. Maintenance mode is accessed via the URL: /maintain.html. Figure 10 shows that page, which is divided into two sections. The first section, with the button label "Access RISQ Maintenance Center," allows the administrator full access to the database. The second is used to review and approve or reject submissions by others.
Clicking on the first button will cause the familiar selection criteria screen (Figure 3, Figure 8) to appear. The only difference is that there are now two buttons: "Add Artifact" and "Modify Artifact".
To add an artifact to the database, the search terms must first be selected using the Maintenance Selection Criteria form. Once the terms are completely defined, the "Add Artifact" button is pushed resulting in a screen that looks similar to Figure 9. The only differences are the lack of a field for an e-mail address, and the addition of a field for a password. The rest of the form is identical and is filled out in the same manner.
Once the form has been filled out (including the current password), pressing the "Add" button will cause the artifact to be added to the database.
To modify an artifact in the database, it must first be located. This is done exactly as if the artifact were being searched for-the search terms are set and the "Modify Artifact" button is pushed resulting in a screen similar to Figure 11.
This screen is almost identical to Figure 5 with the addition of two buttons "Modify" and "Delete". Pressing the "Delete" button will give the maintainer a chance to review the artifact and then delete it. Pressing the "Modify" button will present the maintainer with a filled out form similar to Figure 9. This form can be edited and then submitted to complete the modification process. Of course a password is necessary to complete either of the modify or delete operations.
The other function of maintenance mode is to allow the RISQ administrators to review artifacts submitted by others and to add them to the database with a minimal amount of effort. This process is started when the "Review Submissions" button is pressed on the maintenance home page.
The RISQ Submission Review page is shown in Figure 12. Every artifact submitted is shown in a list. The maintainer can select one of the artifacts and either delete it, or install it. In either event the maintainer will be given a chance to review the artifact (in a form similar to that shown in Figure 9) before proceeding. Either installing or deleting a submitted artifact will result in it's being deleted from the Submission Review page.
We expect that making tool demonstrations available through the RISQ system will be particularly important. There are several sorts of demonstrations that are suitable for inclusion in the RISQ system. We have identified three of them:
A slide-show a sequence of Web pages, containing text, graphics, and perhaps even audio or video clips. These pages are linked together to tell a story in much the same way a speaker would use overhead transparencies.
A canned demo a sequence of Web pages, containing images captured from the tool being demoed. Using the Web's image map facility, the viewer is able to manipulate the images in much the same manner as he would manipulate the real tool.
A ìliveî demo a combination of a live demonstration of a tool on the userís screen, and a set of Web pages which guide the user through the demonstration.
Each of these is implemented in a different manner and we will treat each of them in turn. The key to creating a successful demonstration for use in the RISQ system is a carefully thought out storyboard. The storyboard has its roots in the field of animation. It is a sequence of pictures depicting the key events in the story. It is used by animators to lay out how the story will progress linearly through time. This ensures that the foils created by multiple animators can be assembled into a coherent story. Since a good demonstration tells a story, the storyboard is equally important to the RISQ system demonstration designer.
The storyboard will be different depending on the type of demonstrations being given. For the slide-show it will most likely be based on a traditional slide presentation. For either of the other two kinds of demonstrations, the storyboard will consist of a tree (or even graph) of paths through the operation of the tool.
Given that a presentation already exists, creating a slide-show for the RISQ system is easy. In its simplest form the slide-show consists of on-line copies of the slides already in the presentation. However, to make this demonstration useful, it is important to realize that the slides are not enough. In a verbal presentation, it is usually the case that the slides do not tell the whole story. Rather, it is the combination of the slides and presenter that tell the story. For a slide-show to work on the RISQ system, the words that the presenter adds to the slides must be captured and displayed. One approach to doing this is to attach notes at the bottom of each slide as shown in Figure 13.
For the slide-show to work, the viewer must have a way of progressing through it. The buttons at the bottom of the slide accomplish this, allowing the viewer to move forward, backward, or to a ìhomeî page.
To take full advantage of the capabilities of the Web, the slides can be embellished with links to other relevant items. For instance, the slide in Figure 13 has a reference to POSIX. This reference could easily be made to be a link to an on-line version of the POSIX standard, if it is available. Also, due to the nature of the Web, the slide-show can become multimedia, including both sound and full-motion video.
To make a slide-show presentation suitable for inclusion in the RISQ system, we ask that:
We discuss the creation of a live demonstration first, because in many ways it is easier to create than a canned demonstration. These instructions apply to live demonstrations of X Window System based applications.
The first step in developing a live demonstration is to determine what you would like to show the user. This will usually result in a sequence of actions for the user to perform, and a set of descriptions of what the user is seeing on the screen. The user actions, and the descriptions should be captured on a set of Web pages that take the user through the demonstration step-by-step.
For example, Figure 15 shows a sample screen, while Figure 14 shows the accompanying Web page which describes it. As with the slide-show, the Web pages can be embellished with links to other sites, as well as audio and video segments.
For a live demonstration to be suitable for inclusion in the RISQ system, we ask that
The actual implementation of the live demo is very server and tool specific.
The canned demonstration is a bit of a hybrid. It can be thought of as a slide-show with some live demonstration capabilities added.
The first step in developing a canned demonstration, as for the live demonstration, is to determine what you would like to show the user. This will usually result in a sequence of actions for the user to perform, and a set of descriptions of what the user is seeing on the screen. The user actions, and the descriptions should be captured on a set of Web pages that take the user through the demonstration step-by-step.
The above paragraph is virtually identical to the one for the live demonstration. The difference is that the canned demo relies on the creation of a clickable image, using the Web's image map ability. Figure 16 shows a portion of a Web page from a canned demo. It happens to correspond to the pages show in Figure 15 and Figure 14. As with the slide-show, the Web pages can be embellished with links to other sites, as well as audio and video segments.
For a canned demonstration to be suitable for inclusion in the RISQ system, we ask that
In order to evaluate the success of a particular demo, and the success of the RISQ system, we need to receive feedback from our visitors. The demonstration evaluation form is one mechanism to help accomplish this. To create a demonstration evaluation form for a demo, you must modify a copy of the file ìeval.htmlî which is available from the RISQ system in the templates directory. Specifically you must:
The modified version of the demonstration form is the one which your demo should point to.
World Wide Web (Web) technology has provided access to enormous amounts of information on-line. As such it has a tremendous potential for use as an aid to technology transfer. However, locating relevant information is often difficult partially because of the volume of information available. This report has described a new search engine which is designed to allow the user to do efficient searches for information within a specific domain. The initial implementation of the engine is in the domain of high integrity software systems and is called RISQ which stands for "Reference Information for Software Quality."
The RISQ system is an important part of CHISSA, the Center for High Integrity Software Systems Assurance. CHISSA was organized by NIST's Computer Systems Laboratory in 1994 as a collaborative approach for government, industry, and academia to make available the technology which is necessary for assuring high integrity software in an ever growing number of applications.
The RISQ facility makes available a wide variety of artifacts related to software quality in a highly organized manner. The overarching goal is to help to transition technologies so that the best of them become standard practices when developing software systems with a requirement for high integrity.
The current RISQ facility is located on the World Wide Web ( Web) at hissa.ncsl.nist.gov/risq/. By locating the facility on the Web, we achieve the goal of making the material available to a large and geographically diverse set of potential users.
While the initial database of artifacts is small, the system contains a "submit artifact" capability to enable the user community to nominate artifacts for inclusion. The RISQ administrators will verify that these nominations meet the acceptance criteria.
Future work on the RISQ system may include:
The taxonomy created for the RISQ system represents a distillation of taxonomies from a number of different sources. The resulting taxonomy is a joint creation of the authors and Laura Ippolito of NIST, and Dr. Marvin Zelkowitz of NIST and the University of Maryland, both of whom have provided significant advice on the system design and implementation as well. The original prototype of the RISQ system was designed and implemented by Tony Cincotta of NIST. Many of the ideas incorporated into the current system are based on his original design. Kathy Harvill provided database population assistance.
