I have long been interested in process improvement, both in Development and Testing.
In particular, I have worked to identify a structured approach to designing test cases and scripts. This has led to greatly improved defect detection rates in several trials.
Scope
It is evident to me that both Manual and Automated test cases should be testing the same conditions, therefore this note will be applicable to both.
We first have to appreciate the purpose of scripted testing, which is often mis- understood. It is to demonstrate that the system under test fulfills its design requirements, and not specifically to detect defects – these are discovered as a result of test failures. Often, additional unscripted “exploratory” testing is carried out which specifically looks for bugs. It is usually unrepeatable, and not part of a structured method. This is not covered in this article.
It is possible in most cases to analyse and display the design logic in a hierarchical chart form. The key to the method is in the systematic identification of alternative possibilities. The design document is charted in a top-down, left to right manner, with each branching point (node) owning a number of subordinate nodes corresponding to the identified alternatives. The example below is for a simple login screen, with just a user id and password.
Examination of the design document should identify any conditional phrases, and ensure that all possible alternatives are documented. It is easy to identify illogical or omitted alternatives, e.g., in programming terms, an IF without an ELSE. Using this approach, the test cases are identified very quickly.
When complete, the chart will show, as a separate branch, every alternative (Event Flow in UML terminology) which has been specified in the design, and the conditions needed to navigate to a specific test case are clearly displayed.
Requirements can be attached to the relevant branch of the tree (In blue, below, prefixed *R). This enables test coverage to be demonstrated, and test cases to be easily mapped to requirements.
The hierarchic form of presentation ensures that maximum advantage is taken of commonality between event flows – it is usual for several test cases to be identical up to a decision point, and this part of the script only has to be coded once.
The hierarchy serves to highlight any inconsistencies between these flows. If requirements are inconsistent, it will not be possible to form a proper hierarchy – indicating design errors.
Where tests need to be prioritized, it should be clear which branches of the tree are most important, and similarly, when a defect is found, it will be obvious which branches (tests) are blocked by the failure.
Creation of the diagram needs a skilled resource, but this is far outweighed by the improvement in error detection rate at an early stage of design, preventing the incorporation of errors into code, when they are much more difficult to correct. Value is added by the use of the charts in creating system test cases.
Stakeholders can easily see just what is covered in the test scripts. This is more difficult when they are just presented, for example, with a spreadsheet of test cases.
Unlike common approaches, this method is well-structured, in that it is
- Objective: It will contain only a summary of the relevant facts within the design document.
- Consistent: Different testers should be able to produce similar results.
- Auditable: The resulting diagrams can be archived as proof that the test script has been fully completed, covering all requirements. The archived diagrams clearly show the logic behind the scripting, which is not apparent if just Quality Centre or Excel scripts are archived
- Trackable: The diagrams can be placed in a source repository under change control in the same way as other system deliverables.
- Maintainable: Document updates are easily incorporated into the chart, and can identify new errors introduced by the changes.
- Adds value: When correctly designed, the charts can be directly converted to test cases, which can easily be kept in line with any changes to the documentation.
Software support
I have long believed that we in IT could make more use of our expertise to streamline our own work. Some VBA macros have been written for these VISIO charts to produce test scripts (Manual, e.g. Excel, and Automated, for Selenium or Winrunner). Note that the tree structure is unchanged, because it represents the same tests, needed in all test modes.
Just as important are a number of auxiliary reports, such as cross references of test cases to requirements, and a test coverage report, all of which can simply be regenerated after updates. The macros allow for textual replacement variables for test data, and test data files can be generated which match the scripts.
In conclusion, it has been found that by following this process, highly effective test scripts can be prepared with less effort than using traditional methods.
The design method has been shown to offer the following benefits
- Documentation errors are much more likely to be identified. In one banking project, a significant number of serious defects were identified during the test design activity, before any tests were even run.
- Test Coverage is more complete, as documentation omissions are more easily identified. This is a major contributor to the improvements seen in detection rates in trials.
- It provides efficient creation of test cases, particularly for GUI where it can be difficult to enumerate the total possible interactions.
- Modification for system changes is easy – test scripts are just recreated from an updated diagram.
- Pictorial representation is an aid to understanding and allows tests to easily be reviewed
A Test Manager at a trial site (a Government executive agency) estimated that compared to other similar projects, defect detection rate was improved by 200% to 300%.
One test analyst at a major bank was very impressed and asked me “why didn’t we always do it this way” to which my reply was “no-one showed you how”.
Simple example:
About the Author:
Neil Jennings | Test Manager
I have been an IT worker for my whole career, starting with ICL. I have been a contractor since 1982, when I started my Ltd company.This has taken me abroad, to Saudi Arabia, Dubai, Luxembourg, Gibraltar, The Netherlands, Belgium, and most recently as a Test Manager in Germany, working in banks and insurance.