Trantor Ltd
Copyright © 2006, 2011
Trantor Test Suite
Use of Visio diagrams to create test scripts and data
Overview
It is possible in most cases to design and display test cases in a hierarchicalform, using the Visio “Organization Chart” format.
This has been shown in practice to offer the following benefits
- Functional coverage is more complete, as omissions are easily identified
- Pictorial representation is an aid to understanding the test scripts
- Documentation errors are more likely to be identified
- Design defects are recognised early in the development
- Document updates are easier to apply to test scripts
- Test planning is facilitated
- Testing effort is estimated
- Test creation effort is reduced
- Expected results can be calculated directly from the actual test data
By suitable formatting of the Visio chart, test scripts, test data files and an automated test script can be produced by VBA macros from the Visio text itself. This replaces much of the effort needed to create manual and automated test scripts.
This document should be used in conjunction with the sample Visio diagrams provided.
Contents
Overview......
Process Stages......
Stage 1 example:......
Stage 2 example:......
Stage 3 example:......
Part 1: Test cases and simple test scripts......
Definitions......
Test Case......
Preparing the Visio Diagram......
Conditions, User Actions and Expected results......
Business requirement mapping *R......
Test Planning and dependencies *T:......
Labels :A......
Defects *D......
Test Effort *M......
Test Elapsed time *E......
Useful menu functions......
Computer-assisted Script Creation......
Running the Renumber and Refresh commands......
Licence number......
Reports generated by Renumber and Refresh functions......
Custom Menu......
Default Thread (menu item and command button)......
Select Subtree (menu item and command button)......
Running the Scripts processes to generate Test Scripts......
Script Control Commands within shapes:......
Repeat......
Repeat All......
Go To......
Part 2: Enhanced Test Scripts......
Version Control (optional)......
Creating the Excel Test Script (Manual script)......
Amending a Visio diagram......
Test Results and Metrics (Optional)......
Reports generated by Script functions......
Part 3: Winrunner scripts and Test Data creation......
Defining the Test Data......
Definitions......
Automated Test Actions – Assistant shapes......
Variables......
The EVAL (evaluate) function......
Data and External references......
INCLUDEPATH statement......
INCLUDE statement......
DEFINE and POPULATE statements (in assistant shape)......
Consultant shapes......
DefaultThread, Goto’s, CALL and REPEAT: Advanced features......
Repeat All......
CALL......
Go To......
Data steps......
Use Case precursors......
Test case preconditions and data......
Test Specifications......
Checking the output from Automated tests......
Adding the test data and automation scripting to the chart......
Generating the Test Scripts......
Reports generated by Scripts functions......
Appendix 1: Test Case renumbering problems......
Test cases not numbered left to right in the diagram......
Duplicated test case numbers......
Appendix 2: Structure of the Automated Test Script......
Appendix 3: Automation syntax notes......
Appendix 4: Action Control characters in Visio charts......
Appendix 5: Quality Center restrictions......
Test History......
Reserved characters:......
Appendix 6: MS Visio software defects......
Sheet Shape bug......
Appendix 7: Diagram export/import......
Appendix 8: Chart Preparation – Importing the VBA files......
Appendix 9: Built-in variables......
Process Stages
Trantor Test provides a seamless progression from test design to test execution.
Three stages of increasing functionality have been defined:
- Stage 1 is the minimum required for manual testing.
Charts in stage 1 format can be used directly by testers, who can run the tests by referring to the Visio diagram. The act of creation of a stage 1 diagram is also useful for verifying the design documentation.
- Stage 2 is designed to produce manual test scripts suitable for import to a test repository such as Quality Center.
Charts in stage 1 format can be progressed to stage 2 simply by adding script generation and control information. The script design is not changed.
- Stage 3 is designed to produce automated scripts for import to tools such as Winrunner and Loadrunner. It is also used for creating test data.
Charts in stage 2 format can be upgraded to stage 3 by the addition of data definition and generation items to the existing structure, as well as automation commands.
Stage 1 example:
This shows a test design for a simple Login screen:
Advantages of using this method for test design include:
- Tests are created very quickly
- Test cases can be derived directly from design documents
- Coverage can easily be demonstrated
- Tests are easily modified when the design is changed
- Test status and defects can be recorded in the chart if required
- Tests blocked by known defects are easily identified
- Uses only existing software (Visio)
Stage 2 example:
When planning to create formal test scripts from the chart, the Visio charts should be prepared as specified in stage 2. Codes have been added to the text to drive the script generation, and also markers such as *R for business requirement, and *T for test priority.
Extra advantages of stage 2 are:
- Effort to create and document manual test scriptsis much reduced
- Scripts can be mapped to requirements, and a cross reference is produced
- Script narrative in text format can be created, containing estimated effort
- Scripts can be exported to Excel, for import to test repository such as Quality Center
- Scripts in the repository are easily modified when the design is changed
- Scripts can be selected for execution by priority, aiding risk-based testing
Stage 3 example:
The charts can be further updated to stage 3, in which test data files and automation scripts can be produced. The shapes in blue have been added to the previous diagram to specify the test data and scripting for the tests, leaving the logic unchanged.
Further advantages of using stage 3 are:
- Effort to create test data and automated scripts is much reduced
- Data and Automated scripts are kept in step withtest design and manual cases
- Scripts and data are easily modified when the design is changed
- Each test can be scripted once but repeated with several sets of data
- Expected results can be calculated programmatically using EVAL function
Part 1: Test cases and simple test scripts
Definitions
Test Step
A test step is a single line within a test script. It can be a user action, system response, a pre-condition or a post-condition.
Test Case
A set of test steps for a single test. It is known as a “thread” in the diagram.
Test Set
A set of test cases which are represented as one or more tree structures in a single diagram (even if spread over several pages). They are usually based on a single set of design documents, for example a single Use Case.
Test Script
A manual or automated test script is one or more test sets prepared in Excel format, suitable for input to a test tool or repository.
Preparing the Visio Diagram
To make use of the scripting features, the test case tree must follow a defined structure:
One or more Executive shapes are used as the top level entry points.
Each executable test step, precondition or test result is held in a Manager shape, which must be attached to its owner by using the automated method (i.e. Drop the subordinate shape onto its owner – Visio will then connect it for you. Manually adding connectors will NOT work!).
Unattached shapes will be ignored by the various functions.
The final mandatory box in each thread is a Position box which contains a description of the test case and the test number.
When using the Visio chart for test recording, it is possible to add an assistant shape to each position shape, containing text starting with the word “Pass”, “Fail” or “Blocked”. Test metrics can be calculated by Trantor Test from these assistant shapes.
So a typical structure would look like this (before colouring).
Example of the use of the assistant shape for holding test status:
Test case numbers
All test case numbers are held in the terminal Position shapes, enclosed within brackets ( ). Only the LAST pair of brackets in the shape will be used. If an item longer than three characters is found within these brackets, or no number is found, an error message is given and the test case is not renumbered.
Manager shapes - Test case step descriptions
The test steps should be described in sufficient detail that they are meaningful to a tester. (Remember that within Quality Center, the tester cannot see other test cases when executing a test, so a test should be self-contained.)
Conditions, User Actions and Expected results
You can identify each of these by preceding them, within the Manager shape, by a marker string
for pre-conditions
== for user actions
for expected results and/or system actions.
%% for post-conditions
Pre-conditions are the conditions necessary for the specific test case to be executed. (They can also include instructions for suitable test data.). These have no effect other than to inform the tester. All preconditions of a test case are listed in the test case description within the generated Excel script file.
The classification of an item as a precondition may depend on various factors. Thus
“System performs Use Case 01” may be treated as either a precondition or an expected result, depending on whether the result is observable, and on whether this is part of the specific test case. Basically, it means a step that is necessary, but that the tester does not have to check the output specifically.
User actions are the test steps themselves. For a batch system, they may be the actions which trigger this function, whether or not manually supplied.
Expected results are (observable) data and system actions expected as a result of the user action(s). There is little point in including results which cannot be checked, for example intermediate results which are not saved.
Post-conditions are any significant system states after the test has completed. They may not necessarily be related to system actions, for example, if a requirement is that a particular item is NOT changed by the test.
Examples:
==User clicks cancel
> System displays confirm cancel message
Note: It is recommended for clarity that each action is coded in a separate manager shape. However, there is a performance restriction in older versions of Visio which makes it desirable to be able to include multiple actions in a single shape.
As well as the description of the test step, manager(and assistant) shapes can contain comments enclosed between ^ signs. Comments are intended for use within the chart, and will not be exported to the test scripts.
Business requirement and test chunks can be included in the format noted below.
Business requirement mapping *R
One or more business requirements can be coded within each manager shape, in the format *R xxxxxx. If present in this format, they will be included in the AEF report. (see below) and in the text of the corresponding test steps. Requirements can also be coded in anyassistant shape which is attached directly to a manager.
The format *Rn xxxxxx can also be used, where n is a digit (1 to 4), to assign business priorities to the requirements. A higher number indicates higher priority.
The Renumber, Refresh and Test Script functions will produce a requirements list, with each requirement shown only once, to help verify coverage, and will also produce a cross reference of all requirements to test cases.
If no business requirement is present in a test case, the AEF number within square brackets (such as [AEF0664] above)will be used as a default.
Test Planning and dependencies *T:
Test prioritiescan be represented by using *T n. This works in a slightly different way to business requirements, in that only the LAST one found, working downwards, is reported. Therefore you can put the whole of a test set in priority 1 (Low), and then selected threads can be placed in chunks 2, 3 and so on. The priority n should be a single digit. The program will calculate the overall priority of a test case by multiplying the test priority by the business priority. The test priority can also be coded in an assistant shape attached directly to a manager.
Test Case terminator (Position shape)
The purpose of this item is to mark the end of a test case and to supply its description and reference number. It is NOT an executable test step.
The contents of the position box (Test case terminator) should be a flow reference and test description, enclosed in braces (curly brackets { }).
The flow reference should ideally be in the format MEFxxx (Main Event Flow) or AEFxxx (Alternate Event Flow) which are recognised by the software and included in the AEF reference list.
The flow reference should refer to a specific flow within the design document (e.g. UML Use Case description). If there is no suitable flow reference within the source document, then a reference may be created or assigned directly by the test designer.
The test case description should, where possible, be copied directly from the description in the design document, to avoid errors in interpretation.All text within the braces will be exported to form the test description, but any text added after close bracket will not be exported.
The final item within the position shape is the test case number, up to 3 digits in ordinary brackets ().
Notes:
When initially creating a Visio chart, the test case numbers will automatically be inserted between the brackets by the renumber function.
The text within the braces will be used as the test case description when the Test Scripts are generated.
If the test case was created as a result of a defect report (e.g. against the documentation), then you could use the defect reference.
Important: The position box is not a test script step, and user or system actionsshould not be coded within it, as they will be ignored.
Ensure that the MEF or AEF number is in the correct format (no embedded spaces).
Hint: Where the AEF or MEF number is less than 10, it could be specified as AEF_n, to ensure that you can later sort the AEF report into the correct sequence. For 10 or more, it should be AEF10 etc. This helps to provide a coverage check that all AEF’s have been addressed in the diagram.
Labels :A
Each REPEAT or CALL (see below) must refer to a label. Labels can be up to 12 characters long. They must be prefixed by a colon and be the first entry in their box, so that the software can recognise them. (But no colon is used in the repeat itself). For correct import into Quality Center, especially in stage 2 (see below), it is necessary that each label is placed in a box by itself, with no other content.
Example: If you have “Repeat A” in a consultant shape, then you must have a box whose text is “:A”. All declared labels are shown on the AEF report, to make it easier to find them on the Visio chart.
Defects *D
If a defect (Test Incident) has been raised, but not resolved, it can be coded in manager shapes or attached assistant shapes as *D xxxx where xxxx is the defect reference or number.It should be coded at a suitable point in the hierarchy so that all affected subordinate test cases can be identified.Defects can be listed against cases in the AEF and test statistics reports by picking the appropriate column on the AEF picker form.
Test Effort *M
*M followed by an integer (representing a percentage) can be used (in assistant shapes ONLY) to indicate the complexity of a script as a percentage. For example *M 150 would be used to indicate that this was 1.5 times as complex as a standard script, thus requiring more time to run. Only one factor is taken into account for a particular test, which is the one nearest the bottom of the diagram. This allows a generic factor high in the tree to be overridden by more specific factors for individual tests. The factor is multiplied by the number of steps to indicate total expected effort per test script in the “Test Narrative” report.
Test Elapsed time *E
*E followed by an integer (representing a number of minutes) can be used in manager and assistant shapes to indicate the expected timing of a script step or series of steps. For example, *E 60 would be used to indicate that this step was expected to take one hour. Useful for batch jobs which may take some hours to execute. The total elapsed time for each complete script, calculating by summing all the individual steps, is shown in the Test Narrative report. Default time for each test step is assumed to be 1 minute in addition to any specified times.
Audit Trail information
The executive shape at the top of the Visio chartmay contain the version number and/or Use case version within curly brackets { }. If present, these will be copied into the generated test script.
Shape Fill Colours
The fill colours of the shapes in the chart are set according to the type of shape and, for manager shapes, the text within the shape. This may be done manually, using the menu options, or automatically when the renumbering or test script functions are run. You don’t get to choose the colours – they are preset. If there is sufficient demand, a selection form could be implemented later.
Useful menu functions
A number of menu functions have been provided. They are not part of the test scripting, and are just present to help in creating the Visio diagrams.
On the TRANTOR menu, there are:
For all shapes on current page:
Colour shapes by type
Fills all shapes on the page except Manager with different colours. Only fills shapes if they are currently white or not filled – it does not change colours already assigned.