Agility with Component Based Test DesignPage 1 of 12

Final Paper

Agility with Component Based Test Design

Abstract

Software systems have been constantly growing in complexity. The software development processes has been consistently evolving towards the goal of more agility and shorter release cycles. In light of these developments; handling complexity with ease, nimbleness in accommodating the changes to test inventories, improved turnaround time for automation of Regression cases and improvement in productivity of human resources to be able to make quality releases in short time spans becomes paramount.

In this paper a design approach to create and organize test templates, and a process to create test inventories using test templates to achieve the above mentioned objectives will be discussed.

The process which consists of five phases is discussed in detail. Also, the design approach of templates which starts with the identification of templates, then their development and finally their classification as L2 and L3 to break down the complexity into smaller pieces and achieve flexibility and versatility in design has been discussed. Application of the proposed design approach and process to a sample problem has also been included.

The value proposition and cost benefits will be discussed towards end of the paper.

Table of Contents

Table of Contents

References & Appendix

Intended Audience

Key takeaways for the audience

Field Study1 (Test case Design metric without using Templates)

Current Challenges in test case design in Agile World

Taxonomy

The FITSAA Methodology

Frame

Identify

Templatize

Factors to be considered while designing Templates

Classification of Templates for Test inventory

Guidelines for entering L3 and L2 test templates in HP ALM

Structure

Automate

Adapt

How Templates provide a solution?

Field Study2 (Test case Design metric using L2/L3 Templates)

Conclusion

Author`s Biography

Main Author: Anand Arumilli -

Co-Author 1: Akash Gupta-

Co-Author 2: Satya Surya Naga Prasad Chilukuri-

References & Appendix

•Agile Testing

Book by Lisa Crispen & Janet Gregory

•The Art of Software Testing

Book by Glenford Myers

•HP ALM User Guide

Intended Audience

Test Management Professionals – Managers, Test Leads, Test designers, Test Engineers

Key takeaways for the audience

1)Utilizing the ALM provided templatized tests for designing test cases for “Rapidly Changing System”.

2)A potent heuristic that can harness the power of templates for test design in an Agile World

3)Explore the usage of templatized tests to handle common pain points in test case design and Test Inventory Management.

4)A systematic process for managing test Inventory and automation in AGILE projects.

Field Study1 (Test case Design metric without using Templates)

The following metric was captured with the following considerations without using Templates.

Metric Definition / Without FITSAA
Test authoring (90 Complex Test Cases) / ~90 Hrs
Impact analysis and Incorporating changes / ~40 Hrs
Automation turnaround time / ~130 Hrs

Result

The test engineer took ~90 hrs in understanding the design document and writing the test plan having 90 test cases.

The test engineer took ~40 hrs for impact analysis and updating the regression impacted test cases without using L2/L3 templates.

Current Challenges in test case design in Agile World

•Rapidly changing system

•Test authoring cycle time

•Impact analysis paralysis

•Tedious documentation of tests

•Automation not in synch

Taxonomy

L0 (Flows) High level lightweight test cases for any incoming new feature Example: Verify learner is able to enroll into a course. (L3/L2 not used)

L2 (Transactions )A parameterized transactional flow which has calls to L3/L2. Example: Creating a Course.

L3 (Actions) A parameterized atomic entity which performs a set of actions. Example: searching a course, clicking a menu/button.

L1 (Tests) An executable test which has an objective to validate a workflow. An L1 makes calls to L2s and L3s. It ingests data and has data specific assertions. Example: Admin creates a course (L2). Learner searches for this course (L3) and enrolls (L2). Verify the assignment status.

The FITSAA Methodology

To solve the current challenges in the AGILE world we use a design approach to create and organize test templates, and a process to build test inventory and automation using test templates. This consists of the following stages:

  • Frame
  • Identify
  • Templatize
  • Structure
  • Automate
  • Adapt

Frame

Whenever a new feature is introduced in a milestone, we perform the following actions in this stage

  • Review Functional design
  • Design L0s
  • Review L0s

Identify

Next , we perform the following actions to identify the potential L1 candidates

  • Identify which L0s get promoted to L1s
  • Review

Templatize

What are Parameterized Test Templates?

A parameterized test case can be defined as one where thedata that can take varied range of values;is made a parameter (variable) in thetest templatesto enablere-useof the templates forcreation of end level cases. A templatized and parameterized test inventory has three components viz templates, parameters and end-cases.

A test template is a combination of detailed steps and parameters; which are used very frequently as part of data creation and end-level test case execution. A test template can be"called"in an end level case. The test designer just needs to assign values to the parameter in Test Plan. At the time of running the test case in QC-Test Lab, thecalledtest is displayed Step by Step with the parameters replaced with the actual values that the Test Designer had provided.

Factors to be considered while designing Templates

Following factors should be considered when deciding on and designing the templates:

  1. Test templates should deal with a common functionality that is used in multiple test cases as a prerequisite.
  2. Templates should be designed and parameters should be chosen in such a way that they are versatile and flexible to accommodate changes, variability of data and similar functional flows.

Classification of Templates for Test inventory

The test templates can be classified in to two categories viz. L2 and L3 based on following criteria:

L3 Templates / L2 Templates
-These primarily serve as building blocks for L2 and L1 cases. / -These primarily serve as Building blocks for L1 cases.
-Have steps for a commonfunctionality inparts/components of a workflow or a part of the ; navigation path or actions in workflow. / -Have more specific functionality and usually cover major parts and functionality of a workflow.
-Ca not call an L2 template but can call other L3 templates. / call the generic L3 templates and then based on parameters in the L3 case we create different L2 templates.
-Are parameterized but can't be executed independently of an L1 case. / -Are parameterized but can't be executed independently of an L1 case.
-Examples: Search for a user, Login and Navigate to Learning Administrator page, Login and Navigate to Learner page, Choose type of assignment in Manage Assignment workflow, Navigation to create a class etc. / -Examples: Create a Course, Create an ILT Class, Create a Self Paced Class, Create specialization, Add sections to Specialization, Verify action on Course details page etc.

In this stage we create templatized tests. We follow the following steps:

  • Review the scope for reusability
  • Identify L2/L3s
  • Design the Templatized tests

Guidelinesfor entering L3 and L2 test templates in HP ALM

  • A test templateshould not haveanyhard navigationcontained in it. All navigation should be dealt as part of the L1 case.
  • All variable data should be marked as parameters.
  • All mandatory parameters should be marked with an asterisk e.g. CourseName*
  • The parameters should be given relevant names instead of random names. e.g. variable course name should be named asCourseName* instead of abc or parameter1 etc.
  • All assertions done in test templateshould havea variable named as'Assert'. The assert variable can have value 'Verify' or 'Skip' depending on whether test designer wants to make the assertion or skip it.
  • A test template should have only thenoassertions.
  • Variable names which are commonto multiple templates should be named in such a way that they areeasily distinguishable. e.g. SaveOption is a variable name common to both CreateClass and Create Course templates; hence the variable should be named as 'CreateCourseSaveOption' and 'CreateClassSaveOption'.
  • All 'Assert' variables should have the format:Assert<Test case no. in Test Name> .<Assertion number>. Assertion number is valid only when a test template is making multiple assertions. In case ofsingle assertions "Assert<Test case no. in Test Name>"should suffice.

Structure

In this stage we design L1s using the L2 and L3 components. We follow the following steps:

  • Design L1s
  • Review L1s

Automate

The automation is built on a function based approach that mimics the components used in the manual inventory. We use a hybrid framework to achieve this. In this stage we have the following steps:

  • Develop L2/L3 automated scripts
  • Develop L1 automated scripts

Adapt

Whenever a the system undergoes a change, we have to analyze the impact to existing test inventory and quickly adapt to the changed system. Here we

  • Analyze impact on L3/L2/L1
  • Update/Develop L3/L2/L1

How Templates provide a solution?

  • Reusable test steps
  • Reduced redundancy
  • Quick Impact Analysis
  • Quick Automation Turn Around
  • Quick Estimation
  • Faster Test Case Authoring
  • Maintainability of Test Inventory

Field Study2 (Test case Design metric using L2/L3 Templates)

The following metric was captured with the following considerations using Templates. The same Test Engineer in Field Study1 was assigned a Functional design of equivalent scope and complexity.

Metric Definition / Without FITSAA / With FITSAA / Time Saving With FITSAA
Test authoring (90 Complex Test Cases) / ~90 Hrs / ~50 Hrs / ~40%
Impact analysis and Incorporating changes / ~40 Hrs / ~10 hrs / ~75%
Automation turnaround time / ~130 Hrs / ~ 70 hrs / ~45%

Result

The test engineer took ~50 hrs in understanding the design document and writing the test plan having 90 test cases. This led to saving 40% time.

The test engineer took ~10 hrs for impact analysis and updating the regression impacted test cases without using L2/L3 templates. This led to saving 75% time.

Conclusion

The FITSAA methodology and component based test design helped achieve the following

Testers Predicament / How FITSAA solved the problem
Rapidly changing system / Quick adaptability using L1/L2/L3
Test authoring cycle time / ~40% Time Saved
Impact Analysis Paralysis / ~75% Time Saved
Tedious Documentation of tests / ~430 reusable components
Automation not in synch / ~45% time saved in development

Author`s Biography

Main Author: AnandArumilli -

Anand is a QA Group Manager in Oracle India. He has an avid interest in Test Design and Optimization Techniques in Testing. He has 14 years of experience in QA and a strong development background of 2 years. Anand is a test enthusiast and has been an active participant in the various QA Forums in India and abroad- EXPOQA (Spain), Software Testing Conference (QAI), Step-IN Forum Hyderabad Chapter. His paper on Test Design won the QAI Testing Leadership award in 2011.

Co-Author 1: Akash Gupta-

Akash has 5 years of experience in Quality Assurance. He has done my B Tech in Electrical Engineering from IIT (BHU), Varanasi. He has an avid interest in test optimization and automation.

Co-Author 2: Satya Surya Naga Prasad Chilukuri-

Satya has 9+ years of experience in Quality Assurance. He is M Tech from NIT – Trichy. His past experience as a Tester/Test Lead in Capgeminifor 5+ years and Senior Member Technical in Broadridge for 2+ Years. He is an innovator in test design.

THANK YOU!

Oracle India Pvt Ltd