Challenges Faced by QA Team in DevOps Testing

Abstract

The DevOps is a concept for software development and delivery process that highlights collaboration between software development and operation professionals. The popularity of DevOps has grown in recent years as it helps in releasing a software rapidly, more frequently and reliably. The DevOps practice is being more often adopted for Microservice based architecture application as each service is small, easily tested and deployable through continuous integration.

As per the survey published in January 2017 by RightScale, DevOps adoption has increased from 66% in 2015 to 74% in 2016. As we move on its important to know the different challenges faced while implementing this concept from testing point of view. Hence this whitepaper focuses on providing information on different challenges faced while testing Microservice architecture based application implemented with DevOps concepts.

The testing community and test architects would find this paper very handy.

Business Case and Solution

This business case is related to one of the world leader in financial technology solutions which provides payment solutions for credit cards, debit cards, electronic payments, and check-related services. Their current system was outdated, not scalable and increased with operational expenses. The team was looking for setting up new system fulfilling following goals of business expansion.

  • Faster time to market
  • Allow easy, agile expansion for other countries, and for future acquisitions
  • Reusable services supporting other business workloads
  • Better deployment frequency
  • Improved end user satisfaction
  • Increased ability to build the right product by fast experimentation

In order to achieve the above goals, the team decided to go with a concept of DevOps. The DevOps is more effective when software systems meet set of architectural significant requirements (ASR) such as deployability, modifiability, testability, and monitorability. One of the architectural style falling under ASR is micro service based because the size of each service is small, it allows architecture of an individual service to emerge by reducing a need for big upfront design, allows early software release and continuously.

Hence the business solution team decided to implement new system with agile methodology, DevOps concept, Microservice architecture style and BDD way. The architecture goal was to build an application enabled through use of cloud native capabilities, Microservices and Continuous Delivery.The application was comprised of a number of Microservices to avoid the problems of traditional monolithic applications as they are modelled around business domains.

This implementation achieved the desired goals around availability and maintainability, Microservicesexhibited below traits:

  • discoverable using a service registry
  • software load balancers and circuit breakers to protected from external dependency failure be independently deployable with provision for zero downtime
  • auto scaled up to meet demand and to scale back when demand recedes to optimize cost
  • Be exposed as a RESTful API over HTTPS

Each micro service had a single responsibility. The communication between each Microservice happened either through HTTPs between services or Message Queuing between components. Each service stored in its own source code repository and contain source code, configuration, functional and non-functional tests, documentation, and build and deployment scripts. This enabled effective Continuous Delivery.

Testing Approach

The high level testing approach implemented for this business case and different activities carried out has been explained as mentioned below. The implementation was done in agile methodology with multiple phases. The very first phase was dedicated to pre-work where team had to setup test strategy by performing multiple POCs, test environment enabled for continuous delivery.

The quality engineering approach implemented for this solution has been depicted in the following diagram.

Phase 0 - Testing strategy

Challenges faced while implementing DevOps Testing

The Cultural Shift

DevOps calls for a cultural shift among all stakeholders of the organization to come together and work towards a common goal of creating rapid, reliable and repeatable processes. DevOps encourages development and operations teams to function as one single team, focused on delivering business value across the IT value chain. The traditional team typically is very much used have their own goals, different deliverables and responsibilities. The challenge is to bring all the 3 teams together to make it successful. Between the two differing goals of Development and Operations teams, Quality Assurance (QA) plays an important role as a key enabler in delivering business value.

There are various ways in which we can address this challenge

  • Providing on-job DevOps trainings before the project kick-off
  • Setting up roles and responsibilities of each individual/team based on RASCI chart
  • Setting up objectives focussed more on delivering business value rather than a delivering a piece of feature/code as a team
  • Bringing QA & testing functions as a bridge between all disciplines, from clients and business to development and operations

Skilled Automation Resources:

DevOps implementation expects test automation ~100%. In order to achieve higher percentage of automation, it is important to have skilled resources. Especially when we need to automate the feature which is not having any endpoints, need to have specialized skills in generating Mockito/junit tests to achieve better automation coverage. Also the resource would be expected play “all-rounder” say run performance as well as security tests along with functional automation tests. In reality or in the present situation it is very challenging to staff resources who are good at everything and especially descriptive programming.

This challenge can be addressed in many ways:

  • Identify tools used for all types of testing and provide hands-on training on these to all the testers before the actual development kicks off.
  • Have at least one SME in each of the type of testing who can drive the respective testing activities independently and can handle any technical problems.
  • Be open to take extended help from development team

Deviation from testing mindset:

As like need of skilled resources, testing team having testing as a real user prospective is also a challenge. As the goal of DevOps is to achieve higher number of automated tests at every sprint, team mainly focusses on completing the automation rather than testing the application from user prospective. Most of the time testing team is pressurized to complete the automation in-sprint even though there is a delay in build deployment or feature delivered which is of not good quality.

The real user testing mind-set can be achieved by:

  • Plan for exploratory tests as the normal tests would have been already automated.
  • Create a culture of testing the application along with busyness analysts.
  • If you are having multiple scrum teams, gets the application exploratory tested once by other teams.

CI environment readiness:

As the goal of DevOps is to achieve higher number of automated tests at every sprint, it is very important to plan the CI environment early in the game. It is also important to fit-in the right automation framework to CI pipeline with sample code. If the right test automation solution has not be integrated with CI well before the actual development begins, testing team need to spend additional efforts to execute regression tests in every sprint.

This challenge can be addressed by setting up a right test strategy supporting:

  • Dedicate 2-4 weeks of time before development for setting up automation framework, integration with CI and validate one end to end automation scenario as a proof.
  • Testing team need to work closely with Release Manager for any CI issues appearing in build pipeline and also configure the best tools for test reporting. (Ex: Sonar)
  • Also plan for sample/dry run of performance tests and configuration of security scanners in the dedicated test environment. (Mostly on cloud)
  • Also have the dashboards setup by release engineer to monitor all testing results. (Automation results, security scanning failure report, performance report)

Code Quality:

While that is clearly true for the functional testing and validation of requirements in code, code quality tends to degrade. As the speed increases and the focus on user acceptance and functional validation has increased, the number of defects moving into production is also on the rise. The nature of these defects is significant, including security violations, missing or incomplete code blocks, and unmaintainable code. It almost appears as if the tradeoff in functional testing is with code quality.

DevOps requires that many of the supporting processes change to support the notion of continuous delivery of service. Unlike the traditional processes with “end” states, DevOps features a continuous process. Some of the ways of addressing this challenge are:

  • Specification –With releases being more frequent, design documents should be produced more frequently and with emphasis on business value capture, rather than code. Typically, design documents should be aimed at a bi-weekly/month cycle. Within one month, the tasks can be accomplished predictably. Outside of one month, the accuracy of the estimates begins to degrade and more importantly, the ability to deliver in continuous releases.
  • Coding: In a continuous cycle, there is no way to batch up requests. Instead, automation of processes becomes key. Software quality becomes essential as speed allows code to migrate into production without inspection. Random selection of tools will not achieve the goal. It is important to have a few key tools and standards. Some of the process to achieve the coding standard are:

Function Point counts based on each week’s development. This change is more important than it initial appears. Tracking function points from week 1 through week 4 will provide data for estimation and when coupled with code quality, will provide a key metric around function point by software quality which will become the basis for estimating new projects.

Code Quality: CISQ/OMG have established standards for code quality that have a direct relationship to cost of maintenance. While business functionality is key, cost of maintenance can erode business value.

Transferability/Changeability: In a DevOps environment, code is released more frequently and the emphasis is on multiple streams of development. The ability to integrate quickly becomes essential.

Critical Violations/Security: Without question, Security and “open” code has become a critical topic. Hence having security scanners to identify all the static security testing issues at every sprint could avoid these issues at a later point of time.

Automation Testing challenges:

Test Automation is one of the key factor for DevOps success. There are numerous test strategies available supporting automation. However based on the business case and looking at future needs of client, one need to choose the tool and setup automation framework. In general when we go with monolithic architecture approach, test automation is maintained outside the actual code repository. However, in DevOps and Microservice based architecture it’s very much lined within Microservice. This means to say, the re-usable code exists across different micro services and reusability can be achieved only by manual process. The maintenance of automation code base becomes very challenging as it gets spread across different micro services.

So the solution need to be arrived based on what kinds of tests are being automated.

  • If the automated tests are endpoint/url driven, then the automation code base can be maintained independently and out of micro service code base.
  • Need to configure the automation framework in such a way that, any Microservice can pick up respective service’s test suite to run through its build pipeline.
  • The test suites should be created based on each Microservice
  • Smoke tests should be created separately considering end to end scenarios.
  • As the automation scripts grows higher, it’s better to keep nightly script runs to avoid test environment dependency.
  • The framework should be created in such a way the automation test suite should be able to run in any environment (test/prod) with minimal changes in configuration.
  • If the automated tests have Mockito/junit tests which are at unit level, then one has to maintain this code within Microservice as it cannot be executed outside the codebase.
  • Have the base/master project maintained with basic re-usable code which can be re-used through manual process by any of the tester.
  • Need to plan for end to end automation scripts only in master code base as it will have dependency on other services to execute unless tester has written stubs for dependent services
  • If one plans for end to end test or integrated test automation, this script code base can be maintained separately and run on need basis.

Performance Testing Challenges:

While performing performance testing in DevOps has many challenges like setup, maintenance of performance test environment, time consumption in running the performance test, debugging CI Jobs and result analysis. The challenges along with solution has been explained as mentioned below.

Performance Test Environment: Performance testing environments are complex. Layered applications, multiple dependencies and third-party APIs require different storages and CDNs for testing. This takes up time and resources, which complicates the CI process instead of simplifying it.

Simplify the test environment. We need to trade in being realistic for practicality and the price of build time. This is in line with performance testing in CI goals: using KPIs to examine trends and comparisons over time that let you see if you are improving or not. So stub all 3rd party APIs, run it in the smallest configuration possible, to fit into a single machine and simplify deployment. In order to run the stress or endurance tests, maintain a separate environment which is out of CI.

Time Consumption: Ideally, the complete CI cycle should take no longer than 10 minutes, and we know that up to 30 minutes is good to have. But if the process takes more than an hour, we are losing the “continuous” part, which we all can agree is pretty important. But having a large number of functional tests, or adding performance tests, can turn the CI cycle into a long one.

The best solution is to:

  • Prepare upfront and overnight. Don’t use your day for generating test data sets.
  • Reuse what you can, like generated data sets, deployed environments, etc. Instead if deploying everything at the same time, divide the job and take parts out of the CI cycle.
  • Make short tests - 1-5 minute load tests can reveal many insights and show trends and changes.
  • Don’t put spike and endurance tests in CI. You can use Jenkins as an automation platform, but don’t put them in the CI cycle for every build.
  • Parallelize tests by using Jenkins 2.0 and Taurus, an open-source automation tool. Taurus can run divided tests in parallel. CA Blaze Meter or any other supporting tool can provide resources for parallelizing in the cloud.

Debugging CI Jobs: As can be expected, automating test will result in problems and failing jobs due to issues in the job itself that needs maintaining. This effect should be minimized because it affects and spoils your build history and your ability to gather system health information and analyse trends.

The best solution is to: Take the debugging out of the CI process. Taurus enables you to debug locally and then commit back into the repository. This doesn’t affect the history of the performance testing results because the failures from the debugging trials appear on the local machine.

Results Analysis: Analysing performance testing results is difficult in CI machines, because these machines need to support multiple technologies. Even Jenkins, which is very flexible, has its limits. In addition, the results analysis needs to be available for non-techies and it can’t let decisions be made by a machine.

The best solution is to:

  • Integrate services that can analyse the data, like static analysis, artefact storage, bug tracking integration and cloud services integration.
  • Keep the overview in Jenkins but use the CA BlazeMeter or any other reporting system for rich reporting that can be configured, like comparisons and easier trend lines.

  • Use pass/fail criteria. This is complex because you are examining graph curves and multiple factors rather than yes/no questions. Taurus has a mechanism for displaying and analysing complex expressions of pass-fail.

Security Testing Challenges:In DevOps as like other types of testing we also integrate security programs and perspectives throughout the full software development lifecycle. At the earliest stages, security perspectives can evaluate defined users, software designs and code analysis. The tests can target specific stages of a program’s lifespan, and testing can occur pre-execution (SAST), post execution (DAST). In DevOps concept, SAST is been tried to achieve by configuring SAST tools. This helps to identify and fix the security issues at a code/unit level much more early than later in the software development life cycle.The challenge is to identify, configure the tool, and analyse the report generated. The other challenge is also to plan/fit DAST part of DevOps.

The best solution is to:

  • Identify the security scanner based on technology of the application being used and integrate with CI through build pipeline.
  • Plan for DAST early in the development cycle, so that design level security issues can be identified at the earlier stage.
  • Plan for actual DAST when a considerable end to end scenarios are ready for test.
  • Plan for automation of DAST test as well if possible.

Conclusion