Workload Automation Quality Assurance Strategy Revealed

Digital Transformation is disrupting the way software applications are being developed, amplifying the need for building better software testing capabilities. In fact, the complexities with applications and software are increasing and there is a growing need for third-party integration. The main effect of this transformation is that the patterns, methodologies, and tools for testing are being evolving with the changing software testing needs. For instance, Agile testing and DevOps can be identified as the key drivers for growth in the software testing market, considering the pressure due to the release new software version at speed and with sustain quality at the same time.
​

In this new challenging context, Workload Automation Quality Assurance Team is evolving itself, enabling new processes for which software testing starts way ahead in the development lifecycle instead of being introduced at the end of the development process, as shown by the following picture:
​

Our current quality assurance testing strategy 
Our quality assurance strategy is centered on continuously evolving our testing process to prevent bugs from being created as the software is developed or, worse yet, from being discovered by our customers.  
 
The adoption of the Agile methodology rapidly provides customers what they need to compete while maintaining high-quality deliverables. Using the Agile methodology, we can reach the following goals: 

• Deliver business value early and often in the development cycle
• Validate deliverables with customers and integrate requirements as they emerge 
• Leverage tight, efficient, self-organizing teams
  
  

Functional testing, one of the main quality assurance process that prevents mistakes and defects in manufactured product, is performed during all the sprints applying the following approaches:

• Black box testing that examines the functionality of an application without peering into its internal structures or workings
• Matrix approach where test matrix is created to define the test procedure to be manually and automatically run and the coverage on different devices and platforms
• Component Test in which the product components are isolated and tested as independent from the others using stubs and sniffers 

• Integration Test to verify the product capability to interact with the others components of the product
• Test Driven Development: the adoption of this methodology considers to write test scenarios before writing the code. Developers implement the code and perform unit test using the test scenarios provided by testers. The main benefits of this adoption are:      

                 - Clear design 
                 - Faster feedback 
                 - Radical simplification of the code: developers write code in response to
                    the requirements of the tests 
                 - Identification of the error/problem quickly 
                 - Bugs reduction 
                 - Quality improvement 
                 - Reduction of the development Time-to-Market 
 
We do extensive use of automated test suites to validate that our products function properly allowing the products stabilization of the new implemented features and a continuous regression during all the development phase. Many thousands of automated test cases are daily run to test all the product interfaces and the main product functionalities. Jenkins is used as orchestrator to achieve a continuous testing, according to the following picture: ​

Moreover, we are used to exercise our products in real-world scenarios as they're being actively developed, which helps us to identify problems before our customers do. To enable this approach, we aggressively use customer feedback to evolve our testing plans as we resolve issues identified in our support forums, custom services, and formal support chains.
 
More specifically, the key differentiators of our Testing strategy are: 

Transplant Testing paradigm for System Verification Test (a.k.a. SVT), a methodology to bring customer applications, data, scenarios and procedures into our test labs. ​This paradigm sees its finalization in the implementation of the Corporation methodology to emulate business-driven scenarios in production-like environments.   
     ​
This methodology is built on the following processes/best practices:

• Customer advocacy:  

                 - Observing how customer works 
                 - Learning from customers how to change our test approach  
                 - Providing our consulting for the deployment of our solutions 

• Design thinking and Transparent Development Process: 

                 - Involving customers in our development process    

• Collaborating with L3 product support team:  

                 - Bringing customer scenarios, data and configuration into SVT scenarios 

Continuous Performance Testing based on the following elements:      

• Continuous testing running plan every day on production-like environments with thousands of agents, more than 40k job streams and more than 500k jobs in the plan 

       

• Continuous monitoring with special focus on typical product KPIs (scheduling and mirroring throughput, average delays, internal queues sizes) and on HW main resources (CPU, memory, disk busy) to prevent memory leaks and unexpected HW consumptions during long run workload scenarios

         

• DWC performance@scale running composite user interface scenarios with

                concurrent users to demonstrate how:  
                  -  A single node is able to scale up to 175 concurrent users 
                  -  Clustered solution with 4 nodes is able to scale up to 700 concurrent 
                      users preserving acceptable average page response time for all the transactions

• Continuous environment update/upgrade from previous fixpacks/releases

                to preserve database data consistency over time to emulate better real customer
                environment (for instance, more than 60 * 10^6 rows in the product historical
                (jobs table)

• Continuous focus on customers Not Functional Requirements needs  

                - Translating Performance/Scalability/Reliability pains/needs coming from the
                           field into quality goals to be assessed during each new functional fix 
                           pack and/or main release development cycle 
                -  Publishing periodically Performance Report to provide guidelines on how
                           to configure the product to optimize the performance and the scalability

Security Testing based on the following implementations 

• Vulnerability scan driven by Rational AppScan tool 
• Static code analysis driven by WhiteSource tool 
• Penetration test for Cloud solutions  

 
New challenges for the immediate future 
The main mid-long term goal is to adopt a continuous testing approach to enable a full DevOps development process.  
 
The main benefits of this new shift-left paradigm will be: 

• immediate bug fixing enablement 
• instant test environments provisioning 
• unit, functional, and non-functional tests are run in an automated way, orchestrated by continuous integration and continuous delivery tools. 

 
In this way, the team will improve software quality, gather feedback faster than ever, and manage properly the Workload Automation solutions proliferation due to Hybrid Cloud puzzle.
​  
From a testing perspective, as long as testers have access to those systems, the automation tools needed for continuous testing (test automation, service virtualization, deployment automation, and so on) need to do their robotic magic as well. Understanding this requirement at the outset of a project will help prevent barriers to achieving continuous testing (QA+). 

• Automation+ 
  Based on a shared vision, requirements, architecture, design, coding in some instances, and validation that the automation does what was intended. More focus on unit test, API testing and low-intensity performance testing as soon as new code is available. 
• Environment+ 
  Automated creation and configuration of test environments decreases the time it takes to start testing a new build from hours to minutes, especially if Docker is used, when applicable, to provision test environments. This automation also reduces the number of false errors due to test environment issues, incorrectly installed dependent software, and other manual processes that introduce problems.  
• Data+ 
  Increase test effectiveness having the right sets of test data. A good set of test data also helps the team create exceptions or error situations that might be difficult to test otherwise. ​

If you want to talk more about Quality Assurance strategy and Workload Scheduler, leave us a comment below. We would be happy to discuss. 
 

Elvira Zanin
Software Engineer

Elvira is a Software Engineer in the Workload Automation development team located in HCL Rome Hub. She is currently based in the Web UI development team but she was involved in Test automation team and Add-ons development team also. Elvira has experience with Dynamic Workload Console. She completed her degree in Computer Science at University of Salerno and currently lives in Rome, Italy.

Giorgio Corsetti
​Technical Architect 

Giorgio works as Performance Test Engineer in the Workload Automation team. In his role, he works to identify bottlenecks in the architecture under analysis assessing how the overall system works managing specific loads and to increase customer satisfaction providing feedbacks about performance improvements when new product releases become available through technical documents publications. Starting from April 2018 he added a new responsibility covering the role of Test Architect for Workload Automation Products family. Giorgio has a degree in Physics and is currently based in the HCL Products and Platforms Rome software development laboratory.

​Gianluca Seghetti
Advisory Software Engineer

Gianluca is a Software Engineer with long standing experience in software testing and test automation disciplines. He is currently covering the role of SVT leader for Workload Automation products family. Gianluca has a degree in Chemistry and in Industrial Chemistry and is currently based in the HCL Products and Platforms Rome software development laboratory.