Stress Test vs Load Test

People are often mixing Stress Test and Load Test in software development. So, let’s demystify a bit.

Stress test tries to break the application until its limits and have a clear KPI regarding the SLA. In this stress test, you have 2 aspects: Positive test using all the system working optimally and negative test having some system already broken and what is the limit of the system in degraded mode.

Load test is a scale up and a controlled volume loading test. The main aspect here is to define the bottle necks of the system at different level of volume.

Here a few tools that could do both type of tests:

  • Locust (open source system and Python)
  • JMeter (open source system and Java)
  • LoadRunner (commercial tool)
  • NeoLoad (commercial tool)

Samy Kacem

Test Automation Strategy

Before starting to automatize tests, you need to take a step back and define few aspects.

Consider the test team members

Depending on the test team members you need to think what kind of test scripts you are going to adopt. Depending on the tools like Tosca, UFT, Ranorex or Selenium, there are 3 ways of scripting:

  1. Record : This is a an easy way, but could be expensive to maintain. The profiles adopting this way of working are the Business Users.
  2. Script : This a fair quick way to script, and the maintenance is less expensive then using Record. It requires a Test Automation Expert to develop and to maintain them. Any tool is able to achieve the automation.
  3. Data Driven and Workflow : It requires both profile : business user and a test automation expert. This could take more time to implement but the maintenance is way easier and also scripts could be easily reused by the business user without requiring an expert.
    1. With HP, it will require BPT.
    2. Tosca is able to handle it if you use “modules”.
    3. Selenium and Cucumber or any other BDD are able to handle it. See a test factory as exemple.
    4. Ranorex is supporting the data driven part. But no workflow management.

Consider Manual vs Automated testing

You should not automate everything and keep some test cases for manual testing. To define which one should be automated, keep in mind the following:

  1. How much time the same script need to be repeated? If you are doing more than 4 times, you should automate.

    Y = minutes spent and X= time test execution. We base the figure on a small test scenario.
    Y = minutes spent and X= time test execution. We base the figure on a small test scenario.
  2. If your test are UAT (User acceptance test), exploratory test or complex with many asynchronous processes, you should consider manual test.
  3. Also when you prioritize the test cases based on their business criticality. More a test case is critical, more it should be automatized.

Consider Test Data Strategy

Ask yourself these following questions:

  1. Do you need real data with a state data based on a process? That means your data will change from the beginning until the end of the test. You may need a copy from a production database with data anonymization and a way to reset it. This could be a complicated task. You could also use advanced tool for Service Virtualization to avoid to copy database and have a instant “refresh”. But this path is also quiet expensive in license or in man days.
  2. Does your data drive the test? In other words, do you need more a copy of the database, but also “properties” data to drive all the test scenarios?
  3. Define the right set of data that covers all the business risks. When you generate all the test scenarios for the same workflow, you should ensure to optimize the number of scenarios without being systematic, otherwise it could fall in an exponential number of scenarios. See an example below:

    Define data set scenarios - Gender and t-shirt size are the variable to generate the data set. (*) mean critical data from business point of view.
    Define data set scenarios – Gender and t-shirt size are the variable to generate the data set. (*) means critical data from business point of view.

Consider Test Script Maintenance

When you create automated test you will require later to maintain them for 3 main reasons :

  1. “Repair” test cases based on business changes.
  2. Enhance test cases based on issue found in UAT (User acceptance test) or later in Production.
  3. Optimize or correct test based on false positive errors.

Consider the right scripting tool

Based on all your SUT (System under test) technology. You need to find the right scripting tool depending on the features and your budget.

Comparison of test automation tools between Selenium, Ranorex, UFT, Tosca
Comparison of test automation tools between Selenium, Ranorex, UFT, Tosca

Automated test : UI vs API

When you start to have too many UI tests and it takes hours to complete them, maybe something is wrong in term of automated test strategy.

The first reaction when it takes ages, you start to run nightly test, revise them one by one and find eventually duplicates and finally try parallelize them when it is possible. And if this is still not enough?

Maybe you should define the most critical (20-30%) and modify the other using API calls instead of using UI.

When you test modern application with multiple layer, the UI layer should have no logic and no data. That means if you execute on API or UI level you should have the same results. But on API level the test are more quicker. In fact what you should test on the UI side is if the UI is behaving properly using the API and the security is aligned with the API, then the rest all should be done on API side.

Moving test from UI to API is not obvious because you may also modify all the UI test to cover the alignment between UI and API.

This is why you should think from the beginning to adopt the right test automation strategy with a good balance between API and UI!

For project using open source tool, RestAssured is a good choice to automatized API test. In this Github repository, you could find an example on how to use API test.

Automated Test 3 ways of doing.

In 2017 we could say there are 3 ways of doing Automated test. The typology of the these 3 ways are based on their cost maintainability.

  1. The most easy one but the most complex to maintain is the Record type. All the commercial tools and even Selenium with Selenium IDE, propose this way of doing. But when the SUT (system under test) is changing the automated test could be drastically compromised and need a full rework of the test script.
  2. The second way and more developer oriented is the Scripted type. The maintainability is less complex and could be handle some design change if the script and the UI is properly identified.
  3. The third one is based on Workflow and Data Driven typethere are still scripts but there are more easy to maintain. If a step in the workflow changes, you just need to change those steps. Almost all the commercial tools are proposing these kind of way of doing. With Selenium if you use the BDD with Cucumber will work in that way.

Now in term of implementation the most expensive in maintenance is the most easy to implement.

So when it the time to choose which way you want to implement tests you need forecast how complex,  how many tests you want and how many time they are run.

 

CBTA – Prerequisites

  • An Add-On to be installed on SAP Solution Manager 7.1 SP7 (onwards)
  • An agent which is part of ST-PI
  • A  frontend component to be installed on the user’s desktop.

Useful documentation on Solman 7.2 and 7.1 and Test Automation

Solman 7.2 and Test Suite

CBTA

  • Compatibility Matrix (Mainly for SolMan 7.1 but as long you use the CBTA version for latest version of SolMan 7.1 Sp14, it will work with SolMan 7.2)
  • Configuration (Also available in the How-to Guide below)
  • Prerequisites (Also available in the How-to Guide below)
  • Solman 7.2 Test automation overview on CBTA
  • Component Based Test Automation (CBTA) How-to Guide : This how to is a bit outdated and is targeting SolMan 7.1 SP12 and CBTA 3.0 SP4.
    • Create automated test from Business Blueprint (Transaction : SOLAR02) : from page 35
    • Create automated test from Test Repository (Transaction : SOLMAN_WORKCENTER) : from page 82
    • Create automated test in debug mode : from page 107 (it will execute the test step by step with outputs of the object status)
    • Creating custom library/function : from page 153

OpenStack on a single machine

If you want to go more further in Containers, and you want to have a dedicated machine for it. You will need a few more “thing” above Docker. You will need an Hypervisor and a container orchestrator.
For an Hypervisor you could take LXD and an OpenStack as orchestrator.

At the first glance this could take a lot of configurations steps. But guess what, there is a “Next, Next, Next” installation!

Here is the link and it is super easy and check the tutorial video: https://www.ubuntu.com/download/cloud/conjure-up

 

Service Virtualization – Open source and commercial tools

Introduction

This POC (Proof of concept) tend to analyse different service virtualization available on the market.

Glossary

  • POC = Proof of concept
  • Service Virtualization = “A test double often provided as a Software-as-a-Service (SaaS), is always called remotely, and is never working in-process directly with methods or functions. A virtual service is often created by recording traffic using one of the service virtualization platforms instead of building the interaction pattern from scratch based on interface or API documentation.” (ref1)
  • Stub = “A minimal implementation of an interface that normally returns hardcoded data that is tightly coupled to the test suite.” (ref1)
  • Mock = “A programmable interface observer, that verifies outputs against expectations defined by the test.” (ref1)

Main feature to consider in Service Virtualization

  • Multi protocol (not only HTTP).
  • Record record real transaction to re-play them for testing purpose.
  • Learn and cache responses.

List of Service Virtualization (Open source)

None yet if we consider SV main features.

List of Mocks (Open source)

Name Download link Comments
Citrus Framework https://citrusframework.org Works with a large spectrum of protocols (http, jms, ftp, mail, etc…)
Hoverfly https://hoverfly.io/ Learn more
Wiremock http://wiremock.org/
Mockito http://site.mockito.org/  Java only
Mockey https://github.com/clafonta/Mockey
Mirage https://github.com/InternationalAirlinesGroup/mirage
Wilma https://github.com/epam/Wilma Dockerized solution
Nagual https://github.com/emanuil/nagual

List of Service Virtualization (Commercial)

Name Download link Comments
CA https://www.ca.com/us/products/ca-service-virtualization.html
HP SV http://www8.hp.com/us/en/software-solutions/service-virtualization/ Learn more
Tricentis https://www.ca.com/us/products/ca-service-virtualization.html
ParaSoft https://www.parasoft.com/solution/continuous-testing/
SmartBear https://smartbear.com/product/ready-api/servicev/overview/
Traffic Parrot http://trafficparrot.com/

List of Mocks (Commercial)

Name Download link Comments
Telerik http://www.telerik.com/products/mocking.aspx
Sandbox https://getsandbox.com

References

Samy Kacem

HP Service Virtualization

Install HPE _SV Designer 3.82 CE

  • Windows 2012R2
  • SQL 2012 Express SP1

RestApi

We will use the API from ft-demo-website project => API. We will explore HPE SV and We will virtualize 2 methods from the API:

  • Read list of books
  • Add a book

Read list of books

Original : http://../api/listBooks.php
This GET Rest Api provides a list of books

  • We need to configure HPE Sv to learn the service:
    • Go to Add a new virtual service…
    • Select I don’t have a service description & click Next
    • Select REST & click Next
    • Select agent HTTP Gateway because we want to simulate the endpoint.
    • Add in Real Service > Endpoints the original rest api as specified above & click Next
    • Start to learn by selecting Learn and create a call using the Virtual Service not the Real service
    • System is learning thanks to this unique call. You will see that you have been redirected to the real service. When it the system is learning it captures the call, redirects it, plays as proxy and records the answer.
    • Switch to Simulate now & do again the call using again Virtual Service url. As you can see, you are not redirected and it is providing the same content. Now the service is kind of cached. We can say this service is stubbed as per this definition.

Add books

Original : http//../api/insertBook.php?title=toto&author=tutu&edition=Sun%20Edition
This GET Rest Api adds a book with the content defined in the param. You create a new Virtual service as the step above. Then put the 2 services in Learning mode.
Before starting, reset the db using http//../api/createTable.php (this will not simulated), and start to alternate the call (list, add, list, add etc…) then switch to Simulate and redo the same pattern. You will see the system is redoing the same outputs.
Now lets trick the system. Call only listBooks.php, you will notice he will loop as per the pattern even we don’t use insertBook.php! We you could face some issue for complex Business Process. How you could solve it? By using script or data driven set.

Beyond a simple stub

After recording you can more data scenarios. You have 2 options that could be used even together:

Script (C# or Javascript) to add business logic.
Data driven set managed with an Excel file and you define all the possible request and response required for your test.
Data test strategy

If you are stuck with some poor data or it is hard to use test data. It could be a good tool to put in place and have a new data test set. Based on your test scenarios, you create all the set of data you want to have in requests and answers and you will have fairly quick your data test set. This should be consider when you are doing your test strategy.

Conclusion

It works perfect if you are doing automated test but if you are doing manual explanatory tests it will be wired.
If you need to put more business logic it is still possible. For each service virtualized, you can add scripts with business logic. But how far you want to replicate the business logic? This could be complicated if you need a perfect replication, that means you will have 2 technologies to maintain with the same business logic. You should avoid creating scripts and answer as much as possible with Data driven set. This will be more easy to maintain.

Samy Kacem