This is my favorite url if you need to use SerenityBDD and Sprint Boot starter project : https://github.com/lpicquet/serenity-cucumber-spring
Recently have some trouble automating an Application using Angular using Selenium and SerenityBDD.
I found a component that helps a lot. NGWebDriver
How to search an element and interact with it?
Also it could be handy to sometime wait that Angular load everything by using:
Also I had to revise a bit my Xpath way of working. Here is a good tutorial : https://www.guru99.com/xpath-selenium.html
I had some trouble lately with some Angular interface. So here interesting work around I found.
Scenario 1 : You have a list of checkbox to click, they are not hidden but hidden under an other element. Like hidden behind a pop-up window or a menu button, but they should partially visible for a human user.
The work around : move the checkbox to a visible position in the page. How?
js.executeScript("arguments.setAttribute('style', 'top: 0px');", windowElement);
When a customer website is critical for his business and a lot of users are interacting with the website. Even with a content review process some non-compliant contents or issues may be missed. And above that, sometime the pictures have less attention then the text.
But how could we test picture content?
Here are some non-exhaustive objectives. What are the quality objectives on the pictures displayed on the website?
- Picture object are relevant with the description.
- Avoid certain type of object content in pictures.
- The picture content doesn’t contain unauthorised logo.
- People on the picture are not happy enough.
- The picture colors are not following the graphical chart.
- Misspell on title in the picture.
What are the technology solutions to achieve this goal?
There could be several solutions, but let’s narrow a bit to one particular solution that we are working on it.
We have started from a test factory based on Serenity, Cucumber and Selenium.
Depending on the test scenario, when it is required to validate a picture, we send the picture to an service that allows to validate a picture content based of acceptance criteria like defined in the quality objectives above.
How this picture validation works?
The service is based on Google Vision Api. This Api is able to identify objects on picture. But this Api is not perfect and only provide relevancy percentage on different options. In order or be accurate, the AI need some context and that’s what we provide him to get a better relevancy and have a final answer.
This service solution is not free. Google is charging any transaction to the Api. So some caching process need also be taken in account when the same picture with identical content are analysed several time.
This Google API is using the machine learning and and his big data. This Api is the most popular service even used by the US Air Force to identify objects on picture taken by drones or satellite. Each day it becomes more and more “clever”. That’s why we have chosen this Api to build our service for picture object identification.
The object identification could be items like human constructions, human faces behaviours, animal and vegetal identification and much more.
AI or not AI
Some AI “expert” could be not agree with this, but we want to keep this simple. For us AI means “a branch of computer science dealing with the simulation of intelligent behavior in computers” (merriam-webster). Google Vision API is based on Google Big Data and Machine Learning to be able to simulate this human behaviour. So for that reason we consider as AI.
AI service release
We are still working on this service. This service using AI is in alpha stage and we are preparing for Q1 2019 an initial release. We are in an optimisation and testing phase. Stay tuned for more info on twitter @fanaticaltest.
We are super exited to release today a new web test factory that allows you to create functional automated test of any web application or website. This factory is open to anyone and anyone could contribute.
What’s new in 2.0?
We have remove our own framework and we decide to adopt Serenity BDD framework. We have decide to go with Serenity BDD in order to focus on test delivery and not reinvent the wheel. Also this framework has so many contributors that allows to keep it-self up-to-date.
Also the factory is delivered with a Gradle including some build tasks examples to manage several level of tests.
Last but not least, we provide a docker container for selenium agent in order to have a full selenium grid.
Getting started quickly
- Clone or fork the test factory.
- Open the project in your favorite Java IDE.
- In the terminal window, go to the projects root and run the selenium agent :
docker-compose up -d
- Run the vnc to open the agent in order to see live the test:
- Run the smoke test:
gradle clean smokeTest
- When the test is completed, run in the browser [project-root]/target/site/serenity/index.html and you should see a report similar to this example.
In term of architecture there plenty ways to connect to an IoT devices, at least there are:
- Devices connected to a mobile
- Devices connected to a desktop or a laptop
- Devices connected to a cloud solution
Today we will talk about IoT devices connected to a mobile. How these devices are connected?
Mainly they use Bluetooth. If they use Wifi, then they are using a cloud solution as proxy. That means we could categorize them as devices connected to a cloud solution. We will discuss how we could test them in a future article.
Before exploring how we could test them automatically, let gives a few solution examples of IoT devices connected to a mobile:
- Watch connected to a mobile (Apple Watch, FitBit, etc.)
- Beverage cooler and Freezer
- Coffee machine
- Construction tools
- Healthcare equipment
Last point before starting, here will cover the software part of the test. The hardware will be not covered in this article.
- The first challenge is how we get feedbacks from the IoT devices? Mainly the IoT devices have a very simplified interface with a very limited scope of functionality. But when we test, we need to access to a bit more of functionality, like to have access to details logs, have feedback on functional that return nothing, have a feedback when it changes state. Usually the hardware and firmware vendor of the device need to provide board that simulate the hardware but it uses the real firmware as it is a real device. In fact, what we test here is if the firmware is reacting properly as per the specification. And the firmware will interact with the board as it is a real hardware.
- When we talk about mobile phone device, we need to cover a huge number of devices version (hardware and OS). And sometime, the way the Bluetooth works, it could vary between different mobile phone type.
- There are no End to End solution for this type of architecture. You will need to take what already exists and you have developed, then put them together.
Architecture and test
To test this architecture, we could use a Java Test factory based on Appium, Selenium and Rest-Assured. This factory could be driven by a BDD (behavior-driven development) like Cucumber to define the test cases. The only missing part is the machine controller.
As mentioned before the main challenge is to set a Hardware emulator to test the real firmware and have a full access to all input and output logs. This emulator will be handle by a custom developed machine controller. This controller will interface with the device through a serial port and the machine is expected to implement parsing the MCP messages.
Regarding Appium, the setup is required to test a Native or Web application running in a Mobile device.
To test the backend, let assume they are in majority accessible by Web interface or Rest Api. So, the best tool for that are Selenium for the web and Rest-Assured for the API.
The major effort to set this kind of End 2 End test is to set the machine controller, and that could take more than 50% of the time required to setup this environment. What is missing is a standard protocol to manage device through serial. And also, I’m quite sure any time soon this pain will be solved by any major Automated Test Framework leader (HP, Tosca) or by an open-source community. But in the meantime…good luck!
Before starting to automatize tests, you need to take a step back and define few aspects.
Consider the test team members
Depending on the test team members you need to think what kind of test scripts you are going to adopt. Depending on the tools like Tosca, UFT, Ranorex or Selenium, there are 3 ways of scripting:
- Record : This is a an easy way, but could be expensive to maintain. The profiles adopting this way of working are the Business Users.
- Script : This a fair quick way to script, and the maintenance is less expensive then using Record. It requires a Test Automation Expert to develop and to maintain them. Any tool is able to achieve the automation.
- Data Driven and Workflow : It requires both profile : business user and a test automation expert. This could take more time to implement but the maintenance is way easier and also scripts could be easily reused by the business user without requiring an expert.
- With HP, it will require BPT.
- Tosca is able to handle it if you use “modules”.
- Selenium and Cucumber or any other BDD are able to handle it. See a test factory as exemple.
- Ranorex is supporting the data driven part. But no workflow management.
Consider Manual vs Automated testing
You should not automate everything and keep some test cases for manual testing. To define which one should be automated, keep in mind the following:
- How much time the same script need to be repeated? If you are doing more than 4 times, you should automate.
- If your test are UAT (User acceptance test), exploratory test or complex with many asynchronous processes, you should consider manual test.
- Also when you prioritize the test cases based on their business criticality. More a test case is critical, more it should be automatized.
Consider Test Data Strategy
Ask yourself these following questions:
- Do you need real data with a state data based on a process? That means your data will change from the beginning until the end of the test. You may need a copy from a production database with data anonymization and a way to reset it. This could be a complicated task. You could also use advanced tool for Service Virtualization to avoid to copy database and have a instant “refresh”. But this path is also quiet expensive in license or in man days.
- Does your data drive the test? In other words, do you need more a copy of the database, but also “properties” data to drive all the test scenarios?
- Define the right set of data that covers all the business risks. When you generate all the test scenarios for the same workflow, you should ensure to optimize the number of scenarios without being systematic, otherwise it could fall in an exponential number of scenarios. See an example below:
Consider Test Script Maintenance
When you create automated test you will require later to maintain them for 3 main reasons :
- “Repair” test cases based on business changes.
- Enhance test cases based on issue found in UAT (User acceptance test) or later in Production.
- Optimize or correct test based on false positive errors.
Consider the right scripting tool
Based on all your SUT (System under test) technology. You need to find the right scripting tool depending on the features and your budget.
In 2017 we could say there are 3 ways of doing Automated test. The typology of the these 3 ways are based on their cost maintainability.
- The most easy one but the most complex to maintain is the Record type. All the commercial tools and even Selenium with Selenium IDE, propose this way of doing. But when the SUT (system under test) is changing the automated test could be drastically compromised and need a full rework of the test script.
- The second way and more developer oriented is the Scripted type. The maintainability is less complex and could be handle some design change if the script and the UI is properly identified.
- The third one is based on Workflow and Data Driven type, there are still scripts but there are more easy to maintain. If a step in the workflow changes, you just need to change those steps. Almost all the commercial tools are proposing these kind of way of doing. With Selenium if you use the BDD with Cucumber will work in that way.
Now in term of implementation the most expensive in maintenance is the most easy to implement.
So when it the time to choose which way you want to implement tests you need forecast how complex, how many tests you want and how many time they are run.