The rise of the AI automated testing era

AI automated testing revolution? Participating at the SEETEST 2019 Testing Conference was enlightening on the topic.

The conference was held at the JW Marriot Hotel in Bucharest, between the 25 – 27^th of September and brought up subjects like big data, AI, metrics and more. There were over 400 participants, 30 speakers and an exhibition hall with stands from top companies in Romania. The conference atmosphere was positive, gathering open-minded people from various IT sectors exchanging ideas. From testing to how to make their processes and tools more efficient in today’s dynamic environment.

Data is the new oil!

The first tutorial I attended during day 1 of the SEETEST conference was about the Basics of AI and AI in Testing, by Vipul Kocher. While the explanations about how AI evolved and where it is headed in the test area were quite comprehensive and exciting, the main idea that stuck in everyone’s mind was that Data has become the new oil in the 21^st century.

Quality data for training AIs is so hard to find and even more difficult to produce, that those who invest and focus on creating data will win big in the very near future. It’s becoming difficult to train an effective AI due to a lack of comprehensive inputs that need to be fed to it. Therefore, AIs still make mistakes – i.e. in image recognition, once the background changes to a different color than the one with which most previous data inputs were given, the AI was no longer able to distinguish properly between the image of a cat and a dog. And this problem is just a simple, harmless example – just imagine what can happen in different lighting conditions for self-driving cars!

Additionally, feeding complete, quality data to an AI is just half the problem. Another challenge is choosing the right technique through which it should learn and distinguish the inputs to that it can offer an appropriate output. So, it seems that all progress made until now has merely been scratching the surface of the whole AI paradigm. Many challenges await and we must remember that data will become the new oil! Just like the industrial revolution and the cars we drive today, without oil, this wouldn’t have been possible.

Metrics and happiness

Another presentation I attended at SEETEST 2019 was called Metrics: Reach your goals faster with a Metrics Dashboard by Pablo Garcia Munos. He stressed the importance of not only adding metrics to your projects in order to track bugs, number of tests or test cases automated, but also digging a little bit deeper and finding out how these metrics are linked with each other in order to find the root cause of problems on projects.

An important idea that we were left with, was to try and add happiness metrics on projects. This means that the happiness of the team should be monitored daily so that any frustrations or demoralizing behaviors can be identified and addressed. The success and performance of a team are highly dependent on the mental wellbeing of its members. So, it comes as no surprise that the happiness index can directly influence all other metrics of a project such a number of bugs found, a number of bugs escaped to production, or the number of bugs fixed. As such, managers should really think about introducing this kind of metric in their companies or teams.

The monkey in the machine

You may have heard the expression “Deus ex Machina – the God in the Machine”, well Peter Varhol and Geri Owen talked about the notion of a monkey in the machine that’s bent on destruction. One that is weirding a figurative wrench and can crash random systems at random times. This is the concept behind Chaos Engineering, a concept that has been gaining more and more ground in recent years. The idea started with Netflix’s Chaos Monkey (now called Simian Army) and focuses on failure and disaster recovery.

Peter and Geri argued that teams should prepare for failure, plan for it, and exercise it as much as an airline pilot would for failing. For testers and developer’s system failure should just be another day at the office. This means that system failure and disaster scenarios should be introduced as regular exercises along with plans on how to mitigate and recover from them.

Open-source is the new sexy

I was surprised to see that many speakers at SEETEST touched upon the subject of licensed software versus opensource software. It seems that the general trend is for companies to adopt open-source software and that licensed tools are coming to an end. Even in new areas such as automated testing using AIs, there are several companies developing free, open-source solutions available to the wider public. It is just a matter of time until we will start to see the emergence of free software tools using AI for testing, along with the already existing and continuously extending array of free, open-source testing tools.

Times are changing, and companies are looking towards AI more and more, as well as those tools that incorporate as many functions as possible. For example, several speakers and participants from the conference have said that they’ve started using JIRA as their go-to tool for both bug reporting and test case management. Still, the process of finding a comprehensive tool able to cover all is often difficult.

Final thoughts

SEETEST 2019 was definitely food for thought. About AIs, about current trends, about the future. I have seen several presentations where the speakers would talk about their custom-built solutions over existing test frameworks such as Selenium Webdriver in order to meet the new challenges imposed by JavaScript on UI testing. Finding the right automation tool for the job is becoming more and more difficult, so many speakers have said that they resorted to the customization of their Webdrive framework. Could this too be a sign that we are starting to see the sun setting of Selenium Webdriver and Java?

Time will soon tell as new automated testing tools, especially in the area of AI testing, are starting to emerge.

About the author: Răzvan Vușcan is a Quality Assurance Automation Tester at AROBS on the Travel & Hospitality Division.

If you want to read more about automated testing, find more here.