Test of insanity

  • Dino Vinkovic

If I were to put a basic definition to a test’s job it will be something like “doing the same thing over and over again and expecting different results”. Each tester goes through several phases on a regular basis, such as “enthusiastic phase”, a creative phase where everything is new and waits to be explored, or “bored-out-of-your-mind” phase, a non-creative phase where nothing is new and everything has already been explored, yet something is still missing for testing to be completed. However, all roads lead, eventually, to the “frustrated phase”, a phase where things do not work, but no one seems to know why or even when. And this is the phase that actually tests the tester himself.

For all the testers out there, I have one magic word: Selenium. Whoever has worked with the Selenium tool knows that there is no conventional magic word that evokes such strong emotions that Selenium can. Don’t get me wrong, Selenium is a fine and very useful tool, but it is also a tool that will test your patience, code skills and, finally, your sanity.

Where do I begin? Let’s begin with the famous Selenium flakiness, that is, with the fact that there may be (and the probability is high) a test which will pass and fail at random, without any evident reasons. At the beginning, this will probably happen a lot to you, but no need to worry, a lot of problems can be solved with explicit waits or, perhaps, numerous try-catch blocks which may slow down the testing process but will make test much more stable. Of course, reusing code should also make things much simpler and more trustworthy when executing tests.

However, what does a person do when bizarre things, such as sendKeys method not sending all the keys without any seemingly logical explanation, start occurring? I had to face that very problem myself – Selenium didn’t seem to behave very well when testing Angular (especially when Material library is included), so, for some unknown reason, sendKeys method would sometimes type the whole text in the textbox and sometimes not. To fix the problem, I’ve done what any sane tester would do – I’ve tried to google explanations.

After finding out that it’s a pretty common occurrence with Selenium, I’ve tried to find the simplest solution I could. Of course, it was a try-catch block. Basically, the idea was to try to type all the keys and then verify correct keys were entered via string comparison. All of this was enclosed in a while loop with the boundary set at around 2 or 3, meaning that sendKeys method would work at least once in, say, three attempts. This may not be the best solution, I’m sure some of you may have found a better one, but this was the solution that actually worked and we were pleased with it.

Unfortunately, not all stories end happily like this one. Despite all the love I have for Selenium and its numerous good sides, I had to turn my back on it when I finally entered the “twilight zone”. “Twilight zone” is the name of everything concerned with testing which I have never managed to explained to myself, at least not in the non-supernatural way.

There were several happy days when tests were executed just fine – code was doing what code does best, it was functional, pretty quick and was displaying no exceptions. But, at one point, they started failing again for no reason whatsoever, and not in one place, but randomly. For some reason, visible elements Selenium interacted with suddenly became non-visible, then visible again, same elements located in the same place were sometimes clicked successfully, sometimes not. I know, you’re probably thinking “oh, but that should be easy to handle!”, but, trust me, I’ve already tried what you’re thinking of.

Fact is, a lot of people told me they’re experiencing same issues with Selenium and they are always full of advices on how to create another workaround for this and that. But is it really a way to go? How many compromises does a person have to make before looking elsewhere? Just think, how many people would use Java of C# nowadays if they were a truly flaky programming languages, considering that there are other options just waiting out there? Could it be that Selenium is no longer a “good enough” tool for testing?

Personally, I believe Selenium needs some radical changes to be a 2018 (or 2019, 2020, etc.) tool instead of a 2008 tool. I’ve tried combining with Protractor (another Selenium-based testing tool) and I’ve got a bit better results, less flakiness and quicker tests. However, it is still far from perfect.

Another option, the one we decided to settle within our company, is Cypress, JavaScript-based tool, made for testing JavaScript-based sites to be tester-friendly as much as possible. Probably the best thing about Cypress is the fact that it is not a Selenium-based tool, but it completely independent. It also has a lot of things which make tester’s work much easier, such as explicit waits which are included in methods by default, and seems to be, as far as we know, stable 99% of the time. If it’s not, there is always a logical explanation for everything and most of the times the problem can be solved quickly.

Cypress obviously has its own set of issues. For starters, it’s still in its beta phase, which means that it’s, basically, an unfinished product, and its online community is not nearly as big as Selenium’s. This means that a lot of times you will have to find solutions or try some things by yourself instead of relying on Stack Overflow or something similar. It also means you’ll need more time to get things done. However, the final results may be much more stable and trustworthy..

Let’s go back to the comparison of testers and madmen. A tester’s job will always be finding different things by looking at the same places. A tester will always be driven to insanity. However, true challenge should be finding bugs in your application rather than on the tool that is supposed to help you do it. If the biggest challenge in a tester’s world nowadays is finding a reliable tool for finding bugs, then something has gone really wrong.

The irony is that there is not one stable and bug free testing tools available in market today. Selenium barely fits into that category, it has not followed the sophistication of evolving and future technologies, and perhaps its true place lies in the past, the history.

You may also be interested in these articles

  • Rajiv Anand

What really is IoT Middleware?

A term like “middleware” has been bastardized by software industry in so many ways that it is no surprise when people are rightly confused even by its IETF definition “those services found above the transport (i.e. over TCP/IP) layer set…

  • Rajiv Anand

RFID and Health

If you live in an urban area, chances are that you are continuously bathing in variety of WiFi, Bluetooth, Cellular wireless signals without which modern living in unimaginable. If we could see beyond frequencies of light, we will be blinded…

  • Rajiv Anand

Let’ get physical to Digital

As smarts in machine get closer to human intelligence, life of a worker in manufacturing, logistics and at point of use of physical assets is drastically changing. Now and in foreseeable future, machine learning, artificial intelligence, industrial automation, and ubiquitous connectivity is transforming physical world into digital and in center of this revolution is Internet of Things and location intelligence with sensors such as NFC, UHF RFID, Bluetooth LE, WiFi, UWB and other technologies waiting to break out of their labs.

Want to know more?

Get in Touch