After Selenium Conf 2012

I’m one of the guys who went to the Selenium Conf 2012 in London and it was awesome!

Very good talks and very nice people, congratulations to the organizers and specially to Simon Stewart and Jason Huggings.

I could write lots of things of what i saw there, but i don’t have much time so i’m going to write some brief conclusions:

  • Most of the top tier companies test their software in a ( more or less ) similar way, which indicates that the testing world is going through a single line and this involves very good things such a future standardization and good cooperation between people within this testing world. Great news.
  • In my current job at Tuenti, we are also in the same page. We achieve, in some cases, better results and have a better testing framework/infrastructure than some companies that are in the same vein. That thing made me feel very proud and happy of what we are doing at Tuenti.
  • Everyone has problems with their tests:
    • slowness
    • flakiness
    • brittle tests
    • undeterminism produced by AJAX asynchronous requests
    • DOM changes made by the developers and its consequent test update
    • Third party elements in tests
  • Some of them could (kind of) fix their problems, but there isn’t a good nor standard solution for all these problems. Here, the testing world has a big room of improvement.  I have some ideas for some of them.
  • Surprisingly for me, Cucumber ( or at least BDD ) is being more and more popular.

When i was seeing some of the talks, i though that maybe in the future i can write an article or even do a talk of how the testing at Tuenti has evolved since my first days in October 2009, some points:

  • We ran 400 tests in 3 hours, now we run 11000 in 23 minutes
  • We used Cruisecontrol, now we use Jenkins
  • Every release we had about 40 tests out of 400 failing randomly due to an important flakiness, now, helped with some magic tricks, i’m proud to says that zero tests fail out of 11000
Advertisements

PHPUnit and the Fatal error: Call to undefined method

PHPUnit+Selenium+Fatal errors:

Once upon a time i was writing and executing some Selenium acceptance tests using PHPUnit and i realized that PHPUnit detects fatal errors due to an undefined method and convert them into exceptions. Good feature! i though, if you have a fatal error in your test the process execution  doesn’t stop, the log is generated and the rest of the tests can finalize. But i started to investigate the PHPUnit code and i saw that the detection of the fatal error might be done by sheer chance.

The PHPUnit SeleniumTestCase driver has a well known list of valid commands to send to Selenium, if you send an invalid one from your test it will be detected and the exception is thrown. How PHPUnit detects it? using the magic function “__call”, each command the driver receives goes through this magic function and switch it to see if the command is contained in the list of commands.

Although i don’t like the PHP magic functions and i think this conversion from the fatal error to the exception was not on purpose, the result is good and gives us a better way to handle the “undefined method” fatal errors because the process execution is not stopped and the test result follows the PHPUnit format therefore is easily readable.

But the behaviour is not always like this, what will happen if you are delegating test responsabilities to other external classes? maybe because you are using the Page Object pattern or just using helpers.

Continue reading

AJAX + Selenium: solution for race conditions

There exists a problem with the race conditions testing websites full of AJAX, here is a solution using Selenium.

I’ve been looking for the perfect solution of the pair “AJAX + Web testing tool” and i’ve reached the conclusion that, so far, there isn’t a perfect solution.

Selenium is ready (at least almost ready) to test AJAX websites and apparently, Selenium is the most used tool and has the biggest support community, and therefore, is the better one to deal with AJAX applications.

As probably many of you know, Selenium has methods like “waitForXXX” that can assert most of the AJAX request and if your website renders the code statically or dynamically just some pieces of the site, you won’t have any problems.

But what happens if your website is full of AJAX and all the source code is generated dynamically? I mean, for instance, the JavaScript events.

I’m not an expert in HTML or JavaScript but i’m talking about the onclick, onmouseover, onload, etc. attributes of the HTML tags. Usually, these attributes have JavaScript events that perform a change in the site.

BUT, if these JavaScript events are loaded dynamically and asynchronously using an AJAX request, is Selenium able to perform proper assertions? NO.

Continue reading