It doesn't matter what kind of tool you buy. Implementing and keeping test automation up and running for a long-term period comes with a cost that a lot of people did not expect when going for test automation. Whenever I had the chance to talk to people about test automation or even see what they've done or started to do, I recognized that most people tried to implement their own frameworks (me included), simply because what they bought or downloaded was not good enough or not easy enough to use for the tester. Enriching your tool with a test framework that fits your needs is not wrong per se, but what most of my contacts had in common, is the fact that they started to automate testing on the Graphical User Interface (GUI) only.
That is funny, because the GUI is one of the most difficult and complex interfaces to automate for testing. The scripts are usually slow and the GUI changes often. Developers might embed components of different technologies and newer versions of it and your test tool might not (yet) support it. And then, developers start changing the GUI, re-arranging it, inserting additional dialogs that were not there before, etc. That is when a tester / test automation expert / framework developer / is suddenly more busy adapting existing scripts or the framework itself, than writing new test cases.
Doesn't this sound common: "Uhh, ohh that script does not work anymore because they have changed, this and that, so let us exceptionally execute this one test case manually because I need to have the report tonight. Maybe another test script is affected, too, uhh, ohh and may be a third and fourth test script as well".
I call this the Cretaceous Period of Test Automation. The scripts start to die one by one until you see so much red that you are attempted to question test automation and maybe even banish it and put the expensive tools back to the shelf.
I am not telling you here, that GUI Automation is bad. But there are other ways which can be more effective and easier and cheaper to maintain. For example, testing below the UI. This could be an API or a B2B interface (WebService), or something similar. If you are a lucky quality analyst who tests software components which provide a public interface to customers...use it for testing!
For those, who don't have APIs, ask for it, even if it is just for testing. We do quite a lot of automated testing on a WebService level. BTW, we did NOT abandon GUI automation, although my answer here may sound like we did. Of course not, we need it, but we are always trying to test the feature below the UI if possible. The API-Tests run on a daily basis and can therefore act as an early warning system. Something the big amount of UI scripts cannot do for us.
Of course, you always have to do the sort of testing / test automation that is appropriate for your needs and your situation, but at least you may think about the alternative of testing somewhere else than the UI.
TOJZ
My cartoon about this topic was also published at the "The Testing Planet" Magazine (March 2011)
