Performance Testing

This is a reworked cartoon. Originally I drew the cartoon without any keyboards at his hands/tentacles and it was actually also published like this in "The Testing Planet". But now the joke is a little bit more clear, I guess. Enjoy.

How to do performance testing
What a test manager is interested before going live is often this: "Does the system respond fast enough and still accurately if there are 10,20,30 or more users's working in parallel?

Performance testing tools provide you the ability to develop scripts that fire scripted scenarios and measure their response time when executed isolated or all together using these ramp up numbers. Often, problems already show up with a much smaller number of parallel users using the system. Before you buy expensive experts, solve these problems first. Write either one or more simple Selenium UI scripts or some API-tests like REST/SOAP, build an endless loop around it and ask some colleagues whether they are willing to run the scripts from their workstations. Then find a free workstation where you can do manual testing in parallel to feel how it is not to be alone anymore.

If that reveals no bugs, move to the next stage and hire experts who can do the ramp-up as explained in the first sentence. Usually you should come up with 4-5 typical scenarios which cover at least 1 read, 1 update, 1 create and probably 1 delete operation. Not more, because scripting and providing test data for these scripts can be the most time-consuming and costly part in a performance test session. Also note, that a typical distribution of user's actions is 90/5/5 (90% read, 5% update, 5% create) or 80/10/10 or similar.

When using a professional performance testing tool (or working with an expert company), make sure you test the different scenarios in isolation. That means, let's test what happens if you have 10,20,30 or more users only doing read requests, then do the same only for update and yet another set only for creation. In most cases you will learn that a high number of read requests are nearly noticeable while it is often a different story once you test one of the other scenarios. Combining all scenarios together is more realistic but should be done after you have already learnt about the first tests. A combination makes it always hard to pinpoint "who is the problem"?

Don't forget to have someone manually test the system while you are running the automated performance test scripts. Learning how it feels if all of a sudden the performance goes down, is an invaluable information that numbers can hardly give you.

In a project I coached the performance test experts, it turned out the biggest amount of work was spent during the preparation of meaningful test data for the scripts. I mean, we were firing the tests on a real productive database, but finding accurate records with which one could perform the update scenarios without triggering this or that validation message wasn't easy. It turned out it was probably better if we created these data ourselves.

Not only this, we assigned each virtual users to work on only one particular set of data that others don't write to. If we didn't follow this approach, we had another challenge of locked records or trying to update something that was already updated beforehand that would have resulted in validation error messages. Next was the creation of records that allowed only one instance. A second record wasn't allowed. So we had to expand the script to also always delete right after the record was created. Also for this scenario, each virtual user had a particular set of cases that he/she could add new records to without getting in the way of another.