Sunday, August 16, 2020

333-How much time do you need for testing?

According to Randall Rice [1], about one third of the total test effort should be spent on determining how to test. I’ve usually added another third to prepare reusable test data for regression tests and yet another third to finally execute the test. That’s what the 3-3-3 stands for me. This is for the current sprint where new features are being developed.

But I was often asked, how much time my team needs to perform a full manual regression test. While I could give a rough estimation, I liked to challenge the interviewer by asking whether she could provide me the number of bugs that I am going to find during our tests. I asked because I often experienced tests scheduled for 30-45 minutes, taking up to 2 hours depending on the number of anomalies found during the tests.

How can this happen?

  • When an anomaly is observed, I need to analyse it. I need to understand whether it is a bug, a new feature or a function that I just don't remember working the way it is working now.

  • I also need to prepare a reliable scenario to reproduce it and document it.

  • In case of a defect, it is also helpful to understand, when it slipped in. We need to know because understanding the exact date may impact the priority of the issue. If the defect is there since weeks without anyone to notice, maybe it can't be so important. If instead, it is a recently broken function, the priority might increase. To find answers to these questions, one may need to test the same scenario on an additional test environment that runs an older version of the SUT (software-under-test). Maybe, you need even more test environments to get more accurate information about the history of the observed anomaly.

  • Then follows the documentation for developers. One needs to create screenshots that highlight the situation before/after, adding labels to the pictures so the bug becomes clear for everyone without losing a developer's valuable time reading a lot of text.

  • Sometimes, an issue cannot easily be reproduced and you must examine the exact previous steps you took. Easy, if you followed a script; more challenging if you followed an exploratory approach while testing. The issue may occur in one particular specifity of the workflow while it may still work fine if you execute the workflow from a different starting point.

  • If you are executing a test that someone else documented, you might find that important information is missing for you to effectively execute the test. You may need to ask someone for help or investigate your own, following referenced link (if they exist) and read additional material to finally understand what the purpose of the test was and how that feature-under-test is supposed to work.

  • Testers are often asked for help when a problem occurs on production, because they often know more about the context and how and why a feature was built the way it is in production. The tester is taken away from the planned test to help someone in need. Such tasks cannot be planned.

These are all tasks that impact how far the estimation deviates from the actual time needed to perform a regression test. So, if you get asked how much time you need to test, my suggestion is to provide a rough estimate and add the disclaimer "...if we don't find any anomalies."

T. Zelger, August 2020


[1] Testing Dirty Systems, by Randall W. Rice and William Perry

No comments:

Post a Comment