333-How much time do you need for testing?

According to Randall Rice [1], about one third of the total test effort should be spent on determining how to test. I’ve usually added another third to prepare reusable test data for regression tests and yet another third to finally execute the test. That’s what the 3-3-3 stands for me. This is for the current sprint where new features are being developed.

But I was often asked, how much time my team needs to perform a full manual regression test. While I could give a rough estimation, I liked to challenge the interviewer by asking whether she could provide me the number of bugs that I am going to find during our tests. I asked because I often experience tests that are scheduled for 30-45 minutes, taking up to 2 hours depending on the number of anomalies we find during the tests.

How can this happen?

• When an anomaly is detected, we need to analyze it. We need to understand whether it is a bug, a new feature or a function that we just don't remember working the way it is working now.
  
  
• We also need to prepare a reliable scenario to reproduce it and document it, otherwise product owners and developers tend to close the ticket too fast.
   
• In case the anomaly turns to be a defect, it is helpful to understand, when it slipped in. Understanding when the defect was introduced, impacts the priority of the issue product owners assign this finding. If the defect is there since weeks without anyone to notice, it may be considered with lower priority. If instead, it is a recently broken function, the priority set to this anomaly might be rated differently. Figuring out the exact date of intrusion can be time consuming. If you have automated tests, you can go through all past test results until you find the time when it still worked fine. Otherwise, you may need to run some extra tests on different test environments just to learn more about the history of the anomaly. This costs time, usually not planned in a manual regression test.
   
• What follows next is an accurate documentation for developers. One needs to create screenshots providing evidence of the feature how it worked before and how it behaves now that you found a problem. The more accurate the description the less time they lose understanding your ticket.
   
• Sometimes, an issue cannot easily be reproduced and you must examine the exact previous steps you took before the detection of the bug. Easy, if you followed a script; more challenging if you followed an exploratory approach while testing. The issue may occur in one particular situation of the workflow and may still work fine if you execute the workflow from a different starting point.
   
• If you are executing a test that someone else documented, you might find that important information is missing for you to effectively execute the test. You may need to ask someone for help or investigate your own, following referenced link (if they exist) and read additional material to finally understand what the purpose of the test was and how that feature-under-test is supposed to work.
   
• Testers are often asked for help when a problem occurs on production, because they often know more about the context and how and why a feature was built the way it is in production. The tester is taken away from the planned test to help someone in need. Such tasks cannot be planned.

These are all tasks that impact how far the estimation deviates from the actual time needed to perform a regression test. So, if you get asked how much time you need to test, my suggestion is to provide a rough estimate and add the disclaimer "...if we don't find any anomalies. Everything changes with every new bug we find".

T. Zelger, August 2020

 

References
[1] Testing Dirty Systems, by Randall W. Rice and William Perry