It was a bear, for sure (overvalued bugs)

 

Almost every year we go skiing in the mountains for about a week. We usually rent the same house  which is surrounded by a huge garden. One morning - it's quite a while back now  - we encountered a series of big footprints in the garden.  They were really large. Next to them was a big bloody bone which I assumed to be the mortal remains of a sheep. "What the heck happened here and whose footprints could these be?"

After measuring the size with a scale and analyzing the shape I put a 2-Swiss Frank coin into the track and muttered to myself.."this was clearly a bear".  I must add to this point, until this date, the only tracks I had ever reliably identified were those of a small rabbit. Regardless, everyone agreed, my wife, my kids and my parents in law who were joining us at holiday. It was enough confirmation to take some photos and show them to the local tourist information bureau. She too, was amazed by the photos. She picked the phone and called the forest ranger. While waiting at the counter and watching the officer talking with the ranger, I heard him mentioning a cat. Wait a minute....!

"Are you kidding me?", I protested. "Can you please tell the ranger about my photos and send those to him?", I added mortally offended. 

If at all one can mention a cat, then only because these footprints had the size of a mark from a cat's after-lunch nap in the snow. That's for sure.

The ranger promised to check our garden shortly before dawn and let us know. Of course, he never did and so I showed the photos to a local farmer. He confirmed, he had come to the same conclusion as me. He guessed, the ranger as well as the tourist office may both just have been afraid the story and the pictures could make it to the news and scare the tourists. 

That's good. Finally an expert who confirms what I've thought from the very beginning. Back home, two weeks later, I assorted my photos and decided to send a few of them directly to the ranger who never visited our holiday house. I was confident, once he sees the pictures, he'll be convinced and confirm my theory.

But, the ranger - who responded promptly - came up with a very interesting new theory. He claimed, it is not likely for the tracks being left by a bear. At the time the bears are usually still hibernating. He stated, these tracks were either left by a wolf or a big dog.

He explained, the huge size of the footprints may be caused by atmospheric conditions that let the footprints grow by at least 50% in size at relatively short time.

To be honest, I'd loved to hear something different , but it was an interesting argument and these statements made at lot more sense than the original comparison to a domestic cat. Of course, I am still not really convinced, but at least I am now at a point where I have to admit, there may be more than just my only one explanation.

 

 

 

How is that story related to testing? 

From time to time it happens that we find cool and unexpected bugs that have the potential of being a real big thing; the one killer bug. We might be euphoric and immediately record it and probably all to easy forget to collect more facts before we celebrate the great finding and label it highest priority. Often, when looking closer to an ugly looking bug and spending more time to understand its impact, the probability of the anomaly to occur in real life and the number of affected users, then what's left may be the sobering that we've just found another low, maybe medium priority bug, but we made a mountain out of a molehill upfront. It's great to find bugs, but be careful with the initial rating without having a second set of eyes looking at it.

By the way...., I am still convinced it was a bear, for sure =;O)

333-How much time do you need for testing?

According to Randall Rice [1], about one third of the total test effort should be spent on determining how to test. I’ve usually added another third to prepare reusable test data for regression tests and yet another third to finally execute the test. That’s what the 3-3-3 stands for me. This is for the current sprint where new features are being developed.

But I was often asked, how much time my team needs to perform a full manual regression test. While I could give a rough estimation, I liked to challenge the interviewer by asking whether she could provide me the number of bugs that I am going to find during our tests. I asked because I often experience tests that are scheduled for 30-45 minutes, taking up to 2 hours depending on the number of anomalies we find during the tests.

How can this happen?

• When an anomaly is detected, we need to analyze it. We need to understand whether it is a bug, a new feature or a function that we just don't remember working the way it is working now.
  
  
• We also need to prepare a reliable scenario to reproduce it and document it, otherwise product owners and developers tend to close the ticket too fast.
   
• In case the anomaly turns to be a defect, it is helpful to understand, when it slipped in. Understanding when the defect was introduced, impacts the priority of the issue product owners assign this finding. If the defect is there since weeks without anyone to notice, it may be considered with lower priority. If instead, it is a recently broken function, the priority set to this anomaly might be rated differently. Figuring out the exact date of intrusion can be time consuming. If you have automated tests, you can go through all past test results until you find the time when it still worked fine. Otherwise, you may need to run some extra tests on different test environments just to learn more about the history of the anomaly. This costs time, usually not planned in a manual regression test.
   
• What follows next is an accurate documentation for developers. One needs to create screenshots providing evidence of the feature how it worked before and how it behaves now that you found a problem. The more accurate the description the less time they lose understanding your ticket.
   
• Sometimes, an issue cannot easily be reproduced and you must examine the exact previous steps you took before the detection of the bug. Easy, if you followed a script; more challenging if you followed an exploratory approach while testing. The issue may occur in one particular situation of the workflow and may still work fine if you execute the workflow from a different starting point.
   
• If you are executing a test that someone else documented, you might find that important information is missing for you to effectively execute the test. You may need to ask someone for help or investigate your own, following referenced link (if they exist) and read additional material to finally understand what the purpose of the test was and how that feature-under-test is supposed to work.
   
• Testers are often asked for help when a problem occurs on production, because they often know more about the context and how and why a feature was built the way it is in production. The tester is taken away from the planned test to help someone in need. Such tasks cannot be planned.

These are all tasks that impact how far the estimation deviates from the actual time needed to perform a regression test. So, if you get asked how much time you need to test, my suggestion is to provide a rough estimate and add the disclaimer "...if we don't find any anomalies. Everything changes with every new bug we find".

T. Zelger, August 2020

 

References
[1] Testing Dirty Systems, by Randall W. Rice and William Perry

Empty Promises

...

Going Live with Gobla 1.0

Roast the test reports

Simplified Bug Prioritization Matrix

No cartoon today, it is just an illustration of our internal simplified bug prioritization matrix

Doozy! Only two bugs left.

That's what I thought, until we got the next version...

As James Whittaker wrote in “Exploratory Software Testing”, bugs tend to congregate for a variety of reasons such as complexity of the code, skill of the assigned developer, number of bugs in the past, etc. To make it short; where you see one bug there are likely more near around. You just need to look for them.