Item based collaborative filtering in the OP room

 

See you next year

 

Some facts about my cartoon blog:

- 8000 clicks per month 
- 500'000 (far more) in total 
- most of the visitors are from USA, Asia and South America.

No ads on the blog by purpose. 

Please note, if you are using one of the cartoons for personal use, you are free to do so, but if you use them commercially, you must ask for permission. It's just fair. ThanX.

Obvious Bugs

 

After a series of good quality delivered to the customers, there came the temptation to short-cut testing activities. Under pressure, trade-off decisions have to  be taken. Can we risk delaying the release again due to a weird anomaly that happens from time to time while we have no reproducible scenario at hand? Is it better to start investigation and communicate the bad news to the release manager? What if we ship anyway? Will the testing department look bad if customers find obvious bugs like low hanging fruits? We've gone through all of this, but one lesson we have learnt is to raise a ticket for all anomalies, regardless whether we have a reproducible scenario yet. The meaning is to label our internal findings as known anomalies/findings and ship this information along with the release notes. As soon as we have a reproducible scenario, we extend the ticket accordingly and the product owner can plan the fix. If it turns out to be a flash in the pan, good; and if not, at least, we are not in the line of fire. We found the problem, before the customer did.

Special fears in the world of insects

    

Software made on Earth

This is a remake of my original cartoon which was published at SDTimes, N.Y. in their newsletter as of April 1, 2008

Mutation Testing and why we don't need it, or do we?

 When our kids were still small, every Easter, it was a tradition to hide chocolate eggs, sweets and small presents in the garden, around the house, at the carport and sometimes also within the house.
While the kids were so excited to find all the little things, we parents watched them equally excited.

When Easter was long over, often, one or the other egg was still found by accident in a corner or somewhere in a plant pot; too old to still be edible. In other words, our kids didn't track down all of them at Easter. Over the years, they got better and better. We had to be more creative finding new extraordinary places to hide the little things from them, so they didn't have an easy catch ("low hanging fruits" how testers would say).

While we never spent a thought about our kids' "mathematical" effectiveness of finding all these little presents, this is exactly what mutation testing is all about.

It is a method to measure the effectiveness of unit tests in detecting anomalies in the code. The idea is to inject bugs by purpose and then verify how many of these are found. That's pretty much the same like hiding chocolate eggs in the garden.

A typical example of injected bugs (mutants) could be the change of a comparison operator from something like (x<y) to (x>y) or a boolean value that is changed from an initial value true to false or vice versa. In case of a calculation engine, the computed value could be fuzzed and made return an incorrect result. The point is, that these bugs are implemented by purpose and - in contrary to our annual tradition at Easter - the tool that modifies the code knows exactly how many mutants were added and where.

When executing a unit test, the mutation test tool compares the number of failed tests with and without the modified code. If the number of failed tests is the same for both scenarios, then this is an indication of inadequate tests.

I am not experienced in automated mutation testing, but I find this topic quite interesting, especially because IT-companies tend to measure just the test coverage but often, have no idea whether their unit-tests are really effective. Test Coverage doesn't tell you anything about the quality of the code. You can have 100% test coverage for one method and still fail miserably with uncaught exceptions by applying other valid inputs to the method.

Although mutation testing is usually done as part of automated tests using corresponding plug-ins, you can do mutation testing also manually. When I was drawing the cartoon, I was more focused on the manual aspect and less about the potential of using it to test existing automated tests.

Let's go a few steps back and look at our today's approach. We have a lot of manual tests (>1000), we also have a lot of unit tests (> 20'000), a very effective API test suite (> 3000 tests) and also a few UI tests (ca.100), following the typical test automation pyramid in terms of distribution of the tests, but we haven't integrated any sort of mutation testing yet.

I get emailed automatically whenever our testers find defects either through manual testing and/or by findings in the automated UI test-scripts and/or automated API tests. Based on the amount of emails received daily, I draw the conclusion that we are an effective test-team finding many defects. But, of course, it would be more interesting to learn whether we could even do any better. Are there even more bugs around to catch? Honestly, with the current amount of anomalies reported by my testers, my first reaction was rather defensive. Why I should inject any additional bugs by purpose? We have already enough to do while analyzing all the findings that slipped into the code unintentionally. This was also the original idea behind the cartoon, but..here is my mistake:

We have no facts at hand but simply a certain amount of defects we raise every week. 

Mutation Testing could help us collect more facts. Mutation Testing can not only be applied through tools, it can also be done manually. For example, if you want to understand how long it takes to find out a certain (obvious) bug introduced by purpose, just add it and let's see. You don't even need to inject code, you can also change a configuration that leads to a different (unexpected) behavior.

For example, one of my tested software creates documents with inquiries to doctors. A configuration allows the documents to be fit with a data-matrix code on pages the doctors have to fill out and return. When the letters are returned with the data-matrix code on it, a software-component can automatically identify the original request and related patient, then map it to the answer received. This enables quick access to both, original request-letter and response. 

The configuration could be turned off (by purpose), causing the created letters being sent out without a data-matrix code. How long do you think will it take until our testers notice the missing data-matrix code on the letters? 

I am pretty sure, it won't take long, because such a test is well documented in the regression test suite. But, what if we challenge them more - like making the letters print a hard-coded data-matrix code that is the same for all letters? 

It takes more efforts for a tester to find the problem. 

If the test is not documented, it is likely for the testers to miss the bug. If it is documented, it may still depend on the priority set for the test case whether the test is executed at all. If testers are all too confident that this piece is likely not to fail, they won't test it either.

If you inject such mutants, you need to be clear on your goal. Do you want to test the efficiency of the testers, the accuracy of the test cases or the effectiveness of automated tests?

Nightmare Acceptance Test

 

The final rollout

 

In the need of a Neuralyzer

...

Roast the test reports

Doozy! Only two bugs left.

That's what I thought, until we got the next version...

As James Whittaker wrote in “Exploratory Software Testing”, bugs tend to congregate for a variety of reasons such as complexity of the code, skill of the assigned developer, number of bugs in the past, etc. To make it short; where you see one bug there are likely more near around. You just need to look for them.

The Demo Effect

"Hang on, the demo starts soon, then let's go out and ruin their show".

Originally, the text was "wait until the sprint review is over, then let's go out and show up again", but the scene is less funny with the previous text. The current one turns this cartoon into a more common situation, aka. the demo-effect.

The original text  has its root in a real story. Long time ago (not in the company I work right now), the product owner regularly moved all reported defects to a low priority heap shortly before the sprint review, only to put them back into the next sprint right after the review. The goal was to shine with a good product having non-important bugs. With this approach he kept the release manager quiet, because the release manager was looking at high priority bugs only shortly before the review. If there weren't any left, the product owners were out of the line of fire.

It's like in Patriot Games, where the secret armed forces in the desert knew exactly when the spy satellite flied over their hidden military camp. They tided up everything shortly before it reaches their coordinates and then they rebuilt the camp after it had passed (until next time). Result: pin sharp satellite images of unsuspicious cabins in the desert.

Technical Debt not close-to-balance

I think it is normal in a project to accumulate technical debt somehow. You may have tough deadlines or other reasons to tend get things done more quickly than usual, probably with the mindset of getting it right later.

Other reasons for (quick) workarounds are new awareness (more experience) of how things work better under certain circumstances. The old approach was good for one particular problem, but not good enough for a general solution.

An increasing number of developers working on the same project may force the architect to enforce new coding guidelines or best practices for typical software development problems.

The use of embedded third party software may also trigger changes on your side when updates are delivered.

Technical debt should be avoided as much as possible and yes, there are scenarios where you are simply forced to live with technical debt. But, one should always be aware that, often, the time to improve old code won't be availble. Even if there will be such time, think about the risk of removing technical debt. Someone who works under pressure not only tends to seed ugly workarounds, but he is likely also adding sloppy unit tests (if he adds any at all). If that's the case, removing technical debts with refactored code adds new risks breaking functionality you won't know about until the customer reports them.

The challenge of prioritizing bugs

...

Hiding bugs

Duplicate Bugs Arguing in JIRA

I guess, we all agree, duplicate bug reports are a pain. They cause avoidable time to investigate. But, it is not always easy to find out whether a reported anomaly is already known.

 When a developer believes that a series of bug reports all have the same root cause, she tends to claim these bugs are all duplicates. The test engineer on the other hand would disagree and state “these are all different scenarios from an E2E perspective”.

 At the time of reporting an issue, we usually don’t know the root cause unless we dig deeper into understanding the anomaly. Even if a developer assures the bugs all have the same cause, it still makes no sense to mark these reports as redundant. One can never be sure the developer is right. I have experienced a lot of situations where a developer claimed having fixed a problem and then learnt that only part of the problem or a completely different one was repaired. 

 Michael Stahl [1] makes an interesting note when he states:

"Why would the same tester report the same issue twice? It just adds extra work for the tester, who for sure remembers the first report. Usually, a duplicate bug is reported when two testers identified the same problem and both reported it without first checking if it’s already in the system".

 I personally believe that doing an upfront research in the bug-tracking system doesn’t really help avoiding duplicates completely. We also contact our team mates or submit a question in a group chat.  A screenshot can help setting the record straight, but searching for clarifying pictures is even harder.

 When testers raise duplicate bug reports, we consider it an indication that these bugs are either easy to find and annoying.

 And customers? They don’t care about how many times the same bug was already reported. Some smart techies might "google" for a solution to their problems, but you can hardly avoid duplicate bug reports raised by customers. For example, according to Castelluccio [2], Mozilla receives hundreds of bug reports and feature requests from Firefox users every day. It is clear that in such cases, tools are required that categorize bug reports based on similarities of other bug reports to save a company’s time analyzing such anomalies.

 Per Runeson [3] describes an approach using NLP to support the automatic identification of duplicates. Their conclusion: "Even though only 40% of the duplicates are found using this approach, it still means a substantial saving for a major development organization"

 However, the best way to avoid duplicate bug reports is by fixing a reported anomaly straight ahead. Don’t wait for the duplicates.

---

References:

[1] When Testers Should Consider a Bug a Duplicate (Michael Stahl), Sticky Minds, January 15, 2018
[2] Teaching machines to triage Firefox bugs (Marco Castelluccio), April 2019 at https://hacks.mozilla.org
[3] Detection of Duplicate Defect Reports Using Natural Language Processing (Per Runeson & Magnus Alexandersson), IEEE 2007

Shortly before the Sprint Review

...

Birds love BUGS

Here are the stats. I raised 2500 bugs in 12 years, then moved to a company where I raised a thousand bugs in only 19 months.
Presuming that a typical year has 252 working days, this gives me rate of 2.5 bugs per day or 12 per week (compared to an average 0.8 per day or 4 per week during the last 12 years).

That means the rate of identified defects has increased by the factor of 3.

What do these numbers tell about me or the software-under-test, or the company and what does it tell about the developers who introduce these bugs?
Do these numbers really have any meaning at all? Are we allowed to draw a conlusion based on these numbers without having the context? We don't know which of these bugs were high priority, which ones weren't. We don't know which bugs are duplicated, false alarm and which of those look rather like they should have raised as a change request.
We also don't know what is the philosophy in the team. Do we raise any anomaly we see or do we first talk to developers and fix it together before the issues make it into a bug reporting system. Do we know how many developers are working in the team? How many of them work really 100% in the team or less, sporadically, etc...Also, does management measure the team by the number of bugs introduced, detected, solved or completed user-stories, etc.? May the high number of identified issues be a direct effect of better tester training or are the developers struggling with impediments they can/cannot be held responsible for and these bugs are just a logical consequence of these impediments? Are there developers who introduce more bugs than others?

As is with these numbers, they are important, but they serve only as a basis for further investigation. It's too tempting to use these numbers as is and then draw one's one conclusions without questioning the numbers.

End of holiday season

When developers come back from holidays...

Baffled sequenceIDs

Sorry folks, Insider. I don't know how I can better generalize the cartoon so it is also funny for those who haven't experienced our exciting moments we had with our love-hate relationship-sequence IDs. The counter was reset after each deployment. It resulted in duplicate numbering, hence kept us busy hunting for ghost bugs.