Revise the Test Report

I thought, I'd published this cartoon in 2020 already, but couldn't find it, so I am doing it now, with a 3 years delay...=;O)

Additional note as of December 2025:
funny though, the cartoon is as relevant today as it was when I originally created it. History repeats itself.

 

Plus...there is a similar cartoon of mine as of August 2021 (Accurate Test Reports) that pretty much addresses a similar topic.

 

 

 

 

 

Mutation Testing and why we don't need it, or do we?

 When our kids were still small, every Easter, it was a tradition to hide chocolate eggs, sweets and small presents in the garden, around the house, at the carport and sometimes also within the house.
While the kids were so excited to find all the little things, we parents watched them equally excited.

When Easter was long over, often, one or the other egg was still found by accident in a corner or somewhere in a plant pot; too old to still be edible. In other words, our kids didn't track down all of them at Easter. Over the years, they got better and better. We had to be more creative finding new extraordinary places to hide the little things from them, so they didn't have an easy catch ("low hanging fruits" how testers would say).

While we never spent a thought about our kids' "mathematical" effectiveness of finding all these little presents, this is exactly what mutation testing is all about.

It is a method to measure the effectiveness of unit tests in detecting anomalies in the code. The idea is to inject bugs by purpose and then verify how many of these are found. That's pretty much the same like hiding chocolate eggs in the garden.

A typical example of injected bugs (mutants) could be the change of a comparison operator from something like (x<y) to (x>y) or a boolean value that is changed from an initial value true to false or vice versa. In case of a calculation engine, the computed value could be fuzzed and made return an incorrect result. The point is, that these bugs are implemented by purpose and - in contrary to our annual tradition at Easter - the tool that modifies the code knows exactly how many mutants were added and where.

When executing a unit test, the mutation test tool compares the number of failed tests with and without the modified code. If the number of failed tests is the same for both scenarios, then this is an indication of inadequate tests.

I am not experienced in automated mutation testing, but I find this topic quite interesting, especially because IT-companies tend to measure just the test coverage but often, have no idea whether their unit-tests are really effective. Test Coverage doesn't tell you anything about the quality of the code. You can have 100% test coverage for one method and still fail miserably with uncaught exceptions by applying other valid inputs to the method.

Although mutation testing is usually done as part of automated tests using corresponding plug-ins, you can do mutation testing also manually. When I was drawing the cartoon, I was more focused on the manual aspect and less about the potential of using it to test existing automated tests.

Let's go a few steps back and look at our today's approach. We have a lot of manual tests (>1000), we also have a lot of unit tests (> 20'000), a very effective API test suite (> 3000 tests) and also a few UI tests (ca.100), following the typical test automation pyramid in terms of distribution of the tests, but we haven't integrated any sort of mutation testing yet.

I get emailed automatically whenever our testers find defects either through manual testing and/or by findings in the automated UI test-scripts and/or automated API tests. Based on the amount of emails received daily, I draw the conclusion that we are an effective test-team finding many defects. But, of course, it would be more interesting to learn whether we could even do any better. Are there even more bugs around to catch? Honestly, with the current amount of anomalies reported by my testers, my first reaction was rather defensive. Why I should inject any additional bugs by purpose? We have already enough to do while analyzing all the findings that slipped into the code unintentionally. This was also the original idea behind the cartoon, but..here is my mistake:

We have no facts at hand but simply a certain amount of defects we raise every week. 

Mutation Testing could help us collect more facts. Mutation Testing can not only be applied through tools, it can also be done manually. For example, if you want to understand how long it takes to find out a certain (obvious) bug introduced by purpose, just add it and let's see. You don't even need to inject code, you can also change a configuration that leads to a different (unexpected) behavior.

For example, one of my tested software creates documents with inquiries to doctors. A configuration allows the documents to be fit with a data-matrix code on pages the doctors have to fill out and return. When the letters are returned with the data-matrix code on it, a software-component can automatically identify the original request and related patient, then map it to the answer received. This enables quick access to both, original request-letter and response. 

The configuration could be turned off (by purpose), causing the created letters being sent out without a data-matrix code. How long do you think will it take until our testers notice the missing data-matrix code on the letters? 

I am pretty sure, it won't take long, because such a test is well documented in the regression test suite. But, what if we challenge them more - like making the letters print a hard-coded data-matrix code that is the same for all letters? 

It takes more efforts for a tester to find the problem. 

If the test is not documented, it is likely for the testers to miss the bug. If it is documented, it may still depend on the priority set for the test case whether the test is executed at all. If testers are all too confident that this piece is likely not to fail, they won't test it either.

If you inject such mutants, you need to be clear on your goal. Do you want to test the efficiency of the testers, the accuracy of the test cases or the effectiveness of automated tests?

Downbeat Quality Freak Concerns

 

Cheerful debugging messages and its consequences

Over a year ago, we tested the automated printing of a clerk’s signature on letters being sent to doctors, lawyers, etc. There were some issues where the signature was missing, that’s why we marked the defective signature-template with a debugging text. The idea was to test whether the template is processed at all or whether the problem is within the signature itself.
 

I don't know what came over me when I used "meow-meow" as the debugging text. Probably, Luna is to blame for it. Luna was our 16 year old cat which died shortly before. She's now eternalized in this (less testing related) cartoon.

We quickly found the cause for the missing signature, we fixed the bug and shipped it to the customer along with an updated template. Unfortunately, I forgot to remove the debugging text in the template. The customer found the problem during their internal BETA test. We fixed the template, shipped it, no big deal, over.

But, about half a year later, the customer reported that he'd seen a letter in the productive document management system containing the complimentary close "meow-meow" right below the printed signature.

I could feel my face becoming soaked with blood. How the heck could this happen again? Even though we don't execute all regression tests each time we ship a minor release or hotfix, the print-out is part of all smoke tests. Therefore, my thoughts were "this simply can't be true", 'cause we've never seen any such text being printed on any of the letters that ended on our printers.

A little investigation and it became soon clear what happened. Even though we had originally sent the correct template, there was still a copy of the old "meow-meow" template around. When someone decided to include the template into the git-repository, he took the wrong one. As a result, with the next minor software-version shipped, the old signature-template was deployed again. 

While this explains how the bug was re-introduced, it does not explain, why testers did not find the issue while testing the print-out. 

Further investigation revealed, even though ALL letters processed the "meow-meow" signature-template, NOT all letters were printing the text below the signature. We found out, it depended on several things such as the size of the signature and the configuration of the target letter being sent out. If there was a fixed size between signature and complimentary close, then the text "meow-meow" simply didn't make it in between and wasn't printed. If instead, the letter was configured to dynamically grow with the size of the signature, then also "meow-meow" made it to the printer.

Without going into too much details, only one particular type of letter was affected and fortunately, it was a letter that customers sent only to internal addresses (like an internal email) and not to stakeholders outside the company. Lucky me! 

Well, not really...because...

..one of the customers told me that - despite this letter is sent only internally - it may still be used as an enclosure for other letters going out to lawyers and doctors. The company feared they'd lose credibility when such letters made it to their customers. It was therefore important to remove all identified documents in production before they pay a high price for this slip.

What are the lessons learnt?
First of all, it could have been worse. I mean, if you get a letter that ends with "meow-meow", it's likely to put a smile on your face. In my 20 years career as a tester and developer, I've seen comments and debugging texts that are much worse and sometimes below-the line. My friends told me several stories about similar happenings. 

A colleague told me, they had added an exhilarant test-text for their ATM. The text was triggered when the card was about to get disabled. When in production - their boss wanted to withdraw money from the machine, the card got blocked and showed the text "You are a bad guy. That's why we take your card away". I only know, he wasn't amused about it. 

Looks like, we are not alone.

But, as we have just learnt, it depends on who is affected and who will read it. I can only suggest to never ever type anything like that in any of your tests because you never know where it's going to show up.

Instead of "meow-meow" a simple DOT or DASH would have done the same trick, and didn't raise the same kind of alarm in case such debugging messages make it to the bottom of a letter. 

This was the last cartoon and blog entry for 2021. There are more to come next year. Please excuse, this cartoon here wasn't a pure testing related cartoon, but the story still is.
I wish you all a merry XMAS and a happy new year. I hope you enjoy the blog and continue to regularly visit me here.
 

 

In the need of a Neuralyzer

...

Simplified Bug Prioritization Matrix

No cartoon today, it is just an illustration of our internal simplified bug prioritization matrix

Bad luck !

According to the table below(published by the New York Times [1] today on March 20), New York has almost 7 times the amount of registered infected Corona patients compared to Washington but "only" about 0.5% of them have died so far.

That's quite a good quota (for Corona). Let's have a look at the other side of the continent.   The Corona deathrate in Washington is nearly 6%. That's quite a contrast. What can be the reason for this difference? In most other States the mortality rate is roughly around 2%, similar to other countries in the world.

Here are some superficial (not seriously meant) hypotheses:

• The New Yorkers are doing something extremly right and those in Washington are doing something extremely wrong

• The dirty air in New York is poison for the virus. The more CO2, the better for human beings. Bad news for Greta Thunberg.

• New York may not be a typical state for eldery people to relax...,probably more young and healthy people live there while in Washington may be a pan of sick or eldery people spend the rest of their lives there (again, this isn't meant seriously, okay?)

• New York was better prepared for this predictable disaster

• or...the measures may have been taken inconsistently and/or the numbers are simply wrong

Let's get more curious and read the full article. We find valuable information by reading the details and then very quickly understand what happened. It is noted that 2900 of the reported cases in New York were all registered at the day of publishing the numbers. Interesting, probably it is too early for those new cases to "die". If we ignore these new cases (7102-2900), we still have a quite good quote of 0.8%. Better than in many countries worldwide. But further reading the article reveals “At least 35 of the deaths were connected to a single nursing center in Washington" and "...many of those cases involved older people with other health problems that made them especially vulnerable to coronavirus”.

Aha! So, it was jsut bad luck for the nursing center to get hit so hard. If you ignore these numbers (75-35)…we are back to an almost "normal" rate of 2%.

These numbers make sense now. No need for me to look for creative hypothetical argumentation.

And the link to IT? A similar incident once happend near the end of our software development sprint. In parallel to the development of user-stories, we also performed refactoring tasks to get rid of technical debt. As is the nature of refactoring, it may break previously working functionality. This is what happened: Instead of being ready for the sprint review, we had to reject the demo of 2 buggy user-stories that were working properly just a day before. It cost us valuable story points. We didn't achieve the goal. What followed were long-winded discussions and assumptions of what to do and what not to do in the future, whereas I thought: "WTF, Just bad luck, shit happens".

[1]https://www.nytimes.com/interactive/2020/us/coronavirus-us-cases.html?action=click&module=Spotlight&pgtype=Homepage

The Demo Effect

"Hang on, the demo starts soon, then let's go out and ruin their show".

Originally, the text was "wait until the sprint review is over, then let's go out and show up again", but the scene is less funny with the previous text. The current one turns this cartoon into a more common situation, aka. the demo-effect.

The original text  has its root in a real story. Long time ago (not in the company I work right now), the product owner regularly moved all reported defects to a low priority heap shortly before the sprint review, only to put them back into the next sprint right after the review. The goal was to shine with a good product having non-important bugs. With this approach he kept the release manager quiet, because the release manager was looking at high priority bugs only shortly before the review. If there weren't any left, the product owners were out of the line of fire.

It's like in Patriot Games, where the secret armed forces in the desert knew exactly when the spy satellite flied over their hidden military camp. They tided up everything shortly before it reaches their coordinates and then they rebuilt the camp after it had passed (until next time). Result: pin sharp satellite images of unsuspicious cabins in the desert.

False Positives

When developers want to quickly demonstrate that in principle it works, they tend to hard-code things with the intention to replace that piece of code later. But, once it works, the focus is shifted to other tasks. It is likely that one forgets about these rotten code pieces.

 When Switzerland changed their tax rate from 7.6 per cent to 8 per cent, we were quite surprised that 7.6 was hard coded rather than read from a configuration file. As a result, we had to redeploy a new fix. In another example we found a code statement that made the program wait for 200 seconds. No wonder the application was slow. The developer had a good explanation why he added a wait-statement, but he mixed seconds with milliseconds.

 The term "false positives" has its roots in medical testing. A false positive is an error in which a test result improperly indicates the presence of a disease (the result is positive), when - in reality - it is not present, while a false negative is an error in which a test result improperly indicates no presence of the disease when in reality it is present.

Simply said, never trust any success messages. If the program claims it worked, the program may be wrong. Vice versa is also true. When the system throws an exception, the operation may still have worked correctly, regardless of the error.