The Birthday Paradox - and how the use of AI helped resolving a bug

While working / testing a new web based solution to replace the current fat client, I was assigned an an interesting customer problem report.

The customer claimed that since the deployment of the new application, they noticed at least 2 incidents per day where the sending of electronic documents contained a corrupt set of attachments. For example, instead of 5 medical reports, only 4 were included in the generated mailing. The local IT administrator observed the incident to happen due to duplicate keys set for the attachments from time to time.

But, they could not evaluate whether the duplicates were created by the old fat client or the new web based application. Both products (old and new) were used in parallel. Some departments used the new web based product, while others still stick to using the old. Due to compatibility reasons, the new application inherited also antiquated algorithms to not mess up the database and to guarantee parallel operation of both products.

One of these relicts was a four digit alphanumeric code for each document created in the database. The code had to be unique only within one person’s file of documents. If another person had a document using the same code, that was still OK. The code (eg. at court) was used to refer to a specific document. It had to be easy to remember.

 At first, it seemed very unlikely that a person’s document could be assigned a duplicate code. And, there was a debate between the customer and different stakeholders on our side.
The new web application was claimed to be free of creating duplicates but I was not so sure about that. The customer ticket was left untouched and the customer lost out until we found a moment, took the initiative and developed a script to observe all new documents created during our automated tests and also during manual regression testing of other features. 

The script was executed every once an hour. We never had any duplicates until after a week, all of a sudden the Jenkins script alarmed claiming the detection of a duplicate. That moment was like Xmas and we were so excited to analyze the two documents. 

In fact, both documents belonged to the same person. Now, we wanted to know who created these and what was the scenario applied in this case. 

Unfortunately, it was impossible to determine who was the author of the documents. My test team claimed not having done anything with the target person. The person’s name for which the duplicates were created occurred only once in our test case management tool, but not for a scenario that could have explained the phenomena. The userid (author of the documents) belonged to the product owner. He assured he did not do anything with that person that day and explained that many other stakeholders could have used the same userid within the test environment where that anomaly was detected.

 An appeal in the developer group chat did not help resolve the mystery either. The only theory in place was “it must have happened during creation or copying of a document”.  The most easy explanation had been the latter; the copy-procedure.

Our theory was that a copied document could result in assigning the same code to the new instance. But, we tested that; copying documents was working as expected. The copied instance received a new unique code that was different from its origin. Too easy anyway.

 Encouraged to resolve the mystery, we asked ChatGBT about the likelihood of duplicates to happen in our scenario. The response was juicy. 

It claimed an almost 2% chance of duplicates if the person had already 200 assigned codes (within his/her documents). That was really surprising and when we further asked ChatGBT, it turned out the probability climbed up to 25% if the person had assigned 2000 varying codes in her documents.

This result is based on the so called Birthday Paradox which states, that it needs only 23 random individuals to get a 50% chance of a shared birthday. Wow!

 Since I am not a mathematician, I wanted to test the theory with my own experiment. I started to write down the birthdays of 23 random people within my circle of acquaintances. Actually, I could stop already at 18. Within this small set I had identified 4 people who shared the same birthday. Fascinating!

 That egged us to develop yet another script and flood one exemplary fresh person record with hundreds of automatically created documents.

The result was revealing:

	Number of assigned codes for 1 person
	500	1000	1500	2000	2500
Number of identified duplicates (Run 1)	0	0	5	8	11
Number of identified duplicates (Run 2)	0	0	3	4	6

 With these 2 test runs, we could prove that the new application produced duplicates if only we had enough unique documents assigned to the person upfront.

 The resolution could be as simple as that:

• When assigning the code, check for existing codes and re-generated if needed (could be time-consuming depending on the number of existing documents)
• When generating the mailing, the system could check all selected attachments and automatically correct the duplicates and re-generate these or warn the user about the duplicates to correct it manually.
  

 What followed was a nicely documented internal ticket with all our analysis work. The fix was given highest priority and made into the next hot-fix. 

 When people ask me, what  I find so fascinating about software testing, then this story is a perfect example. Yes sure, often, we have to deal with boring regression testing or repeatedly throwing back pieces of code back to the developer because something was obviously wrong, but the really exciting moments for me are puzzles like this one; fuzzy ticket descriptions, false claims, obscure statements, contradictory or incomplete information, accusations and none really finds the time to dig deeper into the "forensics".

That is the moment where I cannot resist and love to jump in. 

 But, the most interesting finding in this story has not been betrayed yet. While looking at the duplicates, we noticed that all ended up with the character Q. 

 And when looking closer at the other non-duplicated codes, we noticed that actually ALL codes ended up with character Q. This was even more exciting. This fact reduced the number of possibilities from 1.6 million variants down to only 46656 and with it, the probability of duplicates.
 

At the end I could also prove that the old legacy program was not 100% free from creating duplicates even though it was close to impossible. The only way to force the old program to create duplicates was to flood one exemplary person with 46656 records, meaning all codes were used then. But that is such an unrealistic scenario, it was already pathetic.

At the end as it turned out, the problem was a configuration in a third party vendor's randomizer API function. In order to make the Randomizer create random values, you had to explicitly tell it to RANDOMIZE it. =;O) The title of this blog therefore could also be "Randomize the Randomizer" =;O)

 

Further notes for testers
Duplicate records can be easily identified using Excel by going to DataTools and choose "RemoveDuplicates".

 

 

 

Or, what was also helpful in my case was the online tool at 

https://www.zickty.com/duplicatefinder
where you could post your full data set and it returned all duplicates.
 

Yet another alternative is use to SQL group command like in my case

SELECT COUNT(*) as num, mycode FROM table WHERE personid=xxx GROUP BY mycode ORDER BY num DESC;

 

 

 

 

Bug's survival strategy

"Even a seemingly simple program can have hundreds or thousands of possible input and output combinations. Creating test cases for all these possibilities is impractical. Complete testing of a complex application would take too long and require too many human resources to be economically feasible". [1]

A customer reported a problem exporting documents. The system responded with a null pointer exception. We could not reproduce the problem on our internal test environments while playing around with known different document types used by our customers (they used all sorts of TIF for archiving) such as coloured, grey-scale, different DPIs, etc. until we asked for the original document from the customer even though we follow a strict rule not to work with production data containing sensitive information. We accepted it for once. Initially, we still could not reproduce the problem until we experimented with different kinds of annotations. We finally found out that the problem only showed up if a burnt annotation is set on a page other than the first. That was a wonderful finding after some upfront detective work.

 We asked ourselves whether we can expect the testing department to really find such bugs before the customer does. We all came to the same conclusion: In theory Yes, but practically it is impossible to cover all these kinds of variations for every use-case developed in a sprint and given the strict time-constraints.

  Fact is, no matter how hard and how accurate the testing is done, software will never be fully free of bugs. The complexity of software has increased dramatically during the last few decades and the number of needed test cases to gain a minimum level of coverage has grown so much, that it has become impossible to execute all tests and possible variations within the time given. Test Automation helps a lot but also cannot cover all required tests within the given time-frame. 

  Software development today still is a process of trial-and-error. I have never seen a programmer developing a software component from scratch without several times of failed compiling, fixing, re-writing, re-testing and finally shipping it, then failing at the customer and going through the same process again. With this statement

 "Software errors are blunders caused by our inability to fully understand the intricacies of these complex products" [2]

  Ivars Peterson compares today's technology with black magic where engineers act like wizards who brew their magic potion by mixing various ingredients of terminology that only them understand while the public dazzled by the many visible achievements of modern technology often regards engineers as magicians who can solve any problem

References

[1] The Art of Software Testing, Glenford J. Myers
[2] Fatal defect, Ivars Peterson

Astronomy for developers

 

Test Patterns out of control

 In my blog entry Apostrophe, Ampersand, Exclamation mark & Co. early this year, I highlighted how important the use of special characters has become for our team during testing. A closer look at the article demonstrates that these characters aren't really exceptionnel in real life. They are commonly used; actually more common than what most of us initially thought. It's therefore even more important, we don't treat these letters as a special cases used by testers only.

I smirked when I recently noticed how many fake persons were added into our reference test data containing German umlauts or apostrophes, etc. Looks like a lot of testers are crazy to find maniac but still realistic data. It has become much harder to find vanilla data. Mission accomplished.

 

Simplified Bug Prioritization Matrix

No cartoon today, it is just an illustration of our internal simplified bug prioritization matrix

Bad luck !

According to the table below(published by the New York Times [1] today on March 20), New York has almost 7 times the amount of registered infected Corona patients compared to Washington but "only" about 0.5% of them have died so far.

That's quite a good quota (for Corona). Let's have a look at the other side of the continent.   The Corona deathrate in Washington is nearly 6%. That's quite a contrast. What can be the reason for this difference? In most other States the mortality rate is roughly around 2%, similar to other countries in the world.

Here are some superficial (not seriously meant) hypotheses:

• The New Yorkers are doing something extremly right and those in Washington are doing something extremely wrong

• The dirty air in New York is poison for the virus. The more CO2, the better for human beings. Bad news for Greta Thunberg.

• New York may not be a typical state for eldery people to relax...,probably more young and healthy people live there while in Washington may be a pan of sick or eldery people spend the rest of their lives there (again, this isn't meant seriously, okay?)

• New York was better prepared for this predictable disaster

• or...the measures may have been taken inconsistently and/or the numbers are simply wrong

Let's get more curious and read the full article. We find valuable information by reading the details and then very quickly understand what happened. It is noted that 2900 of the reported cases in New York were all registered at the day of publishing the numbers. Interesting, probably it is too early for those new cases to "die". If we ignore these new cases (7102-2900), we still have a quite good quote of 0.8%. Better than in many countries worldwide. But further reading the article reveals “At least 35 of the deaths were connected to a single nursing center in Washington" and "...many of those cases involved older people with other health problems that made them especially vulnerable to coronavirus”.

Aha! So, it was jsut bad luck for the nursing center to get hit so hard. If you ignore these numbers (75-35)…we are back to an almost "normal" rate of 2%.

These numbers make sense now. No need for me to look for creative hypothetical argumentation.

And the link to IT? A similar incident once happend near the end of our software development sprint. In parallel to the development of user-stories, we also performed refactoring tasks to get rid of technical debt. As is the nature of refactoring, it may break previously working functionality. This is what happened: Instead of being ready for the sprint review, we had to reject the demo of 2 buggy user-stories that were working properly just a day before. It cost us valuable story points. We didn't achieve the goal. What followed were long-winded discussions and assumptions of what to do and what not to do in the future, whereas I thought: "WTF, Just bad luck, shit happens".

[1]https://www.nytimes.com/interactive/2020/us/coronavirus-us-cases.html?action=click&module=Spotlight&pgtype=Homepage

Kind of mixed up

Fred: "Lucy, look I bought a pedal tractor for our kids".

Lucy: "No you didn't, you fool! "

Friday afternoon at Green Galactica

...

He just knew too much

THE DAILY BUG. "A software tester was found dead due to an overdose of drugs yesterday evening at his home in lower Manhattan, New York. His body was discovered before dawn on Sunday, October 5. He was a professional software tester and known for his strategies in spotting even hard to find bugs. An interviewed friend on site believes, he was murdered. "The tester was threatened several times by suspicious phone calls", he said. The police states they have found traces left by ugly critters at the crime scene. The police also stated, the main suspect was identified as Dendro Ponderosa, a modified name which has its roots in the Latin name Dendroctonus ponderosae, a mountain pine beetle and a remote relative of Dendro. The police published an appropriate mug shot of the suspect."
Reported by Formica Rufa

BETA Test at Green Galactica

One day the PL asked me to provide a checklist with the most important acceptance criteria and which served as checkpoints to enter the next phase of the project. The list was later extended with a few more items by the PL and PO. When the date came closer, someone worked on the list and changed most of the quality attributes from MUST to NICE-TO-HAVE without upfront warning. Obviously, someone realized one is not going to meet all the goals and changed the attributes beforehand. As a quality manager, I was concerned that tasks labelled NICE-TO-HAVE are never picked up, therefore I protested heavily.

 We then agreed to keep the list with its original priority, and, on D-DAY we would go through the list again and then decide for each unfinished task how bad it really is and what are the risks related if we are not completing the task.

 But, the cartoon fits even better to several similar situations where the testers warned management long in advance about the immaturity of the system. When deadlines were close, the pattern was often the same: concerns were overruled, management had “higher” goals and caused a mess with the decision to rollout the software despite the obvious consequences.