Lindy's Law in Test Automation

by T. J. Zelger, March 12, 2023

When I developed my first "robot" 20 years ago with the goal to automatically test the software so that my team didn't have to do the same manual tests every day, there were at most a handful of products enabling it. There were tools from well-known companies like IBM and Mercury (now HP), and these were extremely expensive. You didn't have much choice. Once you made a decision for a tool, it was almost impossible to revert that decision and go for another. It inevitably resulted in enormous extra costs.

A little later, a few interesting and cheaper alternatives emerged, such as RANOREX, an Austrian company that soon taught the big ones to fear because of their quality and an attractive value for money.

We also experimented with other products that we used for specific tasks and later replaced with other newer/better ones. Among these, I had a Canadian product on my focus which provided us with valuable services when testing a vehicle valuation calculation engine. As far as I know, the product no longer exists today and my memories are patchy, but I believe it was a forerunner of one of the open source systems that are widely used today or it may have been something similar.

20 years later, you will find a flood of even cheaper or free offers. A closer look reveals that most products are based on a few identical core modules. Selenium is currently one of the most popular "engines" on which most tools are based on. I also use Selenium, and because it's actually just an "engine", you have to develop additional methods and modules on top of it to make it a stable and easy to maintain test automation suite.

Nowadays, when building your own framework, most stick to the page-object model. However, we used to have a different approach. Our test data and instructions (action words) were kept in Excel. The idea was to enable testers with no programming skills to write automed tests. At the beginning we even thought, we could convince our business-analyts to maintain their own tests.

The idea to keep data and keywords out of the code was not new. It already existed at the time I was still working with IBM Rational Robot. For example, the SAFS Framework by Carl Nagle [1] was the first framework I learnt about following a similar approach or take TestFrame which is an implementation of Hans Buwalda's so called "Action Words" [2].

Our Excel based framework was quite a success within our headquarters in Switzerland and Germany and our plan to have testers write their own scripts without programming skills went well, but the maintenance of our framework wasn't quite that easy and it needed an expert to maintain all the UI locators and required extensions. This was sometimes a little too tricky for the non-techies. And, we never managed the business analysts to go for it.

Later (in a different company), I used the same approach but realized that it didn't have the same effect if you have testers WITH programming skills. Excel isn't seen cool enough to write automated tests and if you sell such approach to techies, they will raise their eye brows.

So, we removed the Excel part and integrated everything into NUNIT. That was easier to debug also.

And now? A newcomer called Cypress enters the market [3]. As I don't want to get stuck in sweet idleness, we are now starting a new adventure to see what it has to offer. We still keep our Selenium scripts, but they are going into a maintenance mode right now.

But, do we really have to follow each new fashion trend? Who guarantees that new stuff and ideas are really better than what we already have in place?

Fortunately, in my position as QA Manager, I can mostly set the goals myself in this area. If things are going well, you have the dilemma between "don't touch a running system" and ensuring we are not missing something. 

Today, we use Selenium/C# with NUNIT for automated UI tests triggered daily by Jenkins. And, we have an automated test suite that fires requests on an interface level (below the UI) following the Test Automation Pyramid approach [4].

The problem: Everything works smooth since years! Why should I spend time to investigate alternatives?

I am thinking here of the Lindy's Law [5]. If something has proven itself for a long time, there is a high probability that it will continue to prove itself in the future. In my case, this applies in particular to these automated API tests, which are based on a framework we developed ourselves with the aim to keep the test scripts at the highest possible level of abstraction. The technical details such as authentication communication with the backend remain hidden. Also, instead of just having JSON based input/output, we are dealing with deserialized business objects. 

We have at least 3000 automated tests that have identified bugs that were not caught in the deveoper's unit-tests. Simply said, these interface tests are a success story and I don't spend a second thinking about replacing it with a standard product. Why should we scan the market for "better" stuff that maybe isn't?

Because the examination of alternatives does not necessarily lead to the replacement of a tried and tested system. It can become a supplier of interesting and new ideas and extensions for the existing solution. Being open minded also helps recognizing the limits of one's own system and thus to check whether the current version can last not only in the near, but also far future and/or can be supplemented with one or the other useful feature.

The only constraint I am dealing with in this regard is my available time. But that's another story.

References:

[1] SAFS, Carl Nagle

[2] "Action Words" by Hans Buwalda, Software Test Automation (Fewster/Graham), Addison-Wesley

[3] https://www.testautomatisierung.org/testautomatisierung-cypress-vs-selenium/

[4] Test Automation Pyramid, Fowler, https://martinfowler.com/articles/practical-test-pyramid.html

[5] Lindy's Law by Albert Goldman, 1964, https://www.sciencedirect.com/science/article/abs/pii/S0378437117305964 and "Das Magazin", Nr. 10, 10-11. März 2023

Captcha #1

CAPTCHAs are used to prevent automated software from performing actions which degrade the quality of service of a given system and/or to protect the service from attackers trying to hack login credentials using brute-force attacks.

Until now, I never had to test CAPTCHAs but thinking about it, testing CAPTCHAs automatically is impossible if testability isn't considered at all. Testability here could mean for the roboter to offer a backdoor which contains the correct clear-text. Of course such information should only be available to the script and de-activated when deployed live. Sometimes, even I struggle to identify the clear-text of the CAPTCHA, and I am NOT a roboter...

Automated Test Script Desease #1

If it is not a new unexpected confirmation dialog box that shows up, then it may be a unique GUI control identifier that either changed or is even missing. The response could also be slower than originally recorded. That means we have to either extend the script with hard-coded extra wait statements or - if you are smarter - write some extra logic which always waits for the control to load completely before it gets "touched". The object could also be at a different location, let's say in a list that is sorted differently than before. Hope you didn't record the position but negotiated with the developer for a predicable algorithm so your roboter is still able to reach it even though it shows up somewhere else.Poor you if one detects that the developers now use third party controls that your test automation tool can't deal with. You might also experience ugly test script- or module-dependencies. As if this wasn't enough, you may work in a world where testers don't control the test environment where others can change or update configuration without you knowing until you see the script failing. You may also experience a deployment of a new software version in the middle of test execution or while you are playing Sherlock Holmes to investigate a real fat bug. Your script might only work from your workstation but not from the server you're triggering and executing it ...from one day to the other.

And often, even if none of those typical UI test automation challenges is one that you face today, you will still have to sit there watching the script running,  so you're ready for some extra test script babysitting actions. If you weren't there observing your scripts but going out for a coffee instead, don't expect your roboter to have completed successfully its job when you've come back to your desk...

That is just a few of the reasons, why I love testing below the UI so much.

Robots Learning Negative Testing

Also published at the "Automated Software Testing Magazine", Issue March 2011

The Unexpected Ugly Bug

I remember the time where we had a set of only few automated tests and a big set of manual test cases. As the number of features grew also did the number of test cases. The number of available testers was a constant number and we didn't manage to execute all the test cases. At that time I started to rate the test cases based on their likelihood of detecting bugs. Whenever we executed a test case that helped uncovering a bug, we added a link to the defect's ID. Each time the test suite was executed again, the tester was notified about the number of defects and its priorities it found in the past.

That was a practical approach, but also a risky one if I think of what just happened to me last week at my current employer.

I had an automated test script which couldn't be executed due to some problems with the new version of the test automation framework we were using. Since there wasn't enough time to investigate, I just skipped this script, reported "FAILED due to unknown reasons", marked it as low priority and continued with other tests. I did it because the component it was testing never showed up any defects in the past and so I thought, I could easily put it aside for a while until I found enough time to investigate what was wrong.

When I finally had the time and managed to "revive" the script, it uncovered a real ugly bug, and it was almost too late to fix it for the release that we were about to deploy. My luck, the release-deployment had to be shifted by one week due to another issue and it wasn't a big deal for the developer to fix it.

Anyhow I felt uncomfortable for the fact that I found this issue so late in the process whereas it could have been uncovered much earlier. What does it tell me? Even a test case / script which may look useless for many test cycles, may become an important script at a later point in time. Today, while writing or looking at new/existing scripts to determine its priority, I ask myself..., what could go wrong if that script isn't executed rather than thinking of the success rate in terms of "how many defects did it uncover in the past".

This cartoon along with the text is now also published by the STQAMagazine's blog
(and later on, also in the STPMagazine print version of January 2011 (first page)).

Revolt of the Test Scripts

If testers are not invited early enough, they will not have the chance and the time to determine whether or not they have to adapt their existing scripts. As a matter of fact, some scripts will have to be de-activated since they start to report errors that aren't any. Someone smart once called this "False negatives", consequently the opposite exists, too (false positives). see also the blog entry as of July 2009 (The Test Successfully Failed).

...by the way, Test Tools and/or your own test scripts can be buggy, too (here a false negative).

...in the meantime, I found out under which conditions this situation occurs. In the NUnit Test Automation Framework, even if all your test scripts were running fine the whole suite may report a failure if someting goes wrong outside the test suite. The TearDown function is such a method that gets called automatically after each script and/or after the suite is completed. At that point, all scripts reported already a successful run, but the TearDown can still throw an exception if for instance your implementation invokes a call of a null object (null pointeer exception). In such case, NUnit reports a failed suite although all child scripts were passing all tests.

There is nothing like starting young

Test automation plays a key-role in buidling great software, especially in today's rapid delivery cycles. What works today, may be broken tomorrow. But while most managers agree it is important, many struggle investing accordingly or do not understand that they have to support the testers while asking for testable code. Making code testable is often seen as something less important than delivering features quickly.

Testability is a foreign word although it is the key to successful test automation. I have heard phrases like "Oh, yes, that is a nice idea, but we will do this later". Later here means either never or at a time when it is already too late and becomes too costly to implement it.

Some also believe test automation saves tester resources. That's not really true, because test automation enables you doing tests you would never have hired any test resources for anyway. Test Automation enables you doing things that 100 testers couldn't do, that's right, but you would never hire these testers anyway, so you cannot talk about "saving tester resources".

Test Automation actually saves resources, but not really at QA. Test Automation enables you to point out failing code right after it is committed/checked-in. So, this happens at a time a developer still remembers what he just did. He can correct the error with no big deal. Without automation you would notice the same errors days or weeks later. At that time, the developer already moved on with different stuff and hardly anyone may know who broke the code or what piece of code is responsible for the detected anomaly. An expensive analysis and debugging starts where not only one but liklely more people are involved investigating the issue. These are the costs you can avoid with automation, but never say it saves test resources.

The Test Successfully Failed

"The test successfully failed!" This is what my college sitting next to me was yelling lately. I smiled when I heard him, because a malfunctioning operation is typically everything else but a reason to see happy faces here. Of course, such is different for testers, at least it is for me.

My job is done when I broke the software I am testing. If I all my tests become a green status, I am getting distrustful. Can it really be, that all those tests passed? I mean, couldn't it be that I missed some important tests or didn't I test thorough enough? The possibility that the software could simply work as designed rarely crosses my mind. I even don’t trust my own test scripting. I am sure, my scripts have the same kinds of bugs that developers continuously add into our software.

At this point I remember an article where I read something about "False positives, and false negatives". Applied to my daily work as a test automation scripter, how can I find out quickly, whether my new or touched scripts will find errors if they occur and when the feature currently work fine?

We have a lot of scripts which test the API to our service based application and I found a really stupid but effective way to smoketest my own test scripts. I plugged out the network cable and started the test suite. In such scenario, I expect my complete test suite to "successfully fail" for all tests, but it was big surprise seeing one of the test cases to still get the green status icon.

I wasn't online and all my tests were supposed to submit and query information from a service that needs a network connection to get any meaningful results. How can it then be that this test passed without a network connection?

Looking closer at the validation/assertion the script was executing (hereby debugging my own code and trying to understand why this test returned a "passed" status) revealed that I was doing an assertion to an object that becomes the same state, no matter whether you are online or not.

For example, I was querying a list of items in a database which can be of either one or the other state. For simplicity let's take the states ACTIVE or INACTIVE.

The interface I was testing offered 3 parameters to pass to the API. Passing "0" gives me the INACTIVE items. Passing "1" gives me the ACTIVE items. Passing an empty string returns all items.

As our test environment is used by a many other testers, too, hence the exact number of items could not be determined at runtime unless you clean the database to start from an initial state.

That could not be done, because it affected too many people. Therefore I focused on testing the service by passing all 3 parameters in a sequence and then compare the number of items among each other. I verified that the total number of ACTIVE items plus the total number of INACTIVE items are the same as the total number of all items using the following code:

Assert.IsTrue(nNumTotal==(nNumActive+nNumInactive));

That checkpoint becomes (unexpectedly true) if I am offline because in such case, all the variables are filled with zeroes. Having said that, this test actually validated that

Assert.IsTrue(0 == (0 + 0));

The fix in my script then was to add an additional Assert-Statement that tests for the number of items being greater than 0.

Test Automation Side Effect

Lesson learnt Run a test on environment X, collect reference data, then re-run the same test scripts and collect new data from a different environment Y. If you expect those results to be identical.Good luck.They are most likely not. If you run a test and collect reference data, always run these tests on the exact same physical machine and don't compare data with different machines, unless there is a strong reason for this. There are always things that are different on these machines.

Here is the plan 
The Engineering team expected me to perform just 5 exemplary spot tests to verify whether the response is fine for some web service requests after they upgraded a software component on ACCEPTANCE. I was told to get informed by email when ACCEPTANCE environment is ready to do an initial test.Their plan was to mirror the data from PRODUCTION to ACCEPTANCE first.

Engineering would email me, when this is operation is completed and I would start collecting the reference data by running my scripts. Later, when this is done, Engineering would upgrade the system and inform me, so I could re-run the scripts, collect the new data and compare those with the previously collected reference data. At least, this is what I thought was the plan.

Developing the scripts
I prepared my test-scripts and extended those with additional checkpoints that I found useful and necessary. I prepared over 300 test cases which were firing web service requests. That was far more than the team expected me to do. I tested my scripts on a development environment and collected and stored the results. Everything seemed to work fine. I was ready for the big show on ACCEPTANCE.
I didn’t know much about the exact time line. All I knew was, it started around 1030h on a Sunday morning. I didn't know who was involved and I didn't even know what exactly they changed in this new upgraded software. It was no functional change, just something minor on a database level.

Executing the test
On this sunny Sunday morning at 1030h (my kids running around in our garden and waiting for me) I was sitting in front of my laptop, waiting for the starting shot, that is, an email telling me that the productive data had been successfully mirrored to the ACCEPTANCE, so I can run my scripts. Actually, I didn't get any emails. Finally, after I asked what’s going on, I got a note from SYSADMIN, stating that not only the data but also the new software system was already deployed to ACCEPTANCE and that QA could now start their tests....

Wait a minute… wasn't the idea to first collect the initial reference data so we have something meaningful to compare with? Something went terribly wrong here.

Upps, we have an issue
However, I had no other choice than to simply run my scripts against an already upgraded system with no possibility to compare the results against an untouched system. What I had so far was the data collected from the development test environment from Friday when I was still developing and testing my scripts. When I compared the data I was worried because 20 tests failed and some web service requests fired by my scripts reported a remarkable performance difference between the upgraded ACCEPTANCE environment and the old untouched development environment. The upgraded system for some tests responded 7 times slower compared to DEV.

Investigation of failing tests
After some investigation I found the reason for 19 out of 20 failed tests. It was a false negative result. The reference data collected for these tests on DEV returned an empty response. I took that as a reference, but running the same scripts on a productive near environment returned something different. This explained the failing tests except for one for which the root cause was not identified yet; That was one out of 300, and of course, we still had this performance problem with a few tests.

I repeated all my tests several times to gain more assurance that I am on the right track. But for each test run executed, I got the same results. During this analysis I suddenly got another note from SYSADMIN, now telling me that also the final production system was upgraded….

WTF….! They upgraded production while I was still investigating what was wrong and before I could even provide them a report?

Consequence
After I vetoed and presented my results (finally I learnt who is my contact person), some managers got nervous, and I mean really nervous. They wanted an exact explanation of the failing test and the performance issue, but we were unable to provide such information in the required short timeframe.
As a result, the managers decided to stop this exercise and instructed Engineering to roll-back production immediately which actually helped avoid a disaster when the customer used our system the very next morning. To my opinion, this decision was wise, given the circumstances that the communication and coordination of this operation was a disaster.

5 test cases - are you serious?
At the end, what confused me, is this: I was instructed to do only 5 tests. Are you serious?
I developed and executed 300 and it ended up with one failing test. What would the result and decision have been if I only executed 5 tests? And what would have been the reaction of our customers?

The other vexing scenario is that after PROD and ACC system was rolled-back, none asked me to test the rolled-back system….

And what did I learn here?
OK, ..after spending so much energy in summarizing this exercise, I should of course spend a moment and look at myself. How could I have prevented such situation to occur? Probably it helped if I started to collect the initial reference data way before the official test started, and – despite the fact that I wasn't involved in any planning/organization meetings/emails/telco, I could still have communicated my approach of testing to the people of whom I assumed, may be involved. If I had taken the chance to present my approach of testing, it may have helped driving the test into a different direction.

And then it happened again
Many years later btw....I had a similar situation in a different company where a developer asked me to test peformance of single REST requests fired against a system before and after an upgrade. The BEFORE should be tested on environment X while the AFTER should be tested on environment Y.  The developer stated the machines are phyiscally identical, but I didn't like that argument. Actually they were both identical. X and Y where 2 different web pages running on the same physical machine, accessing the same physical database, but...running 2 different settings and code-basis.
X (the old system) was so so terribly slow that each request took 40 seconds and Y (the improved one) responded within 2 seconds. The developer cried "see how the performance increased". Actually X was so badly configured that it was impossible to prove the new version Y is faster because of his changes. When I demonstrated to him that there is yet another environment Z where the results are far better than those of X although Z had no optimization, he realized that X was a corrupt and bad bad environment to compare.