A customer reported a problem exporting documents. The system responded with a null pointer exception. We could not reproduce the problem on our internal test environments while playing around with known different document types used by our customers (they used all sorts of TIF for archiving) such as coloured, grey-scale, different DPIs, etc. until we asked for the original document from the customer even though we follow a strict rule not to work with production data containing sensitive information. We accepted it for once. Initially, we still could not reproduce the problem until we experimented with different kinds of annotations. We finally found out that the problem only showed up if a burnt annotation is set on a page other than the first. That was a wonderful finding after some upfront detective work.
We asked ourselves whether we can expect the testing department to really find such bugs before the customer does. We all came to the same conclusion: In theory Yes, but practically it is impossible to cover all these kinds of variations for every use-case developed in a sprint and given the strict time-constraints.
Fact is, no matter how hard and how accurate the testing is done, software will never be fully free of bugs. The complexity of software has increased dramatically during the last few decades and the number of needed test cases to gain a minimum level of coverage has grown so much, that it has become impossible to execute all tests and possible variations within the time given. Test Automation helps a lot but also cannot cover all required tests within the given time-frame.
Software development today still is a process of trial-and-error. I have never seen a programmer developing a software component from scratch without several times of failed compiling, fixing, re-writing, re-testing and finally shipping it, then failing at the customer and going through the same process again. With this statement
"Software errors are blunders caused by our inability to fully understand the intricacies of these complex products" [2]
Ivars Peterson compares today's technology with black magic where engineers act like wizards who brew their magic potion by mixing various ingredients of terminology that only them understand while the public dazzled by the many visible achievements of modern technology often regards engineers as magicians who can solve any problem
References
[1] The Art of Software Testing, Glenford J. Myers
[2] Fatal defect, Ivars Peterson
