The Search for the Real In Software Quality and Testing

“Eliminate the unnecessary so that the necessary may speak.”
― Hans Hofmann

As AI integration FOMO hurtles us towards even more pervasive technology, the testing of AI models for correctness and most importantly, their potential for harm becomes paramount to their success. That testing has to be underpinned by principles and values to guide the observations and reporting, so I was inspired by Maaike Brinkhoff bravely taking on the meaning of “quality engineering”, as well as my multiple conversations with Michael Bolton on similar lines to try to put into words some views I’ve not published in the past.

When I was at university, I had a fantastic art professor Lyle Salmi who really challenged me to think differently about composition, perception, and the creative process. He turned me on to Hans Hofmann and some other abstract artists which only furthered my mild obsession with Jackson Pollock and exploring the constructing of things creatively.

Hans Hofmann wrote in the “Search for the Real” about trying to move beyond imitation and finding truth through expression. At that time in my life, 20^th century abstract art was more about representing ideas than directly trying to copy life – art was about the experience.

This isn’t an always an easy way to think about the creative process (or life in general), but I feel like trying to represent what it means to be human entirely by the output of the artistic process is not only impossible, it’s misguided. What makes us human or “real” are the intangible things: joy, sadness, love, fear, grief, happiness – all the things that can’t be measured or counted.

Software testing has the same problem. Our business (and the vendors that dominate it) has more than a mild obsession with counting things to check if something “worked”, which over the course of my career has only resulted in reducing what “passes” for testing to a shallow performance. The current AI mania has only accelerated the descent into mistaking running tests for testing and replacing manual, <shudder> human testers with automation, as if quality were a problem to be solved.

Complex systems like financial markets, relationships, or software used by those unpredictable things called people, are ecosystems. Treating them as complicated problems that can be “solved” through deconstructing them into test cases is as useful as trying to judge a painting entirely by how many brushstrokes were used. You can count every movement in a Pollock and miss the entire point.

**Ocean Greyness** – Jackson Pollock, 1953

And that brings us to the endless existential threat from the values displayed in the software testing business.

The current crop of AI bros would have you believe that the mission of testing is to reproduce human behavior through some sort of algorithm and the only important thing learned through this process is a green check. But that view again misses the point entirely – risk to your business isn’t managed through failed automated checks, it lives in empathy, intuition, observation, all things that can’t be scripted through your algorithmic-defect-predict-o-nator.

If testing is a performance than its interpretation can’t focus on mechanics alone and more than ever, our systems must be evaluated for more than if they just “work”, but the impact of that work on the world.

I used to live in New England, and one Father’s Day my family took me to East Hampton for a tour of Jackson Pollocks studio. Standing on that floor, breathing the air, looking at the jars of brushes ticked a big one off my bucket list, and as it was a quiet weekend and we were the only ones there, the guide let me back in after the tour to check it out by myself.

I laid in the middle of the studio for a long time trying to imagine what went on in there – the sounds of his feet shuffling, the wild and controlled movements, the smell of cigarettes and the wood burning stove – all as important to his art as the final “product”. It meant more to me than all the time I’d spent just looking at his paintings and when I see them now, that experience – that expression of his art has deepened my understanding into his and my own humanity.

I don’t know exactly where we’re heading in the testing business with the integration of AI, but software systems are actually alive – alive with human interactions, expectations, and emergent behavior and testing is how we make sense of it in context of what matters – people. I’ve always believed that testing is both art and science and to reduce that to output checking is to deny what makes testing – and what it means to be human – real.

That’s what’s always been true about testing, and in my experience the people and companies who understand that and apply to their strategy will see it through…