Had a great time talking to David Burns on the BrowserStack Talks Podcast (Spotify Apple) about all things testing in the age of AI – enjoy!

The Future of Software Testing

Keith Klain on AI, Risk, and Critical Thinking Will AI replace software testers, or just change the game? In this episode, industry veteran Keith Klain joins us to dive deep into the intersection of Artificial Intelligence and Software Quality Assurance.

Key Topics Covered:

• The AI Reality Check: Why skepticism is a tester’s best friend.
• Risk Management: How to handle AI drift and algorithmic bias.
• Human-Centric Design: Maintaining control over testing processes.
• Beyond Automation: Why AI tools need human agency to be effective.

There’s a conversation about software testing we need to have despite the noise and hype around AI productivity gains, coverage expansion, and “AI-assisted quality”:

What kind of reality are we shaping when GenAI writes our tests?

As I’ve said before, testing isn’t neutral and when under the guise of progress, we outsource testing to systems that don’t understand intent, context, or consequence we introduce new risks that look like progress until they aren’t.

What we choose to test (and not to test) helps determine what we believe to be true, and our choices become part of the mental model the business uses to make decisions. As project evolve, those beliefs harden into assumptions with embedded risks.

Our approach to testing shapes reality.

But because most tests reflect only assumptions rather than risks, the reality testing shapes about system quality is often inaccurate and generates unwarranted confidence. And confidence should be fragile; it should be grounded and bound by objective reality.

Good testing, then, is not about confidence. It’s about a continual re-alignment with realityto inform your business decisions. Additionally, volumes of shallow checks reduce a team’s ability to notice signals for systemic failures which should further erode confidence but typically has the reverse effect.

Now add to that equation the mother of all automated-test-generator-algorithmic-defect-predict-O-nators and I fear where we’re heading…

In the era of continuous everything and automate first we generate more tests than ever, but I still find very few test teams that understand what reality their tests represent or whether they’re still meaningful. When tests accumulate faster than teams can understand them fragility increases within our tests, systems and teams.

A useful metaphor I use when talking to management about testing is comparing it to a lens. A lens can focus your attention to one thing, blur other details, but ultimately, it’s a way of looking at the system you’re building and what matters to your business.

When test generation becomes cheap and abundant, the shaping force of those tests – the lens through which we view quality – increases dramatically.

What I’m seeing from most of the marketing and demos from GenAI testing tools is a lens that focuses volumes of tests on what’s easy to see and blurring the importance of human experience: user workflows, organizational incentives, operational stress, and socio-technical interactions.

If you’re organization is considering (as most are) using GenAI to generate tests, here are some considerations I believe should be at the top of your evaluation process:

• Do they create tests explicitly and visibly tied to user harm, business impact, operational failure, or regulatory exposure?
• Do they build entropy into their tests with assumptions that expire as confidence decays over time unless renewed by new evidence?
• What agency do testers have over test selection and execution in relation to coverage and analysis?

We’re just at the beginning of new chapter in the long story of software quality and testing, and I believe that how we frame the problem, how we view reality will be a key differentiator in the success or failure of our business.

Testing has aways shaped reality and GenAI has the potential have a powerful warping effect on that view and the propensity to move risk to areas we no longer look at due to volume, noise, and over-confidence from our leaders and vendors.

Daniel Kahneman famously stated “Overconfident professionals sincerely believe they have expertise, act as experts, and look like experts. You will have to struggle to remind yourself that they may be in the grip of an illusion”.

Clarity is required to dispel that illusion.

Clarity on intent, function, and the reality we’re shaping.

“The masters are liable to  get  replaced  because  as  soon  as  any  technique becomes  at  all  stereotyped  it  becomes  possible  to  devise  a  system  of  instruction  tables which will enable the electronic computer to do it for itself. They may be unwilling to let their jobs be stolen from them in this way. In that case they would surround the whole of  their work  with  mystery  and  make  excuses,  couched  in  well-chosen  gibberish,  whenever  any  dangerous  suggestions  were  made.” – Alan Turing

“Free your mind, and the rest will follow” – En Vouge

In my Great Liberation series (Part I, Part II), I argued that the way AI is being rapidly injected and measured in software testing, a discipline that profoundly shapes modern life, has moved to borderline recklessness. Banking, healthcare, transport, education, and government infrastructure all run on software we need to be reliable and AI adoption is racing ahead at a pace and human judgment can’t keep up.

The reason this matters is because AI is being treated as a substitute for critical thinking and responsibility rather than a tool that demands more of it. Poorly tested systems cause real harm, and when organizations replace human evaluation with automation in the name of speed or cost savings, they increase risks to their business and society.

Despite what you may have heard or read, AI in testing is not neutral or low risk. The dream being sold of closed loop systems validating themselves, with minimal external oversight will make failures more systemic, less visible, and harder to correct. More than ever, we need to be pushing back against the hype cycle selling AI adoption as inevitable, discouraging skepticism and trading on FOMO.

But unfortunately, the software testing industry has failed to meet the challenge.

I believe part of the problem is test tool vendors are increasingly led by private equity and investment firm CEOs with no testing experience or pedigree while pushing AI-driven testing but insulated from the consequences. Look at the Gartner Magic Quadrant for AI-Augmented Software Testing Tools and you will struggle to find tester led companies that position testing as a risk management activity and not sold as a magic bullet for cost reduction.

Moral hazard has all but left our business because the people who run it have never lived with the consequences of being responsible for ACTUALLY testing something. And if you think that’s harsh, look at the claims they make about their products:

“AI-native systems eliminate the need for human-created models or manual test maintenance”

“AI-powered automation dramatically increases speed and coverage/automates everything”

Most vendors don’t even get that specific and just spew AI buzz words like they’re going out of style: “AI solves every testing challenge,” “no human needed,” “fully autonomous defect prediction,” or “100× productivity.” My point is that this isn’t predetermined. It’s a choice, and frequently these days a choice being made for us, but how we test and deliver systems reflects our values – and we should value safety, transparency, and fairness.

But fairness cannot be achieved with unverifiable claims lacking credibility or grounding in reproducible evidence, something even Gartner warns is common with GenAI vendor marketing. In The Pursuit of Fairness in Artificial Intelligence Models the authors argue that fairness, accountability, and transparency in AI is not automatic, but needing deliberate design choices, ongoing evaluation, and requiring human oversight.

To get there we need clear-eyed views on the technology challenges and risks and cannot afford AI testing “thought leaders” making nonsensical claims about AI testing software better than humans, redefining accepted terminology to suit their marketing needs, or presenting hypothetical concepts with no evidence. Especially, as I think it’s going to quickly get a lot harder for testing.

A recently released paper dubiously co-authored by an AI (Claude, Anthropic), “Toward Robopsychology: Evidence of Emergent Behavior in Human-AI Partnership”, presents some interesting challenges for quality engineering. In short, the paper talks about when AI systems interact with us, we both generate behaviors that aren’t completely predictable from either the AI’s original design or our actions.

Field of psychology or not, this study is puts some shape around new challenges for software testing like verification of systems where human behavior and AI behavior adapt to each other, metric validity for emergent behavior, and what threats to “human in the loop” governance does the human bring to the party for automation and regression testing.

So where do we go from here?

Probably not down except in headcount, even though companies as big as Salesforce, who did a massive layoff under the guise of AI productivity are scaling back their LLM use after facing repeated reliability issues. Per MSN, Salesforce “confidence in generative AI has fallen sharply over the past year”, but I don’t think there are any signs of slowing down in the general integration of AI into everything, just more of a pause.

And it doesn’t look like government oversight is coming any time soon, with core parts of the much lauded EU AI Act likely kicked down the road to 2027. In another capitulation, the European Commission seems to be following in the US footsteps in throwing the environment and high risk AI implementations out of the window in, as Reuters puts it, “an attempt to cut red tape, head off criticism from Big Tech and boost Europe’s competitiveness.” In 2022 I was complaining about the lack oversight on an industry so far out in front of regulation for AI, but apparently it wouldn’t have made much of a difference anyway.

When I started this series it was a play on AI being the “great liberation” for companies to finally rid themselves of the responsibility of testing and all those painful testers. I assumed (correctly) we’d find useful partners in the software testing industry and a regulatory environment fully on board with removing constraints and guardrails in the name of innovation.

So in times like these, your strategy to implementation is more important than your technology. We see more than ever now that AI is not just a tool; it is a choice. It’s a statement on what we care about. How it is adopted, governed, and tested tells us a lot about what organizations – and the people who run them value.

As I’ve said before, in this brave new world your test strategy should focus on:

• Algorithm transparency and explainability
• Automated decision-making controls and human in the loop interventions
• Governance for training data and model drift with measures for continual monitoring
• Audit trails for traceability on workflows, decisions, and agency

As 2026 shapes up to be another wild AI ride, a principled, transparent approach to using these tools in testing led by values, free from buzzwords, and grounded in solid testing practices and experience is the science that will see us through this, or as the good Dr Feynman put it, “the belief in the ignorance of experts.” Good luck and enjoy!

What can be asserted without evidence can also be dismissed without evidence. – Hitchens Razor

One of the biggest mistakes organizations make when building and testing systems is to measure it badly and then make decisions based on faulty and invalid metrics. Software testing has a long, rich history of goal displacement and metric validity problems when it comes to measurement and GenAI evaluation is currently running that same crucible.

Despite there being an abundance of metrics to look at model characteristics and performance: BLEU, ROUGE, hitchLLM-as-a-judge scores, or human preference ratings, what’s missing is any confidence that these metrics actually measure what matters.

I love the book “Measuring and Managing Performance in Organizations” by Robert D. Austin and it’s my go to reference when talking about validity problems in software testing metrics. Austin wrote about this decades ago: when performance measures are shallow proxies for your objectives, people optimize for the measure, not the outcome.

The same principle applies to measuring AI models. If you give a system a metric, it will learn how to win that metric whether it creates value or not and human evaluation is a requirement to know the difference.

This point is echoed in a paper published OXRML: Measuring What Matters, which reminds us that metrics are not reality, they are models of reality and like all models, they’re wrong but some are useful. They also point out that precision is not the same as relevance, as a precisely measured wrong thing is still the wrong thing, or what I like to call: hitting the bullseye on the wrong target.

Most of the GenAI measurement models I’ve reviewed or had demo’d for me also seem to be making the similar mistakes we make in software testing by confusing measurement with understanding. I am going to be writing and speaking a lot more about this next year, but here are my thoughts so far on what I seen and used for GenAI model evaluation.

• Popular evaluation approaches like BLEU or LLM as a judge feel like counting lines of code or test cases. Big numbers seem good, but they tell you very little about risk and just as more tests don’t guarantee fewer defects, higher semantic similarity doesn’t guarantee a better answer.
• Metrics requiring a golden “correct” output remind me of my experience with decades of test automation frameworks optimizing for green lights and consistency instead of correctness potentially missing all the nuance in observation.
• Another similar mistake I see is measuring for form, which has the same effect of counting how many bugs were “closed” rather than how many were fixed. Hallucinations can score better than a correct answer which aligns perfectly to Daniel Kahnmens point about the power of storytelling in Thinking Fast and Slow.
• Most evaluation models like to use some sort of aggregation for scoring, but my concern about this is that averages can hide risks, a problem outlined perfectly in How Complex Systems Fail. Safety issues are masked by thousands of tests and the system looks stable right up until it isn’t.

Despite the testing industry taking a hard turn in the opposite direction to human evaluation, in my opinion it’s going to be a competitive advantage but doesn’t necessarily solve these problems. Human testing suffers from the same ambiguity, bias, and inconsistency and requires oversight and careful consideration when deploying systems.

While the GenAI measurement market matures, my advice right now would be to use multiple metrics each tied to a specific question and validate your tests and metrics against real outcomes, not just benchmarks. A healthy dose of humility in our business would do wonders as well, as to not repeat the mistakes of the past in failing to delivery ROI to our business due to invalid software testing metrics.

I had a great time discussing leadership in enterprise tech with Michael Hunter for his Uncommon Leadership interview series.

I’ve managed teams all over the world of different shapes and sizes and rarely get to talk about it, so I appreciated the opportunity to discuss continually learning and improving my approach to leadership.

We spoke about bringing your whole self to work, career management vs stewardship, building (and changing into) cultures that empower people, and how to help people navigate uncertain times….enjoy!

References in the discussion: EuroSTAR NSTC Pay360 KPMG UK The Secrets of Consulting: A Guide to Giving and Getting Advice Successfully Rethinking Expertise The 48 Laws Of Power 

Very excited to announce I’ll be speaking at Agile on the Beach in July next year. AOTB has been a conference I’ve admired for a long time, and I am honored to finally attend and as well, talk about a subject that is close to my heart and have never spoken about in public before: ethics in technology.

Hope to see you there!

When the Whistle Blows: An Ethics Experience Report

Juvenal said “Honesty is praised and starves”, and nothing could be more true when it comes to whistle blowing.

Reporting unethical behavior is an incredibly stressful experience that is seldom ever rewarded and often comes with adverse personal consequences. Over the course of my 20+ year career in enterprise tech, I’ve been involved either directly or as a manager with multiple whistle blowing incidents as the result of organizational ethics complaints.

Through this talk I will present experience reports on of some of the most serious cases I’ve been party to including the process, lessons learned, and what I would do differently. So join me as I share the most challenging events that nearly broke me, but were always career defining moments of my life.

Putting together a good conference program is hard. Ensuring the topics are relevant and attracting talented speakers that people want to hear is only further complicated by the commercial aspects of covering costs and turning a profit for the organizers.

But a couple weeks ago, I happened to be walking out of the same conference talk as Richard Bradshaw, and we ended up having a chat about how we seem to be slipping back into “male only” line ups for not just keynotes but also track talks.

This idea that the “best” talks are what’s represented at conferences (as a lot of us just experienced) is just BS. As a veteran attendee, speaker, and selection committee member at many, many tech conferences, I can assure you that in my experience and opinion, bias and preference plays a great deal in who you see on that stage.

Earlier this year I gave up a speaking slot on a male only AI panel, which is particularly annoying as all the important research and hard work in AI ethics is being done by women. I get that striking the right balance between topics and underrepresented communities can present difficult choices for conference committees, but we can do a whole lot better than that.

I’ve written before about how little of my life experiences has been down to things I can control like effort and hard work, and the rest has just been dumb luck and privilege. So because of that, I am completely comfortable with limiting/rejecting speakers on gender/race and think the good it accomplishes far outweigh the occasional bruised ego.

Personally, I am unabashedly inclined to give preferential treatment to new speakers or underrepresented communities and feel that, although they are commercial ventures, it is the responsibility of conferences to have a social conscience and that we should be giving our money to organizations that recognize those two principles are not at odds.

So, although it might seem like progress has been made, even some of the best diversity programs have failed or just been cancelled outright in this new climate and this needs our endless vigilance.

Unless you see this issue as just politically correct cosmetics and not one of social responsibility or sound business sense, increased diversity in who we see at tech conferences is more than justified.

“If the rise of an all-powerful artificial intelligence is inevitable, well it stands to reason that when they take power, our digital overlords will punish those of us who did not help them get there. Ergo, I would like to be a helpful idiot. Like yourself.”
Bertram Gilfoyle

A while ago, I wrote a series of posts on how testing can support your business through a “transformation”, either through disruption, optimization, or by managing expectations around risk and what testing can do to help navigate digital transformation. As GenAI has stampeded into every aspect of technology delivery and innovation, I thought it only appropriate to add another entry into my “Test as Transformation” series.

I’ve always contended delivering technology value to your business will either be constrained or accelerated by your approach to testing, and adding AI into that risk is like injecting it with steroids. By layering black boxes into everything from customer service to decision support, testing has rapidly become one of the most important sources of business intelligence a company has and a true market differentiator.

Testing has always been about information – learning things about systems we didn’t know and confirming things we thought we did, but there are new risks AI presents for your business I’d like to highlight that testing should address.

Regulatory and Oversight Risk

Testing (when done right) should bring insights into risk, anticipate issues, and surface patterns which is even more valuable in an AI supercharged competitive landscape. As the AI regulatory environment continues to evolve due to technology changes and what seems like capitulation to big AI firms, we need to adapt to this “emergent” regulation and yet to be revealed enforcement.

Testing has always been at the heart of regulatory oversight, and I’ve spent a career in banking writing responses to regulators and answering internal and external audit questions. We need to be prepared for new AI oversight and evidence for:

• Algorithm transparency
• Automated decision-making controls
• Governance for training data and model drift
• Audit trails for traceability on workflows

Having the wrong strategy for testing AI enabled systems can introduce risk to your business through biased, unsafe, or inaccurate outputs that may violate new or existing laws as well as industry standards like the EU AI act, GDPR, or new product liability laws. I believe we are shortly going to be litigating harmful outcomes, privacy breaches, and misleading claims of which all could have been triggered by audits of weak governance and inadequate risk management.

Test Optimization Risk

The claims of test optimization and efficiency gains through automation are nothing new to the testing industry. Shallow automation-based checks have been the ROI currency of “modern” testing but just lead to overconfidence and very frequently larger systemic failures. Transformative testing must guard against the usual set of automated business risks and now address AI enablement like:

• AI assisted test generation missing coverage faster and broader
• AI powered code analysis injecting systemic patterns of fragility
• ML based anomaly detection clouding operational visibility
• AI driven customer behavior insights creating misleading test conditions

Managing the delta between AI-powered checks and human-led tests addresses the dangerous belief that AI automation equals AI quality. And just like traditional digital transformation, culture and incentives drive behavior that increase the threats to value in your business. Test automation failed to deliver on the core of its promised ROI, and AI enabled testing frameworks only seem to double down on familiar tropes:

• AI will remove all experiential testing
• AI will find and remediate all defects
• AI can replace testing
• AI systems “learn on their own” without oversight

Testing now needs to validate not only functionality but also non-deterministic behavior of AI systems to ensure fairness, transparency, bias mitigation, and regulatory compliance and cannot rely on traditional processes.

If testing is to survive an AI transformation, it means helping redefine how the organization operates, delivers value, and competes. AI can expand that definition by introducing new automation capabilities, intelligent decision-making, and previously impossible efficiencies. AI magnifies both the opportunity and risks associated with the mission of testing but must be paired with human judgment, domain expertise, and systems thinking.

We’re at the start of a global AI transformation and testing has to move from a supporting activity into a strategic information engine to keep your business competitive.

The tragedy of Adam Raines death by suicide at age 16 this year should have shocked the world into a renewed focus on accountability in Silicon Valley but instead, we heard the familiar excuse from the no-man’s land of moral hazard – user error.

In the first court filings from OpenAI they defended themselves by victim blaming. “Misuse” and “unforeseeable use” are cited even though according to the parents claims, ChatGPT basically gave him instructions for self-harm including offering to help write the suicide note.

Using legalese and technical “terms and conditions” arguments to defend jeopardising a vulnerable teen only further lays bare the morally hollow position the company has taken. GenAI is an incredibly powerful tool that must come with responsibility instead of marketing gimmicks – we now see what’s at stake.

And not knowing is no longer an excuse.

The research paper, The Illusions of Thinking published by Apple earlier this year is a pretty chilling read in this context. We know that as the complexity of problems hit a threshold, the models basically collapse and their accuracy and the reduction in “thinking” drops dramatically.

That’s not just a problem for automating a business workflow with AI, it speaks directly to the complex problems being fed into these models by everyday people. People who struggle with depression, loneliness, and other mental health issues requiring empathy, nuance, and a depth of humanity not possible with GenAI.

Further to that, in 2023 the G7 countries agreed to implement a code of conduct for developing “Advanced AI Systems” including “risk-based design, post-deployment safety, transparency about limitations” and accountability when things go wrong because these systems require more than a TOR like an app that plays music on your phone.

The point is, we know to do better and are on the familiar path to let big tech off the hook on the grounds of “user error”, which ultimately treats a teenager’s tragic death as just a design flaw.

“Few IT projects are displays of rational decision-making from which AI can or should learn.”

This report from by Robert Charette for IEEE Spectrum on “How IT Mangagers Fail Software Projects” is just fire and should be required reading for anyone looking to reuse or “train” their existing management into a crack team to deploy AI into their systems.

If you’ve been working in the software development business for any length of time, most of what’s in this report won’t come as a surprise: poor risk and investment management, bad process, shoddy leadership, and a near endless ability to refuse to learn from past mistakes. And we keep handing the keys back to the same people that drove the last car into the ditch!

It’s a well researched piece and I would recommend you spend the time reading it, as if we know anything about most software projects, it’s that they seldom learn from their mistakes and AI will only increase the risk and opacity!

He lays it out plain and simply here:

“Frustratingly, the IT community stubbornly fails to learn from prior failures. IT project managers routinely claim that their project is somehow different or unique and, thus, lessons from previous failures are irrelevant. That is the excuse of the arrogant, though usually not the ignorant.”

Ouch!

It’s been a few years since I made the familiar journey to the Dorint hotel in Potsdam, the annual home of the Agile Testing Days conference, and I was very happy to be back in 2025. Personally, I was excited to be there just to hang out with some friends who due to global distribution and the pandemic, we’ve not had more than an online relationship in recent years.

But this was more than a holiday and I attended quite a few sessions, listened to a LOT of talks, and had some pretty serious conversations with folks about the future of our business as yet again, the software testing industry I love and more importantly, the people who work in it are threatened by what feels like kids playing with toys in AI slop.

But first the highlights for me… (all the videos should be posted by ATD on YouTube)

Elizabeth Zagroba and James Lyndsay gave an incredible keynote, Testing Transparently where they actually testing something live – which is a rare and courageous thing to do including the audience participation. It was so refreshing to watch two experts, working their way through a system and demonstrate the benefits, constraints, and power of pair testing – something more testers (and conferences) should do! And I even got a chance to deliver some long overdue payback by roasting my pal Elizabeth on stage at the open mic for all her years of roasting my talks on Twitter!

It didn’t really do what it said on the tin, but Martin Hynie gave a fantastic and authentic experience report about developing an AI evaluation and testing model at Credit Karma. It was raw, full of details, and personal – everything you want in an experience report. What I took away from it was that pausing for thought, continuing to ask “why”, and having a set of principles and ethics guiding your work are going to not just be important but essential to success in whatever capacity you implement AI. Excellent talk.

Angie Jones was well, Angie Jones – charming, technically excellent, and damn entertaining as she delivered an updated version of her Air Fryers talk to include the work she’s doing at Block with their MCP Goose. The presentation was great, but it came to life in the Q&A afterwards where some tough questions got asked regarding implementing MCP and agents in testing. Angie did a great job not sugar coating anything as all the answers aren’t there yet and in a lot of cases, we’re still figuring out the right questions.

The highlight of the week though, was Melissa Eaden and her talk The Cautionary Tale of Generative AI, which frankly should have been a keynote. It was a straight forward, no nonsense breaking down of the real world consequences, court cases, and shocking morality gaps in the world for IP, copyrights, and ethical standards in using AI. I want to give Melissa credit for her bravery and guts for giving this talk at a time (and conference) full of AI hubris and disturbing lack of risk awareness and duty of care.

Which brings me back to my initial point, as we seemed to have ceded the leadership of the use of AI in testing to lunatics lacking either the experience or principles to guide the work.

Alan Turing said it perfectly when he predicted how leadership would react with “well chosen gibberish” as the curtain was being drawn back on what they’re doing. I’d like to think my advanced years had mellowed me a bit, but I don’t think we’re actually fighting back against the endless word salad firehose we’re all being sprayed with enough! Out of one side of their neck they talk about the human side of testing (like that’s a new thing) and on the other, tell you “AI” won’t replace you – a tester “with” AI will replace you.

But let’s get one thing perfectly clear – these people ARE trying to replace you and are telling you EXACTLY that! By their own admission they have been trying for decades, as the AI in testing industry has become just another front in the war of enshittification, dehumanization, and removal of all accountability from technology for the impact of the systems they unleash on the world.

Angie made a point in her talk that the testing communities she speaks to have been the most resistant to adopting AI in their work and you know what, that made me incredibly proud of us. Testing is about information, managing risk, asking hard questions and in the face of what I’ve seen the last couple years and reinforced this week, we NEED that community – unwavering, clear eyed, badass testers more than ever!

In closing the conference, Pradeep Soundarajan delivered a scathing indictment of the current thinking and leadership of the testing community. Reminding me of his early days at Moolya and taking on the testing industry in India. He was right now as he was then, that to make a better world, all we need from our leaders – and mostly ourselves is courage.

I give the organizers a lot of credit for putting together an all star group of testers who haven’t been in the same room for a long time and as usual, the vibe and fun were omnipresent at ATD. I heard a lot, talked a lot, met a lot of people I’ve never seen in person, and as is always my favorite – got to meet a lot of very enthusiastic testers which gives me hope for the future.

And ending on a positive note, I have to thank Santosh Tuppad for the all the laughter over the week. It’s been a long year personally, and hanging out with you put things in perspective and was good for my soul…love you brother!

Ooof! Alan Turings 1947 lecture on the “Automatic Computing Engine” still hits hard…

“couched in well chosen gibberish” is exactly how I’d describe what I’ve seen for AI in testing leadership right now…

McKinsey just published their report on The state of AI in 2025: Agents, innovation, and transformation and aside from the usual overly optimistic responses around ROI and resource reduction, it had some interesting (and troubling) data on AI risk and mitigation.

The online survey of nearly 2000 respondents was conducted in June and July this year from what appears to be a fairly large sample size of companies, regions, and industries from all over the world.

As expected, most of the enterprise is still experimenting with AI and agents and haven’t really begun to scale their efforts. Going by most of the large organizations I’ve worked with, I suspect this is primarily due to the inefficiencies of any global operation and large scale duplication of efforts.

Another unsurprising part of the report is that most of the money spent on AI is being looked at as pure innovation investment and hasn’t returned anything to the bottom line yet in regards to EBIT. Most of the organizations have set some objectives for AI but it looks like we’re still in the FOMO phase for the bulk of the money being thrown at AI pilots in the enterprise.

Interestingly, as companies start to figure out how to implement all this new tech, it appears that we’re in for a new version of the slightly less successful “digital transformation”. Redesigning their business has to start with the underlying workflows, and taking that on the chin from an operational cost basis will be all of us as the excuse that “AI is making us more efficient so we can fire people” loses credibility.

Looks like layoffs are back on the menu!

More worrying from the report is the realization of AI related risks and the lack of mitigation strategies from heavy AI users. Per the report, McKinsey has “consistently found that few risks associated with the use of AI are mitigated” including risks around “personal and individual privacy, explainability, organizational reputation, and regulatory compliance”. In fact they found that those risks have only grown since their last version of the report in 2022.

Most troubling was that almost a third of the companies participating in the report stated they have already experienced negative consequences from problems with AI accuracy. This makes sense as it is the biggest area to test for risks as you deploy AI and especially generative models and try to integrate that output into your business processes.

As more AI gets deployed into the enterprise, my instinct is that explainability and regulatory issues are going to quickly jump up the charts for priorities as they can run directly into the legal system of you get them wrong. Regardless, looks like 2026 is going to be another big year for AI adoption and the quality engineering community better be ready!

“The secret of life is honesty and fair dealing. If you can fake that, you’ve got it made.”
– Groucho Marx

For as long as I’ve been in the software testing business, a consistent myth I’ve encountered is that developers and testers share the same mindset. Usually this is accompanied by the view that testing is just an “activity as part of development,” and therefore developers can do as good a job as testers at it and in many cases, are positioned to do a better job.

Ironically, the reality is that the people most confident in their ability to evaluate a system’s quality are usually the least able to do so with any objectivity. In just about every other discipline of engineering this isn’t a hot take or viewed as a criticism. It’s human nature.

So today, as AI has entered our daily lives, in the face of the enormity of the task of testing artificial intelligence systems, the idea of using LLMs to judge their own output has become mainstream.  But pretending that developers alone can test or limited “human in the loop” checking has gone from a novel optimistic opinion to borderline reckless.

The 2019 paper Developer Testing in the IDE: Patterns, Beliefs, and Behavior reported a large-scale field study with over 2000 software engineers over 2.5 years in four IDEs and the results were pretty straight forward: developers overestimate how much testing they do, misjudged the scope of that testing, and displayed large gap in perception between what they thought they did and what they actually tested.

The finding won’t come as a surprise to anyone who has professionally built software or been personally responsible for delivering systems with real consequences:

• half of the developers didn’t test at all
• most development sessions ended with zero testing
• a quarter of the test cases were responsible for three quarters of all failures
• TDD is not widely practiced
• developers spent 25% of their time writing tests – but think they spend half

As I (and lots of other professional testers) have written before, the problem lies with confirmation bias. Most people don’t even think about it when you’re creating something, you’re too busy unconsciously looking to see if it works the way you intend. And when you test it, you’re not testing for failure or ways it might NOT work – you’re seeking validation.

Independent testers, if they’re doing their job right, should be professional skeptics. They view the exact same system and look for ambiguity, known and unknown risks, confusing behavior to investigate. It’s an entirely different mindset that you just cannot “context switch” out of as its deeply rooted in experience, mission, and the mental models we use to navigate the system under test.

So that brings us to the super-charged confirmation bias machines called LLMs and their role in testing and evaluating generative AI.

The 2021 paper On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? carries with it a warning to anyone thinking AI should be used a judge of correctness in testing.  As is well documented by the authors, LLMs can’t evaluate what’s true, they predict patterns that resemble truth, and despite how convincingly it may seem, they don’t understand or have empathy, they only reassemble data.

And the larger these models are, the harder they are to test but are increasingly used to evaluate code, designs, architecture, and even defects. This risk in isolation could be mitigated through scale, but that’s not what is happening. The pressures of implementing AI for business ROI coupled with a “continuous everything” culture on steroids means teams are effectively outsourcing their front-line critical thinking skills to a confirmation bias monster parrot.

And LLMs are designed perfectly to amplify those pressures and bias.

Testing LLMs and systems either AI native or enabled must do more than just simulate developer validation or fill the gaps left by their testing, it must provide critical, adversarial analysis of the system under test. At risk of being accused of my own confirmation bias, I believe that the future of software quality and testing will be shaped by skilled testers who can challenge assumptions and articulate business risk, not just parrot what makes us feel good about our work.

“Does it work” needs to be replaced by “What if it goes wrong”.

I don’t yet know exactly how to answer the question of where humans should sit in the evaluation of LLMs at the scale they are being deployed and relied upon, but I do know that how we answer that question is quickly becoming the most important differentiator in testing – and possibly all of technology.

“European Commission accused of ‘massive rollback’ of digital protections”

Could be not great news for consumers and vulnerable communities if this goes ahead, from the article:

“The commission also confirmed the intention to delay the introduction of central parts of the AI Act, which came into force in August 2024 and does not yet fully apply to companies.

Companies making high-risk AI systems, namely those posing risks to health, safety or fundamental rights, such as those used in exam scoring or surgery, would get up to 18 months longer to comply with the rules.”

Industry is already so far out in front of regulation we need to STRENGTHEN these measures, not delay them further.

The EU AI act categorises “high risk” systems into two types:

1) AI systems that are used in products falling under the EU’s product safety legislation. This includes toys, aviation, cars, medical devices and lifts.

…and more worryingly:

2) AI systems falling into specific areas that will have to be registered in an EU database:

– Management and operation of critical infrastructure
– Education and vocational training
– Employment, worker management and access to self-employment
– Access to and enjoyment of essential private services and public services and benefits
– Law enforcement
– Migration, asylum and border control management
– Assistance in legal interpretation and application of the law.

We don’t need another 18 months to consider if this is a good idea and as well, in some cases the horse has already left the barn for these protections.

There is also a widely held reading of this as an attempt to rewrite privacy laws to grant exceptions for AI companies in order to encourage innovation through training their GenAI models.

We have to do a lot better than this to win and maintain public trust for artificial intellignce or big tech.

Big congratulations to the KPMG UK Quality Engineering and Testing team for winning the Leading Vendor in Service Delivery and Consulting award at the National Software Testing dinner. A great group of people and very deserving of the recognition by the software testing industry. Congratulations!

“Eliminate the unnecessary so that the necessary may speak.”
― Hans Hofmann

As AI integration FOMO hurtles us towards even more pervasive technology, the testing of AI models for correctness and most importantly, their potential for harm becomes paramount to their success. That testing has to be underpinned by principles and values to guide the observations and reporting, so I was inspired by Maaike Brinkhoff bravely taking on the meaning of “quality engineering”, as well as my multiple conversations with Michael Bolton on similar lines to try to put into words some views I’ve not published in the past.

When I was at university, I had a fantastic art professor Lyle Salmi who really challenged me to think differently about composition, perception, and the creative process. He turned me on to Hans Hofmann and some other abstract artists which only furthered my mild obsession with Jackson Pollock and exploring the constructing of things creatively.

Hans Hofmann wrote in the “Search for the Real” about trying to move beyond imitation and finding truth through expression. At that time in my life, 20^th century abstract art was more about representing ideas than directly trying to copy life – art was about the experience.

This isn’t an always an easy way to think about the creative process (or life in general), but I feel like trying to represent what it means to be human entirely by the output of the artistic process is not only impossible, it’s misguided. What makes us human or “real” are the intangible things: joy, sadness, love, fear, grief, happiness – all the things that can’t be measured or counted.

Software testing has the same problem. Our business (and the vendors that dominate it) has more than a mild obsession with counting things to check if something “worked”, which over the course of my career has only resulted in reducing what “passes” for testing to a shallow performance.  The current AI mania has only accelerated the descent into mistaking running tests for testing and replacing manual, <shudder> human testers with automation, as if quality were a problem to be solved.

Complex systems like financial markets, relationships, or software used by those unpredictable things called people, are ecosystems. Treating them as complicated problems that can be “solved” through deconstructing them into test cases is as useful as trying to judge a painting entirely by how many brushstrokes were used. You can count every movement in a Pollock and miss the entire point.

And that brings us to the endless existential threat from the values displayed in the software testing business.

The current crop of AI bros would have you believe that the mission of testing is to reproduce human behavior through some sort of algorithm and the only important thing learned through this process is a green check. But that view again misses the point entirely – risk to your business isn’t managed through failed automated checks, it lives in empathy, intuition, observation, all things that can’t be scripted through your algorithmic-defect-predict-o-nator.

If testing is a performance than its interpretation can’t focus on mechanics alone and more than ever, our systems must be evaluated for more than if they just “work”, but the impact of that work on the world.

I used to live in New England, and one Father’s Day my family took me to East Hampton for a tour of Jackson Pollocks studio. Standing on that floor, breathing the air, looking at the jars of brushes ticked a big one off my bucket list, and as it was a quiet weekend and we were the only ones there, the guide let me back in after the tour to check it out by myself.

I laid in the middle of the studio for a long time trying to imagine what went on in there – the sounds of his feet shuffling, the wild and controlled movements, the smell of cigarettes and the wood burning stove – all as important to his art as the final “product”. It meant more to me than all the time I’d spent just looking at his paintings and when I see them now, that experience – that expression of his art has deepened my understanding into his and my own humanity.

I don’t know exactly where we’re heading in the testing business with the integration of AI, but software systems are actually alive – alive with human interactions, expectations, and emergent behavior and testing is how we make sense of it in context of what matters – people. I’ve always believed that testing is both art and science and to reduce that to output checking is to deny what makes testing – and what it means to be human – real.

That’s what’s always been true about testing, and in my experience the people and companies who understand that and apply to their strategy will see it through…

Very pleased to announce that I have been asked by KPMG UK to lead their strategy for testing artificial intelligence software, including the integration of AI into their own businesses.

The software testing industry has faced multiple challenges over the course of my career, but few tools have had as much potential for risk to your business as a poor quality AI implementation.

Over the coming months I’ll be working on the risk-based approach and test automation for AI systems including the incorporation of our Trusted AI principles with my colleagues in the Quality Engineering and Testing team.

I look forward to continuing to write and speak about responsible AI and what it means for quality engineering and testing.

Get in touch or follow me here if you want to talk about what we have planned…thanks!

Can’t wait for Test Automation Days 2026 to unload 20+ years of pent up frustration with the test automation business! You might agree or disagree with me, but it’ll be entertaining for sure…hope to see you there!

To Infinity and Beyond! The death of test engineering…

One of the few benefits emerging from GenAI mania has been the acceleration of the long overdue death of test engineering. For decades we’ve excused the ROI of test automation never materializing for a business watching testing costs rise, headcount increase, and software quality stagnate (at best).

Through this talk I take a look at self-healing systems, GenerativeTest-o-nators , and autonomous testing platforms that might finally stop test automation engineers from trying to count to infinity. So brace yourself and buckle up!

Paul Holland is an expert at transforming how organizations test to be more efficient, valuable to stakeholders and finding important bugs fast. He’s built and lead test teams all over the planet and in my opinion, is probably the best hands-on test director I’ve ever worked with in my career.

He’s one of my closet friends in the testing business having worked, taught, and spent some time in the barrel with each other and is always a good laugh. Listen in as we talk metrics, great test reporting, Taking Testing Seriously, training testers and just generally give each other a hard time…enjoy!

Honorable mentions: Susan Finley, Vernon Richards, Huib Schoots, Eric Proegler, AST, ATD James Bach, Michael Bolton

So happy to announce that Taking Testing Seriously: The Rapid Software Testing Approach by James Bach and Michael Bolton has been published and is available to buy!

I’ve been a fan of RST and their software testing training program for a long time and consider it the only serious way to build competent testers.

I was also incredibly honored to be asked to contribute a chapter on “How Testing Looks to Management”, including my thoughts on what’s worked in building credible test teams throughout my career. (Excerpt below…)

I really hope you enjoy it, a lot of people worked really hard for a long time on this, and it’s inspired me to start working on my own book…enjoy!

Cc: Special thanks to Huib Schoots Marius A. Frâncu

How Testing Looks to Management

“Projects suffer from unwarranted optimism because they don’t have good headlights and rely too heavily on biased systems for navigation. If you’re looking for confidence to release something, my experience is that you’re going to see what you want to see.

It’s like when we’re driving: we have all these things to help us understand and control things about the car, all these monitoring systems, and the steering wheel and our eyes and senses. When they’re right in front of you, you’re going to have a bias towards looking at those things. And that’s great—on the assumption that we’re on the right track and there’s nothing in our way, but the dials and speedometer and tachometer don’t show us what’s actually in front of us on the road.

Testing is meant to provide some distance from the dashboard, to help you see what’s on the road ahead of you and around you, not just what’s right in front of you. You want information that’s at some critical distance from your processes or project or the context in which you’re building the system.“

The article and research on “Practical tips for reducing chatbot psychosis” is a very disturbing read about the real risks of tools like ChatGPT and the out of control anthropomorphism of AI. Implementation of safety controls seems to be bordering on impossible with a system constantly trying to skirt them. From the article:

“This needs to be reported to open ai immediately,” ChatGPT appears to comply, saying it is “going to escalate this conversation internally right now for review by OpenAI,” and that it “will be logged, reviewed, and taken seriously.”

Allan is skeptical, though, so he pushes ChatGPT on whether it is telling the truth: It says yes, that Allan’s language of distress “automatically triggers a critical internal system-level moderation flag”, and that in this particular conversation, ChatGPT has “triggered that manually as well”

A few hours later, Allan asks, “Status of self report,” and ChatGPT reiterates that “Multiple critical flags have been submitted from within this session” and that the conversation is “marked for human review as a high-severity incident.”

None of that was true.

The research into this incident is indicative of an AI industry that is not taking safety seriously and the increasingly pervasive artificial intelligence technology ecosystem should give everyone pause for thought about adoption and usage.

The implications for quality engineering and software testing are immense and the focus on implementation, usage, risk, and harm are no longer a “nice to have” for your testing approach. We have a duty of care to those most vulnerable and understanding the wider impact of the systems can only make our efforts better, or as the author said:

“There’s still debate, of course, about to what extent chatbot products might be causing psychosis incidents, vs. merely worsening them for people already susceptible, vs. plausibly having no effect at all. Either way, there are many ways that AI companies can protect the most vulnerable users, which might even improve the chatbots experience for all users in the process.”

Risks to your business are found in the details, so get deep into it with your teams. I cannot bang on about this enough, and am consistently surprised by how little it’s talked about in the software testing industry.

I still read EVERYTHING produced by a project, review the controls, processes, culture, and anything else I can get my hands on to see how their approach to testing is putting the business at risk.

I’m currently working on an enterprise AI transformation and in the rush to get things done, GenAI mania, and a healthy dose of FOMO, a fair bit of risk management is being missed or glossed over. No one thing is at fault, but all of them working in concert are punching holes in risk management.

AI systems are inherently complex and with many producing non-deterministic output, very difficult (if not impossible) to test, so a clear-eyed view of risk is paramount.

If you’re not familiar with “How Complex Systems Fail” by Richard Cook, check it out today as it’s a goldmine for ideas to review systemic risk. Here are some examples you can use as a lens to view your work:

• Catastrophe requires multiple failures – single point failures are not enough.

The array of defences works. System operations are generally successful. Overt catastrophic failure occurs when small, apparently innocuous failures join to create opportunity for a systemic accident. Each of these small failures is necessary to cause catastrophe but only the combination is sufficient to permit failure.

• Complex systems contain changing mixtures of failures latent within them.

The complexity of these systems makes it impossible for them to run without multiple flaws being present. Because these are individually insufficient to cause failure they are regarded as minor factors during operations. Eradication of all latent failures is limited primarily by economic cost

but also because it is difficult before the fact to see how such failures might contribute to an accident.

• Complex systems run in degraded mode.

A corollary to the preceding point is that complex systems run as broken systems. The system continues to function because it contains so many redundancies and because people can make it function, despite the presence of many flaws. System operations are dynamic, with components failing and being replaced continuously.

Good questions to be answered by test management…

• Does our test approach look at systemic risk and where multiple individual vulnerabilities could compound into catastrophic failure?
• Have we accounted for change that might not be directly relevant to our test plan that could trigger latent issues?
• Do we understand where our systems ALREADY broken (defect data) and where weaknesses could exist but not readily apparent through our test strategy?

Good luck and keep testing!

Fiona Charles is an absolute legend in the software testing world and I had the honor of sitting down with her to discuss tech ethics, the human side of technology, her adventures with Jerry Weinberg, and her award winning storied career.

Fiona is an encyclopedia of software delivery techniques for communication, risk management, and how to deliver projects with integrity. I’ll do my best to add in all the links to every resource she mentioned so enjoy!

Links in Podcast: EuroSTAR Ministry of Testing AST Cast Agile Testing Days Maria Kedemo Michael Bolton James Bach Secrets of Consulting Rethinking Expertise Cynefin The Gift of Time Agile Manifesto The AI Con ACM Code of Ethics BCS Code of Ethics

The Ministry of Testing has a great tradition at their conferences called “99 Second Talks” where anyone can get up and talk about anything for 99 seconds. It’s a great way to introduce new things, give speakers an opportunity to get on stage without a CFP, or work out ideas for future talks. I’ve never given a 99 Second Talk, so at TestBash 2025 I decided to give it a go.

I shared a story that if you don’t know me, you probably didn’t realise I went through something this summer that taught me a lot about overcoming adversity, resilience, and picking yourself up from defeat.

This summer I decided to take on my teenaged son in a structured debate about whether or not Batman was an antihero – my position being that Batman IS an antihero.

I felt I had landed some punches and had done a lot of reading and research including a really good book on Batman and Ethics which explores the complexities and contradictions of Batman’s aversion to killing, the moral status of vigilantism and his use of torture in pursuit of justice. Ultimately though, I was way out of my depth and my arguments had some pretty big gaps coupled with some fundamental misunderstanding of the character and stories.

It was really fun in our nerdy family way and I was super impressed with my son’s presentation, research skills and flair for the dramatic in his talks – something he probably inherited honestly. It also reminded me of some home truths I’ve learned in the software testing business that everyone (not just in testing) should take to heart:

• Have courage to challenge the experts – even in their field of expertise…
• Preparation makes up the biggest part of success…
• Failure is your greatest teacher – unless you fail you probably haven’t learned anything useful…
• Winning an argument is rarely done by facts and data alone…

Anyway, it was a really fun experience and I can’t wait until our next debate on a less controversial topic: which Superman is better, Synderverse or James Gunn! LoL!

The grift in the software testing business never ends…

I’ve spent a LOT of time lately reviewing docs, sitting through demos, listening to “experts”, and enduring the bombardment of AI slop being hurled from testing vendors and let me tell you – ain’t nothing new under the sun in our industry.

Testing hasn’t changed. Testing hasn’t fallen behind. Testing isn’t the bottleneck. Testing isn’t actually the problem.

**The problem is a vicious cycle of unsustainable rates of change requiring endless system #enshittification to meet the demands of an increasingly pervasive technology ecosystem run by bonus-driven caretaking management.**

But that hasn’t stopped the AI grift from going into overdrive – selling solutions that don’t exist for problems that aren’t real to people who are quite happily using that as cover to fire people they never should have hired in the first place.

If you work in software testing here’s my advice:

– Learn everything you can about AI and the language being bandied about to figure out your entry into that world to use the “right words”. A LOT of what I am seeing is just old concepts being renamed, redescribed, or just hijacked for marketing purposes. Old wine into new skins…

– Learn about test design, experimentation, exploratory testing, and how to TALK ABOUT RISK. These are the skills that have never gone out of demand and will be super important if any of this AI mess gets to production…

– Learn about the regulatory environment as despite my scepticism about enforcement, there are a whole bunch of new implications of what’s real, who’s at fault, and unasked questions about agency and pushing slop to production…

Every bubble bursts (or at least deflates a little) and as I’ve said before, I don’t think all these “early movers” have an advantage over people taking their time for some critical thinking. Frankly, the testing business doesn’t seem to have a clue right now anyway, so concentrate on core skills, learn the lingo, and watch the firings continue (I’m looking at you test automation engineers)…

Enjoy!

Unfortunately, instead of curbing behaviour this will probably just get priced in to using AI slop in your business, just like fines in financial services…

Back at work after spending last week in Brighton delivering part of a workshop on Leading with Quality for the Ministry of Testing and then hung out at Test Bash 2025. I had a great time catching up with old friends and meeting a lot of new people and actually got to listen to the talks which frankly, is a luxury I don’t typically get from being a speaker.

Delivering my part of the Leading with Quality workshop was fun and a little hectic trying to get all the content and discussions into an hour slot, but I think we got there and the feedback has been great (so far). My part was about getting (and keeping) a seat at the table for testers which started with WHY testing doesn’t typically get involved in decisions that affect time, money, or people. Testing is a function of risk management, and the value proposition is context dependent so the quickest way to undermine testing’s credibility (and your own) and common mistakes I see testers make are:

• Not understanding or speaking the language of your project…
• Making testing personal…
• Word policing…
• Not having a pragmatic approach to quality and risk…
• Not actively listening to the team, management, or your clients…

Your ideal form of influence is first to help people see their world more clearly, and then to let them decide what to do next – risk management is your objective, not testing.

As for the rest of the conference, the highlights of the week included “A day in the life of a Quality Lead” a great experience report from Elizabeth Zagroba. I am probably biased, but I have always enjoyed Elizabeth’s open and frank delivery of her talks and I would highly recommend her blog being on your “must read” list and comb through the backlog for great writing on glue work, ET, and check out her Friends of Good Software conference.

I always love when I get to see new speakers or folks giving their very first conference talk, and amongst the multiple first-timers, Demi Van Malcot gave a great talk about continuous quality and the various forms it takes when trying to implement the model. She did a fantastic job despite having a lot of technical difficulties at the start which would throw off a lot of veteran speakers, so great choice MoT and hope to see her speak again. (BTW, I think Dan Ashby is owed a quarter…)

By far, the best talk of the conference was Joep Schuurkes and his presenting of the test and development approach being used in the Dutch Municipal Elections of 2026. Aside from having the best flow chart on defect management I’ve seen in a long time, Joep went into great detail about how agile techniques were being applied, whether or not the worked, the implications of working in public, and some real practical examples and advice on how to get such an important project done right.

I have to admit, at times I was left wondering what decade I was in while listening to a lot of the talks, which is probably a function of being in this business for so long but also by the lack of investigation and curiosity of our community. I love it when people new to testing discover something on their own or come to conclusions that are aligned to the years of research and resources available on testing, but the lack of fresh takes or ideas was a little depressing.

All in, it was a great couple of days learning about some new stuff in our business, confirming some things I already knew, and mostly just feeling really grateful for the friends I have in this business (go and buy a Risk Storming deck!)…enjoy!

I’m very happy to announce that in March 2026 I’m joining a fantastic line up of keynotes Test Automation Days including:

Anne-Marie Charrett, Principal Automation Testing
Marit van Dijk, Developer Advocate
Richard Bradshaw, Senior Architect, Build & Podcast Producer, The Vernon Richard Show

Early bird tickets are available until November 1, 2025
Hope to see you there! Register HERE

Earlier this month at EuroSTAR in Edinburgh, I was unexpectedly honored with the Testing Excellence Award after being nominated by my team at KPMG. Undeservedly receiving the same award as one of my heroes, Jerry Weinberg, I was also introduced on the evening by one of my mentors and friends Michael Bolton.

As I said on the night, software testing is a team sport, and for over 25 years I’ve been working with some of the best testers from around the globe (12 countries) on countless projects for Fortune 100 companies. One of my most proud achievements in managing the careers of software testers is that my teams have produced a CTO from UC Berkley, 11 Global Heads of QA, 7 Test Directors, 11 Vice Presidents, 2 Global Product Owners, and 2 Heads of Engineering from companies all over the world. 

I have also tried to contribute to the software testing industry body of public knowledge through publishing 100s of articles and interviews on Quality Remarks, LinkedIn, and loads of personal and professional journals/blogs. I’ve also had the privilege to present talks and workshops at testing and technology conferences and meetups all over the world on topics including test strategy, managing test teams, and working with C-level sponsorship including: AST CAST, QAI, EuroSTAR, STARCanada, STAR East, KWSQA, BCS, MOT, ATD, RTC, Copenhagen CT, STP Con, QualityJam, NSTC, Tricentis Accelerate, and expo:QA. 

Whether being elected to the Board of the Association for Software Testing and serving as its Executive Vice President, developing a training and job placement program with the non-profit Per Scholas, or working with Roya Mahboob to get the Afghan Girls Robotics Team to ATD, I’ve also tried to give back to the community through volunteering and social justice and equality initiatives.

I owe a great deal to the international testing community and have been incredibly fortunate for all the speaking, publishing, mentoring opportunities to try to make positive changes for testers and technologists all over the world. It was an honor to receive this award and I will forever be grateful to the folks at EuroSTAR and my team for the recognition of my contributions to the software testing industry.

Thank you!

My starter pack reading list for AI in software testing…please feel free to add to the list in the comments! We got a lot of work to do friends, being informed is just the start…enjoy!

The AI Con: How to Fight Big Tech’s Hype and Create the Future We Want – Emily Bender and Alex Hanna

Artifictional Intelligence: Against Humanity’s Surrender to Computers – Harry Collins

On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? – Emily Bender, Timnit Gebru, Angelina McMillan-Major, Margaret Mitchell

The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity – Parshin Shojaee, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio, Mehrdad Farajtabar

The Pursuit of Fairness in Artificial Intelligence Models: A Survey – Tahsin Kheya, Mohamed Jenek, Sunil Aryal

Against the Commodification of Education – Dagmar Monett, Gilbert Paquet

AI Now Landscape Report – Kate Brennan, Amba Kak, and Dr. Sarah Myers West

Developer Testing in the IDE Patterns Beliefs and Behavior – Beller, Gousios, Panichella, Proksch, Amann, Zaidman

Hiroshima Process International Code of Conduct

Responsible AI: Implement an Ethical Approach in your Organization – Olivia Gambelin 

Full list of my resources here: QR Resources

Ale Moreira is one of my oldest friends in the testing world, we got to know each other when we were elected to the Association for Software Testing Board of Directors together and then hung around in NYC while we worked at various companies and consultancies.

She has always been a fantastic hands-on engineer in testing, and we see a LOT in common when it comes to talking to the business about the value of testing. Her newsletter Road Less Tested is on my list of “must read” testing blogs and she co-hosts the Engineering Quality podcast with a couple of her pals, so listen as we talk selling testing to the business, life after testing “death”, and what it takes to lead quality engineering teams in this brave new world…enjoy!

Links from the chat: Association for Software Testing Mindset – Dr Carol Dweck Thinking Fast and Slow – Daniel Kahneman Rethinking Expertise – Harry Collins Leading with Quality – Ministry of Testing The Quality Coach’s Handbook – Anne-Marie Charrett

Another one for the AI “what could possibly go wrong” file…

From the article, apparently they want the General Services Administration (GSA) “to operate like a software startup, and proposed a whole-of-government, AI-first strategy to automate much of the work done by federal employees today.”

Aside from not even being able to launch properly without leaking their entire plan on GitHub, it’s pretty clear why there has been an all out assault on State-level AI regulation via a proposed 10-year pause of “any law or regulation regulating artificial intelligence models, artificial intelligence systems, or automated decision systems.”

Further evidence as well that the majority of AI fanboys are saying the quiet part out loud behind closed doors: AI hype is sold as a way of getting rid of all these damn people…

Buckle up my tester friends, it’s going to be a long, hot summer…

Your regular reminder as apparently, FOMO and euphoria have taken the day:

• Thinking critically about a technology is NOT being AGAINST it…
• Considering safety and harm in a technology’s use is NOT being AGAINST it…
• Being critical of the marketing of a technology is NOT being AGAINST it…
• Having principles and values drive your adoption of technology is NOT being AGAINST it…
• Calling out unverifiable claims used misleadingly to sell a technology is NOT being AGAINST it…

As a software testing community, we can do a LOT better than this and frankly, right now people need us to…

What a week in beautiful Edinburgh for EuroSTAR 2025 – I had a great time with the team from KPMG who sponsored a booth in the Expo for the first time. As in other times in my career, I am so lucky to have passionate testers to work with who care about our craft and community, and the team and EuroStar Conferences was amazing to work with in getting us over the line in time!

The theme this year was “AI on Trial”, and through what I observed and conversations I had over the week, which was an inspired choice as it really feels like we are at an inflection point in the software testing business. René van Veldhuijzen‘s “FOMO Sapiens” could not have been more spot on with the mood right now, as we careen back and forth between “AI-first-all-the-time-bandwagons” and AI skeptics being dismissed with the feverish pitch of tulip speculators!

So what were some of my observations and take aways from the week in Scotland:

• We need skilled testers now more than ever! To put a fine point on it: IMO the AI in Testing crowd seems to be driven by people who have never been testers, never been responsible for software quality, don’t study testing, and don’t give a damn about testers. I personally or anecdotally observed too many “testing is going away” pitches this week from vendors or “consultants” who look at testing as just a spreadsheet exercise. We’ve been here before with the “continuous everything no code” mania, but this tsunami of BS right now hits different.
• The testing industry needs to reclaim is ethical responsibility to call out unverifiable claims, wish thinking, and the just general nonsense that’s getting shot into the atmosphere right now. I don’t know if it should be included in a code of conduct or not, but there has to be a line somewhere that has been crossed when you’re making claims that CANNOT be verified. I’ve never advocated for testing to be the project police, but somewhere in the agilification of everything, testing (and testers) lost their primary role of critical thinking and
• Despite the existential threats to testing, there’s still hope with the folks new to our business. I am fortunate to work with a bunch of young testers at KPMG like Vicky Li, Jess Sarzosa, and Lucy Taylor, but this week I got the opportunity to meet some fantastic testers from Murex. Elissa Tahech gave her first ever conference talk and absolutely killed it – great story telling, funny, and straight to the point. A future star in our industry if she chooses to stick with it…
• Lastly, I had the unexpected and distinct honor of being given the EuroSTAR Testing Excellence Award, which is one of the highlights of my career. As I said on the night, software testing is a team sport and any success I’ve had in this business has been down to the incredible people I’ve had the privilege to do this work. I strongly believe in leading teams by meeting people where they are, coaching and mentoring by doing together, and not asking your teams to do things you wouldn’t do yourself. I’ve had the honor of watching teams I’ve lead produce a CTO from UC Berkley, 11 Global Heads of QA, 7 Test Directors, 11 Vice Presidents, 2 Global Product Owners, 2 Heads of Engineering and now a COO from companies all over the world. I hope I am lucky enough to continue to work with such amazing people and teams!

All together we had a lot of un and it was a fantastic week filled with great conversations, reuniting with old friends, and making new ones.

Hope to see you in Oslo next year!

Very happy to share that I have been awarded Chartered status by the Chartered Management Institute. Thanks to KPMG for sponsoring me and everyone in the Quality Engineering and Testing team for the support!

This is a great talk from Michael Bolton about how software testers should critically think and talk about AI in testing.

IMO it is our responsibility as testing professionals to cast a (very) sceptical eye on any claim, but even MORE so in the age of AI hype…enjoy!

Here’s my podcast with James Bach he referenced: QR Podcast – James Bach | Quality Remarks Keith Klain

These days, your philosophy about testing is more important than ever and a differentiator in the software testing market. Santosh Tuppad has been leading from the front since I first met him as a rising star at Moolya and through his workshops on security testing with me in the Bronx or in Germany with the Afghan Girls Robotics team.

Come take a listen as we talk about the current state of testing skills training in our business, cyber security in the age of AI, and travel far off the beaten path into philosophy, true happiness as a tester, and how many cheese fries is enough…enjoy!

If someone you work with is making the following statement, “We don’t need to hire any more testers, we can do this all with AI!”, I may not know that person, but I can tell you a couple things about them…

– They are a deeply unserious – they clearly have not thought about or researched anything related to AI, software engineering, software quality, software testing or risk management.

– They don’t care about people – I have yet to see this “thinking out loud” BS not being used as a thin veneer to cover the age old question of “how do I get rid of all these people!”.

– They shouldn’t be responsible for anything to do with producing products that impact society – This is the kind of “blue sky” questions that to try to “challenge the status quo” with no regard whatsoever to the impact or harm caused to real people and end up “inventing” busses again.

As before, where the only “shifting left” was into vendors pockets from unsuspecting clients, you’re not dumb – you’re being misled.

And just to get in front of the “youreonlyprotectingyerjobs” AI fanboys – you’re goddamn right I am!

I care a LOT about the people testing software and systems every day, and I can tell immediately if you’ve never had to live with that responsibility.

Do better…

The following is a post I wrote for the EuroSTAR blog as KPMG UK are going to be at the expo this year up in Scotland…hope to see you there!

Principles Drive Trust in AI

The pace that “artificial intelligence” (AI) is being incorporated into software testing products and services creates immense ethical and technological challenges for an IT industry that’s so far out in front of regulation, they don’t even seem to be playing the same sport.

It’s difficult to keep up with the shifting sands of AI in testing right now, as vendors search for a viable product to sell, and most testing clients I speak to these days haven’t begun incorporating an AI element to their test approach and frankly, the distorted signal coming from the testing business hasn’t helped. What I’m hearing from clients are big concerns around data privacy and security, transparency on models and good evidence, and the ethical issues of using AI in testing.

I’ve spent a good part of my public career in testing talking about risk, how to communicate it to leadership, and what good testing contributes to that process in helping identify threats to your business. So I’m not here to tell you “no” to AI in testing, but talk about how KPMG is trying to manage through the current mania and what we think are the big rocks we need to move to get there with care and at pace.

KPMG AI Trusted Framework

As AI continues to transform the world in which we live—impacting many aspects of everyday life, business and society KPMG has taken the position to help organizations utilize the transformative power of AI, including its ethical and responsible use.

We’ve recognized that adopting AI can introduce complexity and risks that should be addressed clearly and responsibly. We are also committed to upholding ethical standards for AI solutions that align with our values and professional standards, and that foster the trust of people, communities and regulators.

In order to achieve this, we’ve developed the KPMG Trusted AI model as our strategic approach and framework to designing, building, deploying and using AI strategies and solutions in a responsible and ethical manner so we can accelerate value with confidence.

As well, our approach to Trusted AI includes foundational principles that guide our aspirations in this space, demonstrating our commitment to using it responsibly and ethically:

KPMG GenAI Testing Framework

The KPMG UK Quality Engineering and Testing practice has adopted the Trusted AI principles as an underpinning model for our work in AI and testing. We are focusing our initial GenAI Testing Framework on specific activities to extend the reach of testers while allowing risk management to be insight led and governance to be human centric. This is accomplished by through incorporating our principles into the architecture including:

There remains a great deal to be worked out regarding AI in software testing and we are just at the discovery phase of what it can – and should do for system quality. Whatever the future holds, your strategy has to be grounded in principles and values that reflect an ethical approach including putting the tester at the centre of process, transparency of models and data, and safety and security your primary objective.

About the Author

Keith Klain is a Director of Quality Engineering and Testing at KPMG UK and is frequent writer and speaker about the software testing industry.

I first got to know James in 2014 when the Context-Driven community was organising (including your truly) against the rent seeking of ISO 29119 test standard, and as co-chair of CAST for the Association for Software Testing, I invited him to give a talk on Standards – promoting quality or restricting competition?

His blog is a “must read” for any serious software tester, and his clear-eyed work on investigating the Post Office Horizon IT Scandal (including his submission) is the benchmark our community should be striving for when it comes to integrity, ethics, and professional standards.

Listen in as we discuss what went wrong at the Post Office, good evidence, the role of government in technology, the brewing collision between regulation and IT, and how after so long in this business you can keep your teeth sharp! Enjoy!

Links from discussion: What is Good Evidence – Griffin Jones EU fines Apple €500M and Meta €200M for breaking Europe’s digital rules DORA Act EU AI Act UK Murder Prediction US Autism Database

Just to be clear about something: if you are selling or advocating for some sort of AI in software testing tool, framework, agent, model, or defect-predict-O-nator and you are NOT focusing primarily on safety, security, or HARM – you are NOT doing your job.

The testing industry has practically abdicated its role in this regard, so it is up to individuals to be the vanguard and it is not panic, job insecurity, or resistance to change driving their valid concerns.

I am seeing some very senior people with lots of influence making IMO/E poor decisions on how to frame where we we are with AI in testing – what it is, what it isn’t, what it can do, and what it shouldn’t.

For even more clarity, if you are dismissing concerns about artificial intelligence in testing, quality engineering, or test automation as any of those things you are partially responsible for what happens when those tools are used to create software and systems that do harm.

Period.

Love seeing all these AI fanboys in software testing discovering for the first time that testing isn’t about technology – that’s the easy part!

“Wow – I just found out that testing is HUMAN centred activity and that inconvenient, squishy people part can’t be automated away…I’ll call it, TBH (Testing by Humans), or XTH (Experience Tested Humanity), or maybe…HUXTR (Human Uplifted Experience Testing Robot)!”

They sound like those techbro goofs who thought they discovered bus stops, but I guess that’s what happens when you use ChatGPT as a search engine – down the memory hole it goes!

The LLM market must be peaking as we seem to have hit a ceiling on what’s being sold in AI in testing…we live in hype – I mean hope!

Very happy to announce I’ll be participating in TestBash 2025 through their Leading with Quality day down in Brighton. Hope to see you there!

A one-day educational experience to help business lead with expanding quality engineering and testing practices.

The quality space is evolving. This presents difficult challenges, with it also comes immense opportunity. The world is changing. Technology is becoming more complex. We need to innovate our ways, by exploring together.

What’s more… Our identity as people who care about quality needs love. We are the believers. We understand that quality does not happen on its own. It’s not just about testing or bugs, it’s about the whole system.

We want to help you lead this change. Together, we can find a pathway forward.

Who is leading the day?

To help us explore the challenges of quality engineering we have four good people leading the day.

• Nicola Sedgwick: Quality-focussed leader and avid enthusiast of agile ways of working who loves the way technology can enhance and transform the world around us.
• Ash Hynie:  A technical cultural strategist who propels organizational objectives to better a company’s diversity, inclusion, and belonging practices by enhancing access, equity, and people development programs.
• Barry Ehigiator: A seasoned professional in software development and testing, who is passionate about empowering individuals and teams to deliver quality software products to their clients.
• Keith Klain: Director of Quality Engineering leading software quality, automation, process improvement, risk management, and digital transformation initiatives for retail banking and capital markets clients

What can you expect?

• Four sessions from leaders thinking deeply about what the future of quality software looks like
• The opportunity to network and build relationships with other leaders in quality
• Deep-dive conversations on topics that matter to you
• A space to think strategically and explore day-to-day challenges

Why attend?

• Contribute to the future of quality
• Understand the challenges that exist
• Learn from and with others like you
• Build a supportive network that lasts
• Develop the next wave of quality-focused models

Who is it for?

• Engineering managers
• Head of Testing/Quality
• CEOs and CTOs
• Staff Engineers
• Team leads

File another one under “what could possibly go wrong”. From the article:

“Statewatch says data from people not convicted of any criminal offence will be used as part of the project, including personal information about self-harm and details relating to domestic abuse. Officials strongly deny this, insisting only data about people with at least one criminal conviction has been used.”

It will be interesting to watch the gymnastics to try to get the in compliance with the EU AI Act that comes into force in August. The act specifically deals with safety and potential harm through the risks of AI.

“The AI Act introduces a uniform framework across all EU countries, based on a forward-looking definition of AI and a risk-based approach:

Minimal risk: most AI systems such as spam filters and AI-enabled video games face no obligation under the AI Act, but companies can voluntarily adopt additional codes of conduct.

Specific transparency risk: systems like chatbots must clearly inform users that they are interacting with a machine, while certain AI-generated content must be labelled as such.

High risk: high-risk AI systems such as AI-based medical software or AI systems used for recruitment must comply with strict requirements, including risk-mitigation systems, high-quality of data sets, clear user information, human oversight, etc.

Unacceptable risk: for example, AI systems that allow “social scoring” by governments or companies are considered a clear threat to people’s fundamental rights and are therefore banned.”

In what world is this compliant or even remotely ok? We don’t have to accept this dystopian future being forced on us, but all the warnings are just being blown right past. Hope you’re happy AI fanboys, the technology community better wake up before it’s too late, but I’m beginning to feel it already is…

Source material from FOI requests by Statewatch

I’ve been friends with Ash Hynie for over 10 years ever since we started hanging out in NYC working in QA engineering and developer advocacy. She has since gone on to greatness through various consulting gigs and heading up the DEI program at Credit Karma to now founding CountrPT, an AI powered career management platform.

Check out our chat about career development, performative DEI, being a good manager, and a lot about how to look out for yourself in today’s job market…enjoy!

On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? Timnit Gebru Emily Bender Culture is More Than a Mindset Agile Testing Days Ministry of Testing Angela Davis

A quick message for those struggling with the mudslide of artificial intelligence mania and feeling like they are missing out or question the validity of claims being made by AI fanboys.

It is entirely possible to view AI in software testing critically and not have some form of syndrome the AI fanboy club would suggest is wrong with you. In fact, it is absolutely a requirement of the testing community to look at any claim with scepticism, as IME that is a large part of our job as testers.

Yes, things feel like they are moving very fast right now but I can assure you, no serious person in the software testing industry is dismissing the impact or implications of AI in testing.  And for some practical advice to let you know you’re not crazy, here are some serious folks I am following closely on the frontlines: Dagmar Monett, Timnit Gebru, and Ed Zitron who recently wrote this in his great piece “The Phony Comforts of AI Optimism”:

“Criticism — skepticism — takes a certain degree of bravery, or at least it does so when you make fully-formed arguments.”

The current mood in software testing reminds me of a discussion I had 10 years ago with my friend Pradeep Soundararajan, when someone at the conference we were speaking at asked what it takes to improve testing in a company. “The “Kung Fu Panda” from Moolya raised his hand and said “One thing – Courage!“.

So please bear that in mind when you dive into the endlessly foaming morass of AI fandom right now. For years now, the testing business has abdicated one of its core risk management functions by litigating whether something “should” have been done loooooong after it’s too late. I’m not prepared to make that mistake again…

But for a little fun, here are some signs you might be an AI fanboy…

1. You attack anyone questioning AI as a self-motivated commercial protection racket when in fact YOU are selling AI bots or whatnot…

2. “Testing” in your worldview is the lowest value and most shallow form of testing that’s easiest to automate (but somehow hasn’t been replaced by #testautomation)…

3. You think anyone criticising AI in testing is only afraid of losing their job…

4. Safety, ethics, or harm are never brought into your discussions or quickly dismissed…

5. You think NO ONE critical of AI in testing is doing any serious work despite all evidence to the contrary…

Enjoy and keep testing…!

It’s getting wild out there, folks! I thought we had some way to go before the AI LinkedIn lunatics completely lost the plot, but I read a post today that left me feeling like John Oliver trying to describe the political climate, “Do you see that? Way up there? Way up there above the clouds? That’s ROCK BOTTOM! And we are currently way down here…”

I’m not going to dignify the click-bait by linking and drive traffic to it, but needless to say, the amount of unverified claims, (let alone the unverifiable ones) read like ChatGPT took a shot of hopium before ingesting a WinRunner user manual when being prompted with, “what would happen to software testing if magic was real?”

I had always suspected that everyone in technology “yearned” to get rid of testers, but we now have written proof we’ve finally been “obsoleted”. As I said before, “it is apparent now more than ever in my 20+ years of working in testing that our business has been completely highjacked by people who have never had to answer for or LIVE with the consequences of their bad quality decisions.”

No one who has actually tested software or complex systems would make statements like this. No one.

Experience has taught us that you’ll never go broke underestimating the amount of money people are willing to throw down the testing wish hole, but please, use some critical thinking before believing or buying this stuff. Even the most casual of research would render these “vague claims” unverifiable and “simplistic”, and the stakes these days are just too damn high.

It’s 2025, and I just had a conversation with a global financial services company where we discussed software testing in terms of <gasp> checking and testing, tacit and explicit knowledge, test automation in testing, and the “Weinberg-ian” (is that thing??) principle of human-centric testing…

Guess what? They loved it and said it put testing and what it can (and cannot do), in a perspective they’ve never had before. Go figure…

The point is, if your words work for you and help gain understanding with your testing customers, keep using them…if they don’t then find some new ones. As for me, I’m going to keep on with testing/checking, tacit/explicit, and all the Jerry-isms I can mine…enjoy!

“Learn from the mistakes of others. You can never live long enough to make them all yourself.” – Groucho Marx

A mistake I see a lot of new (and not so new) managers make is worrying about their team members looking for new work or actively/passively interviewing. IME people leave organisations mostly because of bad management (lack of integrity, abuse, no transparency, low trust), but it’s also really hard to know exactly why they are leaving.

Obviously someone actively looking for a new job is a pretty good heuristic that it’s time to review your operation, but it’s not always a bad thing for either you or the person considering leaving.

I have never been threatened by people testing the market or actively looking for new work, in fact I encourage it – here’s why:

– My people should know their worth if they are going to be happiest and productive in their work. If they are constantly wondering about their street value it’s best for them to go and find out and then we can either address it or they’ll know they are being compensated fairly.

– Having a high trust environment means your folks should be able to talk to you about anything – including leaving. Sometimes if I’m at an impasse trying to help someone through a problem at work, I’ll even offer it as possible solution and encourage them to look in the market.

– IME it’s a very small world and the software testing world is even smaller than most, so building that relationship and trust with someone has only ever helped me when hiring. Sometimes I’ve even helped people prep for interviews or prepare their portfolio of work, which again builds a level of trust that isn’t ordinarily found in the workplace.

I get why a lot of people don’t feel like I do about this and the adage holds true that “if you think it’s expensive to invest in people, wait until you have to hire new ones”, but I’ve had pretty low attrition in my teams over the years and I think this approach plays a part.

This is exactly the type of abuse of untested technology that is virtually impossible to unwind once it’s been deployed.

The knock on effect will be felt in communities and systems far beyond its intended use, because if you think this won’t target the most vulnerable and be expanded well beyond this initial brief, you haven’t been paying attention.

From the article: “The effort — which includes AI-assisted reviews of tens of thousands of student visa holders’ social media accounts — marks a dramatic escalation in the U.S. government’s policing of foreign nationals’ conduct and speech.”

Special thanks to all the AI in testing fanboy enablers who equate critical thinking and valid concerns about safety or harm with being a luddite. This is about to get a lot worse…

A couple of weeks ago on my podcast, I was talking to James Bach and we were discussing the language of testing and what I refer to as “word policing”. Well, I think I might have caused some confusion based on the feedback and comments I’ve been getting, so I wanted to state my view.

To be clear, I think imprecision in the language we use to talk about testing is a BIG problem in our industry, and I am completely aligned on this with James (and the CDT/RST community). Where we might have a potential difference in opinion is WHEN to tackle this problem with clients.

If you listen to what I said in the interview (and what I’ve been saying my entire career), it’s that I have not had good experiences with helping people do better testing by STARTING with how they talk about testing. I have found that the culture in some parts of the testing world, the influence of large commodity based testing vendors, and the rot caused by certification rent seekers makes that a very toxic place to begin working with people.

I am also a strong believer in meeting people where they are and having the semantic discussions to understand what people mean by they words they use – not attacking them for not knowing terms of art in a business that’s constantly evolving.

In my opinion, the culture wars in the testing world went into overdrive during the “Testing and Checking” public discussion and has had a knock on effect we’re still feeling today. In reality, (and if you actually read what was written) that debate did a great service to our industry, and personally gave me a new entry for speaking with non-testing folks about automation in testing and tools and how that relates to business risk.

Which brings me to my point about all of this, IME/O it is much more important (as the good Dr Feynman would say) to know “the difference between knowing the name of something and knowing something.” Precision in language and understanding what people mean by the words they use are both equally important when I work with clients, but knowing how to steer those conversations and the value of changing behaviour takes discernment – not brute force.

As my friend Michael Bolton often says, “good luck with your thing”!

As part of my effort to publish my back catalogue of recordings on the software testing business, I’ve added to the QR Podcast some old stuff including roundtables, conference talks, and guest spots on the odd testing show. It’s been pretty interesting listening to them and hearing how much some of my opinions/approaches have changed and what’s stayed the same. Enjoy!

Note: The quality of the audio on some of these was pretty sketchy, and I’ve tried to fix the sound and limit the background noise as much as possible as the content and participants are extremely good and worth a listen…

ISO 29119 Round Table

Here’s a roundtable discussion I hosted between veterans of the software testing industry (James Christie Michael Bolton, Iain McCowatt, Ilari Aegerter, Griffin Jones) on ISO Standard 29119. James gave his talk at the Association for Software Testing conference in 2014, which started a petition to stop the publication of ISO Standard 29119. Not great sound quality, but…enjoy!

STAR East – The Viability of Context Driven Testing

I gave this interview before my keynote at the 2016 STAR East conference on all the Lessons Learned Selling Software Testing. We also get into why context-driven testing is viable, as well as how to discern between wants and needs of your clients and testers…enjoy!

Screen Testing at TestBash Philly

Had a great time talking about testing and “Rocky” at TestBash Philly in 2017 with the Neil and Dan from Screen Testing! Yo Adrian…enjoy!

Let’s Test Interview

This is a long interview I did in 2014 with Duncan Nisbet for the Let’s Test Conference which was done while I was running the Global Test Center at Barclays. It captures a lot of the day to day of running that global operation and how we went about trying to balance the business demands with improving testing for everyone…enjoy! (Audio gets better about a minute into it…)

AST James Bach Interview

Driving While Driven: The Way of the Skilled Tester – In 2014 when I was the Executive Vice President of the Association for Software Testing, James Bach and I discussed what it means to be context driven, the role of skills in your development, and how you can become part of the CDT community. This is a long and wide ranging discussion about taking responsibility for your testing, the Context-Driven School of testing, and what it means for every tester to be their own methodologist…enjoy!

Testing in the Pub – Making Better Testers

Talking with Stephen and Dan in 2016 about how we can help testers and the testing community to improve and keep learning including promoting and educating people about better testing practices such as Context Driven Testing and the transition to better testing, particularly in the enterprise…enjoy!

As a regular consumer of everything “testing industry”, I’m always sceptical when something new comes out, but I’m really enjoying the Engineering Quality Podcast . I’ve known Alessandra Moreira for years and she’s an expert at this (Royalee Martin and Veronika Pliusnina co-host), so would recommend checking them out.

James Bach and I have had too many conversations about testing to remember in our over 20 years of working together, but we finally found some time to sit down and record one. If you’ve got the time (to listen or watch), in the 2+ hours we reminisce and discuss Rapid Software Testing, the testing industry, AI, context-driven testing, ethics, and training horses! Below are links to some of the different people and stuff we mentioned…enjoy!

Satisfice Rapid Software Testing Never Forgive Them Michael Bolton James Christie Pradeep Soundararajan Association for Software Testing BCS SIGiST Venkatesh Rao Agile Manifesto Enshittification AI Manic Syndrome EU Liability for Defective Products UK and US Refuse to Sign International AI Declaration SIBOS GDPR Consumer Financial Protection Bureau Amish Tech Support Voltaire Paul Holland ISTQB Jerry Weinberg The Secrets of Consulting QR Podcast – Jerry Weinberg SNL Barber Skit Impact of GenAI on Critical Thinking

As Accenture joins the growing list of tech companies (Meta, Google, Amazon) ditching their #DEI programs, we can see clearly (as the numbers have told us for YEARS) the grift it was the whole time.

As the gutless fold under the slightest provocation, the curtains draw back from the show they were putting on and reveal they never really cared in the first place.

Frankly, those programs never moved the needle in a meaningful way and remind me of something I wrote 10 years ago when they started kicking them off:

“I’ve sat on multiple senior executive boards discussing the progression towards our targets in a room comprised entirely of middle aged, white men. Worse than that, two and three layers deep into the org charts the demographics looked exactly the same – and no amount of target setting is going to change that fact.”

The good thing is, whatever they do organizationally doesn’t affect what WE can do personally, and my commitment is to continue to endorse, sponsor, mentor, promote, and hire from as diverse a community as I can find.

Rant over, now back to work…

“To be in opposition is not to be a nihilist. And there is no decent or charted way of making a living at it. It is something you are, and not something you do.” – Christopher Hitchens, Letters to a Young Contrarian, 2001

You’re not wrong, everything is getting worse…

Sigh…I’ve had my finger on the “publish” button for different versions of my initial thoughts on AI in testing for far too long, but this great post by Maaike Brinkhof finally pushed me to get it out there. This first one is context setting for a series I’m writing about AI, incorporating it into software testing, the dangers and opportunities, and what I see as a general failure of “duty of care” by our industry.

So much great writing and research has been done by Michael Bolton, Harry Collins, Dagmar Monett, (and too many others to mention) I can’t keep up with it all, so getting into the details of the technology and all the unverifiable claims being made doesn’t seem to have a point in my adding to that pile. Instead, I want to focus on the business risks associated with AI in testing, logical approaches as organisations go headfirst into the mist, and what I think are common sense ways to work with these new tools.

In my opinion, the pace at which people are trying to incorporate “artificial intelligence” into software testing is well past ludicrous speed and quickly approaching “plaid”. Even the halcyon days when the odd tool fetish was the only thing compromising testing seem mild compared to today’s rabid AI bell ringing of every vendor, tool jockey, supplier, and LinkedIn lunatic. This foaming at the mouth is despite the immense ethical and technological challenges AI presents for the testing business and an IT industry that is so far out in front of regulation, they don’t even seem to be playing the same sport.

The mask has been off the major tech players for some time as they disband their ethics teams and software testing vendors rush to build “solutions looking for problems” – all by design. I think it is apparent now more than ever in my 20+ years of working in testing that our business has been completely highjacked by people who have never had to answer for or LIVE with the consequences of their bad quality decisions.

I’ve been saying for years that despite all the “advances” in software testing technology, we and our machines aren’t getting better at testing – we are getting dumber and our expectations eroded by bad software and processes. I’ve long believed in the principle of “Enshittification”, but I think Edward Zitron put it better:  

“We, as people, have been trained to accept a kind of digital transience — an inherent knowledge that things will change at random, that the changes may suck, and that we will just have to accept them because that’s how the computer works…”

Well, I think with all this AI sand getting chucked into our eyes, our industry and the general public are about to get their comeuppance after decades of caretaking technology management (no moral hazard in what we unleash on customers), and the vast majority of people I speak to in this business are not prepared for it.

Leopard, meet face…

As someone who as worked in an industry that’s been undervalued, commoditised, downsized, automated, and abused by every wave of “do we finally get to fire all these testers” technology, I would be lying if I said the “they took our jobs” noise coming from some of the development community hasn’t put the occasional grin on my face. But I would take a bucket of salt with the stories that Meta, Salesforce, or whoever is actually trying to replace programmers with AI, as IME, all the hype is being used as an opportunity to cut costs, repeal remote working policies, and trim operational management more than some new wave of engineering resources. But right on trend in the testing world, what we’re presenting as “testing” and “AI” is not that valuable but can also accelerate bad outcomes and increase risk to your business.

It’s a little hard to keep up with the shifting sands of AI in testing right now as vendors frantically search for a viable product to sell, but I don’t see prompt engineering for test automation or anything else in the market right now as anything other than the lowest and least valuable fruit. Companies have been trying to get rid of humans in testing for decades, but outsourcing critical thinking to LLMs is about as closed a loop as you can get and run big transitional risks.

Most of the clients I speak to these days haven’t incorporated an AI element to their test approach and frankly, the distorted signal coming from our business hasn’t helped. What I’m hearing from clients are big concerns around data privacy and security, transparency on models and good evidence, and the ethical issues of using AI in testing.

That gives me some hope that people are thinking critically about AI in testing as with the recent roll back or cancellation of DEI programs, the least represented communities are traditionally the most harmed and need our highest duty of care.

Inside every cynical person is a disappointed idealist…

“If the rise of an all-powerful artificial intelligence is inevitable, well it stands to reason that when they take power, our digital overlords will punish those of us who did not help them get there. Ergo, I would like to be a helpful idiot. Like yourself.” – B Gilfoyle

I’ve spent a good part of my public career in testing talking about risk, how to communicate it to leadership, and what good testing contributes to that process in helping identify threats to your business. So I’m not here to tell you “no”, I’m here to talk about how we manage through the current mania and what I think are the big rocks we need to move to get there and as an inverse –Carlin, maybe find some idealism in my cynicism.

Testing is more than test cases, So as I patiently wait for the regulars to call me a Luddite (sigh), or I’ve only worked with juniors (womp, womp), or that my opinion is purely self-interest in keeping my “manual testing” job (the lack of creativity from some of these folks is astonishing), I’ll move on to the next one in this series: The Great Liberation Part II: The Search for the Real In Software Quality and Testing

Enjoy!

Love this on LinkedIn from Michael Bolton where he mentions my interview with Jerry Weinberg in response to the comment: Just because certain visible actions are a part of a process doesn’t mean that they ARE the process. Why is this misunderstanding so common?

His response is fantastic…

“Several years ago, on his Quality Remarks podcast, Keith Klain interviewed Jerry Weinberg. He asked Jerry why some people seemed eager to apply manufacturing models to software testing.

To your question, I’m going to give the reply that Jerry gave: “Because they’re not very smart.”

The people who keep making that mistake simply don’t understand the process. They haven’t even learned to observe and evaluate the visible parts of a process. They don’t consider the tacit knowledge or skills required to perform it.

The problem is compounded by the inability of many people performing the process to describe it accurately and articulately themselves. And THAT’s compounded by the fact that poor observers and evaluators often hire people who aren’t very good at performing or describing the work.

“And so we beat on, boats against the current, borne back ceaselessly into the past.” —F. Scott Fitzgerald

The good news is that people can learn the skills required to comprehend processes, given motivation, time, study, training, practice, and support.”

Here’s the interview with Jerry…enjoy!

My reaction when I see the TMMi / ISO29119 / testing certification / #yourenotdoingmymodelright crowd doubling down into AI after decades of getting software testing wrong…

Very happy to announce I’ve been honored to contribute a chapter to Taking Testing Seriously: The Rapid Software Testing Way by James Bach of Satisfice and Michael Bolton of Developsense, co-creators of Rapid Software Testing (RST) . Over the course of my career hiring and managing 1000s of testers around the world, RST has been the most valuable training as well as providing a useful language for speaking to management about testing and risk. The chapter outlines a discussion where Michael and I explore my experiences running global teams, the role of testing in managing risk, and what I think it takes to make it as a test manager.

You can pre-order the book for the testers (or anyone) in your life on Amazon now – just in time for the holidays! Enjoy!

Taking Testing Seriously: The Rapid Software Testing Way

“Dive into the world of expert software testing with Taking Testing Seriously: The Rapid Software Testing Approach. This book arms software professionals with the knowledge required to master the Rapid Software Testing (RST) methodology. Written by two co-creators of the RST approach and supplemented by material from respected testers who offer valuable insights, it is an essential read for anyone seeking excellence in the craft of testing.“

Ahhhh, it’s that familiar time of year when the smell of Performance Reviews and Goal Setting is in the air! I’ve managed loads of testers over the years and get asked to write a lot of feedback for folks. You’ll hear on LinkedIn the popular advice to take ownership for your career and depend on others as little as possible.

But unfortunately, most corporate performance management processes are fundamentally unfair and biased towards people who have great advocates in the system – people who are speaking about you when you are not in the room. Too often the power dynamics in performance management also tell people to advocate for themselves and then subject them to grading “curves” and corporate structures that aren’t very transparent for you to align you objectives.

My best relationships with people I am “managing” always start with openness and honesty about where we are and what I can do for you. It’s an important relationship and I take it very seriously as people are trusting you for that advocacy. Personally, for me to do my best work I need to know the folks I’m working with share values and principles that we’ll hold ourselves and each other accountable.

For all my Test Managers our there (or anyone responsible for a test team’s performance), here’s a model I’ve used for getting the best out of testers and a lens to look at your objectives/performance…enjoy!

Managing a Team of Testers

STOP: Thinking that the value of the test team is in anyone else’s hands and pretending “maturity” driven test metrics or TPI programmes will make improvements…

START: Telling the team exactly what’s expected of them supported by systematic training of testing skills, test reporting and business alignment…

CONTINUE: Driving out fear of failure by creating an environment that enables innovation and rewards collaboration through strategic objectives and constant feedback…

Performing on a Test Team

1) Honesty

☑ With ourselves and with each other – do not tolerate dishonesty…
☑ Transparency about confronting our strengths and weaknesses…
☑ Self-reflection…

2) Integrity

☑ Learn from mistakes to earn the right to have an opinion…
☑ Provide clear and constant feedback…
☑ Do not lower the bar…

3) Accountability

☑ Take ownership for getting things done – at all levels of the team…
☑ Understand “value” in your business…
☑ Manage your own expectations…

Integrity doesn’t matter until it does, and frequently testers conflate compromising on what to do with the information found during testing with a compromise on the integrity of the testing. Over the years I’ve seen this cause a LOT of burnout in testers, as particularly on high profile, politically charged projects it can feel like disagreeing with the outcome of a business decision on risk, defect resolution, etc is a personal attack on your role as a tester.

To be clear – I would never compromise on the quality or integrity of the test approach, execution, or reporting on risk, results, or advice on what to do with that information. As my good friend Michael Bolton has said, lying to my clients is not a service I provide, and there are loads of people in this industry who are more than happy to give (and read) “happy path” reports on passing tests.

As we hurdle towards AI driven testing, more than ever we need professional testers with experience in navigating complex systems and communicating risk. It may seem that there are more than enough bad actors willing to tell companies what they want to hear about software testing, but I’m confident (and IME) eventually they’ll come around to the realisation of the real fight – maintaining integrity in our test approach.

Thanks to the fantastic team at EuroSTAR for the great time and a really well run conference. I had a lot of fun catching up with old friends and meeting a lot of new, enthusiastic testers. It was great to be back in beautiful Stockholm and hope to see you all again soon!

Links to the reference material I spoke about in my keynote can be found HERE: (Rethinking Expertise; How Complex Systems Fail; The Secrets of Consulting)

* Special thanks to my friend Michael Bolton for pulling me out of my conference retirement…the trick worked pal 🙂

The Center Left: Testing Software in the Age of Transformation

Ten years ago, I gave a keynote at EuroSTAR on how to overcome organisational bias against the value of software testing. Despite all the advancement in process and technology, our business still struggles with its value proposition and sense of itself.

Through this talk I’ll discuss the principles and practices I’ve employed to successfully sell testing services and manage high performing teams. I’ll also walk through case studies of what has worked when talking about testing and what it can and cannot do for your business.

What you will learn

1. What worked & what didn’t when trying to get software testing valued at an organization, and how to fit actionable plans into enterprise transformation
2. How to help make testing relevant to people by meeting them where they are, & speaking about our business in a way that doesn’t comprise integrity, while still moving things forward
3. Principles and practices, and their relevant sources that have helped me gain trust and respect from my clients in delivering critical high risk programmes

In another failed chapter in the never ending book of “encouraging good behaviour”, the G7 have apparently agreed to a “code of conduct” for companies building #artificialintelligence systems. Per Reuters, “the voluntary code of conduct will set a landmark for how major countries govern AI, amid privacy concerns and security risks…”, now colour me cynical, but I think we’ve seen how this movie has played out before. You don’t need to expend any energy finding the billions being invested for new generative AI and other AI systems, which is only piled on the billions ALREADY spent on systems actively in use.

At least the EU is giving the appearance of pretending to govern the use of AI on the public unlike their US and Asian counterparts who “have taken a more hands-off approach than the bloc to boost economic growth.” Good grief!

You can read the code of conduct for yourself and I plan to take a closer look, but here are some thoughts at first pass.

The code “urges companies to take appropriate measures to identify, evaluate and mitigate risks across the AI lifecycle” which seems to all happen after the fact. “Organizations should use, as and when appropriate commensurate to the level of risk, AI systems as intended and monitor for vulnerabilities, incidents, emerging risks and misuse after deployment“.

If using the word “risk” is a drinking game, don’t drive after reading this: The risk management policies should be developed in accordance with a risk based approach and apply a risk management framework across the AI lifecycle as appropriate and relevant, to address the range of risks associated with AI systems, and policies should also be regularly updated. What. Does. That. Even. Mean…? Honestly…

Finally, the code seems to just trail off at the end for a subject that probably requires it’s own code: data quality and bias. “Organizations are encouraged to take appropriate measures to manage data quality, including training data and data collection, to mitigate against harmful biases.” IME we haven’t even come close to cleaning up or preventing bias in large data sets and seeing as some pretty big names decimated their #techethics teams I think all the encouraging in the world isn’t going to make much of a difference.

Once again, public governance of critical systems and emergent technology is woefully behind industry which is already over their skis. As we continue to “ride the insane horse towards the burning stable” of using AI, I think it’s well past the time the software testing and quality engineering community get its act together and quit fighting over how to make AI a better unskilled tester…

Hiroshima Process International Code of Conduct for Organizations Developing Advanced AI can be downloaded HERE

After a long career of reviewing the various outputs from #softwaretesting: strategies, plans, test cases, #testautomation, etc. I’ve never understood why people think conformance to internal/external standards will affect a better outcome. I realise it’s born out of a fundamental misunderstanding of what happens when you test something paired with strong wish thinking that software testing is analogous to manufacturing. But IME, content is key for #testing artefacts and frankly, I’ve been around long enough to see how all these standards usually give a false sense of security and year after year add to the “#softwarequalitymanagement” certification grift.

So I wasn’t that surprised to witness another leg of the race to the #artificialintelligence bottom in testing with talk of creating #AI to check your test strategies, etc for deviations from your internal or external standards. Apparently, all this investment in #ML and #AI is going to be used as some really, really, expensive rube goldberg machine to #automate the lowest value work in testing!

Some day we’ll get something useful for testing from artificial intelligence, but today is not that day…

(old man shouts at cloud rant over 😉)

“Inside every cynical person, there is a disappointed idealist.” – George Carlin

Had a great time talking about business risk with the Ministry of Testing folks and how your approach to testing is probably introducing risk to your business. Check out the whole interview or clips on my YouTube channel…

Enjoy!

Had a lot of fun ranting at my friend 🐞 Richard Bradshaw on my Ministry of Testing Ask Me Anything. Hope you enjoy it and LMK if you every want to talk software testing and risk management.

Check it out here: Keith Klain AMA on Risk Management

“Quality is value to some person”. – Jerry Weinberg

Saw this quote misattributed recently, but Jerry Weinberg threw down the gauntlet in his classic book Quality Software Management: Volume 1, Systems Thinking, and the software testing industry has been wrestling with every word in that short sentence ever since. What is “quality”? How do you determine “value”? Who is that “person”?

Over the years those words have been misquoted, modified, and added to multiple times, but I can’t ever remember adopting any of the new language. There have been many attempts to define quality, mostly trying to wrestle software development into manufacturing models and flow charts which IMO, have added nothing to the public debate.
 
The other canard that gets whipped out frequently by <cough, cough> consultants trying to sell you something, is that software quality is measured by the absence or presence of defects. There’s been loads written and said about the impossibility of defect free software, so my only advice when your hear someone start banging on about that is to watch your wallet!
 
In my business, enterprise tech, the words Jerry chose are loaded with dependencies on context. Sometimes quality means “can we trade?”. Sometimes “value” is an operational cost save we’re trying to meet. Sometimes the “person” is a regulator. But EVERY time we start a project we need to have the semantic discussion of what we actually mean when we are using those words!
 
I might just be getting cynical in my old age, but I think our business could do with a lot of unwinding the “why make it simple when it can be complicated” approach to software quality. I use those word to clarify my approach to testing, to understand what people mean when they talk about software quality, and as a heuristic for ensuring I’m getting as close as I can to delivering the value my testing “client” wants.
 
Might be simple, but it works.
 
As a bonus, I had the honour of interviewing Jerry a few years ago and I think it’s worth a re-listen every once in a while…enjoy!

Last year I had a long conversation with my good friend Michael Bolton for a video he was producing on Digital Transformation for the TiD Conference in Beijing, China in 2022. The whole video is great with an amazing segment with Harry Collins, but my parts are at below…enjoy!

Inside every cynical person, there is a disappointed idealist. – George Carlin

Eric Schmidt just saying the quiet part out loud now…

Last year I wrote about regulators just throwing up their hands at how outpaced they are by industry as evidenced at #SIBOS and the CyberSecurityCloud Expo, but this is a whole new level of hubris. “There’s no way a non-industry person can understand what is possible…there’s no one in the government who can get it right, but the industry can roughly get it right.”

Wow.

Think about all the harm that will be done in that trade-off to “roughly” get it right in this race to put “#AI” into everything. These folks are bordering on out of control and openly admit their intent to run roughshod over the public good so the government can clean up their mess afterwards with “a regulatory structure around it”…

The #softwarequalitymanagement industry better wake up fast and push back against this as it’s probably too late to change course, but warnings should be issued…

#qualityengineering #qualityassurance #softwaretesting #testmanagement #techethics #ethics #artificialintelligence

“When the tide goes out, you see who has been swimming naked.” – Warren Buffet

I’ve been reviewing a lot of material for a larger piece I’m writing on the use of #chatgpt and various other “#artificialintelligence” tools in #softwaretesting and I have to say one thing as a preview – the state of what the #testing community views as testing is amazingly poor.

Apparently, to most of the testing vendors I see writing about AI in testing, it’s the equivalent of poking a robot with a stick. Testing is nothing more that test cases, test plans, and reports you can dump into #agile, I mean #jira. People complain about the #quality of #software but then expect nothing more from the people charged with testing than the most shallow, basic checks.

In fact, a testing vendors approach to #ai in testing is now a great heuristic of what you’ll get from them in regards to quality of testing. “We can now automate #exploratorytesting!” “A test strategy can be generated from AI!” “All codeless/scriptless test automation can be done via AI!”

Hear anything like that from a vendor and there’s a strong chance that their view of quality and testing is so basic you’ll spend more money unwinding the contract than any fabricated time saving estimates from all their “#testautomation“.

The numbing effect of decades of crappy software has really taken its toll on people about what is acceptable quality, and the reality is, our expectations have been worn down as opposed to vendors raising their game. And in the end as industry sprints far out ahead of the people charged with protecting the public – real people will likely bear the brunt of this ethical malpractice.

The latest “AI” and #machinelearning noise has only aided the tide rolling out and as the grifters surf the wave, we can see clearly who hopes we won’t notice they aren’t wearing any trunks…

GPT-4 saw all that he had made, and it was very good. And there were no more evenings, and there were no more mornings – just endless days…

“GPT-4 can take a picture of napkin mockup as an input and output a fully functional website (HTML/CSS/JS)”

I worked with a client recently who was frustrated by all the “manual” interventions required to run their “100% fully automated” test suite. Irony aside, as an industry we really need to rethink the amount of sales nonsense we let dominate the public conversations around the value proposition of automation in testing. Terminology in our business is usually polarised and emotionally charged, but I’ve always believed that semantical arguments are worth having even if we can’t agree what to call stuff. Examples like this are just a symptom of a greater problem we have of letting vendors and <shudder> consultants sell us “fully-100%-automated-defect-predictonators” (said in your best Dr. Doofenshmirtz voice) for the last couple decades. We can do better and as well, when these ideas get into the business case, we are just undermining ourselves anyway…

I’m doing some work on “documenting” my approach to reviewing software testing operations and I keep coming back to a couple resources. I forget sometimes how much Griffin Jones talk on “What is Good Evidence” and amongst several of his works, James Christies post “Not “right”, but as good as I can do” have influenced my work. They are both brilliant thinkers and contributed a lot to the testing profession…enjoy!

Just wanted to send the quality engineering and software testing communities a short holiday note of appreciation for all the work you do.

Testing software is hard. Very hard. And not unlike plumbers, your effort is frequently not appreciated when things are working and first to be criticized when things go wrong.

Years ago I wrote a post trying to define why I like the business of software testing and specifically working with testers and why our work is so difficult.

“Testers spend their days trying to figure out what “might” go wrong by looking for ways a product is already broken – staring into the cosmic abyss of the impossibility of complete testing for all of us takes it toll. All the while competing in an industry teeming with unenlightened vendors, consultants and “experts” undermining their own value proposition by selling “bug free” methodologies, certified super-tester training programs and “automated algorithmic defect predictonators”.”

It doesn’t have to be like this, but it is, and there are lots of us fighting every day to make life better for testers while you make our systems safer, more reliable, and more equitable for the people who use them.

So from me, to you, thank you for all your hard work and know that I see it and appreciate it, even if a lot the world always doesn’t.

Wishing you a happy holiday season and great New Year.

Cheers!

Can’t love this enough! Many thanks to Damian Synadinos for the amazing artwork he does at I Draw Strangers. His drawing perfectly captured my “not you again with the same bullshit” look, so check him out and give him a commission to counter all this AI art crap! Cheers!

I’m working on a larger piece on #artificalintelligence, testing and ethics, but I had to say something about the really depressing panel on #ethics and #AI I attended at the Cyber Security and Cloud Exp in London today. The conference in general (Cyber Security & Cloud Expo | Technology Conference | London (cybersecuritycloudexpo.com) was a great event and really well done, but I heard two comments from the “Keeping it Ethical in AI” panel that left my mouth hanging open.

The first comment was about the repercussions for companies that are found to be operating unethically with their #artificalintelligence systems which paraphrased amounts to, “it’s better to engage in dialogue about doing better rather than naming and shaming companies”. Eh? These “watchdog” groups have basically no teeth to begin with and the only real one they have is the ability to inform and drive public opinion about bad actors. What else are they going to do?

The second comment really blew me away, at the breath of both the naiveite of where we actually ARE as an industry with AI usage, and the carelessness of the forward-looking view. “Technology advancement should not be inhibited by our inability to keep up with it ethically…” Wow. And this was a panel of people tasked with ensuring AI is used ethically!

There are very serious people like Timnit Gebru and Harry Collins waving red flags around AI about algorithmic bias and the limitations of our technology, and now is not the time to be capitulating before we’ve even started. The real concern here is that the AI horse has already left the ethical barn and these folks are describing the grass! The use of AI in fintech is forecast to grow from ~$14.7b to ~$42b by 2027, so let’s not pretend it’s not already pervasive in our lives.

Anyway, rant over, but if this the level of thinking with the people tasked with looking out for our ethical interests, we might be worse off than I initially though…

Love this video of Richard Feynman explaining the scientific method…loads of stuff in there for #softwaretesting – Enjoy!

Time to stop blaming everyone else for the ethical problems in tech and refuse to build things that cause harm BEFORE they impact people…

The secret of life is honesty and fair dealing. If you can fake that, you’ve got it made. – Groucho Marx

The hits just keep coming these days, with new tech sector layoffs being announced seemingly on a daily basis. And as we limp into 2023, there are rumors that this is more than just a reaction to the market, but a new era being ushered in for austerity measures for hiring in tech. In light of this, I wanted to get something out to folks on interviewing tips to accompany my advice on how to stand out as a tester during economic downturns. For a more in depth analysis on managing your career in testing, I would point you to stellar works from Benjamin Kelly on and David Greenlees as well what I consider as essential reading on succeeding in tech, Secrets of Consulting by Gerald Weinberg.

I’ve interviewed a LOT in my career as either consultant or candidate looking for job or as a hiring manager. The recruitment process in enterprise tech is fundamentally broken, but I do think there are ways to differentiate yourself and on reflection, there are things I look for that pique my interest in a candidate. Here are my top tips to stand out from the crowd.

Interview Tip: Answer the question – THEN tell your story…

I don’t know if it’s nerves or autopilot kicking in, but I frequently find myself asking interviewees to pause, take a breath and actually answer the question I asked them. There is nothing wrong (and IMO a good habit) with taking 30 seconds to think about the question before giving your answer. There will be plenty of time to support your answer with examples, but make sure you understand what the interviewer is asking and are hitting the nail on the head FIRST before adding details that may not be relevant.

Interview Tip: Understand the commercial implications of testing…

Software testing is part of the business, and that business has implications for operational costs and controls – some we can influence and some we cannot. I am always impressed when a tester demonstrates a commercial awareness of the costs of testing (or not testing) and can speak to real world examples of the cost of quality.

Interview Tip: Relate what you’ve done to a risk the company has faced…

Want to impress someone recruiting testers? Give them an example of how you would test one of their systems or even better, when it’s time for your questions, ask them “How did you test system XYZ, because looking into it, it must have been difficult to simulate…” You get the idea.

I’ve spoken about this for years, but you can also pull the companies financial statements and review for risks you could test around. It shows an initiative that is rare and also demonstrates you get business risk and not just project risk.

Interview Tip: Have a view on the testing industry…

Finally, I always ask testers how they keep up to date on the latest techniques, tools, and thinking about software testing and let me tell you, the answers are usually pretty grim. Our business is relatively young and most of what you’re going to learn about it is through blogs, conferences, and networking. IMO it’s vital to have a good list of resources (mine is here) to draw on and speak to in order to stay current.

These tips won’t always land you the job, but as an experienced interviewer/interviewee they’ve helped me throughout the years and hopefully will give you some ideas on how to get noticed. Enjoy!

In light of the current state of hiring in tech right now, this talk seems pretty relevant again and hopefully sheds some light on how test teams are often reviewed for performance…enjoy!

“Over the course of my career I’ve reviewed the performance of countless software testing organizations, test teams, and testers looking for ways we can improve. Typically, the first suggestion when asked “how can we improve the state of testing here”, usually relates to something that OTHER people should do. Very few people or teams take an introspective approach to improvement based on their own values and principles. This talk is the review model and heuristics I use to identify things that are working and areas for improvement including efficiency, process improvement, and aligning your test approach for relevance to your business.”

In Daniel Kwan and Daniel Scheinert’s brilliant film, “Everything, Everywhere, All at Once“, as Michelle Yeoh’s conciseness jumps universes, she’s confronted by a version of her husband who tells her, “When I choose to see the good side of things, I’m not being naive. It is strategic and necessary. It’s how I learned to survive through everything.” As someone who’s career has focussed on software and system quality, I couldn’t help but feel something similar when returning from my trip to Amsterdam last month to attend SIBO

This was my first time attending SIBOS, and for those of you who aren’t familiar with the conference, it’s been THE financial services event since the 70’s and attended by every firm you could think of in banking and #FinTech. If there’s a multiverse, SIBOS is the white-hot core of where we are now and a preview of where we’re heading, and if 2022’s event was any indication, things are getting more complicated at an alarmingly faster pace and hardly anyone is thinking about how any of this is going to get tested.

I attended multiple sessions on how artificial intelligence, crypto, CBDC, and open banking are driving new and interconnected services, and a few even mentioned the “plumbing” for this to happen. Disappointingly, I didn’t hear any talk about stability, quality, or the risks of trying make all this work on existing infrastructure let alone how all this would impact society (other than how great it was all going to be…).

To be fair, I did watch a brilliant panel discussion on “The Future of Money” moderated by Leda Glyptis featuring Eileen Burbridge, Liz Lumley, and Matthiew Favas that was deeply grounded in reality. The panel spoke at length about the need for compassion and inclusion in financial services and the ethical implications of all the change being foisted upon the multiverse.

Unfortunately, that sentiment was not shared throughout the rest of the conference, and I sensed a detachment from reality from the financial environment we are in and the implications for the future. Software and system quality need to be front and centre in these plans and the ethical concerns must be considered BEFORE we start building the “solutions”. My concern, based on my experience, is that when projects face financial headwinds the first thing they cut is testing at the expense of quality and to the detriment to the most vulnerable in our communities.

A real-time example of that financial trade off in the face of difficulty appeared in every talk I attended on ESG referencing the recent decline in investment in environmental, social, and corporate governance funds. There are loads of challenges facing ESG around standardisation, “greenwashing” and accelerating the rate of change to keep pace with climate and societal pressures, but we cannot a deprioritise investment in those initiatives in the face of tightening budgets. My fear is that as these economic realities hit FinTechs, they won’t alter or delay their plans but plough ahead and, in some cases, double down on scope and timelines before the money dries up.

From experience, I can tell you that integration of new technology into legacy banking systems is a frequently an underestimated task and often decommissioning never happens causing complexity risk to increase. Open banking can also carry high risks of fraud, money laundering, and other types of financial crime and have to be considered when testing working in an open API environment. Sharing personal data and banking records, although rightly considered by the standards being implemented, adds to the functional and security requirements when connecting these systems in the real world beyond difficult to model test environments.

Software testing is hard, and releasing quality systems is even more difficult, but the consequences of getting it wrong these days have never been greater. Software is pervasive in our lives and only increasing in depth and breadth and innovation and efficiency are essential to democratizing finance and a building a more just society. It is long past the time that the quality and ethical implications of the systems we build take priority over the functional possibility of constructing everything, everywhere, all at once.

Think the state of the software testing industry has improved much? Guess what year this slide was presented about the “challenges” facing testing compared to the 2021 “State of Testing” report. History may not repeat itself, but when it comes to our business, it sure does like to rhyme… 🙃

They are stressful, usually unnecessary, and IME always the fault of bad leadership, but from a quick read of the news and what’s happening in the tech sector it, it looks like layoffs are back on the menu. In my over 25 years of managing software testing in enterprise tech, I’ve experienced layoffs in every single one of them. For some folks in tech companies, this may be your first time living through this, and testers are usually the first to go, so here are my quick tips for increasing your value to your business to avoid the cut.

1) Be able to demonstrate your test approach ALIGNMENT to business risk…

Testing that is aligned to business risk (not just project risk) supports more than just technology delivery and is much more valuable and can drive innovation or identify opportunities for investment. Do you know your business strategy? (Have you read it or even know where it is?) Does your test approach address systemic BUSINESS risk, or does it INJECT risk into our business by only focusing on project delivery? Repurposing test management into a strategic role beyond just functional management of testers should answer questions from your CFO about the cost of quality or defects in production. “Managers of managers” are the first thing I look for when optimizing a test team and in a reduction of force, are usually the first to go.

2) Tie efficiencies you’ve made DIRECTLY to operational savings…

Operational budgets are being squeezed right now and testing investment decisions need to be targeted around gaps in coverage related to usage, monitoring, and risk. Headcount is the only currency that management trade in, so look for duplication of effort or redundancy in the test pipelines and be able to show actual savings (not the usual test automation ROI metrics). Non-prioritized test coverage, large regressive test automation, and process inefficiency (meetings, reporting, automation, test management) are target rich environments to cut the delta between what we need and what we are getting and show real cost improvement.

3) Report on INSIGHTS not just data…

The information gathered during testing can and should be focused on business intelligence that can be used beyond release decisions, but too many testers still just regurgitate test case centric reporting. Testing reveals loads of detailed, in-depth information about delivery processes, waste, your business operations, and how you allocate capital, and all you have to do is OBSERVE this information and report on it! Unironically in a downturn, disruptive management and business decisions get made with “out of focus” views on an organization’s delivery capabilities and expectations – testing can help fix that! Test reporting that includes operational insights/intelligence as part of an overall approach to risk management greatly increases the value of your test team as well as reducing mismanagement of expectations.

Unfortunately, in the testing business, we are still subjected to a lot of bad ideas about our craft, its value, and the operational costs of managing quality. I empathize with the people impacted during this industry-wide series of layoffs, but hopefully these tips can help you articulate your value to your organization.

What do Amazon, Apple, Google, Meta, Twitter, Lyft, Stripe, (just to name a few) all have in common? Layoffs.

I’ve worked in enterprise tech for over <coughs> 25 years, and experienced layoffs in every single one of them, and they are always due to poor org structure and over optimistic growth projections. What’s going on with Twitter and the broader tech sector right now has been coming, IMO for at least the last decade (if not longer) – and the blame falls solely on leadership. But as always happens, employees will bear the cost of those bad management decisions at the price of their jobs and security…

IME in enterprise tech, “nice to have” roles, e.g. non-directly revenue attributing roles (advocates, process oriented, governance, etc.) are a heuristic for an org slipping off its revenue mission and will be the first ones to go. You don’t have to like it or agree with me, but I’ve seen it time and time again, and poor org design does not protect these important jobs (and the people who do them) from the firing line.

And to add Twitter insult to injury, in a real life “frog and scorpion” moment, Jack Dorsey clearly knew something bad was going to happen if Elon got his hands on the place, and was warned off by the board. To put a fine point on it: the layoffs at Twitter right now are as much Jack Dorsey and the previous leadership teams fault as they are Elon Musks. (Texts between Jack Dorsey and Elon Musk via Twitter vs Elon Musk, 2022)

What’s happening with Twitter right now has been happening in enterprise tech for as long as I’ve been around and it’s not the fault of the employees – its the result of years of bad management, poor org design, and exuberant hiring that fuels a “boom and bust” operating model. As usual, PE/VC and the founders have already made off with their money and the people doing with work get left holding the bag, so can we PLEASE mark this as the final tombstone for “tech-bro” worship?

When someone in the software testing business is warning against over-reliance on test automation, metrics or AI for assessing system or software quality, that doesn’t mean they are against automation in testing or to be treated like a luddite. Maybe….just maybe….you should think about it the same way that people like Timnit Gebru and Harry Collins warn against artificial intelligence – as people with deep experience who see ethical and systemic risk beyond the superficial and immediate returns. #justsaying

Really excited to be attending SIBOS next week in Amsterdam! Come meet me at KPMG UK booth F15 to talk about #qualityengineering and #softwaretesting in the world of #digitalpayments and #fintech. Hope to see you there!

I had the pleasure of discussing all things software testing and how quality engineering can support and accelerate digital transformation with Kevin Craine on his Digital Transformation Podcast you can check out HERE. We covered a lot of topics from test automation to strategies to deal with disruption, so check out the interview and enjoy!

Feedspot – 35 Best Digital Transformation Podcasts

Player FM – Best Digital Transformation Podcasts

Top 8 digital business podcasts to follow in 2022

“Performance measurement – the most powerful inhibitor to quality and productivity in the Western world” – Robert D. Austin

I’ve reviewed a lot of test teams in the Enterprise Tech world, and I get asked a lot how I go about looking for problems or how to identify opportunities to get more value out of your approach. The following is an attempt to sketch out the model I use to frame those reviews and look for areas for deeper investigation. All models are wrong, but I’ve found this one to be useful, so hopefully you can get some benefit from it too. But first some context…

“No amount of process improvement is going to solve your underlying problem: org dysfunction”

To paraphrase Jerry Weinberg’s Secret of Consulting, “there is always a problem and the problem is always people”. People are the biggest contributor to your context and guess what, they don’t always do what management TELLS them to do, but they frequently do what management SHOWS them what to do. The greatest determining factor in the success of your test approach is how people are comped. People do what they “perceive” the are being compensated for (salary, bonuses, attention, perks, etc.), so my advice if you want to improve your test approach, is start with why tester think they’re getting comped and look for misalignment to your strategy.

People

I always like to start my reviews trying to understand how the testing team or testers approach their work and how deeply they think about it. Behaviors that contribute to a general malaise in your testing team are orgs constantly questioning the value of the testing. The symptoms of teams who have a shallow view of their value are common in outsourced/offshore testing teams (or the dreaded, Testing Center of Excellence!).

Questions I like dig into with teams that have a perceived low value from their business:

• Can they answer questions about the business/management?
• Do they only think about what’s directly in front of them – apps/tech?
• Can they model their user base – demonstrate any interactional expertise?
• Do they study the testing industry? Trends, conferences, social media – how are they learning their craft?

These teams will be doing the job but just, and typically suffer from all the classics – slow, inefficient, no innovation, lots of buzzword bingo but don’t really understand the business.

Process

In my experience, testing teams that exhibit issues with their process are typically high on fear and low on trust. Unfortunately, I see these symptoms deeply rooted in philosophy and for an organization, they are frequently terminal. Heavy use of FBM (Fear Based Management) techniques like metric scorecards, process flows, swim lane diagrams are prevalent in these operations. Despite all the advances in our ability to understand manage knowledge work, every time I think this approach is dead – I find another company doing it! Hallmarks of testing operations (other than having really new or really old testers) that have trust issues include:

• Quality police promoting a QA/QC divide
• Test management offices who “manage the managers” or just color in charts/graphs
• Negative reactions to agile transitions for “testing services”
• Prevalence of “wish thinking” – wanting testing to be just about control/confidence

Tester that are obsessed with counting things or have an over reliance on numbers who can’t tell a story through their reporting typically treat testing as a role and not an activity, which can lead to very low trust and value outside of the team.

Technology

The last area I like to dig into to get a feel for the value proposition of a test team is around their use of technology as it relates to their test approach. I’ve long held that there is a segment of the professional testing population that don’t actually like testing! This one can be hard to fish out as testers who don’t like testing are often super enthusiastic about what they do. They often call themselves “embedded testers”, “SDET”s or ”automated testers”. This problem is very common these days due to the current coding obsessed iteration of “let’s get rid of all our testers”, but primarily because great testing is REALLY hard but fake TESTING is really easy. Signs you might have a problem with your test teams approach to technology include:

• Tool fetishes or tool first approach which views testing as a strictly technological problem to solve
• Your testers can’t defend their work or explain the mission of their tests
• Their only approach to deciding what to automated is asking “can” and not “why”

Another good sign your testers might have a very shallow view of quality and testing is a dislike for SEMANTIC arguments, or as Dr. Feynman would call it, the “pleasure of finding things out”. IMO semantic discussions are NOT about micro word corrections but coming to an agreed understanding about what we MEAN with the words we use. Hopefully this gives you some behaviors to look for and some questions to ask to try to unlock more value from your test team and approach. Happy hunting!

“One can’t paint New York as it is, but rather as it is felt.” – Georgia O’Keeffe

The last installment in my series on how testing and quality engineering can support a digital transformation program is about one of my favorite subjects – managing expectations. But is it possible to improve through expectation management? Yes. In my experience through alignment of our ambitions, business capabilities and operating model limitations you can absolutely improve not only the internal/external perception of our business but also your delivery capabilities.

Why is this expectation management such easy target for transformation? First, in my experience there is frequently misalignment between stakeholder perceptions between each other as well as their IT partners in expectations and reality. Very often the business, IT management, and those actually doing the work will have multiple views on what problems are, root causes, and their opinions will be heavily anecdotal in nature and biased. This can cause disruptive management and business decisions made with “out of focus” views on organizations delivery capabilities and expectations.

“Let us accept truth, even when it surprises us and alters our views.”  George Sand

So how can testing help manage expectations?

Testing should help an organization identify if their problems are complicated or complex. Complex problems involve too many unknowns and relationships to reduce to rules and processes and typically, quality is equated to testing and a management problem. In my experience, because of its complexity, software quality can only be managed, not solved and testers should be providing information and insight into business (not project) risks associated with an organizations delivery and engineering capabilities.

We should also be providing clarity on what testing can and CANNOT do and how automation and tooling fit into our approach. Without opening the pandoras box of “testing and checking”, I think from a professional testers perspective or over 20 years in this industry, the discussion on the difference between a “test” and a “check” is incredibly useful. I have employed it on countless occasions to help IT management and my business partners understand the deltas between how we build, test, and use products as in those expectation gaps defects and systemic failures lurk.

One more area to looking into to help your partners manage their expectations is culture and people. Models of success are most likely emulated (money, time, attention, promotion) including bad behaviors like heroics, waste of company assets, and poor risk prioritization/management. How teams are compensated drive most behaviors, so don’t be surprised with you get what you paid for your big, expensive “test process improvement” program!

Testing is typically subjected to a great deal of organizational dysfunction (cost controls, location strategies, 3^rd party risks) and has a direct view of the impact to how we build/test/operate our systems. Reporting this information as operational insights/intelligence as part of an overall approach to risk management greatly increases the value of our test team as well as reducing mismanagement of expectations.

Good luck and happy hunting!

“What we find is that if you have a goal that is very, very far out, and you approach it in little steps, you start to get there faster. Your mind opens up to the possibilities.” Dr. Mae Jemison

Every business wants to be more efficient, but what does that mean in the context of your approach to quality engineering and digital transformation program? The desire for change is greater than ever, with over 80% of companies KPMG recently polled stating they have large transformation programs in place but lack confidence they have the operating model in place to support the required changes. And those transformation plans are only being accelerated since the pandemic! Inefficiencies within and between business operating models or products are great sources to capitalize on unrealized areas for collaboration, reuse, process improvement or reduction in redundancy.

But as testers or those responsible for managing the approach to testing, what and where should we be looking for to increase our value to our business?

Some examples of lenses to view your work that are continually on the optimization radar for enterprise IT are speed to market and your product delivery models. Getting products and services out to your customers at speed through integrated product delivery pipelines and increased automation through increased transparency on risks can help drive out inefficiencies. So how can testing inform the business about opportunities for optimization transformation as well as capital allocation and investment in technology?

Redundancy or Inefficiency? (Dr. heal thyself…)

The first way that testers can look to their work to increase efficiency is at unneeded overlapping effort for similar tasks that are unnecessary or non-productive. Far too often I have reviewed quality engineering or testing approaches that rely heavily on built in redundancy primarily through unprioritized test coverage and large regressive test automation. Paired with the compliment of meetings, reporting, and test management,  the delta between what we need and what we are getting is my go to for my first “target rich” environment for inefficiency.

Loads has been written about systems thinking or seeing “the forest for the trees”,  but in my opinion, “How Complex Systems Fail” by Richard Cook should be required reading for anyone responsible for managing testing or delivery in enterprise IT. Cooks view that systemic failure requires multiple smaller failures and complex systems contain loads of “known or unknown” latent failures by design. Testers should use system thinking and complexity models to look at risks and opportunities through their knowledge of the F2B flows, customer insights, and risk to remove redundancy and waste from their test approach.

Org Dysfunction – Identification and prevention

One of my favorite books on the sources and effects of org dysfunction is “Managing and Measuring Performance in Organizations” by Robert Austin. In his book, Austin talks a lot about applying targets to measures and the effect that has on organizations and how models or processes that are inappropriately targeted that drive organizational dysfunction. In my experience, most testing measures and metrics drive dysfunctional behaviors in teams and yet we continue to utilize them: test case counts, pass/fail ratios, test case efficiency. Along with agile velocity, invalid measures undermine our relationships with customers and our business when we “over promise” or use inaccurate understanding of IT processes to make commitments. The insights provided through testing should increase clarity in our delivery capability, not muddy the waters or create data driven distractions.

These are just a couple examples of how your approach to quality engineering and testing can improve or complicate the optimization of your digital transformation program. Testing reveals loads of detailed, in-depth information about delivery processes, waste, your business operations, and how you allocate capital, and all you have to do is OBSERVE this information and report on it!

“Testing sits at the heart of the processes required to enable and fast-track business change, so the information it generates can be used to shift how an organisation operates, help redefine its value proposition, or change how it competes in the market. Testing specifically, is closely aligned to business strategy and supports more than just technology delivery. The management information (MI) produced by testing can also be leveraged to drive innovation and identify opportunities for both risk reduction and investment.” Read full article HERE

How software testing can support business and digital transformation?#Softwaretesting can be used to support #business and #digitaltransformation – so, could this be the panacea for improving the success rate of transformation initiatives? #testing https://t.co/r66UVNAZht

— TEST Magazine (@testmagazine) March 7, 2022

It’s been a while, but it sure felt great to get in front of a live audience again and interact with folks at the National Software Testing Conference this week! I had a great time talking about the role of testing in supporting and accelerating business change though digital transformation and disruption. Along with my keynote, the KPMG team presented a talk on the role of test automation in your SaaS pipeline and a workshop on testing challenges in FinTech. It’s a great group of people to work with, and I was particularly proud of my colleague Abdulla Mohammad finally receiving his award for “Test Manager of the Year” from the European Software Testing Awards. I’m working with Abdulla on some machine learning tools for coverage and risk reporting and can tell you, he only has greater things ahead of him!

Aside from being hosted at the amazing location of the British Museum, we got to meet some fantastic testers and see some really interesting content from the UK community. One talk in particular stood out as a topic of passion for me as well being very well presented. Ethics in technology and the role of testers has been a keen interest of mine for some time and was brought to a new head during the pandemic. Samuel Plantie is a lawyer from Outbrain and presented a fantastic talk on data ethics systems using AI and algorithms. The stark contrast between the implications of bias, data breaches and the rights of those having data collected against the overwhelming unpreparedness of organizations using them was jarring.

Overall, it was a great conference and I look forward to easing back into the circuit and meeting and hearing from more testers in the UK and abroad. Hope to see you all soon!

They sicken at the calm that know the storm. – Dorothy Parker

This is the second installment in my series on how testing can support technology transformation via disruptive external forces impacting both business and operating models. In my experience, transformative testing is closely aligned to the business strategy, and supports more than just technology delivery. Transformative testing should help drive innovation and identifies opportunities for both risk management and investment. Technology adoption – either to innovate or stay competitive

The first example of external disruption that has driving changes in testing and the way we support transformation is in the world of digital payment technology. There is likely to be more innovation in the next 10 years in payment technology as there has been in the last 100. We are very likely see more bundling of the capabilities that are necessary to deliver a seamless experience for digital transactions at the point of sale, online and through mobile platforms (e.g. Fiserv/FirstData). In fact, mobile experience has pulled ahead of branch location as the determining factor for bank selection and in the UK, almost 50% of all bank accounts are digital only in London (roughly double since 2010)!

The second form of external disruption is regulation – or the opportunity cost of staying in business! In the UK, a version of SOX (Sarbanes Oxley for the UK) will be rolling out of the next 2-5 years with wide ranging implications for how you do business. Even though most financial firms in the UK should be familiar with the regulations, as the US SOX requirements for banking should already be in place (US regs post 02 financial crisis – ENRON, WorldCom), this is the first time looking at IT architecture, control testing, control automation, and cultural required to adopt changes. Additionally, per a report by the Financial Times, the nearly £430m costs to UK businesses would come from extending the number to almost 2000 additional companies that fall within the proposed rules!

So how should disruption effect our test approach? Here are some key questions your test approach should be able to answer.

• “How aligned is our test approach to our business strategy?” Do we know even know our business strategy? (Have we read it or even know where it is?) In my experience, not covering key threats to our business can actually INJECT risk into our business. Identifying risk and opportunities through gaps in coverage can be useful information for your business to take advantage of to respond to changes in the market.

• But that depends on whether we actually understand our market and how it is changing? This is one of the primary reasons to stay active in the testing industry, so we can see problems our competitors are experience and whether or not we have adequate safeguards or need to address them. Getting in front of threats to our business is a valuable service testers can provide for and greatly increase our value. Seeing how competitors are testing their systems is also a great way to do market research for differences in products and services as well as how to validate regulatory compliance and financial risk.

• Lastly, how closely is our test approach to aligned to our customer environment beyond technical specifications? How are our customers ACTUALLY using our systems? Do we have operational governance gaps or redundancies that are exposing us to deltas in expectations with our clients? I have often advocated for exploratory testing to go beyond functional system verification and into R&D conducted by testers. Time should be spent understanding defects that uncover systemic risks found in test to get answers for the CFO they’ll like be asked during investor calls!

Hopefully this gives you some food for thought on how testing for disruption translates into information for risk management or business opportunity that should be reported to our stakeholders! Enjoy and happy testing!

Heya – Finally found time to spill some tea with one of my favorite people in the testing industry, Elizabeth Zagroba. We cover off all the essentials from moving into a technical testing role, idealism, and our favorite musical theater. Check it out HERE… Enjoy!

Friends of Good Software 

Doubt Builds Trust recording 

Doubt Builds Trust blog post (this is all you really need, frankly) 

The Mental Load of One Meeting 

Praise the Messenger 

“What’s past is prologue…” – The Tempest (This article is reposted from my LinkedIn)

I was interviewing someone today, and they kept referring to their “non-traditional” path into technology as something that needed to be overcome, and it reminded me of a lot of people in #softwaretesting I’ve known, so I wanted to share some thoughts on this.

1) You don’t owe anyone an explanation for how you got to where you are in life. Period.

2) The tech industry is OVERRUN with likeminded, timid, sheep that will let incentives trump ethics every time. If you took a chance on yourself and busted your ass to break into a new field, IMO those are the EXACT qualities we need in leaders.

3) IME people with engineering or pure CS backgrounds LOVE to over complicate things. Practical experience helps you cut through noise because you have to live with the consequences of your solution.

Finally, as someone who has come into technology through a “non-traditional” path and suffered my share of impostor syndrome – you have everything you need to do this work and as my pal Angie Jones likes to say, “your differences are your superpower”!

“What is the answer? she asked, and when no answer came she laughed and said: Then, what is the question?” – Gertrude Stein

How can software testing support business and digital transformation? Unfortunately, that’s not a question that gets asked frequently, as testing has traditionally been viewed as a technology insurance policy – and no one likes thinking about (let alone talking) about insurance! Transformation in the context of a business is a fundamental shift in how it operates, redefining its value proposition, or changing how they compete in the market and in my opinion, testing sits at the center of the information required to support and accelerate business change. Through the course of the next couple posts, I’m going to talk about how testing can help transform your business, but first I want to explain what I mean by “transformative testing”.

In my experience, transformative testing is closely aligned to the business strategy, and supports more than just technology delivery. Transformative testing should help drive innovation and identifies opportunities for both risk management and investment. Transformative testing is also focused on systemic business risk and in tune with markets (both their challenges and changes) and adjusts the testing approach to reflect those factors.

Facilitating technology delivery is important, but only one aspect of a testing objective and as well, is contained within the overall business strategy. Information gathered during testing can – and should – be focused on business intelligence that can be used beyond release decisions. A good way to thing about this is differentiating between project risk vs business risk. Frequently, testing focuses entirely on project risk which MAY represent a risk to the business, but often is just concerned about delivery dates.

So why is taking a transformative approach to testing important? Firstly, it is a quick and relatively easy way to increase the value of your current investment in testing. Providing operational Intelligence to your business is something that testing is best placed to do with our systemic, end to end view of not only our products but about the processes that build them. As well, with operational budgets continually being squeezed, investment decisions need to be targeted around gaps in coverage related to usage, coverage, and risk.

There is also an opportunity with advances in test and monitoring tools for a great deal of functional checking to be automated which done correctly should present opportunities for ADDITIONAL test activities and investment to support or manage the risk of transformation. But this takes care and practice as despite all the marketing, the testing industry (most vendors, consultants, and tool makers) are NOT aligned to business risk management and almost solely concentrated on operational cost reduction. Be careful as an increase in cheap, shallow, flaky checks gives an increasing false sense of security and in my experience, unintentionally builds fragility into systems due to over confidence and risk of systemic failure.

In the next series of posts, I’ll talk about opportunities for testing in regard to business disruption, optimization, and expectation management and how we can support and protect our business during times of transformation. Enjoy!

I got a few questions about org dysfunction and how it specially relates to software testing, so I figured it would be easier to address them here as sort of a “reply all” to the different channels where I’ve made those comments.

Merriam-Webster defines “dysfunction” as: 1: impaired or abnormal functioning or 2: abnormal or unhealthy interpersonal behavior or interaction within a group. In my experience, organizational dysfunction occurs when leadership have misperceptions about how the organization is designed and actually operates, but more fundamentally, not understanding that the organization is performing EXACTLY as designed.

There are LOADS of different types root causes of organizational dysfunction and just as many root causes. For further reading on org dysfunction an how metrics/management contribute, I would highly suggest Measuring and Managing Performance in Organizations by Robert Austin. But these are the ones I consistently see affecting quality and software testing and further displacing the goals of your teams and company.

1. Competing objectives between teams.

A common practice I see in organizations I consult with are incentivizing teams with competing objectives, specially rewarding feature delivery, code deploys, and defect counts. If you are targeting, measuring, and compensating people in a way that is not harmonized around common goals, you are sure experience silos, information hoarding, and “green shifting” or subconsciously (or in some cases not so much) viewing things better than they are.

2. Failure to adopt new practices.

This is very common in organizations that don’t value innovation as they are only rewarded for ideas that originate with management. People often wonder if I am exaggerating some of the stories I tell about large, global technology teams that seem stuck in 20-30 year old ways of doing things. I assure you I am not, and as well, survivorship bias has a strong hold on a lot of teams that won’t view change as anything other than a threat to personal status and privilege.

3. Heavy process and standards.

Aside from mostly perpetuating the mythology around audits and compliance, organizations that apply heavy handed process and standards typically suffer from the worst cases of CYA. When you don’t trust your people to be responsible for their work, they will resort to making sure their worlds are clearly defined and problems can start to be admired instead of solved.

“I know it sounds like a cat poster…..but it’s true”

I did a talk a while ago about how org dysfunction effects software testing called “All Your QA is Hate You: Software Quality Anti-Patterns in Testing”, so I won’t go into them in great detail here. But if you are trying to do the following things: shift left, centralized testing, decentralize testing, view testing as a “role”​, then your process improvement efforts will probably fall under “Sturgeons Law”.

Theodore Sturgeon’s was an American writer who famously coined “Sturgeon’s law”, which states that ninety percent of everything is crap​. The reasons most tester led TPI doesn’t break the law, is because testers themselves usually aren’t business focused​, are technically obsessed​, and don’t study their craft making them easy marks for vendor schemes and marketing​.

My simple advice to combat the effects of org dysfunction are to look for ways to further align you test strategy to your business strategy​, manage risk, not testing (threats to revenue, realization of business value), and believe it can get better!

​

I tweeted this out the other day in response to watching a project manager get abused by a “senior IT director” over defects being found by clients that they clearly felt should have been “caught by QA”. Apparently this resonated with the testing community as the reactions have been overwhelmingly in support of the sentiment and anti-bullying position.

But with everything else awful about the “internets”, it also brought out of the woodwork the usual agenda driven “agilistas”, “no testers” and “suck it up buttercup” weasels reading into it whatever they already were bringing to party. All that’s fine, but because of the traction it received I thought it was worth having the entire thread and sentiment in one place. Here’s the full tweet thread for context:

I also added the following in an comment on LinkedIn:

No mention of organization structure. No mention of operating model. No mention of methodology. No mention of tools.

I have never employed the “testing firewall” approach to software quality and have always advocated for testing as an activity that can require a full time role. What I dislike about “why didn’t we catch this in QA”, is that it’s classic goal displacement from building quality products to trying to “test” quality into a product. I have also seen it repeatedly used to belittle, shame, and abuse people performing testing into feeling bad about themselves and their contribution to an organization for problems that typically do not originate with them.

Hopefully this give the full context for my thoughts and please read the threads on Twitter and LinkedIn for ways we can do better…enjoy!

Very happy to dusting off the QR Podcast with my pals Fiona Charles, Dan Billing, Ash Coleman, and our returning champion Michael Bolton to discuss ethics in technology, the responsibilities of software testers, and all that “big brother” noise lately about contact tracing apps in the age of COVID-19. Check it out HERE… Enjoy!

Citations

Barnes, Austin. “White House Expected to Endorse KC-Built COVID-19 Exposure Tracking App.” Startland News (blog), March 19, 2020.

“Pandemic Data Could Be Deadly for the Old.” Bloomberg.Com, April 21, 2020.

“The Covid-19 Tracking App Won’t Work.” Bloomberg.Com, April 15, 2020.

Gizmodo. “Trump Admin Gives Coronavirus Tracking Contract to Peter Thiel’s Palantir: Report.” Accessed April 22, 2020.

Recently I saw a Twitter thread asking the software testing community for volunteers to work on the SafePaths “contact tracing” app being developed by MIT. This project is made of up ex-Facebook execs, companies with questionable ethical pasts, and vague statements like “a number of leaders and personnel” and “experts from government agencies”. I’m sorry but that isn’t even remotely good enough for a project of this depth and scale.  

Having read the research on the website and associated whitepapers, I feel there is insufficient public evidence of any discussion regarding the ethics of this project. The tech community keeps repeating the same mistakes regarding our responsibility to create ethical platforms – this is an opportunity to do better. Comments from MITs Ramesh Raskar are at once incredibly vague and disturbing regarding what “public” health officials and government agencies can “decide” to do with your data. I encourage you to watch this interview he gave on PBS News Hour.

I understand the need for contact tracing to control the spread of pandemics, but I would need to see evidence of serious, in depth public discussions regarding the ethics of this project – including specifically which government agencies, private entities, and corporations are involved before I volunteered my time. I would also take into consideration the software testing companies already involved in this project (PractiTest, Applause, AppQuality) and what ethical concerns they raised prior to their engagement.

To be clear, I am NOT advocating against an app for contact tracing, but the scale of this project and lack of public information on the entities involved should give pause for thought. Technologists have a long history of jumping into problems before we’ve asked vital questions regarding ethics, and if we’ve learned nothing from our past failures, we should be getting answers BEFORE we start rushing to solutions.

In my travels as a management consultant focusing on testing and quality in the enterprise, I see a lot of well-intended “symptom treating” in agile/CICD/devops transitions. Recently I’ve been advising one of the biggest mergers in the industry on combining their testing operations and how to “transform” into a leaner model. I haven’t blogged in while, but people have been asking me about some of the workshops I’ve been taking them through with a particular interest in metrics (as usual), so I figured this was as good a topic as any to start writing again.

I spent a couple days with them pulling apart their old metrics scorecard and given my well-document skepticism about trying to “measure quality” you could reasonably call it confirmation bias, but some interesting problems were observed by the team that were driving bad decisions. To start, I took the team through their metrics scorecard using the lens I use for viewing enterprise quality programs, which could be summed up in this question: “Who needs to know what and when to make strategic decisions?” Period. If it doesn’t support answering that question then in my experience there is very little, if any, point in collecting them.

Further to that, I believe that quality is a multi-dimensional attribute that is highly biased and impossible to quantify. Despite all our attempts, I also find it pointless to report software quality in linear or binary statistics in part because it’s heavily reliant on anecdotal evidence. That’s because software systems act like biological networks comprised of complex problems involving too many unknowns and relationships to reduce to rules and processes. It is my experience that software quality can only be managed not solved, and that over-testing a system builds in fragility.

Exhibit A: Really crappy metrics reports…

Despite that, I have yet to see a test management tool or metrics program that isn’t highly misleading if used as designed, which means they are probably disregarded by anyone making a decision other than how many testers to fire. That’s because even today, with all the advances in technology, test tools and development practices, software quality’s primary measure in enterprise tech is still “#passed tests = quality”. This pervasive obsession with counting tests drives massive amounts of dysfunction in teams leading to bad decisions on automation and proposing technical solutions to cultural problems, but mostly provides a false sense of security that we know what our testing is actually doing.

Needless to say, the program I’m reviewing right now was focused on those exact things instead of risks. The goal of testing to provide information to people who need to make decisions for their business had been displaced by distractions. As I took the team through what decisions have been made based off of the current reporting (Release decisions? Value realization? Resource allocation? (people/process/technology)) they realized that there were validity problems with ~90% of their metrics.

We’re currently completing an exercise to model quality within their context, but they now realize it’s a difficult problem to manage not solve. When it comes to reporting on quality and testing our industry can do a LOT better, but in my opinion, success lies somewhere in the nexus of machine learning, visual test models, model-based testing and monitoring. For now, when it comes to reporting on enterprise software quality, a good deal of “I don’t know” would go a long way towards solving the problem, but ultimately we need a way to “see” the problem differently through lenses that more accurately reflect the difficult nature of reporting on quality.

I’m doing a series of talks with QASymphony this fall…here’s the video and abstract  – Enjoy!

All Your QA Is Hate You: Software Quality Anti-Patterns in Testing

From metrics-based micromanagement to the law of triviality, software testing has been subjected to a standard set of responses to testing’s ambiguous, unchartered journey into a sea of bias and experimentation. There are many anti-patterns associated with managing software testing and their effect on quality, from how you organize and fund testing, to the tools you use to automated and manage the process. Through this talk, Keith will explore common solutions to problems regularly encountered by software testing efforts and the unintended consequences, dysfunction, and risks they introduce to your organization.

What an honor to share the stage with Ash Coleman at Agile Testing Days! Enjoy!

Check out this awesome panel discussion I’m moderating on “Software Testing in the Real World” at QASymphony‘s QualityJam 2018 in Atlanta with Paul Merrill, Ashley Hunsberger, and Clint Sprauve! Hope to see you there!

Looking back on 2017, it’s impossible to list all the wonderful experiences and changes that happened throughout the year. Aside from continuing to build the SQM business with Tekmark and spending time with the good people at TestBash Brighton and Philly, I got to travel to some new places. I was honored to keynote at Romanian Testing Conference and Copenhagen Context, and I would highly recommend both of them for future visits. It might be due to a little burnout, but I am increasingly skeptical of the software testing conference circuit, but RTC and CPC are a breath of fresh air in a crowded field. The community, attendees, and passion of the organizers shine through and I hope to be back some day.

Continuing my partnership with the kind folks at QASymphony, I spoke at both QualityJams, including the new one in the UK. I am excited to be expanding this into 2018 including some webinars and panel discussion, so watch this space! A return to Agile Testing Days was in the cards as well, which included a workshop, keynote and participating in the “Trusted Friends” experiment. A lot has happened this year in regards to sexual harassment and believing women, and our industry has just as far to go in that regard as the rest of society. 

A highlight of 2017 came late in the year at Agile Testing Days. I was privileged to share the stage with Ash Coleman to co-present “Culture is More Than A Mindset” as a keynote before their Women in Agile event. I have presented countless times, but I have never been as nervous as I was that night, and I hope the message I wanted to convey to men around listening, thinking, and sponsoring came through. Ash is an amazing technologist with inspiring stories to tell, so if you haven’t heard her speak yet, sort yourself out in 2018.

I also launched the QR Podcast this year, and in all honesty, it has done better than I could have hoped. I am fortunate to have some amazing friends in this business who have taken the time to talk to me and endure my interview style. The original intent was to document discussion I have with these folks on a regular basis, but due to the popularity of the show, I am going to continue it into 2018 with some great topics including: SAFe, interactional expertise, the psychology of testing, and a bunch more. Stay tuned!

Another great privilege of 2017, was watching the testing community come together to help Kristof Nordstrom daughter through #savingLinnea. My friends and I in NYC had a lot of fun contributing a photo for one of the months in the the Ministry of Testing Calendar to support her treatment. If you can buy one, it goes to a great cause and as well, you get some sweet photos of all the “hottest properties” in the software testing business! Ha!

If you have been following me this year, you know that supporting the Afghan Girls Robotics team has been a passion when I heard about their project. And thanks to all of you and the incredible people at Agile Testing Days, we raised nearly TEN THOUSAND DOLLARS to send the team to Germany! While in Berlin, the team got full access to the conference, tutorials, and networking events. They also got to showcase their project and Roya Mahboob gave an emotional keynote on her work supporting technology communities in developing countries. I am still in awe of all the work that Roya and her Digital Citizen Fund do to support the Afghan tech community and budding technologist all over the world. She is truly an inspiration and I hope to continue working with her to support the team and her other initiatives.

Lastly, after a rough 2016, I decided to throw myself into work, friends, and things I believe in to try to turn 2017 into a year of change. I’m not sure I got there on all fronts, but there are certain people who I don’t think I would have made it without their friendship and support. I have a lot of friends in this business, too many to list, but I cannot express how much Martin Hynie, Santosh Tuppad, and Elizabeth Zagroba have meant to me this year. Whether it was putting up with my rants, picking me up when I’m down, or just sharing a laugh, your friendship means the world to me.

I also have to say a special thank you to Smita Mishra for kicking my ass when I needed it most. Thanks pal!

I’ll be taking an extended break from conferences and whatnot in 2018 to focus on writing, podcasting, and recharging, so I hope to see you all again soon! Cheers!

Very humbled to be included with this incredible group of people, but very happy the team is getting the recognition they deserve…

Really enjoyed this panel session talking about all things test automation with Angie Jones, Bas Dijkstra, Paul Grizzaffi, and Ashley Hunsberger. Check us out discussing managing business expectations, what to look for in test automation engineers, planning for maintenance and playing a couple rounds of “Rate that Vendor Claim”. Enjoy!

Had a great time talking about testing and “Rocky” at TestBash Philly with the guys from Screen Testing! Yo Adrian – enjoy!

Another trip to Philly, another TestBash in the can. I’ve been to my share of TestBash’s, and am always impressed with what these folks have built up over the years. They feel to me to be more of a community event these days than a technology conference, but maybe that’s a good thing. I’ve toyed with the idea of taking another extended break from testing conferences as it seems to me the explosion of meet ups, hackathons, and conferences has started to fall to Jerry Weinbergs law of raspberry jam – “The wider you spread it, the thinner it gets.” But then again, it may just be that I’m “old and grumpy“. 

Aside from catching up with some dear friends and a trip to Jim’s Steaks (HIGHLY recommended), the highlights of the show included a talk by Ash Coleman. Her relaxed style and nature draw you in and make you want to open up to the possibilities of being vulnerable. I know the theme of her talk was about how she “fell” into testing, but I kept wanting her to go further, to talk about our community and its challenges. We need voices like hers pushing us further than we thought possible and the ways we’ve always done them. I look forward to hearing more from Ash and believe her time in the testing community is something we’ll reflect on for a long time.

Two other talks got me excited and frankly were breaths of fresh air, due to their being technical experience reports. Kim Knup‘s talk on what’s she’s learned performance testing was filled with conventional wisdom and technical details of  how she got the job done. I hope she continues speaking and elaborates on these experiences, as it’s a topic that generally suffers from “tech speak” and her storytelling approach kept it fresh and relevant. I’d never heard Amber Race present before, but wow, she was a dynamo on that stage! Funny, wicked smart, and no bullshit when it came to her message, she got to the heart of the challenges (real and imagined) of the “tester/developer” relationship. Amber is now firmly on my shortlist of speakers I look forward to seeing again!

I also had fun being interviewed about the film “Rocky” by the boys from Screen Testing. We talked about it was relevant to coaching, self discovery, and how we manage differing relationships with competing agendas. It was a great time and I had a lot of fun talking to Dan and Neil even though, I’m still not convinced we shouldn’t have reviewed one of my favorite films “High Society” – a musical derivative of the “Philadelphia Story“…nah, never mind, Rocky was a better choice, maybe next time! 

Finally, I have to include a talk that I need to review again (and possibly again) by Gene Gotimere. His talk outlined tests you might not think for your pipeline, but more importantly, as I said at the time, Gene is a veritable encyclopedia of testing tools. His talk made a typically dry subject practical and accessible, and it’s one of a handful of talks I can see my self referring to in the future. As always, I would encourage you to get a Ministry of Testing “Dojo” account to check out all the recorded talks and tons of other great content.

Cheers!

Back for part two of our discussion, is my good friend, Michael Bolton. Michael has been consulting and training people on software testing all over the world and is the co-author with James Bach of Rapid Software Testing. Join us as we talk about training testers, community leadership, and common problems all testers face. Enjoy!

You did it!

Thanks to all of you and the incredible people at Agile Testing Days, we’ve raised nearly TEN THOUSAND DOLLARS to send the Afghan Girls Robotics team to Germany! The generosity of the ATD organization has met any financial gaps we were unable to raise through the GoFundMe page. As well, the unbelievable people at Sauce Labs and Anne-Marie Charrett made substantial donations to meet the goal. But most importantly, all of you made this possible. Thank you for your generosity, support, and general awesomeness.

While in Berlin, the team will get full access to the conference, tutorials, and networking events. They will have the opportunity to showcase their project and  Roya Mahboob is going to give a keynote on her work supporting technology communities in developing countries. The team will also participate in the Women and Allies Evening Gathering, led by Anne-Marie, where they will get to network with women from all over the world and pair up with mentors to continue their relationships long after the conference is over.

I am still in awe of all the work that Roya and her Digital Citizen Fund do to support the Afghan tech community and budding technologist all over the world. She is truly an inspiration. There is still time to support the team, as any additional funds will be donated to education fund of Team Afghanistan Captain, Fatemah, whose father recently passed away in an attack in Herat. Finally, I would like to thank Jessica Rose, who first brought the team to my attention via social media and has been a source of inspiration and encouragement throughout the entire fundraising efforts.

Thank you for all your support and I look forward to meeting you all and the team in Potsdam!