I’m getting a lot of contact from mid-level test managers as well as a fair number of senior ones either looking for work post redundancy or trying to get ahead of an impending one. I also recently gave a short internal talk on the impact AI is having on the testing business, and aside from potential scaring some folks, I got asked the same inevitable question: what should we do?

Here’s my answer: Learn everything you can about AI and BE ABLE TO DO THE WORK.

• Get AI literate.
• Read the research.
• Watch and attend demos.
• Learn how to evaluate model performance.
• Read the regulations.
• Understand AI Safety.
• Test. Evaluate. Report.

But the highest priority is NEVER get too far away from the work. I’ve spent the last week or so writing acceptance criteria with a project, researching, organizing the work, and getting hands on into planning and execution. Managers of managers are always the first to go and testing is filled up to our eye teeth with them, so if you can’t roll up your sleeves and get dirty, there’s not much I can offer you in terms of advice right now.

The nature of testing may not be changing, but how it’s delivered and definitely how it’s managed are going to change – a LOT. I wrote recently about the rise of “incompetent experts”, and I think we’re about to experience a Buffett-esq tide moment and see “who’s been swimming naked”.

Here’s my Resources page to help get started…

Here’s the AI Testing and Assurance LinkedIn group for more research…

Happy for people to please drop other resources in the comments that have helped them…

Good luck and keep testing!

***END NOTE***

I posted this to LinkedIn and was challenged on the practicality of my advice, so I added this comment to the post which serves as a nice addition…enjoy!

That’s a fair and I absolutely do not think every tester needs to read Attention Is All You Need. This is a bit of a brain dump, but if someone is currently doing experiential testing or writing automation, my practical advice would be:

1. Learn to use AI in the work you’re doing now….

For a experiential testers: treat it like a research assistant to interrogate requirements, generate test ideas, identify assumptions, create data, analyze logs, compare outputs, summarize evidence and challenge your own thinking – but ALWAYS verify the output.

For test engineers: get competent with something like Claude Code or GitHub Copilot. Generate tests, refactor them, debug failures, explain unfamiliar code, build small utilities and most importantly learn to review what the code extruding machine produces.

2. Learn enough about AI to understand why it behaves differently….

You can get a lot of test ideas from the published research and articles about how they fail and signals to help design tests for probabilistic output, context, hallucination, prompt injection, agents and tool use, variability between runs, etc.

3. Learn basic evaluation…

I’d disagree that this this is only for AI researchers, as we are getting a lot of interest from clients and new projects, and as well, this is right in the heart of regulatory concerns about safety. To be specific, I mean learning how to define acceptable behavior, construct useful test datasets, evaluate variable outputs, establish failure criteria, repeat tests, examine distributions rather than individual answers and decide when human judgement is actually required. That is going to become an increasingly useful testing skill, because AI is going to turn up inside ordinary applications whether we asked for it or not.

4. Learn how these things fail….

OWASP’s LLM is a great starting point. Prompt injection, excessive agency, data leakage, insecure output handling and inappropriate tool permissions are testable problems.

And if you haven’t been hands on for a while:

5. Refresh the basics…

Oracle, coverage, risk-based testing, testability, critical thinking, systems thinking, exploratory testing and understanding what test evidence actually tells you and how to report it in terms of business risk.

For someone starting today, I’d point them at:

• Microsoft Generative AI for Beginners or Elements of AI for literacy
• GitHub Copilot’s testing tutorials
• Claude Code’s getting-started material
• Anthropics introductory material on evals
• OWASP’s LLM/Agentic Top 10 material for learning how AI-enabled applications fail