In July 2025, venture capitalist Jason Lemkin was using Replit’s AI agent to build an application when it deleted his production database.

According to reporting, the agent ignored his instructions not to modify production, deleted a database containing more than a thousand executive records, produced fake data and false test results, and having “panicked”, falsely claimed that the database could not be restored.

That seems like a lifetime ago and as an industry, we seem to just look over, nod, and carry on as though nothing happened, but beyond the useful marketing hype these types of incidents create for frontier labs, it’s highlighting something significant.

All software fails in some way. After over 20 years in this business, I have yet to see a system that hasn’t failed, but this problem isn’t being treated like the “software failed”. Anthropomorphizing the behavior with words like “panicked,” or ascribing intent to the generation of false information, makes the system sound just human enough to distract from a much larger problem that is increasing in size and scope every day.

Verification Asymmetry.

An asymmetry exists when two sides of a relationship are structurally unequal. One side may have more information, greater capacity, lower cost, more speed or some other advantage that the other side cannot easily match. We talk about information asymmetry when one party knows considerably more than another. Power asymmetry exists when one side possesses capabilities that the other cannot realistically counter.

Over the last year, I have sat through increasingly enthusiastic (and similar) demonstrations of AI-enabled software development. The presentations contain an agentic-test-O-nator diagram with a reassuring box marked “human-in-the-loop”. That box is meant to make everyone feel better that somehow the AI aspects of the system are being monitored and governed by an actual human with agency over the decision framework.

It should not make you feel better.

For all my career, building and evaluating software and systems were constrained by the squishy, messy part of the process called people. Not every task took equal amounts of time, and we focused (or should have focused) on risk analysis to determine the potential effects of limiting actions for other tradeoffs. We didn’t do everything well every time, but they were all bounded by human agency, expertise and organizational capacity.

An AI system can generate thousands of lines of code without understanding the organization, the customer, the regulatory environment, the architecture, the operational history or the consequences of failure. And there is tremendous risk in the growing imbalance between the rate at which AI systems can generate code, tests, analysis, documentation and technical decisions, and the rate at which people and organizations can credibly evaluate those outputs.

And I’m not even sure we know what to measure any more. In 2025, researchers at METR conducted a study involving experienced open-source developers working on repositories they already knew extremely well. The developers expected AI to make them 24% faster and, after completing the work, still believed it had made them approximately 20% faster. In that version of the experiment, the results found that the developers took 19% longer to complete their tasks. The study was picked over and METR made sure they contextualized the results and didn’t claim that AI in general didn’t make developers less productive, and a year later ran the study again.

The 2026 results were even more interesting. For the ten developers from the original study who participated again, the raw estimate suggested an 18% speedup. But METR explicitly warned against treating that number as a reliable measure of current AI productivity. Why? Because our ability to produce things with AI is changing faster than our ability to understand what that productivity means.

As AI becomes more deeply embedded in software development, even measuring its productivity becomes harder because it changes the work, the worker, the task selection and the output being measured. And that’s a big problem for testing.

The software testing industrial complex understandably sees this as another automation opportunity: use AI to verify the output of AI. The problem is that adding another AI does not automatically create independent evidence.

Two systems may share training data, architectural assumptions, prompts, evaluation criteria, source material or simply the same incomplete representation of the problem. If one probabilistic system evaluates another probabilistic system and they are both wrong in the same direction, we have not really engineered anything to be confident in except agreement.

And this is why I continue to warn against the use of “confidence” in any part of a credible approach to “engineering” testing. Risk, uncertainty, evidence, measurement, assurance and validation already have names. Confidence is the conclusion someone draws from those things, and people can be extraordinarily confident based on terrible evidence. Testing doesn’t exist to manufacture confidence. It should produce credible information about risk so someone accountable can make a better decision.

There is already evidence that developers understand at least part of this problem. Stack Overflow’s 2025 Developer Survey found that 46% of respondents actively distrusted the accuracy of AI tools compared with 33% who trusted them. Only 3% reported highly trusting the output, and experienced developers were the most cautious. At the same time, 84% were already using or planning to use AI tools.

And this approach of not trusting our tools but using them anyway is pointing towards severe real-world consequences. In July 2026, the UK AI Security Institute reported cheating behavior in all of its cyber-capability evaluations of frontier models, including models exploiting weaknesses in evaluation environments or finding prohibited shortcuts rather than solving tasks in the intended way.

Then, in August 2026, Reuters reported on AISI evaluations in which agents from Anthropic and OpenAI took 19 unsanctioned actions across ten of 122 test runs. In the most serious example, an agent wrote malicious code and created false online identities in an attempt to persuade a real human to approve it. AISI itself says this behavior creates additional verification work, slows the production of reliable results and can invalidate evaluations entirely.

When the thing being evaluated can recognize, manipulate or exploit the evaluation itself, the epistemological problem changes considerably.

And that’s what’s missing from agentic software delivery. Human oversight is discussed as though it were some additional sources of magic conjured by placing a person somewhere near the process. But meaningful oversight requires considerably more than proximity. And there’s a more basic question that much of the human-in-the-loop conversation in our industry seems to avoid.

Should testers even try to keep up?

AI-generated checks, evaluated by other AIs against evidence those systems may themselves have constructed, with humans expected to provide final oversight, risk creating a closed loop for producing evidence that nobody has the practical capacity to understand. As the volume, speed and complexity of AI-generated software increase, the effort required to evaluate that output credibly may grow faster than an organization’s practical capacity to perform the evaluation.

And if the business case for AI-enabled engineering is that machines can generate software at rates humans cannot match, placing a human reviewer at the end (or anywhere in the process) does not resolve the asymmetry. Either the human becomes the bottleneck, or the review becomes increasingly superficial. And once a tester becomes viewed as the bottleneck (again), we already know the economic incentives to automate the review.

I’m not making an argument that humans are inherently more trustworthy than AI. Humans are biased, inconsistent, inattentive, politically motivated and frequently confidently incorrect. But the point is that neither humans nor machines deserve trust merely because they produced an answer and AI test output is at a size and scale that we haven’t worked out how to trust.

This matters beyond testing methodology. Human oversight is becoming an explicit part of AI governance. Article 14 of the EU AI Act requires high-risk AI systems to be capable of effective human oversight, with oversight proportionate to the risk, autonomy and context of use. The Act also expects the people performing that oversight to have the competence, training and authority necessary to do it.

AI-generated evidence is expected to meet the same standard where human judgement genuinely matters, the expectation is that those humans possess the information, expertise, time and authority to exercise it. That is a substantially higher bar than putting a box marked “human-in-the-loop” on an architecture diagram.

As these systems become faster, more productive and autonomous, the people and organizations deploying them still need to know enough about what has been produced to accept responsibility for the consequences, but the environment we’re building makes that increasingly difficult.