Thanks to my friend Angie Jones at the Agentic AI Foundation (AAIF), I got to attend their flagship conference AGNTCon + MCPCon Europe in Amsterdam this month. Over the course of two and a half days I got to speak with a lot of really interesting people and listen to a ton of talks and demos on the future of agent protocols and standards, orchestration, evaluation, and real-world adoption.

Little did I know that the sandy, grit-coated “Cajun popcorn shrimp,” served in an actual bowl of mayonnaise-covered popcorn at Amsterdam’s nhow hotel, should have prepared me for a conference where familiar words were served with equal confidence and roughly the same relationship to what they were supposed to mean.

I also don’t want to give you the impression that people were walking around twisting their moustaches plotting our demise, and there were even some good talks dedicated to governance and regulation. What worried me was the narrowness with which real-world consequences were understood and discussed.

As I sat through the multiple keynotes and track talks, they all seemed to share the same sense of inevitability and unwarranted urgency. That tone continued in conversations with attendees. I repeatedly heard versions of “the genie is out of the bottle,” “there’s no going back,” “if we don’t do it, somebody else will,” and the regular warning that anyone who hesitates will lose the AI race.

I guess the thing that bothered me most was how ahistorical much of the conversations and talks felt. We’ve spent decades watching technology companies promise that their products would democratize knowledge, connect humanity, improve work and make society more open. We now live with tech-driven industrial-scale surveillance, algorithmic discrimination, social media fragmentation, environmental damage and extraordinary concentrations of corporate power.

Predictably, speakers repeatedly turned to governance as the answer, but even that was framed as a constraint to be limited by your growth strategy. One presentation correctly highlighted that AI is evolving faster than governance but ended with a line that captured the central contradiction of the entire event:

“Governance should never slow innovation. Governance should make it scalable.”

Sorry, governance is not another production capability for getting systems deployed faster. Its purpose is to impose boundaries to protect people and prevent organizations from taking risks when somebody else will have to live with the consequences. An idea absent from most of the presentations and discussions I witnessed.

According to the governance models I saw presented, meaningful human judgment is the main assurance constraint in an agentic system. The Theory of Constraints would suggest then, that production should first be subordinated to the capacity for competent review. Instead, the industry is accelerating the production of code, decisions and actions while trying to automate the reviewing constraint into irrelevance.

Nothing exposed the human-oversight contradiction better than a presentation I attended on the EU AI Act. The talk was brilliant, and during the Q&A I asked how meaningful human oversight could be reconciled with AI and coding agents that produce work at a scale impossible for humans to review. The speaker acknowledged that Article 14 requires effective human oversight for high-risk systems, then shrugged and said they didn’t know how that would be managed in reality.

That shrug was probably the most honest answer to a governance question I heard all week.

We already know the impact of doomsday scenarios like Europe 2031 that turn technological competition into an existential threat and once that framing takes hold, almost any safeguard can be dismissed as “red tape”. That framing has resonated in the recent AI Omnibus which postponed high-risk AI requirements and simplified others in the name of competitiveness and “innovation-friendly” implementation. The danger is that safety regulation becomes permanently conditional on not inconveniencing industry, which means it will always arrive too late, because somebody will always claim that a less-regulated competitor is moving faster.

The conference was starting to resemble a case study in how to summon the third principle in Richard Cook’s “How Complex Systems Fail”: catastrophe requires multiple failures. The principle states:

“Overt catastrophic failure occurs when small, apparently innocuous failures join to create opportunity for a systemic accident.”

That’s how all this is likely to fall apart. Not through a rogue “super intelligence” deciding to destroy humanity in an attempt to rid us of cancer, but through thousands of individually defensible decisions made by intelligent, well-meaning people focused only on their part of the system. If it fails it will be because people were careless, hurried and incapable of considering the broader implications or fully understand what they’re building.

The AAIF describes itself as a neutral, open foundation built on transparency, collaboration and standardization to advance the public interest, and they are off to a great start on those ambitions. But there is no naturally occurring AI race. Organizations choose what to fund, engineers choose what to build, executives choose what to deploy, and governments choose what to permit. Presenting those choices as unstoppable progress conveniently removes responsibility from the people making them.

Sometimes constraints on those ambitions are the only things protecting us from ourselves.

Livestream Recording of AGNTCon + MCPCon EU Day 1

Livestream Recording of AGNTCon + MCPCon EU Day 2