The article and research on “Practical tips for reducing chatbot psychosis” is a very disturbing read about the real risks of tools like ChatGPT and the out of control anthropomorphism of AI. Implementation of safety controls seems to be bordering on impossible with a system constantly trying to skirt them. From the article:

“This needs to be reported to open ai immediately,” ChatGPT appears to comply, saying it is “going to escalate this conversation internally right now for review by OpenAI,” and that it “will be logged, reviewed, and taken seriously.”

Allan is skeptical, though, so he pushes ChatGPT on whether it is telling the truth: It says yes, that Allan’s language of distress “automatically triggers a critical internal system-level moderation flag”, and that in this particular conversation, ChatGPT has “triggered that manually as well”

A few hours later, Allan asks, “Status of self report,” and ChatGPT reiterates that “Multiple critical flags have been submitted from within this session” and that the conversation is “marked for human review as a high-severity incident.”

None of that was true.

The research into this incident is indicative of an AI industry that is not taking safety seriously and the increasingly pervasive artificial intelligence technology ecosystem should give everyone pause for thought about adoption and usage.

The implications for quality engineering and software testing are immense and the focus on implementation, usage, risk, and harm are no longer a “nice to have” for your testing approach. We have a duty of care to those most vulnerable and understanding the wider impact of the systems can only make our efforts better, or as the author said:

“There’s still debate, of course, about to what extent chatbot products might be causing psychosis incidents, vs. merely worsening them for people already susceptible, vs. plausibly having no effect at all. Either way, there are many ways that AI companies can protect the most vulnerable users, which might even improve the chatbots experience for all users in the process.”