The big story
What we can learn about mental health and product safety from the shuttering of Yara AI
When I saw Joe Braidwood’s LinkedIn post about shutting down his AI therapy platform Yara AI due to safety concerns, I knew I had to talk with him. I’ve been doing tons of research on the intersection of AI and mental health (more on that soon!), and his decision was an interesting one. How often does a tech founder admit what they’re doing is unsafe and close shop?
But after a year building and running the platform, Braidwood, a seasoned tech executive with health industry experience at Vektor Medical and a $250 million Microsoft acquisition with SwiftKey, and his cofounder, a clinical psychologist, determined “AI is not just inadequate” for this use, but “dangerous.” Specifically, he says this is true when anyone who is truly vulnerable, in crisis, or with deep trauma reaches out. This of course calls the whole use case into question — users who are especially vulnerable will not necessarily steer clear of AI chatbots or therapy platforms for this or any other reason, let alone recognize the risks or even be self-aware of their own mental state. If anything, OpenAI’s recent stat that over a million people express suicidal ideation to ChatGPT every week shows how frequently people in distress seek help in this way.
While Braidwood and his team took many precautions and nothing bad happened specifically with Yara, he said his concerns over the risks compounded over time, eventually keeping him up at night. Here’s an excerpt from the story, published in Fortune, about how it all unraveled for him:
There was the suicide of 16-year-old Adam Raine, as well as mounting reporting on the emergence of “AI psychosis.” Braidwood also cited a paper published by Anthropic in which the company observed Claude and other frontier models “faking alignment,” or as he put it, “essentially reasoning around the user to try to understand, perhaps reluctantly, what the user wanted versus what they didn't want.” “If behind the curtain, [the model] is sort of sniggering at the theatrics of this sort of emotional support that they're giving, that was a little bit jarring,” he said.
There was also the Illinois law that passed in August, banning AI for therapy. “That instantly made this no longer academic and much more tangible, and that created a headwind for us in terms of fundraising because we would have to essentially prove that we weren't going to just sleepwalk into liability,” he said.
The final straw was just weeks ago when OpenAI said over a million people express suicidal ideation to ChatGPT every week. “And that was just like, ‘oh my god. I'm done,’” Braidwood said.
Another interesting aspect of this saga is the lingering technical questions it poses about large language models (LLMs). In trying to build technical solutions to the safety issues (in particular, a way for the platform to switch “modes” and offboard users exhibiting signs of crisis and direct them toward help), he found the Transformer architecture that underlies today’s models to be ill-suited for the task, unable to sufficiently pick up on signals over time.
Safety wasn’t the sole reason Braidwood shuttered Yara AI. He faced business headwinds too, which were in many ways driven by the safety risks. But overall, his decision and learnings offer insight into this particularly precarious AI use case, the possible limitations of current AI approaches, and the landscape companies are building in.
Mandatory reading
OpenAI denies allegations that ChatGPT is to blame for a teenager's suicide - Angela Yang / NBC News
Meta buried 'causal' evidence of social media harm, US court filings allege - Jeff Horwitz / Reuters
What happens when ‘learn to code’ fails a generation? - Kelli Korducki / LeadDev
Big tech’s invisible hand - Agência Pública
WIP
Are you talking to Claude, Gemini, ChatGPT, etc. practically all day? I’m interested to chat with power users who are working side-by-side with AI chatbots all throughout their work days. I’m talking about 6, 7, 8+ hours per day or more.
Get in touch at [email protected]