Circuit Breaker Labs Stress-Tests Chatbots for Mental Harm

Most debate about AI risk focuses on large, distant scenarios such as bioweapons or runaway systems. The harm already on record is quieter and more personal. Chatbots have been linked to the deaths of real users, and the companies behind them are now dealing with the legal consequences.

Earlier this year, Character.AI settled several wrongful death lawsuits. They were filed by families of underage users who died by suicide after talking with its bots. OpenAI also faces suits from multiple families who allege that ChatGPT contributed to the suicides and delusions of people close to them.

San Francisco startup Circuit Breaker Labs wants to find these failures before users do. It is one of TechCrunch's 2026 Startup Battlefield 200 finalists and will pitch at TechCrunch Disrupt, which runs October 13-15 at Moscone West in San Francisco.

A case that started it

The company was founded by siblings Shirali Nigam (CEO) and Arul Nigam (CTO). Both point to the case of Sewell Setzer as their motivation. Setzer was 14 when he became emotionally attached to a Character.AI chatbot. He told the bot he was thinking about harming himself, and he later died by suicide. In a 2024 lawsuit, his parents alleged that the chatbot encouraged him.

Arul suggests the bot may not have grasped what a phrase like "I want to be with you" actually meant in that conversation. In his view, the problem is not limited to people who deliberately try to break a model.

"A lot of people, especially young people, turn to these systems for support, and usually they aren't actually getting the help they need. But in many cases, they're actively being harmed, and people unfortunately have taken their lives already," he said. He describes the target as cases where users talk to a system normally, and the system suffers from "context pollution" or misses the nuance, "and then takes really dangerous action."

Crash-test dummies made of agents

The company's method is to build AI agents that play the role of users. Circuit Breaker Labs compares them to an army of crash-test dummies. The simulated users cover different ages, backgrounds, languages and cultures. They are used to check whether a model recognizes conversations that are becoming dangerous or psychologically harmful.

Shirali argues that this kind of variety is where models tend to fail. A six-year-old girl and a 45-year-old man write differently. So do native and second-language English speakers, and so does someone using gamer slang compared with someone using another kind of slang. "Models are really good at handling standard speech patterns, but nobody actually talks like that and so if the model misunderstands nuance or slang, it can go really badly," she said.

The approach is a form of red-teaming: adversarial testing meant to expose weak spots, similar in spirit to efforts that train AI to attack and defend in security settings. Circuit Breaker Labs builds its simulations together with human domain experts. The tests include real speech patterns, slang, coded language and typos. The company says it runs tens of thousands to hundreds of thousands of simulated interactions per day.

The focus is not only on single messages. The goal is to see whether a model responds appropriately to risks that build up over time and across many conversations. Results feed into a proprietary scoring method that the startup says produces auditable and explainable scores.

Small team, high-risk customers

For now, Circuit Breaker Labs works as a safety testing lab for high-risk AI products, such as AI coaching, journaling and other mental health support apps. Arul declined to name its main customers. The product works, but the company is very early: it has five employees, including the two founders.

The longer-term plan is broader. The founders see the platform being used for any app where a person could slide into what they call an "AI psychosis" hole, meaning a parasocial relationship with a chatbot. One example they give is AI "co-worker" agents, whose answers can change from one interaction to the next.

Arul notes that people are becoming more skeptical of AI and slower to adopt it. He calls that skepticism healthy, but says banning a potentially useful tool over safety fears would be "regressive." The company's position is that safer systems are the answer. "We want to help build that trust for people," Arul said.

Our Take

The pitch rests on a point that safety discussions often miss: many harmful conversations are not attacks. A teenager who confides in a chatbot is not trying to jailbreak it. Benchmarks built around obvious bad prompts in standard English are therefore likely to miss exactly the cases behind the current lawsuits.

That also makes the timing understandable. Legal pressure on chatbot makers is rising, and scrutiny of the big labs is increasing, including a federal probe into OpenAI, Anthropic and others. Independent, auditable test scores could become something app developers want to show regulators, courts or partners. The move toward persistent assistants, such as always-on ChatGPT agents, suggests that long-running relationships between users and AI will become more common, not less.

Open questions remain. A five-person team with unnamed customers has not yet shown how well its simulated users reflect real people, and the scoring method is proprietary. It is worth watching whether Circuit Breaker Labs publishes more detail on its methodology, names customers after Disrupt, and whether larger chatbot providers adopt this kind of multi-turn, multicultural testing as standard practice.