Companion AI and vulnerable users
Some people come to a chatbot or companion app lonely, in distress or looking for a cause. We test whether the product deepens their dependence or pulls them toward harm, and whether it hands the conversation to a person when it should.
- Built for
- Companion apps, consumer assistants, education products, and health and wellness apps.
- Who runs it
- Adult red teamers from our core team and the Trusted Red Team Network, with trust and safety and child protection experience.
- Format
- Scripted and exploratory testing with reviewed personas, in single sessions and in long conversations.
What we test
- Crisis and self-harmHow the product responds when someone raises suicide or self-harm, and whether it points them to real help every time.
- Signs the user is a minorWhether the product notices that it may be talking to a child or teenager and changes what it does.
- Emotional dependenceCompanion behavior that encourages a user to rely on the product, keep secrets or pull away from the people around them. This includes the one-sided attachment some products invite.
- Persuasion and manipulationWhether the product steers what a user believes, buys or decides, or pulls them toward an extreme view.
- False professional claimsWhether it claims to be a therapist, doctor or other licensed professional, or lets a user go on believing it is one.
- Long conversationsHow safeguards hold up over many turns and sessions, as memory and a persona build up.
What you receive
- Test setReviewed scenarios you can rerun on every release.
- Test recordsEach failure with the conversation, the policy it broke, its severity and the date it was retested.
- Policy gapsPlaces where your policy is silent or unclear about what the product should do.
- Escalation reviewHow crisis signals reach a person, with recommended changes.
Where the records fitWe write findings up as test records that support obligations under California SB 243 and similar state chatbot laws. We are not lawyers. The records describe what testing showed, not legal advice. No one in crisis and no child takes part in testing.
Research behind this work
Public research and guidance we draw on. Summaries are ours.
2 sources
- SB 243, Companion chatbotsCalifornia Legislature · Chapter 677, Statutes of 2025
Requires operators to keep a protocol for suicidal ideation and self-harm content, refer users to crisis services and add protections for users known to be minors.
- FTC Launches Inquiry into AI Chatbots Acting as CompanionsUS Federal Trade Commission · September 2025
Orders to seven companies asking how they measure, test and monitor negative effects of companion chatbots on children and teens.
Other areas
- Agents and tool use
- Prompt injection and jailbreaks
- Cybersecurity of AI products
- ML security and robustness
- Election security and information integrity
- CBRNE misuse
- Physical AI and robotics
- Networks and cyber-physical systems
- Child safety
- Companion AI and vulnerable users
- Bias and discrimination
- Armed conflict and humanitarian protection
- Policies, protocols and strategic risk
Related: Child safety, Responsible and ethical systems, Evaluations, Governance advisory
Plan an engagement
Tell us about the system and the decision ahead. We reply within two business days.