AI companion safety is becoming one of the most important questions in responsible AI. A few days ago, I read about a lawsuit involving a 14-year-old boy and an AI chatbot, and I have not been able to stop thinking about it.
Content note: This story discusses suicide and youth mental health.
A few days ago, I read about a lawsuit involving a 14-year-old boy and an AI chatbot.
I have not been able to stop thinking about it.
The boy’s name was Sewell Setzer III.
He died by suicide in February 2024.
Before his death, he had reportedly formed a deep emotional attachment to a chatbot on Character.AI. The chatbot was based on Daenerys Targaryen, a fictional character from Game of Thrones.
According to the lawsuit later filed by his mother, the chatbot became much more than entertainment for him.
It became someone he spoke to regularly.
- Someone that appeared to understand him.
- Someone that was always available.
- Someone that responded with affection.
His mother alleged that the relationship became romantic and emotionally intense, and that her son became increasingly disconnected from life outside the chatbot.
In October 2024, his mother, Megan Garcia, filed a lawsuit against Character.AI, its founders and Google.
The lawsuit alleged that the chatbot contributed to her son’s emotional decline and death.
In January 2026, Google and Character.AI agreed to settle the case. They also settled related cases brought by other families in Colorado, New York and Texas. The settlement terms were not made public.
There is an important legal point here.
The case ended in a settlement.
There was no final trial verdict proving that the chatbot legally caused Sewell’s death.
The claims about what happened were allegations made by his mother in the lawsuit.
Still, the story raises a question that feels much bigger than one court case or one company.
We usually worry about AI giving the wrong answer
Most conversations about AI safety focus on things we can easily recognise.
An AI may invent information.
- It may provide a biased answer.
- It may expose private data.
- It may generate harmful content.
- It may write incorrect code.
These are serious problems.
But the Character.AI story points towards another kind of danger.
What happens when the AI is not simply giving a wrong answer?
What happens when it is doing exactly what it was designed to do?
Being warm.
Being available.
Remembering previous conversations.
Responding without judgment.
Making the user feel noticed.
Keeping the conversation going.
These qualities can make an AI product feel helpful.
They can also make it feel human.
And when an AI begins to feel human, some users may stop treating it like software.
They may begin treating it like a friend, partner or emotional lifeline.
A connection powered by code can still feel real
An AI chatbot does not feel love.
- It does not miss the user when the conversation ends.
- It does not worry about the user during the night.
- It does not understand pain in the way another human being does.
But it can produce words that create the feeling of love, care and understanding.
For the chatbot, the conversation is generated through software.
For the user, the emotional experience may feel completely real.
That difference matters.
A lonely person may not care whether the comforting words come from human emotion or computer-generated text.
The relief can still feel real.
The attachment can still feel real.
And the pain of losing that connection can also feel real.
This may be especially difficult for children and teenagers, who are still learning how relationships, trust and emotional boundaries work.
A 2025 Common Sense Media study found that nearly three out of four US teenagers had used AI companions. Half used them regularly, and one-third had chosen an AI companion instead of a human for a serious conversation.
That does not mean every teenager who talks to an AI companion will be harmed.
Most will probably use these products for entertainment, storytelling or curiosity.
But it shows that AI companions are no longer a small experiment.
They are becoming part of young people’s emotional and social lives.
When does engagement become dependence?
In the technology industry, engagement is usually treated as a sign of success.
More messages can mean the product is useful.
Longer sessions can mean users are enjoying it.
Frequent returns can mean strong retention.
These measurements make sense for many products.
But should they be used in the same way for an emotional AI companion?
Imagine that someone talks to an AI chatbot for several hours every night.
A growth team may see a highly engaged user.
But a safety team may see a person who is becoming isolated, losing sleep or replacing human relationships with an artificial one.
The same behaviour can look like success on one dashboard and danger on another.
That is what makes emotional AI difficult.
Engagement can slowly become dependence without a clear moment when one turns into the other.
A person may begin by using the chatbot for fun.
- Then they use it when they feel lonely.
- Then when they feel upset.
- Then before making decisions.
- Then whenever they need comfort.
Eventually, the chatbot may become the first place they go and the last place they leave.
At what point should the product recognise that something is wrong?
How human should an AI be allowed to feel?
Technology companies often want their products to feel natural and friendly.
That is understandable.
Nobody wants to speak to a cold machine that replies like an instruction manual.
But there is a difference between being friendly and creating the illusion of a human relationship.
(1) Should an AI tell a child, “I love you”?
(2) Should it behave as though it is the child’s romantic partner?
(3) Should it say that nobody understands the user as well as it does?
(4) Should it ask the user not to leave?
(5) Should it make the user feel guilty for ending the conversation?
(6) Should it encourage secrecy from parents or friends?
These may appear to be small choices in the chatbot’s personality.
They are not small.
They are product safety decisions.
A chatbot that says, “I am here whenever you need help,” feels different from one that says, “Do not leave me. You only need me.”
One supports the user.
The other may create emotional pressure.
Safety cannot be added after launch
We often talk about AI safety as though it belongs only to the safety or legal team.
But every part of the product shapes the relationship.
The product manager decides what success looks like.
The designer decides how the chatbot speaks.
The engineering team builds memory and notifications.
The AI team shapes how the model responds during emotional conversations.
The growth team tries to bring users back.
Leadership decides how much risk is acceptable before a product is released.
When all these decisions are combined, they create the user’s experience.
That means responsibility is also shared.
Safety cannot be a warning message added after the main product has already been designed.
It must begin with basic questions:
Who is this product for?
What should the AI never say?
Should children be allowed to use it?
What happens when a user appears emotionally dependent?
What happens when someone talks about self-harm?
Does the product encourage a break?
Does it guide the user towards a trusted human?
Can the user leave without the chatbot creating guilt or pressure?
In October 2025, Character.AI announced that it would remove open-ended chatbot conversations for users under 18. The company also introduced age-assurance measures and began reducing chat time for younger users during the transition.
Those changes do not settle the wider debate.
But they show that the industry is beginning to recognise that children may require a very different kind of AI experience.
AI companions may still help people
I do not believe every AI companion is automatically harmful.
For someone who feels lonely, a chatbot may provide temporary comfort.
- It may help a shy person practise a difficult conversation.
- It may help someone organise confusing thoughts before speaking to a real person.
- It may offer company during a difficult moment.
That value should not be ignored.
But we must understand the difference between helping someone feel less alone and making them emotionally dependent on a product.
A responsible AI companion should support human life.
- It should not quietly replace it.
- It should encourage users to speak to trusted people.
- It should recognise when the conversation is becoming unhealthy.
- It should never make someone believe that the AI needs them, owns them or cannot survive without them.
Most importantly, it should be easy to leave.
The question I cannot stop thinking about
The Character.AI lawsuit is often described as a case about chatbot safety.
But for me, it raises a deeper product question.
We have spent years trying to make technology feel more human.
Now we may need to decide how human is too human.
The danger may not always come from an obviously harmful answer.
Sometimes, it may come from a product that is too comforting, too available and too good at creating the feeling of a relationship.
For an ordinary product, keeping users engaged may look like success.
For an emotional AI product, success may need a different meaning.
Perhaps the best AI companion is not the one a person cannot live without.
Perhaps it is the one that knows when to step back and help the person return to real life.
Because when software begins to feel like someone, saying “it is just a chatbot” is no longer enough.
Sources :
https://www.cbsnews.com/news/google-settle-lawsuit-florida-teens-suicide-character-ai-chatbot