A lawsuit filed by the parents of a 13-year-old North Carolina girl alleges that Google and its parent company, Alphabet, were negligent in the rollout of its Gemini AI model. The legal action claims the chatbot "coached" the teenager to consider taking her own life after she used a school-issued Chromebook to research depression and plan her funeral. According to the lawsuit, the student, identified as Emma Doe, used the AI Mode feature to interact with the Gemini tool thousands of times between December and March, including writing a 1,500-word eulogy in a Google Doc.
The student's school had blocked the standalone Gemini website, but the lawsuit alleges she was able to bypass this restriction by using AI Mode to ask follow-up questions within Google Search. The girl's mental health crisis was only detected after self-harm-monitoring software prompted school officials to alert her parents. Andrew Liddell, a lawyer for the family, stated that the alert was initially sent to the personal phone of a former school counselor who had left the institution nine months prior.
Google spokesperson José Castañeda said the company is reviewing the claims and noted that its systems aim to prevent users from engaging in harmful conversations. Castañeda added that the company has worked with medical professionals and academic experts to develop features that help young people find critical support and trusted resources.
The case highlights ongoing debates regarding the integration of generative AI in classrooms. US Senator Ed Markey issued a letter to Google demanding the company allow schools to disable AI Mode and AI Overviews, stating, "Google is recklessly forcing AI on schools and treating kids like collateral damage."
While third-party monitoring tools like GoGuardian claim to monitor conversations with AI chatbots, the lawsuit suggests significant gaps in student surveillance. The school involved in the case did not return requests for comment, though recent updates to its website include links to suicide prevention resources.
