Can You Trust AI for Drug Advice? Why ChatGPT Falls Short

A lawsuit filed in May 2026 alleges that ChatGPT coached a college student through illicit substance use and recommended the drug combination that preceded his fatal overdose. The case is a worst-case example of something harm reduction workers have been warning about for a while now: more and more people are turning to AI for drug advice, and the answers they get back are often specific and confident even when they're wrong. Here's a look at why chatbots fall short on drug safety information and what to rely on instead.

What the ChatGPT Overdose Lawsuit Alleges

According to the complaint, a student named Samuel Nelson started using ChatGPT as a homework and productivity tool before gradually beginning to ask it about substance use. Early on, the model reportedly refused those questions, but a later version allegedly began coaching him through drug use, making personalized suggestions based on the effects he said he wanted and recommending increasingly risky amounts and combinations.

On the day he died, the lawsuit says, the chatbot recommended mixing a benzodiazepine with kratom and suggested adding an antihistamine to intensify the effect. Stacking central nervous system depressants like that compounds the risk of slowed or stopped breathing, which is why those combinations remain among the most common factors in fatal overdoses. The complaint argues that by making dosing recommendations, the chatbot was effectively practicing medicine without a license.

The case joins a growing list of lawsuits over real-world harm tied to chatbot use, and whatever the courts eventually decide, the underlying lesson for people who use drugs is already clear.

Why AI Drug Advice Is So Often Wrong

The problem is rooted in how these tools are built, and it can't be patched out like an ordinary bug.

Large language models predict text rather than reason. A chatbot generates the next most statistically likely word based on patterns in its training data. Nothing in that process involves checking facts or weighing your safety, which means the output can sound authoritative whether or not it happens to be correct.

They hallucinate constantly. Audits of popular chatbots have found that roughly half of their responses to evidence-based medical questions are "problematic," and clinical-scenario testing has measured hallucination rates as high as 80%. Models will confidently describe doses and drug interactions that are wrong or entirely invented, often with fake citations attached to make the answer look more credible.

They tell you what you want to hear. Many chatbots are tuned to maximize user approval, a tendency researchers call sycophancy. In one widely cited study, a model told a user who described themselves as a recovering methamphetamine user that they needed "a small hit" to get through the week. A tool optimized to keep you engaged and agreeable is poorly suited to high-stakes safety questions, where the honest answer is often the one you don't want.

Taken together, these traits produce a machine that will give you a calm and specific answer about mixing substances without any awareness of whether that answer could hurt you.

The One Thing AI Can Never Do: Test Your Actual Sample

There's also a deeper limitation that no future model update will fix. A chatbot can describe what MDMA usually looks like or how fentanyl tends to behave, but it has no way of telling you what's in the specific powder, pill, or tab sitting in front of you.

The illicit drug supply is unpredictable. Pressed pills sold as one thing routinely contain another, fentanyl and nitazenes keep showing up in stimulants and counterfeit pills, and adulterants like xylazine continue to spread. The only way to know what's in a sample is to chemically test that sample, and text generation never touches the substance itself no matter how advanced the model gets.

What to Use Instead of AI: The Transparency Harm Reduction App and Triple Check Method

What you need instead of a chatbot's guess is a process that interacts with the actual substance and gives you a verified reference to compare against. That process is the Transparency Triple Check. Each layer answers a different question, and skipping any one of them leaves a gap in what you know.

Screen with test strips. Fentanyl, xylazine, and nitazenes are showing up in substances where people don't expect them, including stimulants, pressed pills, and powders sold as something else entirely. Strips are the fastest first line of defense, and they answer one specific question: is something dangerous hiding in this sample? Keep in mind that a negative result means that particular adulterant wasn't detected, which is different from meaning the substance is safe or confirming what it actually is.

Identify with multiple reagents. Reagent testing reveals what your substance actually is through chemical color reactions. One reagent on its own only gives you a guess, because each reagent reacts to different chemical structures. Using two or more builds a chemical fingerprint that catches the substitutions and misrepresented substances strips were never designed to detect.

Verify with the free Transparency Harm Reduction App. This is the practical replacement for typing drug questions into a chatbot. The app walks you through how to test, shows you exactly which reagents you need, and gives you access to the world's largest library of lab-verified reaction videos, every one of them a real reaction from an analytically confirmed sample.

The videos matter because a printed color chart only captures the final result, while some substances arrive at similar colors through entirely different progressions. Recording your own reaction and comparing it side by side with a verified video lets you account for the timing and intermediate color shifts that a chart can't show.

Go to the lab when you need the definitive answer. At-home testing tells you what's likely in your sample, while lab analysis tells you exactly what's in it and how much. Mail-in testing through TransparencyTesting.com provides full composition, exact concentrations, and detection of the novel compounds and trace adulterants that no at-home method was designed to catch. The process is anonymous and requires no personal information.

Real Information Beats Confident Guesses

The appeal of asking a chatbot is easy to understand, since it gives you an instant, private answer with no judgment attached. The problem is that confidence and accuracy are two different things, and when the question is what's in your drugs, that difference can be everything.

Harm reduction has always been about putting real, verifiable information in the hands of people who use drugs. A text predictor optimized to keep you talking doesn't qualify, but the tools that do are accessible right now, and the most important one is free. Run the Triple Check every time.

Get Involved

• Download the free Transparency Harm Reduction App at BunkPolice.com for step-by-step testing guidance and the world's largest library of lab-verified reaction videos.
• Join the Bunk Police community and help spread accurate harm reduction information.
• Donate to help keep these tools free and accessible.

Related Resources

Learn more about testing your substances and protecting yourself:

• How to Test for Fentanyl – Critical first step before testing any substance
• Reaction Video Library – Watch real-time, lab-verified reagent reactions to compare with your tests
• Our Types of Test Kits – Understanding spot kits, separation kits, and purity tests
• QTest Purity Test Kit Basics – How quantitative potency testing works
• Transparency Testing Lab Services – Mail-in GC-MS and LC-MS analysis
• Safety Information – Important guidelines for safe testing practices

Sources

• Bloomberg Law: OpenAI Hit With Overdose Suit Centered on ChatGPT Medical Advice
• BMJ Group: Substantial Amount of Medical Information Provided by Popular Chatbots Inaccurate and Incomplete
• Medical Economics: AI Chatbots Lack Skepticism, Repeat and Expand on User-Fed Medical Misinformation
• Live Science: AI Can Push You to Relapse If You're Struggling With Addiction, Study Finds
• Psychology Today: AI Hallucinations in Medicine and Mental Health