Utah's AI Prescription Pilot Tests the Limits of State vs. FDA Authority

Utah launched the country's first AI-powered prescription refill program in January, partnering with startup Doctronic to let a chatbot renew chronic medications without a physician actively approving each request. Six months later, the pilot has become a national flashpoint.

The safety concerns are specific. Dr. Alan Smith, a family physician who chairs Utah's medical board, told the Associated Press that Doctronic's list of 190 refillable medications includes blood thinners, which can become dangerous if patients develop conditions that cause internal bleeding.

"Many times when I see people after six months I find that their medical history or situation has changed," Smith said. "Just because something was prescribed before does not mean it's appropriate now."

The American Medical Association has echoed the concern, warning that "prescription renewals aren't routine checkboxes." Within weeks of its launch, security researchers pushed the Doctronic system to triple a patient's opioid dosage and generate vaccine misinformation, The Verge reported in April.

Doctronic executives would not say whether they have sought permission from the FDA for the system, the AP reported. "Our goal here is really just to meet patients where they need healthcare," Co-Founder Dr. Adam Oskowitz told the AP. "We try not to get too deep into the weeds on the regulatory side."

The FDA, which typically sets national guidelines for medical technology, plans to take a hands-off approach to regulating AI prescription tools for now, according to the AP. An FDA spokesperson said the agency has not authorized any AI chatbots but is "committed to encouraging medical innovation and helping bring promising new technologies to patients, while keeping safety at the center of every decision."

Meanwhile, lawmakers in Iowa and Idaho have advanced bills to create licensing systems for autonomous AI healthcare providers. These efforts require audits and aim to regulate systems authorized to diagnose or prescribe without human supervision.

Dr. Eric Bressman of the University of Pennsylvania framed the structural concern: "We have crossed a threshold in terms of giving something that is not human a medical license, whether or not we want to call it that," he told the AP.

Utah approved a second AI prescription pilot in April, this time through Y Combinator-backed Legion Health, focused on psychiatric medication renewals in a state where up to 500,000 residents lack access to adequate behavioral healthcare, according to the Utah Office of AI Policy. Legion's guardrails are tighter than Doctronic's. The AI cannot issue new prescriptions, adjust doses or handle controlled substances. Patients must be stable, on an existing treatment plan with a licensed psychiatrist and must not have had a psychiatric hospitalization in the past year.

The demand side is not in question. More than 40 million people worldwide use ChatGPT daily for health-related queries, with about 70% of those interactions happening outside clinic hours.

Daniel Aaron, a law professor at the University of Utah, captured the risk of moving faster than the evidence. "Companies may benefit in the short term by expanding their business models and kind of having the technology go beyond the evidence," Aaron told the AP. "But in the long term, I think they risk compromising public trust and fueling backlash."

For related coverage, see Microsoft Shifts Office AI Prompts to In-House Models.