Why AI Literacy Isn’t Enough to Fix Chatbots: 6 Questions with Medical AI and Health Law Scholar Sofia Palmieri
Berkeley Law Life Sciences Law & Policy Center Director Vincent Joralemon recently sat down with Sofia Palmieri, a health law scholar working at the intersection of medicine, artificial intelligence, and regulation on both sides of the Atlantic.
Published
Author
Share
Berkeley Law Life Sciences Law & Policy Center Director Vincent Joralemon recently sat down with Sofia Palmieri, a health law scholar working at the intersection of medicine, artificial intelligence, and regulation on both sides of the Atlantic. Palmieri is a Post-Doctoral Fellow in Medicine, Artificial Intelligence, and the Law at the Petrie-Flom Center at Harvard Law School and Managing Editor of the European Journal of Health Law. She holds a PhD in health law from Ghent University and a law degree from the University of Bologna, and her work runs from how the EU AI Act treats medical AI to the public-health case for regulating digital addiction. Their conversation covered why stem-cell-based embryo models are the most underrated development in the life sciences, why longevity medicine is overhyped, why AI literacy is no substitute for regulating chatbot design, and the one design safeguard she would want the United States to adopt.
This interview has been edited and condensed from its original recording.
1. What’s one underrated idea in life sciences right now?
Stem-cell-based embryo modeling — the idea that developmental biology itself can become an experimental platform. I’m thinking of the work coming out of Jacob Hanna’s lab at the Weizmann Institute, where researchers have grown stem cells into embryo-like structures further along than is permitted almost anywhere else in the world.
[Editorial note: Stem-cell-based embryo models are structures grown in the lab from stem cells that mimic the organization of an early embryo. They are not created from egg and sperm, which is precisely what makes their legal status uncertain.]
The research is highly controversial and I’m not advertising for it — when I first read about it I was in shock for a couple of hours. But what grips me are the questions it forces open. At what point does something built in a lab from stem cells, never derived from a human, become an entity the law has to care about? If it breathes like a human, and it looks like a human … at what point does it become a human life?
2. What’s overhyped?
Longevity treatments. They are not well tested, we have little idea what most of them do over the long term, and right now they function as a luxury good rather than as medicine.
But the interesting part is what happens if any of it works. Our whole society is built around a predictable lifespan — how long we stay in school, how many years we work, when we retire, how pensions are funded. Change that and the economics collapse. There’s a José Saramago novel, Death with Interruptions, about a country where people stop dying; it sounds like a triumph and turns out to be a catastrophe.
Which points at the real question: what are we extending life for? In a way — if I am allowed a stream of consciousness — it connects directly to AI, where we’re automating away everything tedious to manufacture more free time. More time, more time, more time — and for what? I don’t have an answer, and I find that fascinating.
3. What should Congress or the courts be doing about AI and medicine that they aren’t?
Regulate AI chatbots, for two reasons. The first is validation: Chatbots answer medical questions they have no business answering. States are starting to move — Illinois has a law now, and there’s litigation over chatbots dispensing health advice as the unlicensed practice of medicine. But this calls for federal intervention.
[Editorial note: Illinois enacted the Wellness and Oversight for Psychological Resources Act in August 2025, barring AI systems from providing therapy services without oversight by a licensed professional.]
The second reason is closer to my own and most recent work: What about chatbots whose design changes our behavior? I’ve advocated a lot for AI literacy and I’m no longer sure it’s enough. I’ve been reading about sycophantic AI — systems engineered to please you, to simulate affection — and the finding that stopped me is that people who know perfectly well they’re being flattered don’t disengage anyway. So, warning labels are not the answer. We have to write rules about what these systems may do by design, rather than leaving the user responsible for using the product safely.
I think my European background might shape how I view this. What strikes me about the American debate is that fundamental rights get invoked as a reason not to regulate. In Europe, rights are the thing that calls for regulation. You can say a duty to warn sits in tension with the right to privacy and then balance the two. But rights as a blanket argument against acting at all — I can’t parse it.
4. What book, paper, or case most shaped how you think about this field?
I recently read Nudge (by Richard Thaler and Cass Sunstein), which is now in my husband’s hands. The irony is that it nudges the way you think about your own decision-making — about everything, and especially about new technology.
What stayed with me is how constant it is. A friend and I were talking about assisted reproduction recently, and the feed fills with people saying: I had a successful cycle, here’s the clinic, here’s 50 percent off. So, the question isn’t whether I’m being nudged or manipulated. The question is: toward what? And meanwhile the system is learning from how I react.
Honestly, the book is generous to “nudging”; I’d call much more of it “manipulation.” I’ve been looking at how the EU AI Act defines manipulation in the practices it prohibits outright, and it’s vague and pitched very high. Manipulation starts much lower than that, and that gap will have to close.
5. You’ve argued that gambling is the closest regulatory fit for digital addiction — allow the product, but engineer the addictiveness out of the design. If the U.S. could realistically pass one design safeguard, what should it be?
Removing anything tied to reward signals. Concretely: get rid of the continuous, endless feed, and heavily limit personalization — which in the U.S. depends on a far more relaxed privacy regime than Europe’s. It’s the combination that does the damage, a feed both infinite and over-personalized. Take away one or the other. Ideally both.
Europe is doing well here, at least on the largest platforms. I have a paper coming out arguing that the Digital Services Act — the EU’s platform-regulation regime — already screens for the practices that ought to be banned; the difficulty is operational rather than conceptual. The European Commission has said out loud that certain designs are addictive, so it’s reasonable to expect they’ll act.
Think about how good it felt when Facebook used to tell you that you’d reached the end of the day’s content. The trap now is that when I hit my limit on Instagram, I just switch over and start a conversation with Gemini instead.
6. One class, book, or piece of advice you wish you’d paid more attention to?
Something a colleague told me years ago. I was dreading a conference, convinced I’d be the stupidest person in the room. And they said: You only learn when you’re the person doing most of the listening. If you’re the one talking, you’re generally saying something you already know.
It sounds lovely, but it’s hard to feel stupid and stay open, and academia makes it worse — nobody wants to be the person who doesn’t know. One of my PhD students told me recently that she feels dumb all the time. I told her: Me too. If we ever convinced ourselves we were the smartest people in the room, we’d stop listening. When my colleague first said it to me, I hated it — precisely because it was uncomfortable.
