The Use of AI by Government Healthcare Agencies: Is It in the Public Interest?

Federal and state departments are rapidly expanding their use of artificial intelligence, and healthcare agencies are no exception. According to the U.S. Department of Health and Human Services (HHS) AI Use Case Inventory, between FY 2024 and 2025 the U.S. Food and Drug Administration (FDA) experienced a 148 percent surge in AI use, the U.S. Centers for Disease Control and Prevention (CDC) an 87 percent increase, the Centers for Medicare and Medicaid Services (CMS) a 78 percent rise, and the National Institutes of Health (NIH) a 51 percent jump. States are following suit.

These developments are being pursued under the banner of boosting efficiency, accuracy, and cost savings. But the accelerating adoption of AI in public healthcare agencies raises a more fundamental question: Are there some governmental decisions for which AI should not be used at all or at least not be used without human oversight? In high‑stakes domains like healthcare, the answer may be yes. As the technology becomes more complex, opaque, and autonomous, the potential for harm may increase. Greater complexity can make errors harder to detect and when an error occurs, tracing its source and correcting it can be difficult, increasing the risk that mistakes go unnoticed. In healthcare, even rare errors can have serious consequences for patients.

Types of AI used by health care and other agencies have progressed over the last several years from rule-based algorithms, to predictive machine-learning systems, and most recently, generative and agentive systems. Less than a decade ago, health agencies relied primarily on rule‑based algorithms. They were transparent, deterministic, and relatively easy to audit. For example, in 2020, as the nation confronted a deadly and uncontrolled COVID‑19 virus and no vaccine yet existed, states allocated scarce supplies of monoclonal antibodies available under an FDA emergency use authorization. To maximize lives saved, some states, such as Utah and New York, adopted algorithms that prioritized individuals by race and/or sex. These were simple algorithms; their logic was explicit, and their goals transparent. This transparency also enabled accountability via legal challenges under anti‑discrimination laws. 

Fast forward to 2023 and the Medicaid “unwinding.” Again, states relied on rule‑based algorithms — this time to determine who remained eligible for Medicaid. But programmer errors, data flaws, and inadequate oversight led to hundreds of thousands of improper disenrollments. A successful class action suit against TennCare alleged violations of the Medicaid Act, the ADA, and due process, stemming in part from TennCare’s computerized Eligibility Determination System (TEDS). Around the same time, the National Health Law Program filed a complaint with the FTC alleging that Deloitte Consulting’s defective eligibility software caused widespread improper terminations in Texas, although the software was used by 20 other states. The FTC has not acted, but the allegations underscore the risks of automated eligibility systems even before the arrival of more advanced AI.

Since 2023, AI capabilities have progressed at rapid pace, with major leaps in model performance, computing capacity, and real‑world deployment. Agencies are increasingly adopting machine‑learning models, “black box” systems, and generative large language models (LLMs). A recent U.S. Government Accountability Office (GAO) report found that while overall AI use cases across 11 federal agencies doubled from 2023 to 2024, generative AI use cases increased nine‑fold, with health-based agencies leading the way. States are also experimenting with generative AI and producing guidance on using the technology.

While documented harms associated with predictive machine learning systems and generative AI remain a small fraction of their use in healthcare, a growing body of evidence highlights serious risks. Studies reveal troubling vulnerabilities including diagnostic errors, unsafe triage advice, privacy breaches, and questionable applications in public sector decision‑making. One striking example involves Medicare Advantage preauthorization. Beneficiaries filed a class action suit alleging that UnitedHealth Group used an AI program to deny medically necessary post‑acute care. In March 2026, a federal district court ordered the defendants to disclose the AI’s error rates, which plaintiff’s alleged were 90 percent, meaning “nine of 10 appealed denials were ultimately reversed.” 

These examples illustrate why government healthcare agencies must conduct comprehensive evaluations before deploying AI systems that could cause serious harm. Existing federal frameworks require AI impact assessments for “high‑impact” uses affecting individual rights or safety. But these frameworks assume that AI will be deployed and focus on mitigating risks, not determining whether AI should be used in the first place. The assessments require agencies to explain the system’s purpose and benefits; evaluate data quality and model fitness; assess impacts on privacy, civil rights, and civil liberties; analyze costs; and obtain independent review. These steps, however, do not answer the threshold question: Should AI be used for this decision at all?

Agencies currently lack a method for determining when AI use is justified, what level of human judgment is required, and how the nature of the governmental function should shape the design and deployment of automated systems. Without a consistent framework, agencies risk making AI deployment decisions that fail to satisfy administrative law principles of arbitrariness review and procedural regularity.

To fill this gap, I propose a Five Factor Framework (FFF) requiring an inquiry into several interrelated dimensions of the decision environment and the doctrinal commitments that shape administrative decision‑making. The framework calls on agencies to:

1. identify the interests at stake — who may be affected and what harms or rights are implicated.
2. examine the nature and normative structure of the decision itself: its functional demands, procedural requirements, and value‑laden qualities, situated within the APA’s requirements of reasoned decision‑making. 
3. assess the characteristics of the technology proposed to perform the task, including its capabilities, limitations, and error modes, considering doctrines governing arbitrary and capricious review. 
4. evaluate the role of humans in the decision‑making process — how human judgment, oversight, and discretion are integrated or displaced, and whether the allocation of authority is consistent with nondelegation principles, due process requirements, and statutory expectations.
5. identify the legal constraints governing the decision, from due process and civil rights protections to sector‑specific statutory requirements.

Together, these five inquiries provide a comprehensive, doctrinally grounded foundation for determining whether a technological intervention aligns with the public interest. If an agency concludes that the benefits of using AI for a high-impact decision outweigh the risks, it should seek public input on its assessment — through notice and comment, town halls, and/or engagement with advocacy groups representing affected communities. Democratic legitimacy requires more than technical compliance; it requires public participation in decisions that shape access to healthcare.

In the end, the question is not whether AI can make government healthcare systems more efficient; it is whether efficiency is worth the price of AI errors that cost people their health, their rights, or their lives. Without a principled framework guiding when AI should or should not be used, agencies risk automating harm at scale. The public deserves better, and the law demands it.