The Disability Gap in Healthcare AI: From Bias to Inclusive Co-Design, Testing, Deployment, and Governance

As artificial intelligence becomes increasingly embedded in healthcare, its implications for people with disabilities warrant particular attention. While AI can expand access to health information, services, and support, systems designed around assumptions of a “typical” user or patient may create new barriers. When people with disabilities are underrepresented in datasets, these systems may perform less accurately for individuals whose speech, movement, communication, or health data differs from the presumed norm. Algorithmic accuracy is therefore closely connected to broader concerns about accessibility, discrimination, privacy, safety, and participation.

This tension is playing out today as federal health agencies accelerate pilot programs deploying autonomous AI agents tasked with diagnosing and prescribing treatment to patients. While more than 1,600 AI-enabled medical devices have received FDA approval for specific tasks such as diagnosing skin cancer from imaging or estimating a patient’s probability of a heart attack, it has yet to adopt a regulatory framework for the latest wave of AI, namely foundation models and agentic AI, which are starting to take on roles once reserved only for physicians. Deploying autonomous tools to “expand access” to medical care without prior accessibility and bias safeguards risks locking disabled patients into automated, lower-quality care pathways. The main issue is not simply weighing the benefits of AI against its risks, but ensuring that these systems are designed, tested, deployed, and governed in ways that account for the diversity of the populations they affect.

Several characteristics of AI systems may create specific risks for individuals with disabilities. Algorithmic systems generally rely on pattern recognition and make determinations based on patterns represented as typical within their datasets. Disability-related characteristics — including differences in gait, speech, or eye movement — may fall outside these patterns and consequently produce inaccurate or discriminatory outcomes, particularly when biometric information is involved. These problems may be compounded by training datasets that underrepresent disabled people or contain inaccurate or improperly classified disability-related information. Moreover, disability-related bias may already be embedded in the data on which AI systems are trained, including clinical notes and other records. As a result, even systems that appear neutral in their assessment may reproduce or amplify biased assumptions already present in the training data. People with disabilities with intersectional identity characteristics such as women, Indigenous People, LGBTQI+, or youth, may also be multiply marginalized and therefore face overlapping forms of algorithmic discrimination.

These risks are not merely theoretical. STAT’s Denied by AI investigation documented cases in which senior patients enrolled in Medicare Advantage had their rehabilitation coverage cut off based on algorithmic predictions, despite evidence that they still needed care. The investigation also found that UnitedHealth pressured its medical staff to follow algorithmic predictions for rehabilitation stays and cut off payments for seriously ill patients, including older and disabled Americans. While these examples involve predictive algorithms, similar risks are emerging with large language models and agentic AI in healthcare. Dr. John Whyte, Chief Executive of the American Medical Association, warned that autonomous AI models are “not ready for prime time,” as the supporting evidence relies mainly on simulations. Meanwhile, the FDA has yet to finalize its approach to overseeing advanced AI models in clinical settings, especially amid concerns that autonomous systems could go rogue.

At the same time, AI and other emerging technologies may improve healthcare for patients, including those with disabilities. In a survey, patients, patient advocates, and caregivers, identified improved disease screening, reduced human error and bias, better access to care, more personalized treatment, more accurate diagnoses, and better prediction of future health issues as potential benefits. Access is one area where these tools hold promise. AI agents such as Limbic for mental health therapy and Curai Health for primary care could reduce wait times and lower costs, extending services to underserved populations.

The challenge is therefore not whether these technologies should be used, but how accessibility, discrimination, bias, privacy, and safety can be addressed while preserving their benefits. This requires considering disability throughout the AI lifecycle rather than treating accessibility as an issue to be addressed only after deployment. The process could incorporate co-design by drawing on the lived experience and knowledge of people with disabilities alongside relevant disability expertise, as well as increasing transparency about the use of AI in healthcare decisions or communications, especially in high-risk use cases that significantly affect privacy, safety, or legal rights.

Existing guidance reinforces these practices. The CDT-AAPD report recommends disability-bias audits of algorithmic systems already used by federally funded hospitals. The World Health Organization (WHO) guidance on the ethics and governance of large multi-modal models (LMMs) similarly calls for post-deployment auditing and impact assessments that disaggregate outcomes by characteristics including disability, as well as the involvement of patients and other affected stakeholders from the early stages of development. More recently, a report by WHO/Europe identified robust governance frameworks, co-design with patients and clinicians, transparency, and equitable access to data and tools as important enablers of responsible AI in health.

Applying these principles concretely, a diagnostic decision-support system that recommends additional testing or specialist referral, for instance, could be evaluated using otherwise comparable patient scenarios that vary relevant disability-related characteristics, including characteristics affecting how symptoms are presented or recorded. If the system produces different recommendations, evaluators could examine whether the variation reflects relevant circumstances or whether the system fails to adequately account for a disability-related presentation, directly addressing the concern that systems trained on typical patterns may produce inaccurate outcomes when disability-related characteristics fall outside those patterns.

As AI becomes more deeply integrated into healthcare, the challenge is ensuring its benefits reach patients with different disabilities without introducing new forms of exclusion or discrimination. Patients should also have greater control over their health information, particularly given that their willingness to share data may depend on its purpose and recipient. Incorporating these safeguards could help preserve AI’s potential benefits while addressing concerns about error, bias, discrimination, privacy, safety, and transparency for patients with disabilities.