Evaluation of Human Factors-Related Risks in AI-Enabled Medical Devices - A Practical Guide
Rebecca Mathias, M.Sc., Anne Schmitt, M.Sc., Mateo Campos, Ph.D., Baptiste Vasey, D.Phil., Sebastian Lorenz, Dipl.-Ing., Peter McCulloch, M.D., and Stephen Gilbert, Ph.D. Abstract
Artificial intelligence (AI)�enabled medical devices show promise for improving health care outcomes, yet real-world effectiveness depends not only on technical performance, but also on how effectively and reliably humans interact with the device and interpret its outputs. In clinical practice, AI systems can introduce unique human factors challenges because of their probabilistic outputs, limited explainability, adaptive behavior, potential for misclassification, and generation of large volumes of data. These characteristics can amplify risks of misperception, misinterpretation, trust miscalibration, automation bias, deskilling, technostress, indication creep, and change- or mode-related errors. Although such risks are increasingly acknowledged in research and guideline development, they are not yet fully embedded in regulatory or health technology assessment processes.
Codeveloped by clinical, regulatory, and human factors experts across health care, academia, and industry, this work addresses the �why, what, and how� of AI usability. Seven human factors�related risks are synthesized into practical guidance aligned with established usability and risk management standards, including defining intended users and exclusions, designing outputs for trust and comprehension, demonstrating workflow integration, supporting training to prevent deskilling, providing safe fallback pathways, monitoring real-world use, and clearly communicating updates. To operationalize these principles, we outline implementation and validation steps compatible with existing regulatory usability deliverables, including use specification, use-related risk analysis, formative and summative evaluation, labeling and training documentation, postmarket surveillance, and change control.
AI-enabled devices will deliver safe and meaningful value only when the interplay between system capability and AI-specific human factors is explicitly addressed across the product life cycle. By pairing premarket usability evidence with postmarket monitoring and highlighting the need for clear allocation of responsibilities among manufacturers, health systems, and assessors, this framework offers a practical, regulator-compatible first step toward embedding AI-specific human factors considerations into routine regulatory and health technology assessments. In doing so, it aims to reduce preventable harm while supporting innovation. (Funded by the European Commission and others.)