Context.—

Machine learning applications in the pathology clinical domain are emerging rapidly. As decision support systems continue to mature, laboratories will increasingly need guidance to evaluate their performance in clinical practice. Currently there are no formal guidelines to assist pathology laboratories in verification and/or validation of such systems. These recommendations are being proposed for the evaluation of machine learning systems in the clinical practice of pathology.

Objective.—

To propose recommendations for performance evaluation of in vitro diagnostic tests on patient samples that incorporate machine learning as part of the preanalytical, analytical, or postanalytical phases of the laboratory workflow. Topics described include considerations for machine learning model evaluation including risk assessment, predeployment requirements, data sourcing and curation, verification and validation, change control management, human-computer interaction, practitioner training, and competency evaluation.

Data Sources.—

An expert panel performed a review of the literature, Clinical and Laboratory Standards Institute guidance, and laboratory and government regulatory frameworks.

Conclusions.—

Review of the literature and existing documents enabled the development of proposed recommendations. This white paper pertains to performance evaluation of machine learning systems intended to be implemented for clinical patient testing. Further studies with real-world clinical data are encouraged to support these proposed recommendations. Performance evaluation of machine learning models is critical to verification and/or validation of in vitro diagnostic tests using machine learning intended for clinical practice.

This content is only available as a PDF.

Author notes

All authors are members from the Machine Learning Workgroup, Informatics Committee, Digital and Computational Pathology Committee, and Council on Informatics and Pathology Innovation of the College of American Pathologists, except Souers, who is an employee of the College of American Pathologists. Hanna is a consultant for PaigeAI, PathPresenter, and VolastraTX. Krishnamurthy is a consultant on the breast pathology faculty advisory board for Daiichi Sankyo, Inc. and AstraZeneca and serves as a scientific advisory board member for AstraZeneca. Krishnamurthy received an investigator-initiated sponsored research award from PathomIQ Research, sponsored research funding from IBEX Research, and an investigator-initiated sponsored research award from Caliber Inc. Raciti has stock options at Paige (<1%) and has employment and stock compensation at Janssen. Mays’ affiliation with the MITRE Corporation is provided for identification purposes only, and is not intended to convey or imply MITRE’s concurrence with, or support for, the positions, opinions, or viewpoints expressed by the author. The other authors have no relevant financial interest in the products or companies described in this article.