AI-based tools have failed to reach the full diagnostic potential in COVID-19, study finds

Published by the Journal of Radiology: Artificial Intelligence, a prospective observational study across 12 hospital systems from the University of Minnesota Medical School evaluated the real-time performance of an interpretable artificial intelligence (AI) model to detect COVID-19 from chest X-rays.

Participants with COVID-19 had a significantly higher COVID-19 diagnostic score than participants who did not have COVID-19. However, researchers found the real-time model performance was unchanged over the 19 weeks of implementation.

The model sensitivity was significantly higher in men, while model specificity was significantly higher in women. Sensitivity was significantly higher for Asian and Black participants compared to white participants. The COVID-19 AI diagnostic system had significantly worse accuracy as compared to radiologist predictions.

This study, which represents the first live investigation of an AI COVID-19 diagnostic model, highlights the potential benefits but also the limitations of AI. While promising, AI-based tools have not yet reached full diagnostic potential."

Christopher Tignanelli MD, MS, FACS, FAMIA, Associate Professor of Surgery, University of Minnesota Medical School

Tignanelli is also a general surgeon at M Health Fairview.

The research findings were informed by an AI algorithm developed by Ju Sun, an assistant professor at the U of M College of Science and Engineering, and his team in collaboration with M Health Fairview and Epic.

• COVID-19 diagnostic models perform well for participants with severe COVID-19 effects; however, they fail to differentiate participants with mild COVID-19 effects.
• Many of the early AI models in the pandemic that were published boasted overly optimistic performance metrics using publicly available datasets.
• The AI model's diagnostic accuracy was inferior to the predictions made by board-certified radiologists.

"We saw the same overly optimistic performance in this study when we validated against two publicly available datasets; however, as we showed in our manuscript, this does not translate to the real world," Dr. Tignanelli said. "It is imperative moving forward that researchers and journals alike develop standards requiring external or real-time prospective validation for peer-reviewed AI manuscripts."

Researchers hope to develop a simpler diagnostic AI model by integrating data from more than 40 U.S. and European sites and multi-modal models that leverage structured data and clinical notes along with images.