Bringing doctors together

or

A free, private network for verified physicians

Leading voices

Dr. Ram Vaidhyanath

Neuroradiology

View

Dr. Judith Gadde

Neuroradiology · Lurie Children's Hospital

View

Dr. Wende Gibbs

Neuroradiology · Barrow Neurological Institute

View

Dr. Daniel Gewolb

Neuroradiology · University of Miami

View

Dr. Sumir Patel

Neuroradiology · Emory University

View

Dr. Julie Bulman

Interventional Radiology · BIDMC

View

Dr. Monica Sheth

Breast Imaging · NYU Langone

View

Dr. Victoria Chernyak

Abdominal Imaging · CUIMC

View

Dr. Laura Hayes

Peds Radiology · Children's Healthcare of Atlanta

View

Dr. Amy Patel

Breast Imaging · UKHS

View
  • Discover your community

  • Stay current

  • A profile that represents you

  • Find your people

About RoonPrivacyTerms

    Overview

    Assessing the Performance of Artificial Intelligence Models: Insights from the American Society of Functional Neuroradiology Artificial Intelligence Competition.

    Sep 9, 2024DOI: 10.3174/ajnr.a8317Journal ArticleAJNR

    Written by: Bin JiangBurak B OzkaraGuangming ZhuDerek BoothroydJason W AllenDaniel P BarboriakPeter ChangCynthia ChanRuchir ChaudhariHui ChenAnjeza ChukusVictoria DingDavid DouglasChristopher G FilippiAdam E FlandersRyan GodwinSyed HashmiChristopher HessKevin HsuYvonne W LuiJoseph A MaldjianPatrik MichelSahil S NalawadeVishal PatelPrashant RaghavanHaris I SairJody TanabeKirk WelkerChristopher T WhitlowGreg ZaharchukMax Wintermark

    Find these authors on Roon

    KH
    Dr. Kevin Hsu
    Neuroradiology · NYU Langone

    Summary

    BACKGROUND AND PURPOSE: Artificial intelligence models in radiology are frequently developed and validated using data sets from a single institution and are rarely tested on independent, external data sets, raising questions about their generalizability and applicability in clinical practice. The American Society of Functional Neuroradiology (ASFNR) organized a multicenter artificial intelligence competition to evaluate the proficiency of developed models in identifying various pathologies on NCCT, assessing age-based normality and estimating medical urgency.MATERIALS AND METHODS: In total, 1201 anonymized, full-head NCCT clinical scans from 5 institutions were pooled to form the data set. The data set encompassed studies with normal findings as well as those with pathologies, including acute ischemic stroke, intracranial hemorrhage, traumatic brain injury, and mass effect (detection of these, task 1). NCCTs were also assessed to determine if findings were consistent with expected brain changes for the patient's age (task 2: age-based normality assessment) and to identify any abnormalities requiring immediate medical attention (task 3: evaluation of findings for urgent intervention). Five neuroradiologists labeled each NCCT, with consensus interpretations serving as the ground truth. The competition was announced online, inviting academic institutions and companies. Independent central analysis assessed the performance of each model. Accuracy, sensitivity, specificity, positive and negative predictive values, and receiver operating characteristic (ROC) curves were generated for each artificial intelligence model, along with the area under the ROC curve.RESULTS: Four teams processed 1177 studies. The median age of patients was 62 years, with an interquartile range of 33 years. Nineteen teams from various academic institutions registered for the competition. Of these, 4 teams submitted their final results. No commercial entities participated in the competition. For task 1, areas under the ROC curve ranged from 0.49 to 0.59. For task 2, two teams completed the task with area under the ROC curve values of 0.57 and 0.52. For task 3, teams had little-to-no agreement with the ground truth.CONCLUSIONS: To assess the performance of artificial intelligence models in real-world clinical scenarios, we analyzed their performance in the ASFNR Artificial Intelligence Competition. The first ASFNR Competition underscored the gap between expectation and reality; and the models largely fell short in their assessments. As the integration of artificial intelligence tools into clinical workflows increases, neuroradiologists must carefully recognize the capabilities, constraints, and consistency of these technologies. Before institutions adopt these algorithms, thorough validation is essential to ensure acceptable levels of performance in clinical settings.
    KHDr. Kevin Hsu

    Dr. Kevin Hsu

    Radiology · Neuroradiology
    NYU Langone Health
    kevinhsu