Clinician Confidence and Diagnostic Accuracy Significantly Improves With Support of Artificial Intelligence in Melanoma Detection

Image courtesy of Derm Net

A recent study published in Nature Communications explored the potential of explainable artificial intelligence (XAI) to play a role in improving accuracy in distinguishing between melanoma and nevus.

Study authors noted that while artificial intelligence systems in dermatology have emerged to aid in diagnostic accuracy of melanoma and nevi, deep neural networks (DNN) lack transparency in decision-making.

Researchers then introduced a multimodal XAI that works to close interpretation gaps, in an effort to assess the way dermatology clinicians engage and interact with XAI. They created an expert-annotated dataset and conducted a phase 3 reader study with 116 participants.

The reader study, conducted between July and December 2022, consisted of three phases. In phase 1, 120 international clinicians specializing in dermatology diagnosed 15 lesions, annotated characteristics using an explanatory ontology, and indicated their confidence.

Phase 2 involved the same participants diagnosing the same lesions with AI support, considering their confidence and trust in the AI.

In phase 3, participants, unaware they diagnosed the same lesions before, used an explainable AI with AI-generated explanations. Participants' confidence and trust in AI were assessed in each phase.

The XAI, trained with annotations from dermatologists, demonstrated a significant balanced accuracy of 81% on the test set, which was greater than a ResNet50 baseline classifier by a large margin.

This significantly greater level of accuracy extended across convolutional neural network (CNN) architectures, with the XAI consistently outperforming 6 out of 8 models tested.

Additionally, the study compared the XAI with 2 state-of-the-art approaches, revealing a competitive balanced accuracy of 81.5%.

Researchers conducted an in-depth evaluation of the XAI model's interpretability by assessing its attention allocation to lesions versus surrounding skin. The XAI demonstrated a substantial advantage over the baseline classifier, focusing on regions within the lesion with a ratio of 35.9 compared to the baseline's 4.1.

The study also assessed the overlap in ontological explanations between clinicians and the XAI, revealing an average explanation overlap of 0.46 for melanoma and 0.23 for nevus. Furthermore, the XAI's region of interest overlap with clinicians outpaced the baseline classifier.

While the XAI significantly elevated clinicians' confidence and trust in their diagnoses, it did not yield a statistically significant improvement in diagnostic accuracy over plain AI support. Additionally, clinicians with more experience in dermoscopy achieved the greatest benefit from XAI support.

Results indicated a positive correlation between trust and overlap, particularly in the context of diagnosing melanoma. Clinicians exhibited heightened trust in machine decisions when their reasoning overlapped.

Study limitations included the testing of the system under artificial conditions and tailoring to a domain-specific ontology.

"Our XAI design was guided by the end user perspective and can iteratively be improved with the results of this study, further clinician feedback, and future work on XAI-human collaboration," according to study authors Chanda et al. "The findings of our reader study illustrate the ability of XAI to enhance clinicians’ diagnostic confidence and furthermore suggest that it has the potential to increase the likelihood of considering machine suggestions."

Reference

Chanda T, Hauser K, Hobelsberger S, et al. Dermatologist-like explainable AI enhances trust and confidence in diagnosing melanoma. Nat Commun. 2024;15(1):524. Published 2024 Jan 15. doi:10.1038/s41467-023-43095-4