眼科画像におけるGPT-4Vの評価

タイトル: Unveiling the Clinical Incapabilities: A Benchmarking Study of GPT-4V(ision) for Ophthalmic Multimodal Image Analysis

概要: PurposeTo evaluate the capabilities and incapabilities of a GPT-4V(ision)-based chatbot in interpreting ocular multimodal images. MethodsWe developed a digital ophthalmologist app using GPT-4V and evaluated its performance with a dataset (60 images, 60 ophthalmic conditions, 6 modalities) that included slit-lamp, scanning laser ophthalmoscopy (SLO), fundus photography of the posterior pole (FPP), optical coherence tomography (OCT), fundus fluorescein angiography (FFA), and ocular ultrasound (OUS) images. The chatbot was tested with ten open-ended questions per image, covering examination identification, lesion detection, diagnosis, and decision support. The responses were manually assessed for accuracy, usability, safety, and diagnosis repeatablity. Auto-evaluation was performed using sentence similarity and GPT-4-based auto-evaluation. ResultsOut of 600 responses, 30.6% were accurate, 21.5% were highly usable, and 55.6% were deemed as no harm. GPT-4V performed best with slit-lamp images, with 42.0%, 38.5%, and 68.5% of the responses being accurate, highly usable, and no harm, respectively. However, its performance was weaker in FPP images, with only 13.7%, 3.7%, and 38.5% in the same categories. GPT-4V correctly identified 95.6% of the imaging modalities and showed varying accuracy in lesion identification (25.6%), diagnosis (16.1%), and decision support (24.0%). The overall repeatability of GPT-4V in diagnosing ocular images was 63% (38/60). The overall sentence similarity between responses generated by GPT-4V and human answers is 55.5%, with Spearman correlations of 0.569 for accuracy and 0.576 for usability. ConclusionGPT-4V currently lacks the reliability needed for clinical decision-making in ophthalmology. Our study serve as a benchmark for enhancing ophthalmic multimodal models. SynopsisOnly 30.6%, 21.5%, and 55.6% responses about ocular multimodal images generated by GPT-4V(ision) were considered accurate, highly usable, no harm, respectively. Currently, GPT-4V lacks the reliability required for clinical decision-making and patient consultation in ophthalmology. O_LIWhat is already known on this topic: First, GPT-4V(ision) exhibited significant advantages in fine-grained world-knowledge-intensive visual question answering. Second, the performance of GPT-4V in the multimodal medical diagnosis domain had been evaluated through case analysis, involving 17 medical systems and 8 modalities used in clinical practice. However, ophthalmic-related images were not included in the study. C_LIO_LIWhat this study adds:Being the first known evaluation of GPT-4Vs capabilities in processing ophthalmic multimodal images, our study adds valuable insights to the existing body of knowledge. Our study highlight the incapabilities of GPT-4V, demonstrating that it currently lacks the essential reliability required for clinical decision-making and patient consultation in ophthalmology. C_LIO_LIHow this study might affect research, practice or policy: The evidence gathered in this study show that continued refinement and testing remain crucial for enhancing the effectiveness of large language models in medical applications. This work provides a benchmark for further investigation in building large language models for processing ophthalmic multimodal images. C_LI