Weike Zhao (赵唯珂)

In this study, we quantitatively evaluate the free-text reasoning abilities of various state-of-the-art LLMs, such as DeepSeek-R1 and OpenAI-o3-mini, in assessment recommendation, diagnostic decision, and treatment planning.

RaTEScore is a novel, entity-aware metric to assess the quality of medical reports generated by AI models. It emphasizes crucial medical entities such as diagnostic outcomes and anatomical details, and is robust against complex medical synonyms and sensitive to negation expressions. The evaluations demonstrate that RaTEScore aligns more closely with human preference than existing metrics.

In this paper, we build up an academically accessible, large-scale diagnostic dataset that encompassing 5568 disorders linked with 930 unique ICD-10-CM codes, containing 39,026 cases (192,675 scans). Also, we present a novel architecture that enables processing arbitrary number of input scans from various imaging modalities and initialize a new benchmark for multi-modal multi-anatomy long-tailed diagnosis.

In this report, we evaluate GPT-4V for multimodal medical diagnosis at case studies, covering 17 human body systems, across 8 clinical imaging modalities. As the cases shown, GPT-4V is still far from clinical usage.