PURPOSE: This study evaluates the performance of four large language models (LLMs) in classifying malignant lymphoma stages using the Lugano classification from free-text FDG-PET reports in Japanese Specifically, we assess GPT-4o, Claude 3.5 Sonnet, Llama 3 70B, and Gemma 2 27B in their ability interpret unstructured radiology texts.

MATERIALS AND METHODS: In a retrospective single-center study, 80 patients who underwent staging FDG-PET/CT for malignant lymphoma were included. The "Findings" sections of their reports were analyzed without pre-processing. Each LLM assigned Lugano stages based on these reports. Performance was compared to reference standard stages determined by expert radiologists. Statistical analyses involved overall accuracy, weighted kappa for agreement.

RESULTS: GPT-4o achieved the highest accuracy at 75% (60/80 cases) with substantial agreement (weighted kappa κ = 0.801). Claude 3.5 Sonnet had 61.3% accuracy (49/80, κ = 0.763). Gemma 2 27B and Llama 3 70B showed accuracies of 58.8% and 57.5%, respectively, all indicating substantial agreement.

CONCLUSION: GPT-4o outperformed other LLMs in assigning Lugano classification from Japanese FDG-PET free-text reports. This demonstrated the potential of advanced LLMs to interpret clinical texts. While the immediate clinical utility of automatically predicting a Lugano stage from an existing report may be limited, these results highlight the value of LLMs for understanding and standardizing free-text radiology data.