A Benchmark for Multi-Task Evaluation of Pretrained Models in Medical Report Generation

¹ School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China
² School of Foreign Languages, Guangdong Polytechnic Normal University

^* Corresponding author: a wangruix5@mail.sysu.edu.cn b lcx@gpnu.edu.cn

Abstract

MRG for medical images has become increasingly important due to the growing workload of radiologists in hospitals. However, current studies in the MRG field predominantly focus on specific modal- ities or training foundation models with a notable lack of research evaluating the impact of pre-trained models on performance across different tasks, particularly their cross-task capabilities. This study introduces a novel benchmark for medical multi-task learning that encompasses four medical modalities: CT, X-ray, ultrasound, and pathology. We believe this benchmark can provide a robust comparative basis for future research in this field. More importantly, we conduct an in-depth analysis comparing modality-specific pre-trained models, natural domain pre-trained models, and medical foundation pre-trained models. Our findings indicate that medical foundation pre-trained models generally outperform other pre-trained models across all tasks, while natural domain pre-trained models exhibit superior performance in cross-modality tasks. Our source code is available at https://github.com/Reckless0/MT-Med.git.