Towards a robust healthcare prediction model using an adaptive multimodal fusion based on hierarchical transformers

Computer Science Research Laboratory(LRI), Faculty of Sciences, Ibn Tofail University, Morocco

Abstract

The Healthcare field has applied multimodal learning that combines diverse data types to improve the quality of predictions in clinical settings in terms of precision. The limitations of such multimodal approaches are highly related to heterogeneous structure, modality relevance, noise, and data size. In this paper, we present an adaptive multimodal fusion model that assigns distinct importance weights to modalities via attention-based pooling within a hierarchical transformer architecture. Indeed, the proposed model derives features for each modality independently and then aggregates cross-modal features using a hierarchical attention mechanism. We evaluate our architecture in a controlled setting across multiple simulations, and we demonstrate that our model performs effectively across four data types for predicting health. Besides, our experiments revealed clear improvements in validation AUC and F1 scores, especially when the data is limited or noisy, which supports the strength and reliability of our hierarchical transformer-based fusion approach.