TY - GEN

T1 - A Privacy-Preserving Framework for Cross-Institutional Medical Image Analysis Using Vision-Language Models

AU - Станкова, Елена Николаевна

AU - Жукова, Наталия Александровна

AU - Yang, Jiafeng

PY - 2025

Y1 - 2025

N2 - The healthcare industry faces significant challenges in leveraging patient data across institutions while maintaining privacy, particularly when third-party organizations like insurance companies and banks require medical information for risk assessment. The rapid advancement of large-scale multimodal models, such as Contrastive Language-Image Pre-training (CLIP), holds immense potential for medical applications by enabling cross-modal alignment of visual and textual data. This paper presents a novel framework that combines vertical federated learning with CLIP to enable privacy-preserving medical image analysis across institutional boundaries. Our framework allows secure analysis of distributed medical data without raw data sharing, while optimizing CLIP’s performance for medical applications through Context Optimization. Experimental validation on a dataset of 7023 brain MRI scans demonstrates the framework’s effectiveness, achieving 93.1% accuracy in classifying four types of brain conditions (glioma, meningioma, pituitary, and no tumor) - a substantial improvement from the original pre-trained CLIP model’s 26.3% accuracy. These results establish a practical solution for secure, cross-institutional medical data analysis that maintains patient privacy while enabling critical business decisions in healthcare, insurance, and financial sectors.

AB - The healthcare industry faces significant challenges in leveraging patient data across institutions while maintaining privacy, particularly when third-party organizations like insurance companies and banks require medical information for risk assessment. The rapid advancement of large-scale multimodal models, such as Contrastive Language-Image Pre-training (CLIP), holds immense potential for medical applications by enabling cross-modal alignment of visual and textual data. This paper presents a novel framework that combines vertical federated learning with CLIP to enable privacy-preserving medical image analysis across institutional boundaries. Our framework allows secure analysis of distributed medical data without raw data sharing, while optimizing CLIP’s performance for medical applications through Context Optimization. Experimental validation on a dataset of 7023 brain MRI scans demonstrates the framework’s effectiveness, achieving 93.1% accuracy in classifying four types of brain conditions (glioma, meningioma, pituitary, and no tumor) - a substantial improvement from the original pre-trained CLIP model’s 26.3% accuracy. These results establish a practical solution for secure, cross-institutional medical data analysis that maintains patient privacy while enabling critical business decisions in healthcare, insurance, and financial sectors.

KW - Cross-institutional analysis

KW - Prompt tuning

KW - Vertical federated learning

KW - Vision-language model

UR - https://www.mendeley.com/catalogue/0c491cf3-c102-3464-bf9b-5b4ce9e37681/

U2 - 10.1007/978-3-031-96997-3_20

DO - 10.1007/978-3-031-96997-3_20

M3 - Conference contribution

SN - 978-3-031-96996-6

T3 - Lecture Notes in Computer Science

SP - 320

EP - 331

BT - Computational Science and Its Applications – ICCSA 2025. ICCSA 2025. Lecture Notes in Computer Science,

PB - Springer Nature

T2 - Computational Science and Its Applications – ICCSA 2025 Workshops

Y2 - 30 June 2025 through 3 July 2025

ER -