TY - JOUR

T1 - SSL-MEPR: A Semi-Supervised Multi-Task Cross-Domain Learning Framework for Multimodal Emotion and Personality Recognition

AU - Ryumina, Elena

AU - Axyonov, Alexandr

AU - Koryakovskaya, Darya

AU - Abdulkadirov, Timur

AU - Egorova, Angelina

AU - Fedchin, Sergey

AU - Zaburdaev, Alexander

AU - Ryumin, Dmitry

N1 - Ryumina, E., Axyonov, A., Koryakovskaya, D., Abdulkadirov, T., Egorova, A., Fedchin, S., Zaburdaev, A., & Ryumin, D. (2026). SSL-MEPR: A Semi-Supervised Multi-Task Cross-Domain Learning Framework for Multimodal Emotion and Personality Recognition. Machine Learning and Knowledge Extraction, 8(3), 56. https://doi.org/10.3390/make8030056,

PY - 2026/2/27

Y1 - 2026/2/27

N2 - The growing demand for personalized human–computer interaction calls for methods that jointly model emotional states and personality traits. However, large-scale multimodal corpora annotated for both tasks are still lacking. This challenge stems from integrating diverse, task-specific corpora with divergent modality informativeness and domain characteristics. To address it, we propose SSL-MEPR, a semi-supervised multi-task cross-domain learning framework for Multimodal Emotion and Personality Recognition, which enables cross-task knowledge transfer without jointly labeled data. SSL-MEPR employs a three-stage strategy, progressively integrating unimodal single-task, unimodal multi-task, and multimodal multi-task models. Key innovations include Graph Attention Fusion, task-specific query-based cross-attention, predict projectors, and guide banks, which enable robust fusion and effective use of semi-labeled data via a modified GradNorm method. Evaluated on MOSEI (emotion) and FIv2 (personality), SSL-MEPR achieves a mean Weighted Accuracy (mWACC) of 70.26 and a mean Accuracy (mACC) of 92.88 in single-task cross-domain settings, outperforming state-of-the-art methods. Multi-task learning reveals domain-induced misalignment in modality informativeness but still uncovers consistent psychological patterns: sadness correlates with lower personality trait scores, while happiness aligns with higher ones. This work establishes a new paradigm for extracting cross-task psychological knowledge from disjoint multimodal corpora, demonstrating that semi-supervised multi-task cross-domain learning can bridge annotation gaps while preserving theoretically grounded emotion–personality relationships.

AB - The growing demand for personalized human–computer interaction calls for methods that jointly model emotional states and personality traits. However, large-scale multimodal corpora annotated for both tasks are still lacking. This challenge stems from integrating diverse, task-specific corpora with divergent modality informativeness and domain characteristics. To address it, we propose SSL-MEPR, a semi-supervised multi-task cross-domain learning framework for Multimodal Emotion and Personality Recognition, which enables cross-task knowledge transfer without jointly labeled data. SSL-MEPR employs a three-stage strategy, progressively integrating unimodal single-task, unimodal multi-task, and multimodal multi-task models. Key innovations include Graph Attention Fusion, task-specific query-based cross-attention, predict projectors, and guide banks, which enable robust fusion and effective use of semi-labeled data via a modified GradNorm method. Evaluated on MOSEI (emotion) and FIv2 (personality), SSL-MEPR achieves a mean Weighted Accuracy (mWACC) of 70.26 and a mean Accuracy (mACC) of 92.88 in single-task cross-domain settings, outperforming state-of-the-art methods. Multi-task learning reveals domain-induced misalignment in modality informativeness but still uncovers consistent psychological patterns: sadness correlates with lower personality trait scores, while happiness aligns with higher ones. This work establishes a new paradigm for extracting cross-task psychological knowledge from disjoint multimodal corpora, demonstrating that semi-supervised multi-task cross-domain learning can bridge annotation gaps while preserving theoretically grounded emotion–personality relationships.

KW - multimodal and multi-task knowledge extraction

KW - multimodal emotion recognition;

KW - multimodal personality traits assessment

KW - cross-domain learning

KW - multi-task learning

KW - multimodal and multi-task knowledge extraction

KW - multimodal emotion recognition

KW - multimodal personality traits assessment

KW - semi-supervised learning

UR - https://www.mendeley.com/catalogue/9669dce3-9427-3154-b729-ed51823333ff/

U2 - 10.3390/make8030056

DO - 10.3390/make8030056

M3 - Article

VL - 8

JO - Machine Learning and Knowledge Extraction

JF - Machine Learning and Knowledge Extraction

SN - 2504-4990

IS - 3

M1 - 56

ER -