TY - JOUR

T1 - MM-PSYCHE: Multimodal Multitask Psychological Characteristic Estimation Through Cross-Domain Semi-Supervised Learning

AU - Егорова, Ангелина Денисовна

N1 - E. Ryumina et al., "MM-PSYCHE: Multimodal Multitask Psychological Characteristic Estimation Through Cross-Domain Semi-Supervised Learning," in IEEE Access, vol. 14, pp. 124759-124778, 2026, doi: 10.1109/ACCESS.2026.3723020. keywords: {Modeling;Labeling;Multitasking;Erbium;Videos;Learning (artificial intelligence);Conferences;Transformers;Machine learning;Visualization;Affective computing;personality computing;psychological characteristic recognition;multimodal learning;semi-supervised learning;cross-domain adaptation;multitask learning;human-computer interaction

PY - 2026/7/23

Y1 - 2026/7/23

N2 - Psychological characteristic estimation from multimodal in-the-wild behavior is usually studied using separate corpora, each annotated for a single target task. Such annotation fragmentation limits cross-task learning and cross-domain generalization across affective, dispositional, and interactional phenomena. To address this problem, we use emotion, apparent personality trait, and ambivalence recognition as representative tasks and introduce MM-PSYCHE, a framework that unifies heterogeneous single-task corpora through cross-domain semi-supervised learning (SSL) with GradNorm-based adaptive task weighting. The framework jointly trains models for emotion, apparent personality trait, and ambivalence recognition. Its multimodal fusion model contains three task-aware components: (i) Task-Specific Projectors, which produce task-conditioned prediction embeddings and re-embed them into a hidden space; (ii) Task-Specific Query-Based Cross-Attention Fusion, which uses the prediction embeddings as task-dependent queries and shared modality embeddings as keys and values; and (iii) Guide Banks, which are learnable task-specific prototypes that anchor the final predictions. Experiments on the task-specific corpora show competitive supervised learning (SL) performance and more pronounced gains in cross-domain SSL, where heterogeneous corpora can be exploited without joint annotations. On the unseen Multimodal EmotionLines Dataset (MELD), multitask SSL improves generalization in emotion recognition, increasing the macro-F1 score by 7.5 percentage points (35.0% vs. 27.5%) over single-task SSL. The results indicate that cross-domain SSL is useful for fragmented multimodal corpora with heterogeneous psychological annotations, while also showing why single-task SL, single-task SSL, and multitask SSL should be evaluated separately. The code will be made available upon publication.

AB - Psychological characteristic estimation from multimodal in-the-wild behavior is usually studied using separate corpora, each annotated for a single target task. Such annotation fragmentation limits cross-task learning and cross-domain generalization across affective, dispositional, and interactional phenomena. To address this problem, we use emotion, apparent personality trait, and ambivalence recognition as representative tasks and introduce MM-PSYCHE, a framework that unifies heterogeneous single-task corpora through cross-domain semi-supervised learning (SSL) with GradNorm-based adaptive task weighting. The framework jointly trains models for emotion, apparent personality trait, and ambivalence recognition. Its multimodal fusion model contains three task-aware components: (i) Task-Specific Projectors, which produce task-conditioned prediction embeddings and re-embed them into a hidden space; (ii) Task-Specific Query-Based Cross-Attention Fusion, which uses the prediction embeddings as task-dependent queries and shared modality embeddings as keys and values; and (iii) Guide Banks, which are learnable task-specific prototypes that anchor the final predictions. Experiments on the task-specific corpora show competitive supervised learning (SL) performance and more pronounced gains in cross-domain SSL, where heterogeneous corpora can be exploited without joint annotations. On the unseen Multimodal EmotionLines Dataset (MELD), multitask SSL improves generalization in emotion recognition, increasing the macro-F1 score by 7.5 percentage points (35.0% vs. 27.5%) over single-task SSL. The results indicate that cross-domain SSL is useful for fragmented multimodal corpora with heterogeneous psychological annotations, while also showing why single-task SL, single-task SSL, and multitask SSL should be evaluated separately. The code will be made available upon publication.

KW - Affective computing

KW - cross-domain adaptation

KW - human-computer interaction

KW - multimodal learning

KW - multitask learning

KW - personality computing

KW - psychological characteristic recognition

KW - semi-supervised learning

UR - https://www.mendeley.com/catalogue/61566b88-6af5-3bdd-83ca-5757464bd545/

U2 - 10.1109/ACCESS.2026.3723020

DO - 10.1109/ACCESS.2026.3723020

M3 - Article

VL - 14

SP - 124759

EP - 124778

JO - IEEE Access

JF - IEEE Access

SN - 2169-3536

ER -