TY - GEN

T1 - Phonetic and Visual Characteristics of Cognitive Load

AU - Evdokimova, Vera

AU - Maksimova, Maria

N1 - Conference code: 27

PY - 2026

Y1 - 2026

N2 - The study of speech in different emotional, psychophysiological and cognitive states is an important task for the development of speech systems. Cognitive load is the load on a person's cognitive system when performing a task. This paper analyses speech characteristics and facial features that can serve as the markers of cognitive load. Previous research revealed that cognitive load is associated with increasing fundamental frequency (F0), laryngealization, narrowing F0 range, changing articulation rate. Cognitive load can also be recognized using head pose, eye gaze and facial expressions. Two experiments were conducted in order to study speech and facial movements under cognitive load. During the first experiment, the participants played a driving simulator game and answered general knowledge questions simultaneously. Audio and videosamples were recorded. The information about action units (facial muscle movements) was obtained using Open Face 2.2.0. The results revealed that the most frequent visual characteristics of cognitive load are turning eyes to the right (AU62) and dimpler (AU14, the contraction of the buccinator muscle). In the second experiment, three episodes of a talk show were studied. The interviewer was driving a vehicle and conducting an interview as a dual task. The results showed that the most common visual markers of cognitive load are AU01 (inner brow raising), AU02 (outer brow raising), AU05 (upper lid raiser), AU10 (upper lip raiser), AU15 (lip corner depressor). The findings in both experiments suggest that cognitive load could be recognized by movements in the eye area and lip area.

AB - The study of speech in different emotional, psychophysiological and cognitive states is an important task for the development of speech systems. Cognitive load is the load on a person's cognitive system when performing a task. This paper analyses speech characteristics and facial features that can serve as the markers of cognitive load. Previous research revealed that cognitive load is associated with increasing fundamental frequency (F0), laryngealization, narrowing F0 range, changing articulation rate. Cognitive load can also be recognized using head pose, eye gaze and facial expressions. Two experiments were conducted in order to study speech and facial movements under cognitive load. During the first experiment, the participants played a driving simulator game and answered general knowledge questions simultaneously. Audio and videosamples were recorded. The information about action units (facial muscle movements) was obtained using Open Face 2.2.0. The results revealed that the most frequent visual characteristics of cognitive load are turning eyes to the right (AU62) and dimpler (AU14, the contraction of the buccinator muscle). In the second experiment, three episodes of a talk show were studied. The interviewer was driving a vehicle and conducting an interview as a dual task. The results showed that the most common visual markers of cognitive load are AU01 (inner brow raising), AU02 (outer brow raising), AU05 (upper lid raiser), AU10 (upper lip raiser), AU15 (lip corner depressor). The findings in both experiments suggest that cognitive load could be recognized by movements in the eye area and lip area.

KW - Facial Action Coding System

KW - Phonetic and Visual Markers of Cognitive Load

KW - Phonetics

KW - Speech Acoustics

UR - https://www.mendeley.com/catalogue/d94dc79f-bad8-3374-915b-5f7257528d1c/

U2 - 10.1007/978-3-032-07959-6_22

DO - 10.1007/978-3-032-07959-6_22

M3 - Conference contribution

SN - 9783032079589

T3 - Lecture Notes in Computer Science

SP - 302

EP - 317

BT - Speech and Computer. SPECOM 2025.

PB - Springer Nature

T2 - 27th International Conference on Speech and Computer

Y2 - 13 October 2025 through 14 October 2025

ER -