Результаты исследований: Научные публикации в периодических изданиях › статья › Рецензирование
Automated Assessment of Creativity in Multilingual Narratives. / Luchini, S.A.; Moosa, I.M.; Patterson, J.D.; Johnson, D.; Baas, M.; Barbot, B.; Bashmakova, I.; Benedek, M.; Chen, Q.; Corazza, G.E.; Forthmann, B.; Goecke, B.; Said-Metwaly, S.; Karwowski, M.; Kenett, Y.N.; Lebuda, I.; Lubart, T.; Miroshnik, K.G.; Obialo, F.-K.; Ovando-Tellez, M.; Primi, R.; Puente-Díaz, R.; Stevenson, C.; Volle, E.; Zielińska, A.; Van Hell, J.G.; Yin, W.; Beaty, R.E.
в: Psychology of Aesthetics, Creativity, and the Arts, 2025.Результаты исследований: Научные публикации в периодических изданиях › статья › Рецензирование
}
TY - JOUR
T1 - Automated Assessment of Creativity in Multilingual Narratives
AU - Luchini, S.A.
AU - Moosa, I.M.
AU - Patterson, J.D.
AU - Johnson, D.
AU - Baas, M.
AU - Barbot, B.
AU - Bashmakova, I.
AU - Benedek, M.
AU - Chen, Q.
AU - Corazza, G.E.
AU - Forthmann, B.
AU - Goecke, B.
AU - Said-Metwaly, S.
AU - Karwowski, M.
AU - Kenett, Y.N.
AU - Lebuda, I.
AU - Lubart, T.
AU - Miroshnik, K.G.
AU - Obialo, F.-K.
AU - Ovando-Tellez, M.
AU - Primi, R.
AU - Puente-Díaz, R.
AU - Stevenson, C.
AU - Volle, E.
AU - Zielińska, A.
AU - Van Hell, J.G.
AU - Yin, W.
AU - Beaty, R.E.
N1 - Export Date: 01 November 2025; Cited By: 6; Correspondence Address: S.A. Luchini; Department of Psychology, Pennsylvania State University, United States; email: skl5875@psu.edu; R.E. Beaty; Department of Psychology, Pennsylvania State University, United States; email: rebeaty@psu.edu
PY - 2025
Y1 - 2025
N2 - Researchers and educators interested in creative writing need a reliable and efficient tool to score the creativity of narratives, such as short stories. Typically, human raters manually assess narrative creativity, but such subjective scoring is limited by labor costs and rater disagreement. Large language models (LLMs) have shown remarkable success on creativity tasks, yet they have not been applied to scoring narratives, including multilingual stories. In the present study, we aimed to test whether narrative originality—a component of creativity— could be automatically scored by LLMs, further evaluating whether a single LLM could predict human originality ratings across multiple languages.We trained three different LLMs to predict the originality of short stories written in 11 languages. Our first monolingual model, trained only on English stories, robustly predicted human originality ratings (r=.81). This same model—trained and tested on multilingual stories translated into English—strongly predicted originality ratings of multilingual narratives (r≥.73). Finally, a multilingual model trained on the same stories, in their original language, reliably predicted human originality scores across all languages (r ≥.72).We thus demonstrate that LLMs can successfully score narrative creativity in 11 different languages, surpassing the performance of the best previous automated scoring techniques (e.g., semantic distance). This work represents the first effective, accessible, and reliable solution for the automated scoring of creativity in multilingual narratives. © 2025 Elsevier B.V., All rights reserved.
AB - Researchers and educators interested in creative writing need a reliable and efficient tool to score the creativity of narratives, such as short stories. Typically, human raters manually assess narrative creativity, but such subjective scoring is limited by labor costs and rater disagreement. Large language models (LLMs) have shown remarkable success on creativity tasks, yet they have not been applied to scoring narratives, including multilingual stories. In the present study, we aimed to test whether narrative originality—a component of creativity— could be automatically scored by LLMs, further evaluating whether a single LLM could predict human originality ratings across multiple languages.We trained three different LLMs to predict the originality of short stories written in 11 languages. Our first monolingual model, trained only on English stories, robustly predicted human originality ratings (r=.81). This same model—trained and tested on multilingual stories translated into English—strongly predicted originality ratings of multilingual narratives (r≥.73). Finally, a multilingual model trained on the same stories, in their original language, reliably predicted human originality scores across all languages (r ≥.72).We thus demonstrate that LLMs can successfully score narrative creativity in 11 different languages, surpassing the performance of the best previous automated scoring techniques (e.g., semantic distance). This work represents the first effective, accessible, and reliable solution for the automated scoring of creativity in multilingual narratives. © 2025 Elsevier B.V., All rights reserved.
KW - automated scoring
KW - creative writing
KW - creativity assessment
KW - large language models
KW - narratives
UR - https://www.mendeley.com/catalogue/fbfef7b1-5998-3136-bea5-1025907981b4/
U2 - 10.1037/aca0000725
DO - 10.1037/aca0000725
M3 - Article
JO - Psychology of Aesthetics, Creativity, and the Arts
JF - Psychology of Aesthetics, Creativity, and the Arts
SN - 1931-3896
ER -
ID: 143614000