Multilingual dyadic interaction corpus noxi+ j: Toward understanding asian-european non-verbal cultural characteristics and their influences on engagement
Marius Funk, Shogo Okada, Elisabeth André
Proceedings of the 26th International Conference on Multimodal Interaction, pp. 224–233, 2024.
Abstract
Non-verbal behavior is a central challenge in understanding the dynamics of a conversation and the affective states between interlocutors arising from the interaction. Although psychological research has demonstrated that non-verbal behaviors vary across cultures, limited computational analysis has been conducted to clarify these differences and assess their impact on engagement recognition. To gain a greater understanding of engagement and non-verbal behaviors among a wide range of cultures and language spheres, in this study we conduct a multilingual computational analysis of non-verbal features and investigate their role in engagement and engagement prediction. To achieve this goal, we first expanded the NoXi dataset, which contains interaction data from participants living in France, Germany, and the United Kingdom, by collecting session data of dyadic conversations in Japanese and Chinese, resulting in the enhanced dataset NoXi+J. Next, we extracted multimodal non-verbal features, including speech acoustics, facial expressions, backchanneling and gestures, via various pattern recognition techniques and algorithms. Then, we conducted a statistical analysis of listening behaviors and backchannel patterns to identify culturally dependent and independent features in each language and common features among multiple languages. These features were also correlated with the engagement shown by the interlocutors. Finally, we analyzed the influence of cultural differences in the input features of LSTM models trained to predict engagement for five language datasets. A SHAP analysis combined with transfer learning confirmed a considerable correlation between the importance of input features for a language set and the significant cultural characteristics analyzed.Links
Paper: funk2024multilingual.pdf
Paper Access: https://dl.acm.org/doi/abs/10.1145/3678957.3685757
BibTeX
@inproceedings{funk2024multilingual,
title = {Multilingual dyadic interaction corpus noxi+ j: Toward understanding asian-european non-verbal cultural characteristics and their influences on engagement},
author = {Funk, Marius and Okada, Shogo and Andr{\'e}, Elisabeth},
booktitle = {Proceedings of the 26th International Conference on Multimodal Interaction},
pages = {224--233},
year = {2024},
doi = {10.1145/3678957.3685757},
url = {https://dl.acm.org/doi/abs/10.1145/3678957.3685757},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA}
}