SATO Wataru Laboratory

Development of machine learning-based facial thermal image analysis for dynamic emotion sensing

(Tang, Sato, & Kawanishi: Sensors)

Information on the relationship between facial thermal responses and emotional state is valuable for sensing emotion.

Yet, previous research typically relied on linear analysis methods based on regions of interest (ROIs), which may overlook nonlinear pixel-wise information across the face.

To address this limitation, we investigated the use of machine learning (ML) for pixel-level analysis of facial thermal images to estimate dynamic emotional arousal ratings.
We collected facial thermal data from 20 participants who viewed five emotion-eliciting films and assessed their dynamic emotional self-reports.

Our ML models, including random forest regression, support vector regression, ResNet-18, and ResNet-34, consistently demonstrated superior estimation performance compared to traditional simple or multiple linear regression models for the ROIs.
To interpret the nonlinear relationships between facial temperature changes and arousal, saliency maps and integrated gradients were used for the ResNet-34 model.
The results showed nonlinear associations of arousal ratings in nose tip, forehead, and cheek temperature changes.

These findings suggest that ML-based analysis of facial thermal images can more effectively estimate emotional arousal, pointing to potential applications of non-invasive emotion sensing for mental health, education, and human-computer interaction.

Return to Recent Research.
Return to Main Menu.