Heterogeneous Graph-Based Facial Pain Modeling with Ordinal Boundary Analysis and Cross-Dataset Transfer

Abstract

Automatic facial pain assessment remains challenging due to the subtle and continuous nature of facial expressions and the substantial variability between experimental and clinical datasets. While recent deep learning approaches report strong within-dataset performance using fine-grained pain labels, it remains unclear whether such representations generalize across domains. In this work, we present a diagnostic study of facial pain representations using a structured temporal graph model based on a Heterogeneous Graph Transformer (HGT) with temporal convolution and Long Short Term Memory (LSTM) layers. Pain-relevant facial Action Units and head pose features are explicitly modelled as heterogeneous graph nodes with clinically motivated connections. Fine-grained multiclass pain modelling (levels 0–4) is first evaluated on the BioVid Heat Pain dataset and compared with state-of-the-art methods. Although competitive performance is achieved, prediction errors occur predominantly between neighbouring pain levels, while boundary-wise AUC analysis shows greater separability at higher pain intensities. Based on this analysis, three binary pain-label formulations are evaluated for cross-dataset transfer to UNBC-McMaster. The results show that pain-label grouping influences cross-dataset discrimination. The Extreme formulation achieves the highest discrimination but excludes intermediate BioVid pain levels, while the Coarse formulation retains the complete severity range with improved discrimination over the Presence formulation. Correlation analysis further shows that stronger binary discrimination does not necessarily correspond to a stronger association with the original clinical pain ratings. Overall, the findings highlight the trade-off between pain-level separability, severity coverage, and cross-dataset discrimination.