Introduction

Facial expressions provide cues to emotions being experienced by mammals and can yield valuable information about their internal states.¹⁷ Macaques are used extensively for research worldwide,^10,18 and negative experiences can significantly affect their physiologic, psychologic, and behavioral responses during or after an experimental procedure.⁴² According to the Association of Primate Veterinarians, pain is a debilitating condition that affects an animal’s quality of life and, as a consequence, may negatively impact scientific results and increase the variability of animal-based research data.⁵ While legislation to enforce the ethical treatment of research animals has improved over the years, it still varies by country and relies heavily on self-regulation.³⁷ The ethical debate on animal experimentation and the 3Rs (Reduction, Refinement, Replacement) principle emphasizes the importance of assessing and treating pain to minimize the suffering of research animals. Recognizing pain and evaluating its severity are critical components of this ethical framework as they guide the treatment and assessment frequency for pain. However, the assessment of pain in nonhuman primates is greatly derived from anecdotal evidence due to a lack of comprehensive assessment tools.^11,35,40 Other reasons include the lack of time and resources for intensive monitoring and inherent difficulties in recognizing pain in nonverbal beings. Therefore, an evaluation method that does not require complete human management and does not increase workload is desirable.

Macaques live in societies with frequent competition among group members and may hide behaviors associated with weakness from conspecifics and potential predators. Among captive nonhuman primates, the presence of an observer has been shown to influence the spontaneous behaviors of the animals, making the animal appear to be healthier than its actual status.²¹ This suggests that direct human observation may alter an animal’s spontaneous actions, thereby influencing the observer’s assessment of their condition. Pain evaluation may include facial expressions, as it has been reported that they can be an important indicator of pain in several species.^16,20,27,33 Methods such as grimace scales¹³ or geometric morphometrics^20,22 are used to evaluate facial expressions in mammals, but they depend on a human coder. The disadvantages of having people performing this task include the need to extensively evaluate video records, the time-consuming and labor-intensive image observation or annotation,³ the need to train observers to use the system correctly, and the inherent human bias in the evaluation process. For example, proficiency in the human facial action system (FACS) requires approximately 50 to 100 h of training, and experts require about 2 h to code each minute of video.²⁹ Although reports of facial expressions as indicators of pain in macaques are scarce, they are helpful resources and complement other existing indicators.^16,17,22,39

Automated recognition systems have the potential to objectively assess pain in nonverbal humans^6,7 and other animals.⁸ These systems have been particularly useful for identifying pain in horses undergoing castration, allowing for efficient treatment.^2,28 Artificial neural networks (ANNs) are a branch of machine learning loosely inspired by the brain, consisting of thousands of interconnected nodes, organized into layers, that conduct information. ANNs are particularly useful in computer vision tasks, speech recognition, and medical image analysis.¹ To perform a task, ANNs typically use training examples (that is, previously identified data), such as images. In the case of object recognition, the system can be trained with thousands of labeled images (for example, house, truck, tree). By adjusting its parameters, the network can learn to match input images with corresponding output labels. Similarly, the system can be trained to recognize and classify images for the assessment of pain states in animals. However, recognizing pain from animal facial expressions, especially from macaques, is complicated by their subtle signals and potential masking behavior in the presence of observers.^21,22,39 To automate the classification of pain from facial images in macaques, videos must be recorded in the absence of observers; the macaque face must be located in the video frames and then used to train the system to perform the classification task.

The use of deep learning for image analysis provides several advantages, such as significantly reducing the need for manual image capture and annotation while also minimizing evaluation bias. This study aims to evaluate 2 models for facial recognition and frame extraction, as well as a training model for classifying pain in Japanese macaques using deep learning techniques.

Materials and Methods

Face detection and frame extraction.

Facial images were captured from 30 to 60 min of video footage of the macaques under 2 different conditions: before surgery (No Pain [NP]) and before receiving analgesic medication on the morning after surgery (Pain [P]), by using cameras (GoPro HERO6Black, HERO7Black, and HERO8Black) attached to the cage bars (Figure 1). The observer was not in the room during the recording session. The video recording was taken before the daily administration of analgesics to capture images at the time considered to have minimal analgesic benefit based on the pharmacokinetics of the analgesics.³⁶ Pain was considered to represent the most informative condition for facial pain changes, while NP was categorized as pain free. Automatic sequential video processing was performed to localize the region of the frame that contained faces to build the dataset.

View Figure

• Download as Powerpoint
• Download Figure

Two facial location and frame extraction systems were compared: box extraction and contour extraction. For box extraction, we employed RetinaFace,¹⁵ which localizes the face and yields a bounding box around it. For the Contour extraction, we used Mask R-CNN,²³ which marks the specific pixels in the image that belong to the face as compared with using coarse bounding boxes during object localization. Therefore, the image resulting from contour extraction is a polygon outline of the face (Figure 2).

View Figure

• Download as Powerpoint
• Download Figure

Box extraction.

In the first set of experiments, RetinaFace was used to detect and capture the macaque face in the frame. A total of 68 videos were processed, resulting in 70,852 images. The extracted images were labeled based on the macaque’s condition (P/NP). Three frames per second were automatically extracted, allowing the capture of different versions of the face without intensive computation or bias. Redundant data were removed based on a high incidence of pairwise similarities as detected by the histogram of oriented gradients (HOG) (Figure 3). HOG is based on feature descriptors, which help to extract useful information while discarding the unnecessary parts. The HOG’s 0.9 threshold resulted in 15,987 images, which were then classified by the pretrained neural network ResNet50.²⁴ After experiments 1 and 2 (E1 and E2, described below), we manually excluded profile and blurred and occluded images or images containing elements other than the face resulting in a dataset of 11,445 pictures.

View Figure

• Download as Powerpoint
• Download Figure

Contour extraction.

In the second set of experiments, Mask R-CNN was used for object recognition and frame extraction. The same 68 videos were analyzed, resulting in 54,542 images. Masking was used to allow capture of only the face in the images, excluding any background or nearby objects. The presence of unnecessary data in a machine-learning model can be detrimental to its performance. For example, if the model uses irrelevant information, such as background and objects in the cage, to classify the images, it may mistakenly assume that all other pictures containing that background and objects belong to the same category. This can lead to inaccurate predictions and decreased accuracy. The extracted images were also converted to grayscale, brightness was equalized, and images were manually selected for suitability (Figures 4 and 5). After redundant data reduction with HOG and manual selection, the dataset comprised 19,216 images.

View Figure

• Download as Powerpoint
• Download Figure

View Figure

• Download as Powerpoint
• Download Figure

Neural network training.

ResNet50 was used for image classification.²⁴ A backpropagation algorithm was used to train the multilayer networks, thereby minimizing the loss function that quantifies the difference between the model outputs and correct labels, NP or P, for images representing NP or P. To increase the amount of training data, real-time data augmentation was applied by using minor random modifications of the images, such as rotations, zoom, shifting, and horizontal flipping. We did not use predefined pain indicators, and the classification algorithm relies only on the current image being presented to the ANN.

The experiments were conducted to study the influence of 3 factors: the number of trained layers, image preprocessing, and generalization to nontrained datasets. We assessed the model’s overall performance using accuracy, recall (sensitivity), precision (positive predictive value), and F1-score. The accuracy indicates the proportion of pictures correctly classified by the ANN. However, the accuracy does not indicate the degree of bias of the ANN. For example, 50% accuracy may result from classifying images as P or NP 100% of the time. Therefore, ANN performance in binary classification can be described in more detail using recall, precision, and F1-score. Recall indicates how many times the model was able to detect a specific category (that is, of all pain images, what fraction is correctly detected). Precision indicates the fraction of samples classified as pain that are truly pain images. The F1-score summarizes the precision and recall by taking their harmonic mean.$$\text{Accuracy}=\frac{\text{True} \text{Positive}+\text{True} \text{Negative}}{\text{Total} \text{of} \text{images}}$$$$\text{Recall}=\frac{\text{True} \text{Positive}}{\text{True} \text{Positive}+\text{False} \text{Negative}}$$$$\text{Precision}=\frac{\text{True} \text{Positive}}{\text{True} \text{Positive}+\text{False} \text{Positive}}$$$$\text{F}1=\frac{2\times \text{Precision}\times \text{Recall}}{\text{Precision}+\text{Recall}}$$

Six experiments (E) were conducted after box extraction of the face from the videos using RetinaFace. Each experiment comprises a session of training and testing of images. The first layers of an ANN are usually not unlocked during training because they recognize the basic geometrical structures of the objects. The last layer is the one typically modified and refined to recognize a new set of classes, such as P or NP in facial expressions. We compared the number of trained layers in E1 and E2 when using the whole dataset. In E1, only the last layer (a fully connected layer with 256 ReLU units) was modified to classify images, and E2 permitted the training of all 50 layers. For E3, E4, E5, and E6, images were manually selected. We also compared the number of trained layers in E3 (only the last layer was modified) and E4 (all layers were modified). For E5 and E6, the dataset was further refined and contained only paired data (that is, training data included only images of the same individual before and after surgery). E5 and E6 also excluded the datasets of 2 animals to test the model’s generalization (that is, an estimate of how well the system can classify novel data). The generalization test set comprised 1,088 images: 544 images each of P and NP classes.

E7 to E30 were conducted after masking extraction of the face from the videos using Mask R-CNN (for an outline of the study, see Figure 6). We compared the number of trained layers in E7 (the last layer modified) and E8 (all layers modified). E9 (the last layer modified) and E10 (all layers modified) used only paired data, and the datasets of 2 animals were not used for training to test the model’s generalization. The goal of E11 to E30 was to improve accuracy for generalization. Therefore, we ran 20 trained ANNs using 2-stage training; this allowed fine-tuning of learning. The generalization test set contained 1,586 images (793 images each of P and NP), using the parameters shown in Table 1.

View Figure

• Download as Powerpoint
• Download Figure

Table 1.Architecture and hyperparameters of the neural network ResNet50

Architecture	ResNet-50 - 256 FC (0.5 dropout) - 512 FC (0.5 dropout) - 256 FC (0.5 dropout)
Hyperparameters	l2_reg = 0.0001 lr_decay = lr × sqrt(batch_size/(train_size × epochs))
Two-stage training	(decreasing the learning rate at the second stage to fine-tune)
Stage I	All layers updated, lr = 5 × 10-6 EarlyStopping (1) config: monitor = “val_accuracy,” patience = 10
Stage II	All layers updated, lr = lr/100 EarlyStopping (2) config: monitor = “val_accuracy,” patience = 20

View Fullscreen

Results

Tables 2, 3, and 4 show classification performances across experiments. The results of the tests using RetinaFace for facial image capture and classification with ResNet50 are as follows: E1, which considered only the modification of the last layer of the ANN and contained all images, resulted in an accuracy of 69%. After excluding unsuitable images, E3 resulted in an accuracy of 70%. The exclusion of unsuitable images and generalization test to 2 novel macaques in E5 resulted in an accuracy of 48% (Table 2).

Results for tests using Mask R-CNN for facial image capture and classification with ResNet50 are as follows: E7, which excluded unsuitable images, resulted in an accuracy of 72%. E9, which excluded both unsuitable images and tested the generalization to 2 novel macaques, resulted in an accuracy of 55% (Table 3). Excluding unsuitable images and fine-tuning the ANN resulted in accuracy between 57% and 64%, with an average ±SD of 60 ± 2% for the generalization test to 2 novel macaques (Table 4).

Discussion

Machine learning techniques have been used to decode animal emotions with less risk of anthropocentric biases and comparable performance with human evaluators.^2,28,43 The current study provides information on 2 face detection methods and one ANN model (ResNet50) to classify pain in Japanese macaques without hyperparameter or architecture modification. The methods were tested to identify which performed the classification of facial expressions of pain in Japanese macaques with the greatest accuracy. Using RetinaFace for face detection and image extraction resulted in an overall test accuracy between 48% and 98%, depending on the experiment. E1 used all images extracted by RetinaFace, without manual selection, while E3 used the dataset after manual selection, which excluded profile and blurred or occluded images. Despite rigorous image exclusion, the test accuracy rose from 69% to only 70%, suggesting that the excluded images did not extensively impact the classification system. For E3, modification of all 50 layers was permitted for training, resulting in an accuracy of 94%. However, the high accuracy per se does not mean that the ANN is highly efficient in the classification task. This could result from overfitting training images, as confirmed in E6 in which generalization to novel subjects resulted in 48% accuracy. In E6, the ANN likely recognized and incorporated features that were irrelevant to pain, such as color and background, resulting in low accuracy. Tailoring the dataset by removing background and obstructing objects, converting to grayscale, and normalizing the brightness improved accuracy for E7 and E8 as compared with E3 and E4 from 69 to 72% and 94 to 97%, respectively.

The results of the generalization tests in E9 and E10 had unsatisfactory performance levels of 55 and 44%, respectively. Ensuring that training and test subjects do not overlap is crucial to avoiding classification and learning of individual-specific features by the model.¹⁹ Tests to evaluate generalization are essential to classification systems;⁴⁶ therefore, a subset of the data not used for training was used to determine whether the model could be applied to other Japanese macaques. Our results indicate that the tests conducted after box extraction did not perform well for generalization. The dataset from box extraction included a significant amount of “noise,” such as background and different illumination, that could have interfered with the learning. Because these features are unrelated to pain, and yet this information might have been incorporated by the model, it resulted in poor generalization. Contour extraction excluded a significant proportion of these potential interferences. In this study, P is the classification of greater importance because incorrectly classifying P as NP (false negative) is worse than classifying NP as P (false positive). Therefore, P recall and precision values are important components in assessing the model. In a study on pain classification in cats using a pretrained ResNet50 network, the overall accuracy reached 72%. In the present study, the NP recall values in E9 and E10 were close to 100%, while for P was near 0%. In tests of generalization, the model will likely classify images as NP due to an unbalanced training set that has significantly more NP than P images. A model with good accuracy should be able to distinguish features from a small number of pain images. The model’s performance was improved by fine-tuning, and the best model achieved a 69% recall and a 65% F1-score for P.

Even when controlling for a small number of images and an unbalanced training dataset, the classification of pain images is difficult. The facial features that indicate pain in Japanese macaques are usually subtle and vary in intensity.²² Also, our system does not use predefined regions of the face, action unit annotation, or geometric features to indicate pain areas but learns only from full-face images and their associated labels. Therefore, we view our model results as satisfactory for this dataset. We stress that our classification algorithm relies only on the current image being presented to the ANN. However, a potential avenue for further research is to use images within a predefined time window, classify each image separately, and, if the fraction of images classified as pain exceeds a user-defined threshold, classify the image as P. This approach is similar to recent research on pain categorization based on the facial expressions of mice.⁴⁵ We hypothesize that this approach will provide higher accuracy because facial expressions change over time, and some frames are more representative of pain facial expressions than others. Furthermore, this method would prevent false positives that can occur when a brief facial expression similar to a pain expression is misclassified as P. However, this method may require larger datasets because multiple images are used for a single final classification. A more sophisticated approach to detect pain could be the use of Convolutional Long Short-Term Memory (C-LSTM) ANN, as used to detect pain in horses.⁹ C-LSTM integrates both temporal and spatial information, thereby using facial expressions and behavior for the classification and outperforming convolutional neural networks (CNN) and CNN followed by an LSTM NN.⁹

Recently, automated recognition of pain has been extensively applied in horses to determine the presence^9,28 and level²⁸ of pain. Most efforts to identify facial indicators of pain rely on the FACS, which decomposes expressions into individual facial muscle movements or “action units” (AUs). AUs have been identified in research species and provide the anatomic foundation for the development of grimace scales and other tools used to evaluate pain.^17,25,27,43 AUs can be classified and used to train an ANN, tested either alone or together for detecting the presence and intensity of pain.^28,31,32,41 Although FACS was recently published for Japanese macaques,¹⁴ a grimace scale has not yet been developed for this species. The classification model may benefit from specific facial features that contribute to the detection of pain in primates, such as orbital tightening, cheek tightening, and eyebrow lowering.^22,39 When detecting pain in human faces, fusing the best-performing AUs associated with pain achieved a slightly better accuracy (78%) than extracting features from the whole face (75%).³² Focusing on specific areas of the face is likely to reflect pain more accurately and could improve classification accuracy. For example, using the Mouse Grimace Scale for mice, an automated method achieved an overall accuracy of 89% for pain classification after anesthesia and surgery.⁴ In sheep, a multilevel approach with detection of faces, localization of facial landmarks, normalization, and extraction of facial features provided an overall accuracy of 67% of AUs classification.³¹

The position of the ears is a common indicator of pain in mammals.^25,27,43 In Japanese macaques; however, the ears are covered by fur, making them hard to see. The boxing and masking extraction methods were able to include the ear area, but this area probably had no significant impact on the classification results. Ears that are forward or flattened, as compared with being in a neutral position, have been associated with silent threatening and affiliative behaviors.^38,44 However, information on ear changes associated with pain has not been reported. Pain expression can vary widely among species, which complicates the extrapolation of these external cues. Currently, lip tightening and squeezed eyes are considered potential pain indicators in macaques, while ears were not found to be associated with pain.^17,39

In addition to facial expression, behaviors are also important when evaluating pain. Smart devices have been used to record behavior patterns and activity changes associated with pain in humans¹² and animals.⁴⁷ Smartwatches and wearable sensors can provide information in real-time and facilitate the medical approach to the condition. Devices that have contact with the patient’s body may be ill-suited for captive wild species and induce stress or be damaged. Therefore, video recording is still among the least expensive and most viable options for objective and continuous monitoring in captive or naturalistic scenarios. Markerless motion capture was developed from video-recorded macaques, facilitating the study of macaque behavior with accuracy comparable to that of humans.²⁶ In experimental surgical settings, data processing can reduce observation and training bias by monitoring the body parts that indicate the patient status.

Limitations of this study include the limited number of images for training and testing, which differs from human pain and object detection datasets that are more easily accessible in open libraries, containing many images compared with those used in animal studies. In addition, housing macaques in pairs with their conspecifics could have influenced the facial expressions of some individuals. The experience of pain can vary among individuals, and different surgeries can result in different types of pain. Sedation may also affect facial expressions and impact pain scoring, as observed in rats anesthetized with isoflurane.³⁴ Because our recordings began the day after the surgery, sevoflurane anesthesia was not likely to have affected the frames captured. Finally, the DL approach used in this study uses “black-box” reasoning, which means that the model’s decision-making process may not be easily understood by humans, limiting its use in clinical applications.^8,30

Assessing pain in research macaques is essential for animal welfare and helps to reduce bias in research outcomes. However, manual annotation of facial expressions and behaviors is labor- and time-intensive. Our study has shown that ANN-based algorithms can be used for automated facial recognition and classification of pain in Japanese macaques. Further studies might improve overall performance by expanding the training set, focusing on specific areas of the face, and using sequential models that consider video dynamics for classification.⁹