Identifying fetal yawns based on temporal dynamics of mouth openings: A preterm neonate model using support vector machines (SVMs).

Fetal yawning is of interest because of its clinical, developmental and theoretical implications. However, the methodological challenges of identifying yawns from ultrasonographic scans have not been systematically addressed. We report two studies that examined the temporal dynamics of yawning in preterm neonates comparable in developmental level to fetuses observed in ultrasound studies (about 31 weeks PMA). In Study 1 we tested the reliability and construct validity of the only quantitative measure for identifying fetal yawns in the literature, by comparing its scores with a more detailed behavioral coding system (The System for Coding Perinatal Behavior, SCPB) adapted from the comprehensive, anatomically based Facial Action Coding System for Infants and Young Children (Baby FACS). The previously published measure yielded good reliability but poor specificity, resulting in over-representation of yawns. In Study 2 we developed and tested a new machine learning system based on support vector machines (SVM) for identifying yawns. The system displayed excellent specificity and sensitivity, proving it to be a reliable and valid tool for identifying yawns in fetuses and neonates. This achievement represents a first step towards a fully automated system for identifying yawns in the perinatal period.

Introduction

Fetal yawning has been the subject of increasing interest over the last decades, due to its clinical implications for early neurobehavioral assessment [1-3] as well as for its theoretical insights into the ontogenetic origins of a wide arrays of phenomena, including auto-regulation [4, 5], mirror-like behaviors [6], interoception and arousal [7], consciousness [8] and communication [9].

Being able to identify fetal yawns could serve various potential clinical interests. In fact, it has been proposed that increased rates of yawning might help to identify high risk fetuses [2, 3], while lack of fetal yawn can be predictive of brainstem dysfunction after birth [7]. Therefore, having an accurate, reliable method for coding fetal yawns is crucial not only for the study of the development of perinatal behavior, but also for allowing research results to be implemented in actual clinical practices.

However, the intrauterine development of yawning is still poorly understood, as indicated by inconsistencies among published studies in the estimates of yawning frequencies within the same windows for gestational age (GA). In particular, as shown in Table 1, during the third trimester of pregnancy, the average number of yawns observed per hour varied from zero [10] to 14 [11]. The sharp differences between these results might be partially explained by different factors, including fetal circadian rhythms and pathological conditions [2, 3]. However, all studies included healthy fetuses [3, 10–19] and most had US scans performed during the afternoon [11, 12, 13, 14, 15, 18, 19]. Therefore, the inconsistencies shown in Table 1 suggest that the measures used in these studies lacked adequate reliability or validity.

Table 1

Spontaneous yawning frequencies of healthy fetuses.

Reference	USTechnique	N	GA range(w)	Duration(h:min)	Yawns/h (SD)
Van Woerden et al. (1988) [12]	2D	19	38–40	01:00	4.3 (NA)
Petrikovsky et al. (1999) [3]	2D	16	36–40	01:00	5.0 (4.0)
Kurjak et al. (2003) [13]	4D	10	30–33	00:15	4.0 (NA)°
Kurjak et al. (2004) [11]	4D	10	33–35	00:15	14.0 (NA)°
Reissland et al. (2012) [10]	4D	14	24	00:10	11.6 (13.0)
Reissland et al. (2012) [10]	4D	15	28	00:10	8.4 (12.2)
Reissland et al. (2012) [10]	4D	15	32	00:10	4.4 (5.8)
Reissland et al. (2012) [10]	4D	14	36	00:10	0.0(0.0)
Yan et al. (2006) [14]	4D	10	28–34	00:15	11.6 (7.2)
Kanenishi et al. (2013) [18]	4D	24	25–28	00:15	4.5 (4.3)
Sato et al. (2014) [19]	4D	23	20–24	00:15	2.6 (3.5)
Yigiter and Kavak (2006) [16]	4D	63	11–40	00:30	7.0 (3.5)
AboEllail et al. (2018) [15]	4D	25	30–31	00:15	0.0(NA)°
AboEllail et al. (2018) [15]	4D	43	32–35	00:15	0.0(NA)°
AboEllail et al. (2018) [15]	4D	43	36–40	00:15	4.0 (NA)°

Note: Values in parentheses represent Standard Deviations; N = Sample Size; Duration = Duration of a single observation session;° values calculated based on reported medians; NA = Not Available.

Most studies adopted qualitative criteria in identifying yawns based on the de Vries et al. definition (Table 2). An alternative approach defined yawns in a more operational and quantitative way as those mouth openings where the time to maximum opening was longer than the time from maximum opening to closure [10].

Table 2

Operational definitions of fetal yawning.

Reference	Definition
Van Woerden et al. (1988) [12]	Similar to the yawn observed after birth; a prolonged wide opening of the mouth followed by a quick closure, and mostly combined with a retroflexion of the head (p. 99)
Petrikovsky et al. (1999) [3]	Yawning was defined as a prolonged wide opening of the mouth followed by a quicker closure of the mouth (pp. 127,128)
Kurjak et al. (2003) [13]	Slow and prolonged wide opening of the jaws followed by quick closure with simultaneous retroflexion of the head and sometimes elevation of the arms of exoration (p. 499)
Yan et al. (2006) [14]	Yawning [was defined] as a slow, wide, prolonged opening of the jaws followed by quick closure with simultaneous retroflexion of the head (p. 110)
Reissland et al. (2012) [10]	We defined a yawning event to be those mouth openings where the time to maximum opening of the mouth was of a longer duration than the time from maximum opening to closing (p. 3)
Kanenishi et al. (2013) [18]	Video Sample
Yigiter and Kavak (2006) [16]	Slow and prolonged wide opening of the jaws followed by quick closure with simultaneous retroflexion of the head and sometimes elevation of the arms of exoration (p. 708)
Sato et al. (2014) [19]	Video Sample
AboEllail et al. (2018) [15]	Yawning represented prolonged wide and slow jaw opening followed by quick closure with simultaneous head retroflexion (p. 2)

Reliability and validity issues

In order to overcome the observed inconsistencies in coding yawns from ultrasound scans, we need to address two distinct issues: the precision or inter-rater reliability of measures used to code yawns and the accuracy or construct validity of the measures, i.e., whether the measures used can discriminate yawning from other fetal behaviors involving mouth opening, such as swallowing, mouthing or distress expressions.

Kanenishi et al. [18] and Sato et al. [19] mentioned the reliability issue in relation to discrepancies between their findings and those from other studies. They concluded that the subjectivity of methods used to identify yawns and other behaviors from US scans might have resulted in low inter-rater agreement. To the best of our knowledge, however, only the Reissland et al. [10] study included two independent coders. In that study, calculation of Cohen’s Kappas showed good reliability. On the other hand, the issue of construct validity, i.e., the accuracy of identifying yawns and distinguishing them from other actions involving mouth opening, has not been addressed in any published study.

The criterion proposed by Reissland et al. [10] is the only formalized, quantitative system for assessing fetal yawns. It can therefore be reproduced and tested for both reliability and validity. Moreover, since it relies only on temporal cues pertaining to mouth movements, it has the advantage of being particularly simple and easy to apply, especially when dealing with US scans, often characterized by limited spatio-temporal resolution, partial accessibility of the face to observation, and imaging artifacts [20].

However, this simplification might carry the risk of sacrificing accuracy, because the authors did not address the crucial construct validity issue of whether the system can distinguish yawning from other movements involving mouth opening. Reissland et al. [10] justified their methodological choices by citing Petrikovski et al. [3], who reported that the opening phase of yawns is longer than the closing phase, but they did not offer any evidence for the complementary assumption on which the system is based, i.e. that yawns are the only mouth opening actions that meet this criterion. Consequently, this method could be prone to type I errors (false positives).

Neonatal yawns as a model for coding fetal yawns

One undisputed feature of yawning is the apparent stability of its behavioral pattern throughout life [21, 22]. Fetal yawns, in particular, have been reported to be similar to the behavioral pattern exhibited after birth by neonates and infants [12, 17, 23, 24]. Therefore, the analysis of yawns in neonates represents a good model for testing and developing methods for coding yawns in fetuses, especially because the superior spatio-temporal resolution and clarity of video-recordings allow a more accurate and precise identification of yawns. The population of healthy preterm neonates, in particular, seems to be the best suited for this purpose, since they are the closest to fetuses with regard to developmental level.

Research strategy

The aim of Study 1 was to assess the construct validity of the criterion for coding yawns proposed by Reissland et al. [10]. We first identified mouth openings in videos of preterm infants adopting the same Baby FACS-based criteria used to identify mouth opening in their study, and then, using the same timing-based criterion they proposed, we categorized mouth openings as yawns or non-yawns.

We then tested the agreement between yawns identified according to their criterion and according to a behavioral description, contained in the System for Coding Perinatal Behavior (SCPB) [25]. SCPB is a coding system that focuses on complex motor patterns occurring in the face, head, and upper trunk adapted from Oster’s Facial Action Coding System for Infants and Young Children (Baby FACS) [22], a comprehensive, anatomically based coding system with established reliability and validity and earlier studies in the literature. Baby FACS Action Units (AUs) represent discrete, minimally distinguishable actions of the facial muscles, allowing any facial movement to be precisely and objectively identified in terms of combinations and sequences of its constituent facial muscle actions.

In Study 2 we developed a new machine learning system for identifying yawns based on temporal dynamics of mouth openings and tested its validity on the same sample of videos analyzed in Study 1. We adopted the description from the SCPB to train and test support vector machine (SVM) algorithms. In order to enhance replicability, the methods for both studies are available on protocols.io at 10.17504/protocols.io.739hqr6.

Methods—Study 1

Participants

Seventeen preterm neonates admitted to the Neonatal Intensive Care Unit (NICU) at SS. Annunziata Hospital of Cosenza (Italy) participated in the study upon informed consent from parents.

The study sample included 17 healthy preterm neonates (5 males and 12 females) born between 26 and 33 weeks GA (M = 29.61; SD = 2.08), with birth weight appropriate for gestational age (AGA) observed between 28 and 35 weeks PMA (M = 31.41; SD = 1.98). Exclusion criteria were: congenital anomalies, heart or metabolic disorders, fetal infections, clear teratogenic factors, Apgar at five minutes < 6 and grade III or IV hemorrhages.

Procedure

Neonates were observed while they were lying supine in a cot. Behavior was video-recorded (24 frames per second) for 10 to 30 minutes (M = 18.63, SD = 6.31), at a midpoint in the feeding cycle, when the neonates were not receiving any stimulation through routine nursing or medical care.

Coding methods

Identification of mouth opening

Reissland et al. [10] identified mouth opening as movements in which the mouth was stretched widely open and the mandible was pulled down vertically. This definition is based on the appearance changes for AU 27 described in FACS [26] and Baby FACS [22], as illustrated in Reissland et al. [10]. For this reason, we coded mouth opening in the preterm neonates every time the lips parted (AU 25) and the mouth stretched widely open (AU 27) simultaneously [22]. Frame by frame coding of the video-recordings was performed by two independent coders expert in FACS, Baby FACS, and micro-analysis of neonatal behavior. The secondary coder did reliability coding for 41% of the video recordings. Videos were coded using ELAN, professional software for the creation and management of complex annotations on video and audio (Max Planck Institute for Psycholinguistics, The Language Archive, Nijmegen, The Netherlands; http://tla.mpi.nl/tools/tla-tools/elan/).

Identification of yawns based on timing of mouth opening and closing (Criterion A)

In order to replicate the Reissland et al. [10] procedure for identifying mouth openings as yawns, coders scored two different time intervals for each mouth opening:

total duration was coded from the onset of mouth opening, the first frame where the mouth opening motion was visible, to the offset, the last frame where mouth opening was visible.

plateau duration was coded from the first frame (plateau onset) to the last frame (plateau offset) during which maximum mouth opening was maintained.

Mouth openings were subsequently categorized as yawns or not yawns based on whether or not they satisfied the Reissland et al. [10] criterion defining yawns as mouth openings in which “the time to maximum opening of the mouth was of longer duration than the time from maximum opening to closing” [10; p. 3].

Consistent with the definition provided in the Fetal Observable Movement System [27; FOMS], the plateau (the portion of the episode where mouth opening remained at its apex), albeit scored separately from the opening and closing phases, was considered “to be part of the opening rather than the closing phase” (p. 169), and the closing phase was timed from the end of the plateau.

Identification of yawns based on SCPB (Criterion B)

Two independent expert FACS and Baby FACS coders identified mouth openings as yawns according to the following definition from The System for Coding Perinatal Behavior (SCPB) [25] based on the AUs described in the comprehensive, anatomically based Facial Action Coding System for Infants and Young Children (Baby FACS) [22] and previous studies in the literature [17, 21].

Yawning (AU 94) is a stereotyped behavior characterized by a slow mouth opening with deep inspiration, followed by a brief apnea and a short expiration and mouth closing. One of the characteristic features of yawning is its timing, with a gradual acceleration followed by an abrupt deceleration of the facial actions involved. Yawning usually emerges from a relaxed face, initially involving mouth stretching widely open (AUs 25 + 27) and upper eyelids drooping (AU 43). Although the specific AUs accompanying yawns vary, at apex they may include tightly closed eyelids (AUs 6+7+43), flattened tongue shape (AU 76b), and swallowing (AU 80). During the plateau, brow knitting (AU 3), brow knotting (AU 4), nose wrinkling (AU 9), lateral lip stretching (AU 20), nostril dilatation (AU 38) and head tilting back (AU 53) may occur. In this phase, the expansion of the pharynx can quadruple its diameter, while the larynx opens up with maximal abduction of the vocal cords [21]. Yawning is often accompanied by limb stretching [17] and other bodily movements.

The SCPB [25] is a recently developed coding scheme based on frame-by-frame analysis of video-recorded material. The system focuses on fetuses, preterm and full-term neonates and aims to identify and reliably code the repertoire of complex patterns of behaviors observable from the last trimester of pregnancy to the first month of age. The SCPB is a system based on the Action Units (AUs), Action Descriptors (ADs), Miscellaneous Actions and Supplementary Codes described in Baby FACS [22] for identifying discrete facial muscle actions and more complex, configurationally defined patterns of facial behaviors.

Inter-rater reliability

Cohen’s Kappa was used to assess inter-rater reliability between the primary and secondary coders. Reliability was separately assessed for identifying mouth opening and for identifying yawns. The assessment of inter-rater reliability for identifying mouth opening, using a time window of one second for both onset and offset, resulted in an acceptable agreement between coders (kappa = .72).

Inter-rater reliability in identifying yawns was then assessed for Criterion A and criterion B. Acceptable reliability was obtained using Criterion A (kappa = .64), while perfect inter-rater agreement was found using criterion B (kappa = 1).

Data analysis: Distinguishing yawns from non-yawn mouth openings

Each mouth opening episode was categorized dichotomously as a yawn (1) or non-yawn mouth opening (0) according to Criterion A and Criterion B. In order to assess the accuracy of the measure adopted by Reissland et al. [10], we used a contingency table to assess its sensitivity (i.e. the true positive rate, calculated as the proportion of yawns that were correctly classified as yawns) and specificity (i.e. the true negative rate, calculated as the proportion of non-yawns that were correctly classified as non-yawns), and compute Cohen’s Kappa.

Results—Study 1

Over the 17 video-recordings, 130 mouth opening episodes were scored. The average duration of mouth openings was 2.48 s (SD = 1.41), and the average duration of the plateau was 0.63 s (SD = 0.69). Eighty-eight mouth openings (67.7% of total episodes) were classified as yawns according to criterion A [10], while only 15 (11.5% of total episodes) were recognized as yawns according to criterion B.

All 15 of the yawns identified according to criterion B also satisfied the temporal criterion specified by Reissland et al. [10] (criterion A). However, 73 mouth openings identified as yawns by criterion A were not coded as yawns according to Baby FACS [22], the SCPB [25], and earlier behavioral descriptions in the literature (criterion B). The remaining 42 episodes were classified as non-yawn mouth openings according to both criteria. As a consequence, despite a perfect sensitivity (1.00), the estimated specificity for Criterion A was 0.36, resulting in a reliability barely above chance (Cohen’s Kappa = 0.12).

Discussion—Study 1

The results of Study 1 revealed that the Reissland et al. [10] criterion for classifying yawns (Criterion A) had low specificity, producing a high rate of false positives (73 of 130 mouth openings, 56% of total episodes). The criterion for yawning proposed by these researchers was satisfied by 88 (67.7%) of the 130 episodes of mouth opening in our sample, excluding only mouth openings where the closing phase was longer than the sum of the mouth opening and plateau phases. It is notable that the high rate of yawns coded by criterion A is consistent with the results reported in the original study by Reissland et al. [10], who classified 56 out of the 83 (67.5%) mouth openings they observed in fetuses as yawns. The performance of Criterion A reflects the limited power of the criterion based on only two temporal landmarks to distinguish yawning from non-yawn mouth opening. In sum, it is clear that we cannot consider the overly simple quantitative criterion for identifying yawns proposed by Reissland et al. [10] to be an accurate, valid method for distinguishing between yawns and non-yawn mouth openings in fetuses or preterm neonates.

Despite these results, it should be possible to develop a quantitative criterion for identifying yawns based just on the temporal dynamics of mouth opening and closing. The perfect sensitivity exhibited by the Reissland et al. [10] criterion confirms that yawns are characterized by distinctive temporal dynamics. However, their criterion overlooked additional temporal variables that could be relevant for identifying yawns. In Study 2 we examined the potential usefulness of several derived measures for distinguishing yawning from non-yawn mouth opening in addition to those analyzed in Study 1.

Methods—Study 2

In Study 2 we conducted a more detailed analysis of the temporal dynamics of yawning to assess the feasibility of distinguishing yawning from non-yawn mouth opening based solely on temporal cues coded from videos or US scans of mouth and jaw movements.

Following a first exploratory phase, we adopted a support vector machine (SVM) approach that was cross-validated on the same sample of preterm neonates used in Study 1. SVMs are supervised learning models with associated learning algorithm that analyze data used for classification and regression analysis, supporting high dimensional data. These methods are widely used in different research fields such as bioinformatics, text mining, face recognition and image processing and are regarded, along with neural networks and fuzzy systems [28], as a state-of-the-art tool for machine learning. SVM-based systems have also been used for automated facial behavior analysis in some pioneering studies [29-33]. Participants and settings were the same as described for Study 1.

Coding method

In addition to coding the total duration of mouth opening and duration of the plateau, as defined in Study 1, we calculated the following variables:

Duration of the opening phase: from mouth opening onset to plateau onset

Duration of the closing phase: from plateau offset to mouth opening offset

opening/closing asymmetry: difference between the durations of the opening and closing phases

opening/closing ratio: ratio of the duration of the opening phase to the duration of the closing phase

Reliability

In order to establish the reliability of these measures, we compared the duration of the three phases (opening, plateau and closing) as scored by the two independent coders for Study 1. The differences between the two coders were deemed acceptable for the duration of each of the three phases: opening (Median = .08 s), plateau (Median = .10 s) and closing (Median = .12 s). Adopting a tolerance window of .5 seconds, the percentage agreement between coders was 93% (39 out of 42) for both opening and plateau duration and 95% (40 out of 42) for closing duration.

Hierarchical logistic regressions

Preliminary analyses were carried out, via hierarchical linear regression models, in order to identify specific features of yawns compared to other mouth openings in terms of their temporal dynamics, and to investigate the relations between different parameters for the two classes of episodes.

Machine learning algorithm

Based on findings from the exploratory analysis, the use of Support Vector Machine (SVM) classifiers was deemed appropriate to maximize the classification margin and minimize the risk of type I (false positives) as well as type II errors (false negatives) in distinguishing yawns and non-yawn mouth openings. We used LibSVM [34] with radial basis functions (RBF) kernel [32] in R, version 3.5.2 (package "e1071"), to build our models and to generate predictions for our test cases. Grid search based on 10-fold cross-validation error was employed to optimize the parameters C and gamma using the function “tune.svm”, within the interval [10–3, 103]. Model A was optimized with C = 100 and gamma = 0.1 while Model B was optimized with C = 10 and gamma = 1.

Classification performance was calculated via percent agreement and Cohen’s Kappa calculation for two different SVM models by changing the subset of episodes used respectively for training and testing, in order to identify and cross-validate the optimized model. Since yawns are expected to share distinctive temporal dynamics, in addition to the full model including main effects and interactions of all variables (total duration, plateau duration, duration of the opening phase, opening/closing asymmetry and opening/closing ratio; Model A), a second model was tested including only the three-ways interaction effects of total duration, plateau duration and opening/closing asymmetry (Model B).

We compared the classification performance of the two SVM models via hold-out validation. On each iteration, two thirds (n = 87) of the sample (N = 130) were randomly assigned to training, while the remaining 43 episodes were used for testing. For each model, we computed overall agreement, Cohen’s Kappa, sensitivity (true positive rate) and specificity (true negative rate). The candidate model with the highest Cohen’s Kappa coefficient was deemed the best model for yawn identification.

Results—Study 2

Hierarchical linear regressions highlighted several differences between yawns and non-yawn mouth openings involving the study variables (see Table 3 for descriptive statistics). In particular, total duration was longer for yawns (Mean = 5.123, SD = 1.252) compared to non-yawn mouth openings (Mean = 2.14, SD = 1.01), t(126) = 9.93, p < .001, β = .652. Moreover, yawns had a longer opening phase (Mean = 2.01, SD = 0.65) than non-yawn mouth openings (Mean = 0.77, SD = 0.56), t(126) = 7.66, p < .001, β = .561. Plateau was also longer for yawns (Mean = 2.10, SD = 0.83) compared to other mouth openings (Mean = 0.44, SD = 0.36), t(126) = 13.88, p < .001, β = .779. The duration of the closing phase, however, did not differ significantly between the two classes of behavior, t(126) = -.35, p = .726, β = -.054 (see Fig 1). As expected, yawns also presented a stronger opening/closing asymmetry than non-yawn mouth openings, t(126) = 6.34, p < .001, β = .496, as well as a larger opening/closing ratio, t(126) = 4.12, p < .001, β = .349.

Table 3

Variables related to temporal dynamics of yawns and non-yawn mouth openings.

Variable	Yawns(n = 15)				Non-Yawn Mouth Openings(n = 115)
	Mean (SD)	Median	Min	Max	Mean (SD)	Median	Min	Max
Duration	5.12 (1.25)	5.23	3.00	6.97	2.14 (1.01)	1.92	0.71	5.95
Plateau Duration	2.10 (0.83)	2.24	1.04	3.20	0.44 (0.36)	0.36	0.04	1.88
Opening Duration	2.01 (0.65)	1.98	1.12	3.67	0.77 (0.56)	0.59	0.13	3.84
Closing Duration	1.01 (0.36)	0.96	0.53	2.01	0.92 (0.58)	0.81	0.11	3.39
Asymmetry	1.00 (0.43)	0.92	0.39	1.66	-0.15 (0.70)	-0.15	-2.64	2.33
Opening/Closing Rate	2.06 (0.53)	1.99	1.38	3.42	1.11 (0.96)	0.78	0.11	4.94

Note: Values are expressed in seconds

Fig 1

Temporal dynamics of yawning and non-yawn mouth openings.

Box plot of the duration of the opening, plateau and closing phases of yawn and non-yawn mouth openings. The lower and the upper hinges represent, respectively the first and third quartiles; the whiskers extend from the hinges to the most extreme value no further than 1.5* from the hinge. Points represent outliers.

SVM models evaluation

The hold-out validation procedure revealed overall good performance, with both models showing an agreement above 98% with the SCPB-based classification. In particular, the full model (Model A) highlighted the best classification performance, showing almost perfect specificity (M = 1.00, SD = 0.01) and good sensitivity (M = 0.93, SD = 0.10). Having achieved the highest Cohen’s Kappa coefficient (M = 0.94, SD = 0.08), Model A was deemed the best model for classifying yawns and non-yawn mouth openings. Model B, defining the probability of a mouth opening to be a yawn as a function of the interaction of total duration, plateau duration and opening/closing asymmetry, performed only marginally worse in terms of Cohen’s Kappa (M = 0.92, SD = 0.12), and even outperformed Model A in terms of sensitivity (M = 0.95, SD = 0.11), while also displaying good specificity (M = 0.99, SD = 0.02).

General discussion

The ability to identify yawns in fetuses and to distinguish yawns from non-yawn mouth openings is of interest because accurate detection of this widely observed behavior can have potential clinical importance for identifying early signs of neurodevelopmental abnormalities and other conditions such as placental diseases [35] in fetuses, very early preterm infants, and other populations at risk.

We reported two studies that examined the temporal dynamics of yawning in preterm neonates comparable in gestational age to fetuses observed in previously reported ultrasound studies of fetal yawning [2, 3, 10–16, 18, 19]. In Study 1 we tested the reliability and construct validity of the only quantitative measure for identifying fetal yawns in the literature by applying the same time-based criterion to yawns showed by preterm neonates coded with a more detailed behavioral coding system (The System for Coding Perinatal Behavior, SCPB) [25] adapted from the comprehensive, anatomically based Facial Action Coding System for Infants and Young Children (Baby FACS) [22]. The results of Study 1 revealed that the Reissland et al. [10] criterion for distinguishing between yawns and non-yawn mouth opening had low specificity, producing a high rate of false positives.

In Study 2, we developed and evaluated two Support Vector Machine (SVM) models for classifying perinatal yawns and non-yawn mouth openings. Our results demonstrate the potential of SVM for identifying yawns based on a quantitative analysis of the temporal dynamics of mouth openings. In particular, the fact that the partial model (Model B), only including a three-way interaction effect of total duration, plateau duration and opening/closing asymmetry (defined as the difference between opening and closing durations) performed only marginally worse than the full model is consistent with previous knowledge. In fact, yawning is known to be a stereotyped behavioral pattern characterized by a prolonged mouth opening, including an extended plateau and a short closing phase [3, 12–14, 22], and this description can be formalized as an interaction of the three variables best representing these features. Moreover, the fact that the episodes coded as yawns occupy a specific region of the three-dimensional space defined by duration, plateau duration, and opening/closing asymmetry is consistent with the prevalent view of yawning as a distinctive type of mouth opening, therefore indirectly confirming the accuracy and precision of the identification criteria based on SCPB that we adopted (Criterion B). Further research is needed in order to confirm these results on bigger samples, as well as to provide additional training for the SVM algorithm.

The proposed machine-learning system might represent a crucial asset for the study of fetal yawning, making it possible to distinguish between yawns and other behavioral patterns involving mouth opening, e.g. swallowing, mouthing, or distress expressions. In fact, it represents the first cross-validated and easily reproducible method for coding yawns at early developmental stages, and, because it is based on the temporal analysis of mouth openings, it can be used with 4D US scans. In particular, this method–developed using preterm neonates as a training population–enables us to overcome the interobserver reliability limitations of descriptive approaches to yawn recognition, while also being able to avoid the risk of construct validity issues resulting from the adoption of an oversimplified criterion like the one proposed by Reissland et al. [10].

In order to establish this method as a viable option for identifying fetal yawns in clinical and research settings, additional work should be done to test it on fetal behavior. In particular, the suboptimal framing and variable quality that characterize ultrasonographic scans represent a ubiquitous issue in the study of fetal behavior. However, the fact that the proposed method only relies on the timing of mouth openings highlights a key advantage of this approach, that was specifically conceived in order to overcome the issues of identifying a complex facial behavior such as yawn from ultrasonographic scans. Another potential reason for concern regards the different conditions of fetuses compared with preterm neonates; for example, fetuses are immersed in amniotic fluid and the yawn of the near-term fetus might occasionally be mechanically constrained because of its position. However, there is unanimous agreement in the literature on the stability of the yawning behavioral pattern throughout life [12, 17, 21, 22, 23, 24]. Therefore, the advantages of using preterm neonates as a model for training classificatory algorithms far outweigh these potential issues.

A strength of the current study was the use of the SCPB [25], a rigorous coding system based on Baby FACS [22], to identify yawns. Because Baby FACS coding is based on multiple and redundant cues to each facial action, facial muscle actions can be reliably identified in fetuses [36], preterm and full-term neonates [37], and infants with facial anomalies [38] as well as typically developing infants. And because the basic coding units of Baby FACS are exhaustive and mutually exclusive, any complex facial movement can be precisely and unambiguously identified in terms of combinations and sequences of its constituent facial muscle actions. Therefore, Baby FACS, which has been referred to as the “gold standard” for coding infants’ facial expressions [39], is especially suited for coding yawning, sensory and perceptual responses [37], pain [40], and other fetal, neonatal and infant behaviors that don’t fit simplified templates for a limited set of emotional expressions.

The reliability of the proposed approach for fetal yawning classification is conditional on a preliminary evaluation of the temporal resolution of the 4D US scans used and of the resulting errors in assessing the duration of the parameters of interest. However, the recent development of ultrasonographic machines performing up to 25 frames per second [14], together with the specificities highlighted by yawns with regard to temporal dynamics are encouraging for future applications.

Because coding micro-analytically mouth openings and their plateaus is time-consuming, the utility of this system for coding neonatal yawns would be greatly improved by the implementation of a machine-controlled method for tracking neonatal mouth openings, which would make the identification process fully automated. This accomplishment would guarantee a quick, valid and reliable system for investigating yawns in fetuses and neonates.

In conclusion, the development of a reliable and valid method for identifying yawning can provide a potentially valuable tool for studying perinatal behavior and for assessing fetal and preterm infant wellbeing. Future research using machine-learning systems could contribute to the development of sensitive measures for early diagnosis of neurological impairments and other disorders.

Dataset.

(CSV)

Click here for additional data file.

24 Sep 2019

PONE-D-19-18762

Identifying fetal yawns based on temporal dynamics of mouth openings: A preterm model using support vector machines (SVMs)

PLOS ONE

Dear Prof. Dondi,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

The manuscript and the reviewers’ comments were carefully evaluated. The manuscript was appreciated by the Reviewers. Nevertheless, as suggested, the manuscript requires some improvement before to be considered for publication. Further suggested revisions are in detail reported in the Reviewers’ comments.

We would appreciate receiving your revised manuscript by Nov 08 2019 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Simone Garzon

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at http://www.journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and http://www.journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2.  We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For more information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially sensitive information, data are owned by a third-party organization, etc.) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

Additional Editor Comments (if provided):

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: I Don't Know

Reviewer #3: No

Reviewer #4: No

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

Reviewer #3: No

Reviewer #4: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: I have no serious objection to any part of the paper. The conceptual design of the paper and its intent are clear. The observation of fetal movement in general is important to clinicians as well as those that study fetal development. This paper makes a solid contribution to this field of study.

The authors assume that the preterm neonate is a valid model for the fetus. My personal experience supports this assumption and I agree that it is certainly the best way to approach the problem.

The authors demonstrate the need for an improved definition of neonatal yawning. They define quantifiable features that can be measured during neonatal yawning that can be used to train a support vector machine (SMV). Anyone should be able to duplicate the work to the extent that a given set of data and a SMV with allow. In Fig 1, the degree to which the distributions are separated suggest that any number of machine learning techniques would be successful. Therefore, I'm satisfied with the sample size and methods they have used.

I do have concerns:

The neonates in this study were placed in a convenient position for observation during data collection, but the fetus will not be so co-operative. Moreover, the yawn of the near term fetus might occasionally be mechanically constrained because of its position. Further, the neonate is yawning in air, while the fetus will be immersed in amniotic fluid. The timings and definitions may need to be adjusted.

The paper would benefit from more data, sufficient to create an independent test set of equal size. The observation that your Model B is the best performing may be a statistical artifact, or a bit of over training, or it may be real. I can't say.

I wish the authors well in their future work.

Reviewer #2: The article deals with the problem of detecting yawning in fetuses (preterm neonates) based on temporal dynamics of mouth openings. The authors conducted two studies. In the first, the accuracy of detection based on the method from the literature [10] was compared with the behavioral description (SCPB). In the second study the machine learning technique was applied: five SVM models were considered. According to the Authors, the second approach (SVM) is better.

The paper represents good quality, detailed comments are given below:

1) In the title it is probably better to replace “preterm model” with “preliminary model” or “initial model”. Unless the adjective “preterm” refers to neonates.

2) How were SVM parameter values determined (C=100, gamma=1, line 316) and what does "type 1 error" and "type 2 error" (lines 313-314) mean?

3) It would be worth describing more precisely how sensitivity and specificity were determined in study 1.

4) The abbreviation "AU" is explained later (line 190) than its first use in the text (line 156).

5) What does "t(126)" (lines 346-353) mean?

6) English can be slightly improved in some places, e.g.:

- “construct validity of the measure” -> “validity of the measure” or “correctness of the measure”

- “risk of sacrificing accuracy for precision” -> “risk of lowering accuracy at the expense of precision” (line 96)

- line 385: “having achieved the highest” -> “having the highest” or “achieving the highest”

- line 389: “did not just display a longer” -> “not only showed a longer”

- line 394: “according to one specific iteration” -> “in a given iteration”

- line 156: “et al” -> “et al.”, “F For” -> “For”, line 383: “011” -> “0.11”

The above minor comments do not affect the generally high quality of the article. In my opinion, after these minor corrections, the article should be published in the PLOS ONE Journal.

Reviewer #3: The paper concerns the problem of identifying and modeling of fetal yawns from ultrasonographic scans by analyzing temporal dynamics of fetal mouth openings. The results of two main studies were presented. In the first, the validity of the criterion for coding fetal yawns was investigated while in the second, the SVM based classifiers to distinguish the perinatal yawns from non-yawn mouth openings were investigated. Five different SVM models were proposed and the presented results shown high efficiency of RBF based kernel SVM of fetal yawns recognition.

The addressed issues are interesting and provides good insight into the problems of constructing the automated systems to support the early assessment of fetal neurobehavioral development. The work is written carefully, with clearly defined objective. The following are general and more specific comments

1) The abbreviation usage needs improving. Each abbreviation shall be explained at the first use. The “symbol” t(126) (page 10) needs explanation as well?

2) There are some typos, e.g.: “F For” -> “For” (line 156), “011” -> “0.11” (line 383). The text should be checked thoroughly.

3) How were SVM parameter values (C and γ) determined?

4) Is the classification performance defined on the testing sets only? If yes, the testing set would contain 5 yawing cases (on average). As the generalization ability estimation is of the crucial importance, is this number enough for making the conclusions about fetal yawn models?

5) It seems that the figure 2 shows the SVM classification result for Model A (which is not explicitly stated). But what is its purpose? Why plateau duration was fixed at 1.05 s. (half of the mean of plateau duration related to fetal yawns)?

6) As the classification results are so high, did Authors consider to model the fetal yawns by using only two temporal dynamics variables of the highest discrimination ability (like opening and plateau phases – see Figure 1)?

Considering the above, my recommendation is to accept the paper after these minor corrections and comments are addressed.

Reviewer #4: I read with great interest the Manuscript titled “Identifying fetal yawns based on temporal dynamics of mouth openings: A preterm model using support vector machines (SVMs)” (PONE-D-19-18762), which falls within the aim of PLOS ONE. In my honest opinion, the topic is interesting enough to attract the readers’ attention. The manuscript is well written, methodology is accurate, and conclusions are supported by the data analysis. Nevertheless, authors should clarify some points and improve the discussion citing relevant and novel key articles about the topic.

Authors should consider the following minor recommendations:

- Introduction. Regarding the difference among reported yawns frequencies, which is the role of circadian changes in yawns activity of the fetus. Does yawns activity change over time? I would suggest discussing this point.

- The Authors did not mention the sample size calculation for their study. I would suggest reporting how the Authors chosen the number of included neonates.

- The authors have not adequately highlighted the strengths and limitations of their study. I suggest better specifying these points. The Authors use the Baby FACS coders as the gold standard for the evaluation of yawns, is it actually the gold standard? – The newborns yawns in air and fetus yawns in water, do Authors expect some changes?

- As appropriately stated by the authors, fetal yawning has been associated with proper neurological development. In this regard, the possibility to detect these fetal movements may play a pivotal importance as additional tool to assess fetal wellbeing in utero. I would suggest stressing the possible underlying causes of fetal neurological impairment, such as for example placental diseases (refer to: PMID: 28282763; PMID: 28466013; PMID: 28243732).

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

Reviewer #4: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

6 Nov 2019

Reviewer #1

Q1: The neonates in this study were placed in a convenient position for observation during data collection, but the fetus will not be so co-operative. Moreover, the yawn of the near term fetus might occasionally be mechanically constrained because of its position. Further, the neonate is yawning in air, while the fetus will be immersed in amniotic fluid. The timings and definitions may need to be adjusted.

RESPONSE: We thank the reviewer for their appreciation of our work, and we agree with their remarks. However, as we argued for in a newly inserted paragraph in the general discussion (lines 455-478), the mentioned differences between fetuses and newborns should not hinder the applicability of the proposed method to fetuses, especially as yawn is universally recognized as a stereotyped behavioral pattern. Here is the new paragraph addressing the reviewer’s remarks:

“In order to establish this method as a viable option for identifying fetal yawns in clinical and research settings, additional work should be done to test it on fetal behavior. In particular, the suboptimal framing and variable quality that characterize ultrasonographic scans represent a ubiquitous issue in the study of fetal behavior. However, the fact that the proposed method only relies on the timing of mouth openings highlights a key advantage of this approach, that was specifically conceived in order to overcome the issues of identifying a complex facial behavior such as yawn from ultrasonographic scans. Another potential reason for concern regards the different conditions of fetuses compared with preterm neonates; for example, fetuses are immersed in amniotic fluid and the yawn of the near-term fetus might occasionally be mechanically constrained because of its position. However, there is unanimous agreement in the literature on the stability of the yawning behavioral pattern throughout life [12, 17, 21, 22, 23, 24]. Therefore, the advantages of using preterm neonates as a model for training classificatory algorithms far outweigh these potential issues.

A strength of the current study was the use of the SCPB [25], a rigorous coding system based on Baby FACS [22], to identify yawns. Because Baby FACS coding is based on multiple and redundant cues to each facial action, facial muscle actions can be reliably identified in fetuses [36], preterm and full-term neonates [37], and infants with facial anomalies [38] as well as typically developing infants. And because the basic coding units of Baby FACS are exhaustive and mutually exclusive, any complex facial movement can be precisely and unambiguously identified in terms of combinations and sequences of its constituent facial muscle actions. Therefore, Baby FACS, which has been referred to as the “gold standard” for coding infants’ facial expressions [39], is especially suited for coding yawning, sensory and perceptual responses [37], pain [40], and other fetal, neonatal and infant behaviors that don’t fit simplified templates for a limited set of emotional expressions.”

Q2: The paper would benefit from more data, sufficient to create an independent test set of equal size. The observation that your Model B is the best performing may be a statistical artifact, or a bit of over training, or it may be real. I can't say.

RESPONSE: We fully embrace the reviewer's suggestion and inserted this consideration as a new sentence (lines 441-443, reported below).

“Further research is needed in order to confirm these results on bigger samples, e.g. regarding the best performing model, as well as to provide additional training for the SVM algorithm.”

Reviewer #2

Q1: In the title it is probably better to replace “preterm model” with “preliminary model” or “initial model”. Unless the adjective “preterm” refers to neonates.

RESPONSE: We thank the reviewer for their appreciation of our work. Preterm indeed refers to neonates: To make it clear we changed the titles as follows: "Identifying fetal yawns based on temporal dynamics of mouth openings: A preterm neonate model using support vector machines (SVMs)".

Q2: How were SVM parameter values determined (C=100, gamma=1, line 316) and what does "type 1 error" and "type 2 error" (lines 313-314) mean?

RESPONSE: SVM parameters were set at the default value for the package adopted. Since results were optimal, and the aim of the Study 2 was to test the feasibility of the proposed SVM-based method, they were reported only for replicability purposes. Type I and II errors mean respectively false positives and negatives. A specification has been introduced (lines 326-327)

Q3: It would be worth describing more precisely how sensitivity and specificity were determined in study 1.

RESPONSE: The methods adopted to calculate sensitivity and specificity were explicitly reported at their first occurrence (lines 237-242, reported below).

“In order to assess the accuracy of the measure adopted by Reissland et al. [10], we used a contingency table to assess its sensitivity (i.e. the true positive rate, calculated as the proportion of yawns that were correctly classified as yawns) and specificity (i.e. the true negative rate, calculated as the proportion of non-yawns that were correctly classified as non-yawns), and compute Cohen’s Kappa.”

Q 4: The abbreviation "AU" is explained later (line 190) than its first use in the text (line 156).

RESPONSE: The "AU" abbreviation was explained at its first occurrence and all abbreviations were checked throughout the manuscript (line 129).

5) What does "t(126)" (lines 346-353) mean?

RESPONSE: The t-statistic is the ratio of the departure of the estimated value of a parameter from its hypothesized value to its standard error. The value in parentheses represents degrees of freedom.

Q6: English can be slightly improved in some places, e.g.:

Q6.1: “construct validity of the measure” -> “validity of the measure” or “correctness of the measure”

RESPONSE: Construct validity, defined as the degree to which a test measures what it claims, or purports, to be measuring, is one of the key concepts in psychometrics. Because this is technical terminology, we maintained "construct validity of the measure" in the manuscript, after consulting the English mother tongue author.

Q6.2: “risk of sacrificing accuracy for precision” -> “risk of lowering accuracy at the expense of precision” (line 96)

RESPONSE: In our understanding, the reviewer’s proposed sentence does not retain the original meaning, as precision is not affected. In order to make the phrase more immediately understandable to the reader and not redundant, we deleted "for precision" from the original formulation. (line 99)

Q6.3- line 385: “having achieved the highest” -> “having the highest” or “achieving the highest”

RESPONSE: In our understanding, the original wording correctly uses the “perfect participle” tense. “The perfect participle indicates completed action and was therefore maintained in the revised manuscript. (https://www.learnenglish.de/grammar/participleperfect.html)

Q6.4: line 389: “did not just display a longer” -> “not only showed a longer”

RESPONSE: The suggested version was adopted (line (403)

Q6.5: line 394: “according to one specific iteration” -> “in a given iteration”

RESPONSE: The original wording refers to a specific iteration taken at random, not to any iteration. However, Figure 2 was deleted from the new version of the manuscript because of its limited informative value.

- line 156: “et al” -> “et al.”, “F For” -> “For”, line 383: “011” -> “0.11”

RESPONSE: The suggested revisions were adopted.

Reviewer #3

Q1: The abbreviation usage needs improving. Each abbreviation shall be explained at the first use. The “symbol” t(126) (page 10) needs explanation as well?

RESPONSE: We thank the reviewer for their appreciation of our work. Abbreviations were checked and explained at their first use. The t-statistic is the ratio of the departure of the estimated value of a parameter from its hypothesized value to its standard error. The value in parentheses represents degrees of freedom. We propose to maintain the original, more concise formulation.

Q2: There are some typos, e.g.: “F For” -> “For” (line 156), “011” -> “0.11” (line 383). The text should be checked thoroughly.

RESPONSE: The manuscript was checked for typos and corrected.

Q3: How were SVM parameter values (C and γ) determined?

RESPONSE: SVM parameters were set at the default value for the package adopted. Since results were optimal, and the aim of Study 2 was to test the feasibility of the proposed SVM-based method, they were reported only for replicability purposes.

Q4: Is the classification performance defined on the testing sets only? If yes, the testing set would contain 5 yawing cases (on average). As the generalization ability estimation is of the crucial importance, is this number enough for making the conclusions about fetal yawn models?

RESPONSE: Classification performance was calculated based on testing sets only, but the hold-out validation included 30 iterations, and for each different training and testing sets were randomly selected. Moreover, both according to previous literature and our results, yawns seem to represent a very specific population of mouth openings, in terms of temporal dynamics. Therefore, we believe that the proposed sample is sufficient to show the feasibility of such method.

Q5: It seems that the figure 2 shows the SVM classification result for Model A (which is not explicitly stated). But what is its purpose? Why plateau duration was fixed at 1.05 s. (half of the mean of plateau duration related to fetal yawns)?

RESPONSE: The purpose of figure 2 was simply to illustrate how SVM models work in establishing hyper-planes for classifying mouth openings. Plateau duration was fixed around the shortest duration coded for yawns, in order to show more informative decision boundaries. The displayed model is Model D, because the adopted R package (e1071) did not allow for graphical representation of interaction-only models. In consideration of these limitations, and of its limited informative content, we propose to delete the figure from the new version of the manuscript.

Q6 As the classification results are so high, did Authors consider to model the fetal yawns by using only two temporal dynamics variables of the highest discrimination ability (like opening and plateau phases – see Figure 1)?

RESPONSE: Other methods were tried out in a preliminary phase. However, the difference between opening and closing (opening/closing asymmetry) also had high discrimination ability, and is directly related to the description, ubiquitous in the pertinent literature, of yawning as characterized by a long opening and a short closing phase.

Reviewer #4

Q1: Introduction. Regarding the difference among reported yawns frequencies, which is the role of circadian changes in yawns activity of the fetus. Does yawns activity change over time? I would suggest discussing this point.

RESPONSE: We thank the reviewer for their appreciation of our work and the valuable comments and suggestions. We inserted a new paragraph in the introduction (lines 54-59) in order to briefly discuss the potential effects of circadian rhythms and other variables (see below).

“The sharp differences between these results might be partially explained by different factors, including fetal circadian rhythms and pathological conditions [2, 3]. However, all studies included healthy fetuses and most had US scans performed during the afternoon [11, 12, 13, 14, 15, 18, 19]. Therefore, the inconsistencies shown in Table 1 suggest that the measures used in these studies lacked adequate reliability or validity.

Q2: The Authors did not mention the sample size calculation for their study. I would suggest reporting how the Authors chosen the number of included neonates.

RESPONSE: Because yawn is universally known as stereotyped behavioral pattern (see Table 1), we concluded that even a small sample (130 mouth openings) would be sufficient for our purposes, i.e. to assess the validity of the method proposed by Reissland and colleagues (in Study 1) and to test the feasibility of more sophisticated SVM-based methods for identify yawns (in Study 2). The results of Study 2 confirm this assumption; models highlighted good performances in distinguishing yawns and non-yawns. It is noteworthy that preterm neonates, despite representing a good model for fetal behavior, represent a very difficult population to sample and study. Because, in our experience, sample size calculation is generally not reported in studies adopting SVM or other kinds of classification algorithms, and classification performance is generally intended as the criterion for determining the required sample size for training (see https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3307431/), we propose not to introduce this specification.

Q3: The authors have not adequately highlighted the strengths and limitations of their study. I suggest better specifying these points. The Authors use the Baby FACS coders as the gold standard for the evaluation of yawns, is it actually the gold standard? – The newborns yawns in air and fetus yawns in water, do Authors expect some changes?

RESPONSE: We fully embrace the reviewer's suggestions and inserted the pertinent considerations as a new paragraph in the discussion (lines 455-478, reported below).

“In order to establish this method as a viable option for identifying fetal yawns in clinical and research settings, additional work should be done to test it on fetal behavior. In particular, the suboptimal framing and variable quality that characterize ultrasonographic scans represent a ubiquitous issue in the study of fetal behavior. However, the fact that the proposed method only relies on the timing of mouth openings highlights a key advantage of this approach, that was specifically conceived in order to overcome the issues of identifying a complex facial behavior such as yawn from ultrasonographic scans. Another potential reason for concern regards the different conditions of fetuses compared with preterm neonates; for example, fetuses are immersed in amniotic fluid and the yawn of the near-term fetus might occasionally be mechanically constrained because of its position. However, there is unanimous agreement in the literature on the stability of the yawning behavioral pattern throughout life [12, 17, 21, 22, 23, 24]. Therefore, the advantages of using preterm neonates as a model for training classificatory algorithms far outweigh these potential issues.

A strength of the current study was the use of the SCPB [25], a rigorous coding system based on Baby FACS [22], to identify yawns. Because Baby FACS coding is based on multiple and redundant cues to each facial action, facial muscle actions can be reliably identified in fetuses [36], preterm and full-term neonates [37], and infants with facial anomalies [38] as well as typically developing infants. And because the basic coding units of Baby FACS are exhaustive and mutually exclusive, any complex facial movement can be precisely and unambiguously identified in terms of combinations and sequences of its constituent facial muscle actions. Therefore, Baby FACS, which has been referred to as the “gold standard” for coding infants’ facial expressions [39], is especially suited for coding yawning, sensory and perceptual responses [37], pain [40], and other fetal, neonatal and infant behaviors that don’t fit simplified templates for a limited set of emotional expressions.”

Q4: As appropriately stated by the authors, fetal yawning has been associated with proper neurological development. In this regard, the possibility to detect these fetal movements may play a pivotal importance as additional tool to assess fetal wellbeing in utero. I would suggest stressing the possible underlying causes of fetal neurological impairment, such as for example placental diseases (refer to: PMID: 28282763; PMID: 28466013; PMID: 28243732).

RESPONSE: We thank the reviewer for this important remark. References to this aspect were inserted at the beginning of the discussion and at the end of the discussion (respectively lines 414-417 and 493-496, reported below).

“The ability to identify yawns in fetuses and to distinguish yawns from non-yawn mouth openings is of interest because accurate detection of this widely observed behavior can have potential clinical importance for identifying early signs of neurodevelopmental abnormalities and other conditions such as placental diseases [35] in fetuses, very early preterm infants, and other populations at risk.”

“In conclusion, the development of a reliable and valid method for identifying yawning can provide a potentially valuable tool for studying perinatal behavior and for assessing fetal and preterm infant wellbeing. Future research using machine-learning systems could contribute to the development of sensitive measures for early diagnosis of neurological impairments and other disorders.”

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

2 Dec 2019

PONE-D-19-18762R1

Identifying fetal yawns based on temporal dynamics of mouth openings: A preterm neonate model using support vector machines (SVMs)

PLOS ONE

Dear Prof. Dondi,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

The Reviewers appreciated the manuscript improvements. Only one point was questioned by Reviewer 3. I would suggest addressing this point or introducing a comment in the methods explaining the chosen method instead of the recommended as an answer to the Reviewer. Further details are provided in the Reviewer's recommendations.

We would appreciate receiving your revised manuscript by Jan 16 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Simone Garzon

Academic Editor

PLOS ONE

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #2: All comments have been addressed

Reviewer #3: All comments have been addressed

Reviewer #4: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #2: (No Response)

Reviewer #3: Yes

Reviewer #4: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #2: (No Response)

Reviewer #3: No

Reviewer #4: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #2: (No Response)

Reviewer #3: Yes

Reviewer #4: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #2: (No Response)

Reviewer #3: Yes

Reviewer #4: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #2: (No Response)

Reviewer #3: R3, Q3: I cannot agree that default values of SVM parameters ensure optimal classification results. In my opinion, for each of the considered models it is necessary to experimentally find C and γ values ensuring the best model performance. For example the first 10 iterations of the hold-out validation can be used to find the parameters (from a reasonable range of C and γ) providing highest values of performance indices. Since it may be difficult to analyze several performance indicators simultaneously, F-Score (or similar) can be employed. However, even in this case the claim of the C and γ optimality is questionable.

Reviewer #4: Authors have performed the required changes, improving significantly the quality of the article.

I have no further concerns.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: No

Reviewer #3: No

Reviewer #4: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

4 Dec 2019

Reviewer #3: R3, Q3: I cannot agree that default values of SVM parameters ensure optimal classification results. In my opinion, for each of the considered models it is necessary to experimentally find C and γ values ensuring the best model performance. For example the

first 10 iterations of the hold-out validation can be used to find the parameters (from a reasonable range of C and γ) providing highest values of performance indices. Since it may be difficult to analyze several performance indicators simultaneously, F-Score (or similar) can be employed. However, even in this case the claim of the C and γ optimality is questionable.

RESPONSE: We really thank the reviewer for insisting on this important point. As suggested, we adopted grid search to optimize C and γ values (see lines 329-332, see below).

“Grid search based on 10-fold cross-validation error was employed to optimize the parameters C and gamma using the function “tune.svm”, within the interval [10-3, 103]. Model A was optimized with C = 100 and gamma = 0.1 while Model B was optimized with C = 10 and gamma = 1.”

Moreover, because once we optimized these parameters within the interval [10-3, 103] the full model (originally model E, now model A) highlighted the best performance (as is generally expected), we simplified the SVM analysis by deleting three models and retaining only the full model (Model A) and Model B (including the three-ways interaction of total duration, plateau duration and opening/closing asymmetry). The rationale for maintaining Model B is that it shows that the interaction of these three variables is responsible for a significant part of the ability to discriminate between yawns and non-yawn mouth openings. On the other hand, because the full model is now the best performing one and differences between models are smaller once C and gamma are optimized, we think that including all five original models would only contribute to make results less clear. Methods, results and discussion of Study 2 were modified accordingly.

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

10 Dec 2019

Identifying fetal yawns based on temporal dynamics of mouth openings: A preterm neonate model using support vector machines (SVMs)

PONE-D-19-18762R2

Dear Dr. Dondi,

We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements.

Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication.

Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at https://www.editorialmanager.com/pone/, click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

With kind regards,

Simone Garzon

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #3: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #3: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #3: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #3: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #3: No

11 Dec 2019

PONE-D-19-18762R2

Identifying fetal yawns based on temporal dynamics of mouth openings: A preterm neonate model using support vector machines (SVMs)

Dear Dr. Dondi:

I am pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

For any other questions or concerns, please email plosone@plos.org.

Thank you for submitting your work to PLOS ONE.

With kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Simone Garzon

Academic Editor

PLOS ONE