Multimodal neonatal procedural and postoperative pain assessment dataset.

This paper presents the first multimodal neonatal pain dataset that contains visual, vocal, and physiological responses following clinically required procedural and postoperative painful procedures. It was collected from 58 neonates (27-41 gestational age) during their hospitalization in the neonatal intensive care unit. The visual and vocal data were recorded using an inexpensive RGB camera while the physiological responses (vital signs and cortical activity) were recorded using portable bedside monitors. The recorded behavioral and physiological responses were scored by expert nurses using two validated pain scales to obtain the ground truth labels. In addition to behavioral and physiological responses, our dataset contains clinical information such as the neonate's age, gender, weight, pharmacological and non-pharmacological interventions, and previous painful procedures. The presented multimodal dataset can be used to develop artificial intelligence systems that monitor, assess, and predict neonatal pain based on the analysis of behavioral and physiological responses. It can also be used to advance the understanding of neonatal pain, which can lead to the development of effective pain prevention and treatment.

Specifications Table

Value of the Data

These data are useful to understanding neonatal pain more holistically in the case of both procedural and postoperative pain.

NICU healthcare professionals and researchers can effectively use the dataset to develop any automated system for neonatal pain monitoring.

These data can be used to analyze the visual, vocal, and physiological signal changes in response to different types of pain (i.e., procedural and postoperative). It can also be used to compare how these two types of pain differ from each other. In addition, multimodal pain indicators and their corresponding relationship can lead to knowing how pain prevention and treatment can be performed more effectively.

To the best of our knowledge, we are the first to introduce a publicly available neonatal pain assessment dataset. It was recorded following procedural and postoperative painful procedures. This dataset contains data recorded from neonates with diverse gestational age and weight as well as race and ethnicity distributions. We believe any development towards understanding the pain or building any automated system using this dataset would greatly enhance neonatal healthcare and treatment outcome.

Data Description

In this paper, we introduce the first multimodal dataset, which we call University of South Florida Multimodal Neonatal Pain Assessment Dataset (USF-MNPAD-I). The dataset was recorded from neonates during procedural and postoperative procedures while being hospitalized in the NICU, which we call University of South Florida Multimodal Neonatal Pain Assessment Dataset (USF-MNPAD-I). Our dataset is the first publicly available dataset that contain both behavioral signals in addition to vital signs and cortical activity, and it was assessed using two clinically validated pain scales. This dataset was collected as part of an ongoing project that focuses on developing innovate automated methods for monitoring and assessing neonatal pain. USF-MNPAD-I dataset was collected from 58 neonates (27-41 gestational weeks [GW]) following clinically required procedural and postoperative procedures. Procedural pain occurs as a result of a quick tissue breaking during routine heel lancing and immunization procedures. On the other hand, postoperative pain occurs in connection with a newly created wound following a major surgical procedure. Our USF-MNPAD-I dataset includes video (face, head, and body), audio (crying sound), vital signs (heart rate, blood pressure, oxygen saturation), and cortical activity. In addition, our dataset contains continuous pain scores for each pain indicator and the medical notes for all neonates.

Our USF-MNPAD-I dataset, which contains behavioral and physiological data in addition to medical notes and ground truth labels (pain scores), can be downloaded as described in the project website (see “Specifications Table”). Because the dataset includes human subjects, agreement and sharing forms are required to be signed prior to any sharing of the data. The researchers who request and access the dataset can use it only for research purposes. Photos of the neonates can not be added to any publications unless proper de-identification is performed (e.g., covering the eyes as shown in Figure 1). We obtained photo consent from the parents of only two neonates, and hence, photos/videos of these subjects can be included in publications or as supplementary materials in research articles.

Fig. 1

Overall data collection procedure. Data collection includes video, audio, vital signs, and cortical activity signals. The lower part of the figure shows the timelines for recording procedural and postoperative pain. and represent heart rate, blood pressure, blood oxygenation, and regional oximetry value. The red box in the postoperative timeline indicates that no recording was performed during the surgery.

The current procedural dataset consists of 36 subjects including 44 procedures. Continuous video (mp4 format), audio (with video) signals, and physiological signals (mp4 format) and the contextual data (csv format) are recorded). Pain scores following the NIPS (Neonatal Infant Pain Scale) [5] scale are provided for every minute. Postoperative dataset collection includes 31 raw videos from 9 subjects. It has continuous video and audio (mp4 format) recording of the neonates, the regular bedside record of the physiological signal (mp4 format), and the contextual data (csv format). Pain scores are provided with a timeline using the N-PASS (Neonatal Pain, Agitation and Sedation Scale) [6] pain scale. Finally, NIRS (Near-infrared spectroscopy) [10] or the cortical activity data include 23 procedures from 13 subjects. The recorded data have the access time and corresponding values (xlsx format) in addition to the contextual and medical data (xlsx format).

Experimental Design, Materials and Methods

Participants

We recorded the data from fifty-eight neonates while being hospitalized in the NICU. Our data collection took place in Tampa General Hospital (Florida, United States) as part of an ongoing project that is expanding to multiple data collection locations both domestic within the United States and internationally. The project was approved by the University of South Florida (USF) Ethics Internal Review Board [1], [3] (IRB # Pro00014318). Any neonate receiving care in the NICU was eligible to enroll in data collection. We excluded from the enrollment neonates with cranial facial abnormalities and Down’s syndrome as well as neonates who were too sick or unstable. Prior to the recording of each subject, written informed consent was signed by the guardians of the subject.

In total, our dataset consists of 58 neonates with a gestational age that ranges from 27 to 41 GW. Our dataset contains 36 (63.16%) preterm neonates with a gestational age 37 GW. The remaining 21 neonates (36.84%) are full-term with a gestational age 37 GW. The dataset covers a wide range of ethnicities (26.79% Hispanic, and 73.21% Non-Hispanic), races (36.36% White, 36.36% Caucasian, 20% Black, and 7.27% Asian), and 47.27% of the participants are males while the rest are females. All the neonates were recorded during procedural and postoperative pain. The procedural pain usually occurs as a result of a short painful stimulus such as immunization and it ends as soon as the cause of pain is removed. Postoperative pain has a clearly defined beginning point and expected end point, and it occurs after a known stimulus such as a surgical procedure. Behavioral, vital signs and cortical activity were recorded during the procedural pain. In case of postoperative, we only recorded behavioral responses and vital signs. The neonates demographics for both procedural and postoperative pain are summarized in Table 1. Table 2 provides the demographics of neonates who show cortical activity in addition to the behavioral responses and vital signs during procedural pain.

Table 1

Demographic information of neonates recorded following procedural and postoperative procedures. The collected data include behavioral responses and vital signs.

Attributes	Types	Procedural	Postoperative
Gestational age	Preterm (< 37 GW)	18	5
	Term (≥ 37 GW)	17	4
	Mean age (GW)	35.94	35.32
Birth weight	Low (< 2500 gm)	13	4
	Normal (≥ 2500 gm)	21	5
	Mean weight (gm)	2823.24	2733.33
Gender	Male	17	3
	Female	17	5
Ethnicity	Hispanic	9	5
	Non-Hispanic	26	4
Race	White	17	3
	Black	6	2
	Caucasian	7	4
	Asian	4	0
Total subjects - 45 (36+9)		36	9

* 3 subjects participated in both procedural and postoperative data collection.

Table 2

Demographic information of neonates who show cortical activity.

Attributes	Data Distribution
Gender	Male	6
	Female	7
Ethnicity	Hispanic	1
	Non-Hispanic	12
Race	Black	3
	Caucasian	9
Birth Weight (gm)	Min	1090
	Max	2360
Birth GA (weeks)	Min	27
	Max	34
Head Circumference (cm)	Min	26
	Max	34.5
Total subjects		13

* NIRS data is collected only for premature subjects with procedural pain.

* 4 subjects participated in both procedural and NIRS data collection.

Data collection design

Both behavioral (facial expression, body movement, crying) and physiological (vital signs and cortical activity) responses following procedural and postoperative stimuli were monitored using an RGB camera (behavioral), pulse oximeter (vital signs), and near-infrared spectroscopy INVOS 5100C (cortical activity). All the data were collected during routine clinical procedures and carried out in the normal clinical environment that was only modified by the addition of the cameras. This makes our dataset highly representative of a real-world condition. Examples of the routinely performed procedures that induce procedural and postoperative pain are presented in Table 3.

Table 3

List of clinically performed procedures that induce procedural and postoperative pain.

Pain Types	Painful Stimuli
Procedural	Heel stick, Accu check, Immunization, Blood drawing
Postoperative	G-tube, Gastroschisis, Intubation, Omphalocele repair, Inguinal hernia repair

To determine the pain levels of the recorded stimuli, trained nurses used the clinically validated NIPS [5] (procedural) and N-PASS [6] (postoperative) pain scales. The documented scores were stored in addition to the relevant medical information and pain history for each neonate.

Equipment setup

The equipment setup for our data collection is shown in Figure 1. As shown in the Figure 1, we installed an RGB (GoPro Hero) on a camera stand facing the neonatal incubator to record the frontal face and body of the neonate. The neonates were lying in a supine position facing the camera in most cases. We used the camera to record both visual and vocal signals. The camera was configured to record video data using a 1080p resolution with a frame rate of 30 FPS and audio data with a sampling rate of 48KHz. We used the GoPro smart remote to start and end the camera recording, and to mark specific events.

In addition to the behavioral responses, we used a bedside Phillips MP-70 device to monitor and record several vital signs including heart rate (HR), oxygen saturation rate (SpO2), and blood pressure (BP). To collect the neonate’s cortical activity, we used an INVOS 5100 NIRS device. The NIRS probe was positioned on the contralateral side of the forehead. We connected both the vital signs monitor and the NIRS device to the Vital SyncTM device as shown in Figure 1. This device is used to synchronize the vital signs and cortical activity by inserting timestamps that indicate the start and end of a specific event (e.g., baseline event, pain event, post-pain event, etc).

To ensure synchronization between the behavioral and physiological signals, we marked the start and end points of the data collection for a specific event (e.g., baseline prior painful procedure) by simultaneously inserting a timestamped-event to the VitalSyncTM monitor and using the GoPro smart remote (or a clapperboard) with the GoPro camera. We also marked, using the same method, the time of assessing pain (providing the pain scores) by the bedside caregivers. It is important to note that we used an additional GoPro camera to collect vital sign data in cases where the access to the bedside vital signs monitor was denied. In such cases, we placed another GoPro camera in front of the vital signs monitor and converted the recorded videos of vital signs into numeric values using OCR (Optical Character Recognition). The complete setup of the equipment and neonates is shown in Figure 1.

Routine clinical procedures for pain stimulation

We recorded, in the presence of clinicians, bedside nurses, and the principal investigator, behavioral and physiological responses during procedural and postoperative pain as follows.

Procedural pain procedures

A total of 36 neonates were recorded during procedural pain. Procedural pain is a short-lived acute pain which occurs as a result of a quick tissue breaking during routine procedures. The painful procedures in our work include heel lancing, accu check, immunization, and vaccinations (see Table 3). The recording for procedural pain consists of the following time periods: baseline prior to the procedure (1-2 minutes), the complete painful procedure, and post the completion of the painful procedure during recovery (3-8 minutes). As for the recording of the cortical activity using the NIRS device, we collected the data from only premature neonates ( 37 GW) during the following periods: baseline prior to the procedure (10 minutes), the painful procedure, and after the completion of the procedure for 10 minutes. Figure 1 shows a visualization of the procedural pain recording using the camera (video and audio), vital signs monitor, and the NIRS device. We synchronized the recording of all devices as described in the previous “Equipment Setup” section 2.3.

Postoperative pain procedures

A total of 9 neonates were recorded during postoperative pain. Postoperative pain occurs as a result of tissue injury caused by any major surgery. Postoperative pain has a clearly defined beginning point and expected end point. Neonates’ behavioral responses to the postoperative painful stimulus are usually less intense as compared to their response to procedural pain [8], [9]. This can be attributed to the low physical reserves of a neonate to sustain a response and the level of sedation/analgesia. The major surgeries that cause the postoperative pain in our work include gastrostomy tube and bowel obstruction (see Table 3). The recording for the postoperative pain consists of the following time periods: fifteen minutes baseline period prior to the surgery and the period after the completion of the surgery for up to three hours. Figure 1 shows a visualization of the postoperative pain recording using the camera (video and audio) and the vital signs monitor. Similar to the procedural recording, we synchronized the recording of all devices as described in the previous “Equipment Setup” section 2.3.

Recorded behavioral and physiological data

Video and audio recording

As reported by several neonates studies [2], [4], [7], facial expression is one of the most common and specific responses to pain. Facial expression of pain is defined as the movements and distortions in facial muscles associated with a painful stimulus. The facial movements associated with pain in neonates include: deepening of the nasolabial furrow, brow lowering, narrowed eyes, vertical and horizontal mouth stretch, lip pursing, lip opening, tongue protrusion, taut tongue, and chin quiver. In addition to facial expression, neonates tend to move their head, extend their arms/legs, and splay their fingers when they experience pain. Finally, crying [4] is a common sign of discomfort, hunger, or pain in neonates. It conveys information that helps caregivers to assess the emotional states and react appropriately.

In our work, we captured and monitored all these behavioral signals using a portable, inexpensive ( US$ 350), and contactless RGB camera. The camera was installed on a stand that faces the neonate’s incubator as illustrated in Figure 1. We recorded the neonates while they are sleeping in supine and prone positions. Some neonates were wrapped by a blanket that covers the majority of their bodies. In these cases where one of the signals is missing (e.g., covered body or blocked face), we rely on other signals for analyzing pain. The vocal data was recorded using the camera’s microphone. The recorded audio includes the the sounds of neonates as well as background noise such as sounds of equipment, nurses, and other neonates in the room. The recorded behavioral data were synchronised with a clipperboard placed in front of the camera or with GoPro smartremote labeling, which allows to mark specific time periods in the video.

Vital signs recording

Beat-by-beat heart rate (HR), blood oxygenation (SpO2), and blood pressure (BP) were monitored and recorded for each neonate. This data was recorded using Phillips MP-70 bedside device and transferred to the The Vital Sync which is used to synchronize the vital signs with the cortical activity and to label specific time periods by inserting time-stamps. Prior to analyzing the vital signs data, we exported the recorded numbers from the the Vital Sync as Excel files.

Cortical activity recording

We collected the cortical activity data as an objective indicator of pain for verification. To measure the deoxyhemoglobin () and the oxy-hemoglobin () concentrations, the INVOS Near-Infrared Spectroscopy (NIRS) device was used following a sample rate of 30 seconds. Finally, total hemoglobin concentration () was measured by adding the and . The regional oximetry value () was also measured following . To export the NIRS data with the timestamp and log the events of start or end of the painful procedure, the Vital Sync was used.

Prior to collecting the data, the bedside nurse verified the location of the painful procedure. Then, a member of our team placed the NIRS probes on the contralateral side of the forehead. At first, a connection was established between the INVOS and Vital Sync. Then, we collected the data for approximately 10 minutes as a baseline. After the baseline, the nurse started the painful procedures. We also collected post-procedural data for approximately 10 minutes.

As the majority of parents refused to attach any probe/sensor to the body of a neonate who went through any major surgery (i.e., G-tube), we only collected cortical activity data during procedural pain.

Contextual and medical data

Several contextual and medical data were collected apart from the video, audio, and physiological data. This data includes pain types (i.e., procedural vs postoperative), demographics (i.e., gestational age, race, ethnicity, and gender), medication pattern (i.e., type, and dose), and non-pharmacological interventions (i.e., mother’s presence, sweeties, swaddling, holding/rubbing, or pacifier use). This information was documented by an experienced nurse and kept in the records. Any missing information or diagnoses requiring follow-up tests were collected from the hospital record at a later date whenever possible.

Pain score labeling

Ground truth labels or pain scores were documented by two expert nurses during procedural and postoperative pain. The inter-observer reliability was established using Kappa coefficient and Pearson correlation to measure the inter-observer agreement. The nurses in the NICU at Tampa General Hospital (FL, United States) use NIPS [5] and N-PASS [6] scales to assess procedural and postoperative pain, respectively.

Procedural pain scoring via NIPS

In case of procedural pain, the nurses performed the pain scoring at bedside (on-site) as well as off-site using the recorded videos. The on-site bedside assessment of pain provides realistic scoring of the current standard while the off-site video assessment allows repeated viewing and hence more accurate scoring. The nurses provided pain scores at bedside (on-site) for the time periods shown in the scoring sheet. These time periods are baseline, preparation of the procedure, at the start of the painful procedure, and after the completion of the painful procedure.

NIPS [5] scale generates the total pain score based on the scores of the following pain indicators: facial expression, crying sound, breathing patterns, movements of arms/legs, and the state of arousal. Each indicator has a score of 0 (no-pain) or 1 (pain) with the exception of crying sound which has three scores 0 (no-pain), 1 (mild pain), and 2 (pain). Finally, all the scores are aggregated to generate the final pain score. To determine the final pain label of the neonates, thresholding is performed as follows: score 0-2 (no-pain), score 3-4 (moderate pain), and score 4 (severe pain).

Postoperative pain scoring via N-PASS

In case of postoperative pain, the nurses performed the pain scoring at bedside (on-site) as well as off site using the recorded videos.

N-PASS [6] scale assesses the pain and sedation states of the neonate based on the scores of the following indicators: crying irritability, behavior state, facial expression, extremities tone, and vital signs (HR, BP, SpO2). Each of these indicators has a score range from -2 to +2. Three score intensities represent sedation (-2 to -1), normal (0), and pain/agitation (1 to 2). The final pain score, which ranges from -10 to +10, is determined by summing up all the individual indicators. This final score is adjusted by adding additional points (1 to 3 points) based on the gestational age; i.e., the score is adjusted for pre-term neonates due to their limited ability to express behavioral signs of pain.

These levels of pain provide the ground truth labels that are used to train the machine learning classifiers. The nurses provide the pain score at the bedside (on-site) every 15 minutes during the postoperative period in addition to the score prior to the surgery. As for the off-site scoring, at first potential segments from the raw unprocessed videos are selected which are then viewed and scored by the nurses.

Ethics Statement

Our study was approved by the University of South Florida (USF) Ethics Internal Review Board [1], [3] (IRB # Pro00014318) and conforms to the standards set by the Declaration of Tampa General Hospital (TGH, Tampa, FL, United States). Prior to the data recording for each neonate, written informed consent was obtained from the guardians in addition to another consent for the use of identifiable images or videos in publications. The guardians also must agree to allow researchers access to contextual and medical data from the patients’ record.

All the data was recorded according to clinical standards by two experienced research nurses. We adopted a protocol optimized for the acquisition of vital signs and cortical activity within the hospital ward environment and used a portable and inexpensive camera that can be easily adopted and integrated into any clinical environment. The research nurses wore anti-static nursing clogs (ToffeIn) and aprons. The clinical scientists monitored the collection of the behavioral signals from the RGB camera and the physiological signals from the bedside monitors. They notified the nurses when 1) the recording starts and ends and 2) the nurses need to provide ground truth scores. At the end of each recording session, the video/audio data were exported from the camera in addition to the physiological numbers which were exported from the Vital SyncTM as excel files. These excel files have a column that indicates the start and end for a specific time period (e.g., start recording, pain score 1, etc). These same events are marked in the video by the clinical scientist using GoPro smart remote.

Pain history and relevant medical information were obtained from the patient notes and collected by an experienced nurse. Whenever possible, missing information or diagnoses requiring follow up tests were collected from the notes at a later date.

CRediT Author Statement

Md Sirajus Salekin: Conceptualization, Methodology, Data Curation, Writing - Original Draft, Writing - Review & Editing, Visualization; Ghada Zamzmi: Conceptualization, Methodology, Data Curation, Writing - Original Draft, Writing - Review & Editing; Jacqueline Hausmann: Data Curation, Writing - Review & Editing; Dmitry Goldgof: Conceptualization, Methodology, Writing - Review & Editing, Supervision, Funding acquisition; Rangachar Kasturi: Conceptualization, Methodology, Writing - Review & Editing, Supervision; Marcia Kneusel: Investigation, Data Curation; Terri Ashmeade: Investigation, Supervision, Project administration; Thao Ho: Supervision, Project administration, Funding acquisition; Yu Sun: Conceptualization, Methodology, Writing - Review & Editing, Investigation, Supervision, Project administration, Funding acquisition.

Declaration of Competing Interest

The authors declare no competing interests.

Subject	Biomedical Engineering and Computer Science
Specific subject area	Neonatal Intensive Care, Artificial Intelligence and Machine Learning Applications
Type of data	Video, audio, and medical information
How data were acquired	RGB camera (GoPro Hero), vital sign monitor (Phillips MP-70), near-infrared spectroscopy (INVOS 5100C)
Data format	Raw
Parameters for data collection	Multimodal data including behavioral (video and audio), physiological (vital signs and cortical activity), contextual, and medical. All the data were scored by trained nurses using clinically validated pain scales.
Description of data collection	Data were collected from 58 neonates while undergoing any procedural or postopertive observation in NICUs.
Data source location	Tampa General Hospital, Tampa, Florida, United States
Data accessibility	To advance the research in clinical and automated pain assessment, we introduced in this paper a publicly available, well-annotated, and multidimensional neonatal pain dataset. To access our dataset, interested researchers must send a request to the Principal Investigator (PI) of this project. The PI will then send back data agreement and sharing forms that have to be signed by the authorized person according to the provided instructions. After receiving the properly signed agreement, we will share the entire dataset via a protective cloud storage. Further details can be found in the following project website. https://rpal.cse.usf.edu/project_neonatal_pain/.
Related research article	Salekin, M.S., Zamzmi, G., Goldgof, D., Kasturi, R., Ho, T., Sun, Y., 2021, Multimodal spatio-temporal deep learning approach for neonatal postoperative pain assessment, Computers in Biology and Medicine 129, 104150. https://doi.org/10.1016/j.compbiomed.2020.104150