Evaluation of a novel computer vision-based livestock monitoring system to identify and track specific behaviors of individual nursery pigs within a group-housed environment.

Animal behavior is indicative of health status and changes in behavior can indicate health issues (i.e., illness, stress, or injury). Currently, human observation (HO) is the only method for detecting behavior changes that may indicate problems in group-housed pigs. While HO is effective, limitations exist. Limitations include HO being time consuming, HO obfuscates natural behaviors, and it is not possible to maintain continuous HO. To address these limitations, a computer vision platform (NUtrack) was developed to identify (ID) and continuously monitor specific behaviors of group-housed pigs on an individual basis. The objectives of this study were to evaluate the capabilities of the NUtrack system and evaluate changes in behavior patterns over time of group-housed nursery pigs. The NUtrack system was installed above four nursery pens to monitor the behavior of 28 newly weaned pigs during a 42-d nursery period. Pigs were stratified by sex, litter, and randomly assigned to one of two pens (14 pigs/pen) for the first 22 d. On day 23, pigs were split into four pens (7 pigs/pen). To evaluate the NUtrack system's capabilities, 800 video frames containing 11,200 individual observations were randomly selected across the nursery period. Each frame was visually evaluated to verify the NUtrack system's accuracy for ID and classification of behavior. The NUtrack system achieved an overall accuracy for ID of 95.6%. This accuracy for ID was 93.5% during the first 22 d and increased (P < 0.001) to 98.2% for the final 20 d. Of the ID errors, 72.2% were due to mislabeled ID and 27.8% were due to loss of ID. The NUtrack system classified lying, standing, walking, at the feeder (ATF), and at the waterer (ATW) behaviors accurately at a rate of 98.7%, 89.7%, 88.5%, 95.6%, and 79.9%, respectively. Behavior data indicated that the time budget for lying, standing, and walking in nursery pigs was 77.7% ± 1.6%, 8.5% ± 1.1%, and 2.9% ± 0.4%, respectively. In addition, behavior data indicated that nursery pigs spent 9.9% ± 1.7% and 1.0% ± 0.3% time ATF and ATW, respectively. Results suggest that the NUtrack system can detect, identify, maintain ID, and classify specific behavior of group-housed nursery pigs for the duration of the 42-d nursery period. Overall, results suggest that, with continued research, the NUtrack system may provide a viable real-time precision livestock tool with the ability to assist producers in monitoring behaviors and potential changes in the behavior of group-housed pigs.

INTRODUCTION

Rapid detection of changes in behavior patterns of individual pigs within a group-housed environment is essential for ensuring health and wellbeing. Because observable behaviors reflect internal states, such changes to behavior may be indicative of sickness, stress, and injury. Rapid and accurate observation of changes in specific behaviors associated with stress, disease, or injury may lead to earlier detection. Early detection may allow for rapid treatment or implementation of management strategies that can improve the health and welfare of livestock and enhance production efficiency. Currently, contemporary swine production systems rely upon human observation (HO) for identification of pigs exhibiting atypical behaviors (Friendship, 2005). While effective, HO is hindered by two primary factors: 1) visible clinical symptoms must be present at the time of evaluation and 2) the tendency of animals to mask disease/weakness in the presence of humans. Caretakers must rely upon visual observation of changes in behavior that may be indicative of disease (clinical symptoms), such as reduced feed intake, coughing, lethargic/depressed behavior, and/or gaunt/emaciated physical appearance (Gemus-Benjamin & Kramer, 2014). In addition, the presence of producers/caretakers to conduct HO can have a significant impact on the adaptive behaviors of the pigs. As a prey species, recognition of a perceived threat/danger may result in pigs masking mild to moderate symptoms of vulnerability (sick/injury) to avoid perceived potential predation (Underwood, 2002; Weary, et al., 2009; Turner et al., 2019). Another hurdle of HO is the requirement of significant training and an attention to detail/focus which tapers off with fatigue. In addition, HO at contemporary facilities is labor intensive and time consuming.

Over the past decade, the pork industry has made significant strides to improve production efficiency (Stalder and Stock, 2017). However, it seems that improvements in reducing the rate of mortality have been out of reach. The 2012 Pork Industry Productivity Analysis (Stalder, 2013) reported that from 2007 to 2012 the average mortality rate of nursery pigs was 4.5% with an overall mortality rate for weaned-to-finish pigs of 7.1%. For the nursery period alone, the rate of mortality per year fluctuated from a low of 3.8% in 2012 to high of 5.8% in 2008. The 2017 Pork Industry Productivity Analysis (Stalder and Stock, 2017) reported that from 2012 to 2017 the average mortality rate of nursery pigs was 5.1% while an overall mortality for wean-to-finish pigs was 7.5%. Stalder and Stock (2017) reported the lowest rate of mortality occurring during 2012 (3.8%) and the greatest rate during 2014 (5.8%). Based upon these two Pork Industry Productivity Analysis (Stalder, 2013; Stalder and Stock, 2017), over the past 14 yr the average nursery mortality rate has hovered around 4.5% with the average wean-to-finish rate at 7.3%. Both rates of mortality are obfuscated by a year-to-year fluctuation in nursery mortality rate ranging from 5.8% to 3.8% and wean-to-finish mortality ranging from 8.3% to 6.3%. While there are numerous factors that contribute to increased mortality, there is little doubt that the limitations of HO contribute to unnecessary deaths.

A potential solution to overcome some of the limitations of HO is the development of precision livestock technology (PLT) that can assist and enhance a producer/caretaker’s ability to detect changes in pig behavior. Utilization of PLT may facilitate early and accurate identification of sickness, stress, or injury of group-housed pigs. One of the most sought-after objectives for the development of PLT is to create a real-time/on-line system capable of continuously monitoring behavior and activities of individual livestock within group-housed settings (Berckmans, 2006). To accomplish this, PLT must overcome the hurdle of tracking a Complex Independent Time varying Dynamic system (i.e., the individual animal Berckmans, 2017 and meet three conditions Berckmans, 2006). The three conditions that it must be capable of are: 1) maintaining continuous ID of the individual animal, tracking/measuring the individual, and providing an accurate measure for each variable of interest, 2) providing a reliable prediction and/or expectation on how the variable will change for an individual and 3) coupling of the reliable prediction/expectation to the continuous measurements of the chosen variable to monitor and automatically provide reliable information (Berckmans, 2006).

To achieve the above-mentioned criteria, researchers have investigated a wide range of technologies. Gómez et al. (2021) reported identifying more than 80 different existing technologies being investigated to evaluate pig welfare. This wide array of technologies includes load cells, microphones, accelerometers, thermal and infrared cameras, and radio frequency identification ear tags, just to mention a few (Gómez et al., 2021). With the rapid advancement of artificial intelligence and machine learning, researchers have added computer vision-based platforms to this list of technologies. The advancements in computer vision have led to our team to develop the NUtrack Livestock Monitoring System (NUtrack). NUtrack is a deep feature-based identification and tracking platform. The NUtrack system has been validated across several pig sizes and housing environments. Initial results suggest that this system may have the capability of autonomously identifying individual pigs, maintaining this identification for an extended period, and classifying the behaviors of group-housed pigs. (Mittek et al., 2017; Psota et al., 2019, 2020a, 2020b). Based upon our previous research, this study was designed to evaluate the hypothesis that the NUtrack system has the capabilities to autonomously detect the ID of individual nursery pigs within a group setting, maintain the ID of individuals, track the behaviors (i.e., lying, standing, walking, distance traveled, ATF, and ATW), and detect potential changes in these behaviors over time for group-housed nursery pigs.

MATERIALS AND METHODS

Animal and Experimental Design

All experimental procedures adhered to the Guide for the Care and Use of Agricultural Animals in Research and Teaching. All procedures were reviewed and approved by the University of Nebraska – Lincoln Institutional Animal and Care and Use Committee (IACUC #1409).

The NUtrack system consisted of a depth-enabled camera (Kinect v2, Microsoft, Redmond, WA) connected to mini-PC (NUC, Intel, Santa Clara, CA) and a 4-TB Fantom hard drive (Fantom Drives, Torrance, CA). The depth-enabled camera captured color, infrared, and depth images at a rate of approximately 5 frames/sec. The method for tracking the location and orientation of each target processes depth images and extracts each individual instance by fitting ellipsoidal shapes to the resulting 3D point cloud belonging to the inside of the pen (Mittek et al., 2017; Psota et al., 2020b). By aligning the point clouds to the pen floor space, the height of each pig’s body was determined, and a threshold of height was used to classify each pig as either lying or standing (Mittek et al., 2017). Movement (walking) was calculated by measuring the distance traveled between frames (5 frames/sec). If an individual pig was determined to be moving at ≥ 10 cm/sec, the behavior was classified as walking (Psota, 2020b).

Utilizing the location and orientation of each pig allowed the NUtrack system to approximate the position of the individual pig’s head. This position was used to approximate behavior “At The Feeder” (ATF) and “At The Waterer” (ATW) behaviors. Within each image space, a polygonal shaped zone defined the location of the feeding trough (0.35 × 0.62 m) and the water source (0.17 × 0.16 m; Figure 3a). When the pigs head and shoulders resided within that polygonal image space, oriented towards the feeder or waterer (Figure 3a) and not determined to be moving (≤ 10 cm/sec), the pig was classified as ATF or ATW. Classification of ATF or ATW was not considered an indication feed or water intake, only a classification being within the defined polygonal shaped zone and not moving. Finally, the identity of each pig was determined by observing the location of each pigs’ left and right ear, cropping provided ear tag images from the video frame, and classifying it with a convolutional neural network. Greater details related to ear localization and ear tag classification was previously reported by Psota et al. (2020a and Psota 2020b).

Figure 3.

Still images of video collected and processed by NUtrack system: a) annotated area for the location of the waterer (blue) and feeder (green), b) example of the identification error classified as Lost Label; NUtrack label for blue 1 not correctly associated with pig, c) example of identification error classified as Label Swap; NUtrack label between red 22 ear tag and green 333 ear tag have been swapped, and d) example of classification of activity error, NUtrack activity classification label for yellow 333 ear tag and green 22 ear tag displayed as walking when visual evaluation determined activity classified as eating.

Twenty-eight newly weaned group-housed pigs (6.64 ± 1.94 kg) were used to evaluate the ability of the NUtrack system to autonomously identify, maintain identification, and track the behavior of group-housed nursery pigs. Pigs were housed within the Animal Science Complex’s Swine Nursery Facility at the University of Nebraska – Lincoln. The nursery facility is comprised of two nursery rooms, each room containing six pens (1.22 × 2.13 m). For this trial, four pens within one room were utilized (pens 1–4). Each pen was equipped with a three-hole feeder and provided ad libitum access to feed and water. Nursery diets were formulated to meet or exceed NRC requirements (NRC, 2012; Table 1) and water was provided via a nipple waterer (pens 1 and 2) or bowl waterer (pens 3 and 4). Within the room, two NUtrack systems were installed. For each system, a Kinect v2 camera was positioned overhead at a height of 2.7 m above the pens to capture behavior data from two adjacent pens (Figure 1). The Kinect v2 cameras were centered based upon the length and width of the two adjoining pens (Figure 1). One camera covering pens 1 and 2 and the other camera covering pens 3 and 4. Cameras were programmed to initiate recording and analysis immediately upon placement of pigs into the pens.

Table 1.

Diets fed to 28 newly weaned nursery pigs during a 42-d nursery period

Ingredient(% of diet)1	Starter	Nursery1	Nursery22	Nursery3
Corn	43.0	43.6	60.0	57.0
Soybean Meal	14.7	32.0	33.8	34.8
Dried Whey	22.5	15.0
Tallow
Fish Meal	8.00	4.0
Animal Plasma	6.00
Corn Oil	3.00	3.0	3.00	3.0
Di-calcium Phosphate	0.40	1.0	1.70	1.7
Limestone	0.25	0.35	0.60	0.30
Salt	0.30	0.30	0.30	0.30
Vitamin Premix3	0.25	0.25	0.25	0.15
Swine TM Premix4	0.15	0.15	0.15	0.15
Mecadox	1.0
Zinc Oxide	0.4	0.3
DL-Methionine	0.05	0.025	0.025	0.025
L-lysine HCL			0.040	0.040
Denegard				0.180
Aureomycin 50				0.400
Phytase
Composition, %
Net Energy	2,631	2,534	2,515.0	2,531.0
Crude Protein	16.69	20.02	17.92	17.95
Lysine	1.33	1.21	1.03	1.03

All ingredients reported on as a percentage as fed basis.

Medicated Nursery 2 diet included Denegard and Auromycin 50.

Vitamin Premix ingredients: Vitamins A, D, E, K, Niacin, Panothenic Acid, Riboflavin, Vitamin B12.

Swine TM Premix ingredients: Copper, Iodine, Iron, Manganese, Selenium, Zinc.

Figure 1.

Schematic of nursery pens and placement of NUtrack system within the research nursery room within the Animal Science Complex at the University of Nebraska – Lincoln.

Prior to the start of the trial, pigs were randomly selected from four litters (Duroc sire × Yorkshire × Landrace sows), stratified by gender, litter, and body weight. Pigs were randomly assigned to either pen 1 or 3. Prior to placement into the pens (day 1), pigs were weighed, and tagged with a Hog Max ear tag (Destron, South St. Paul, MN) in both ears. Five different ear tag colors (blue, green, red, white, and yellow) and a numerical pattern of 1, 22, and 333 were used to create 15 unique identification markers, allowing the NUtrack system to recover from tracking errors and provide visual identification that could easily be identified by trained evaluators. To allow for identification recovery by the NUtrack system, a deep classification network was trained to identify pigs based on small image crops of the ear tags (Psota et al. 2020b). Integration of the deep classification network with the system’s detection and tracking capabilities made it possible for the NUtrack system to recover from tracking errors that occurred.

Immediately following processing pigs were placed in pens 1 and 3 (Figure 2) and the NUtrack systems initiated continuous capture of video data. On days 7, 14, 21, 28, 35, and 42 at 0800 h, body weights and feed disappearance were determined and hard drives for each system were replaced. Replaced hard drives were transported to the Perceptual Systems Research Group within the Electrical and Computer Engineering Department at the University of Nebraska – Lincoln to download and analyze video. On day 23, seven pigs were randomly selected from pen 1 and transferred to pen 2 and seven pigs from pen 3 were transferred to pen 4. The nursery phase concluded on day 42 all pigs were moved to grow/finisher pens.

Figure 2.

Still images of from video frame collected and processed by NUtrack system with overlying labels for individual identifications1 (ear tag color/number), classification of activity,2 and polygonal shaped zone defined for feeder (transparent green rectangle) and water (tranparent blue rectangle). Left image captured from pen 1 during the first week of the nursery phase and the right image captured includes both pens 1 and 2 during the fourth week of the nursery phase.

Manual Determination of Accuracy of Identification and Determination of Activity

To determine the NUtrack systems ability to accurately identify individual pigs and correctly classify behaviors, 800 frames were randomly selected frames across the 42-d nursery phase. Four-hundred frames from video were chronologically selected captured during days 1–22 (pens 1 and 3;14 pigs/pen) and 400 frames from video captured during days 23–42 (all four pens; 7 pigs/pen). From the 800 randomly selected frames a total of 11,200 individual observations were evaluated (14 pigs/frame) by two trained observers to determine individual identification (visual verification of the colored/numbered ear tag) and behaviors of lying, standing, walking, ATF, and ATW. Within each frame, an individual label (Figure 3) generated by the NUtrack system was overlayed onto each pig (Figure 2). The label assigned by the NUtrack system displayed the identification of the pig (ear tag color and number) and the classified behaviors (L = lying, S = stand, W = walking, E = at the feeder, and D = at the waterer). Ethogram for classification of these behaviors by trained evaluators is reported in Table 2. Of the 11,200 observations, 10,655 were used to confirm ID as 545 observations were removed due to inability of evaluators to confirm ID. From the 11,200 observations, 10,682 were used to determine accuracy of behavior as 518 observations were removed due to inability of evaluators to determine behavior. For each observation, ID and behavior displayed by the label generated by NUtrack System (Figure 3) were compared to the ID and behavior determined by trained evaluators. Observations were scored with only two outcomes: incorrect (errors) or correct. For accuracy of identification, two types of errors were observed: 1) trained observation determined that the NUtrack System had swapped the identification label of two pigs within a frame (label swap) and 2) pigs within a frame either did not have a NUtrack label or a generated NUtrack label was present but not associated with any pigs (no label). Correct classification of behavior was defined as agreement between the NUtrack behavior classification and trained evaluators behavior classification. Errors were documented when the NUtrack behavior label was not in agreement with that of the visual observation of the trained evaluators (ethogram reported in Table 2).

Table 2.

Ethogram for classification of behaviors utilized by trained evaluators to determine the accuracy of the NUtrack system

Activity	Description of behavior for visual classification
Lying	♦ Anterior and posterior aspects of body in contact with floor♦ Minimum of one leg visible in frame♦ Legs either straight, bent, or tucked up under the body
Standing	♦ No part of the body in contact with flooring♦ Minimum of one leg visible in frame♦ Legs perpendicular to flooring
Walking	♦ Standing as defined above and visual movements between previous and post selected frame
At the Feeder	♦ Standing as defined above with head located in the annotated area of the feeder and aligned towards the feeder
At the Waterer	♦ Standing as defined above with head located in the annotated area of the waterer and aligned towards the nipple/cup

Statistical Analysis

Accuracy and behavior data were analyzed using the GLIMMIX procedure of SAS specific for repeated measures (SAS Inst. Inc., Cary, NC). Week, sex, and litter were included as fixed effects, pen as a random effect, and individual pig served as the experimental unit. When main effects or interactions were significant (P ≤ 0.05), specific comparisons were made using the PDIFF; P = 0.06–0.10 was considered a tendency. Accuracy and behavior data are presented as LSMeans ± SEM. Two and three-way interactions were non-significant (P > 0.10), thus excluded from the results.

RESULTS

Accuracy of Identity and Classification of Activity

NUtrack system’s accuracy for the detection and identification of pigs reported in Table 3. Overall accuracy for detection of pigs within a frame was 98.8% and accuracy for correctly identifying individual pigs was 96.8%. Errors related to detection (lost label; Figure 3C) was greater (P ≤ 0.001) during the first 22 d (116 errors) when compared to the final 19 d (14 errors). The same observation for errors related to correct identification (label swapped; Figure 3D) was present. The greatest (P ≤ 0.001) number of labeled swap errors occurred during the first 22d (255 label swapped errors), when compared to the last 19 d (82 label swapped errors).

Table 3.

Identification errors (lost label and label swapped) associated with the NUtrack systems compared to visual evaluation of 10,655 individual observations from 800 randomly selected frames of 28 nursery pigs during a 42-d of the nursery period

	Days 1–22	Days 23–42	P - value	Overall
Individual Observations	5,171	5,484		10,655
Detected1	5,055	5,470		10,525
Lost Label2	116	14	≤ 0.001	130
% Detected	97.8	99.7		98.8
Observations for ID	5,055	5,470		10,525
Correct ID3	4800	5,388		10,188
Label Swap4	255	82	≤ 0.001	337
% Correct ID	94.6	98.5		96.8

Number of observations in which the NUtrack system correctly detected and labeled a pig as determined by trained evaluators.

Number of observations in which the NUtrack system did not correctly detect and label a pig as determined by trained evaluators.

Number of observations where the NUtrack system correctly detected a pig and correctly determined the individual identification as determined by trained evaluators.

Number of observations where the NUtrack system correctly detected a pig but did not correctly determine the correct individual identification as determined by trained evaluators.

Association of detection/identification errors and the classification of behaviors as determined by trained evaluators reported in Table 4. During the 42-d nursery period, the greatest (P ≤ 0.001) occurrence of lost labels was noticed when pigs were classified as lying, when compared to all other behaviors. Of the lost label errors observed, the fewest (P < 0.001) errors were observed when pigs were classified as ATW. Of note is that there were no lost label errors associated with the behavior of walking based upon the method by which the NUtrack system determines the activity of walking (>10cm/sec). Similarly, during the 42-d nursery period the greatest (P < 0.001) occurrence of label swap errors occurred when pigs were classified as lying (76.9%), when compared to all other behaviors. Furthermore, label swap errors were greater (P < 0.001) when NUtrack classified pigs as standing, when compared to ATF or ATW.

Table 4.

Overall association of detection/identification errors (lost label and label swap) with the classified behaviors of individual pig as classified by the NUtrack system when compared to visual annotation of 10,655 randomly observations from 800 selected frames of 28 nursery pigs during a 42-d of the nursery period

Error × Behavior	Overall
	n	%
Lost Lable1
Lying	100a	76.9
Standing	14b	10.8
Walking	0c	0.0
ATF2	13b	10.0
ATW3	3b	2.3
P-value	≤ 0.001
Label Swap4
Lying	277a	74.7
Standing	38b	10.2
Walking	9c	2.4
ATF	11c	3.0
ATW	2c	0.5
P-value	≤ 0.001

Denoted lost label in which the NUtrack system did not correctly detect and label a pig as determined by trained evaluators.

Pigs were classified as at the feeder (ATF) when the head of an individual pig was determined to be within the annotated location of the feeder (Figure 3a).

Pigs were classified as at the waterer (ATW) when the head of an individual pig was determined to be within the annotated location of the waterer (Figure 3a).

Denotes label swap error in which the NUtrack system correctly detected a pig but did not correctly determine the correct individual identification as determined by trained evaluators..

Denote differences (P < 0.05) within columns between behavior classifications as classified by visual evaluation.

Change over time in the rate of detection/identification errors by behaviors as determined by trained evaluators is reported in Table 5. For both types of errors (LL and LS), the majority (P < 0.05; 79.4%) occurred during days 1–22. For LL, 116 errors occurred during the first 22 d and only 14 during the final 19 d (P ≤ 0.001). The greatest (P ≤ 0.03) rate of LL during the first 22 d occurred when pigs were lying, compared to all other behaviors. During the final 19 d there was no difference (P ≥ 0.56) in errors associated with LL across the behaviors. The greatest number of LL errors for lying and ATF occurred during the first 22 d, as compared to the last 19 d. There was no difference (P ≥ 0.36) for the behaviors of standing, walking, and ATW.

Table 5.

Change over time in rate of detection/identification errors (Lost Label and Label Swap) with the classified behaviors of as classified by the NUtrack system when compared to visual annotation of 10,655 randomly observations from 800 selected frames of 28 nursery pigs during a 42 d of the nursery period

Error × Behavior	Days 1–22	Days 23–42	P-value
	n	n
Lost Label1
Lying	92a	8	≤ 0.001
Standing	9b	5	0.36
Walking	0c	0	−
ATF2	12b	1	0.03
ATW3	3b,c	0	0.52
P- value	≤ 0.03	≥0.56
Label Swap4
Lying	212a	65a	≤ 0.001
Standing	24b	14b	0.04
Walking	9b	0c	≤ 0.001
ATF	8b	3b,c	0.03
ATW	2c	0c	0.05
P- value	≤ 0.05	≤ 0.04

Denoted lost label in which the NUtrack system did not correctly detect and label a pig as determined by trained evaluators.

Pigs were classified as at the feeder (ATF) when head of an individual pig was determined to be within the annotated location of the feeder (Figure 3a).

Pigs were classified as at the waterer (ATW) when head of an individual pig was determined to be within the annotated location of the waterer (Figure 3a).

Denotes label swap error in which the NUtrack system correctly detected a pig but did not correctly determine the correct individual identification as determined by trained evaluators.

Denote differences (P < 0.05) within columns between behavior classifications

For LS, 255 errors occurred during day 1–22 d and only 82 during the final 19 d (P ≤ 0.001). Regardless of behavior, the majority (P ≤ 0.05) occurred during day 1–22. During the first 22 d, more LS errors were associated with the behavior of lying (P ≤ 0.05), compared to all other behaviors. Similarly, more (P ≤ 0.04) LS errors were associated with the behavior of lying than all other behaviors during the final 19 d.

Overall classification of accuracy of the NUtrack system for identification of behaviors reported in Table 6. For the 42-d nursery period, the NUtrack system correctly classified the behaviors at a rate of 98.7%, 89.7%, 88.5%, 95.4%, and 79.9% for lying, standing, walking, ATF, and ATW, respectively. Overall rate for the classification of behaviors was determined to be 96.3%. A total of 424 behavior classification errors occurred during the 42-d nursery period. Most (P ≤ 0.001) of these errors occurred during days 1–22. The greatest (P ≤ 0.001) percent of accuracy during days 1–22 was related to lying and ATF and the least (P ≤ 0.001) percent of accuracy occurred for the behavior of ATW. Standing and walking during days 1–22 were intermediate (Table 6).

Table 6.

Overall classification accuracy of the NUtrack system for determining behaviors of 28 group-housed nursery pigs during a 42-d nursery period.

	Total	Days 1–22	Days 23–42	P- value
Total Observations	10,682	5,263	5,419
Lying (n)	6,893	3,157	3,736	0.63
Standing (n)	1,279	848	431	0.03
Walking (n)	937	606	331	0.02
ATF (n)	1,414	582	832	0.18
ATW (n)	159	70	89	0.56
Total Observed Errors	424	300	124	<0.001
Lying (n)	87b	76b	11bc	<0.001
Standing (n)	132a	101a	31b	<0.001
Walking (n)	108a	104a	4c	<0.001
ATF (n)	65b	7c	58a	<0.001
ATW (n)	32c	12c	20b	0.46
P-value	<0.001	<0.001	<0.001
Overall % Correct	96.3	94.6	98.0
Lying (%)	98.7a	97.7	99.7
Standing (%)	89.7b	88.8	91.4
Walking (%)	88.5b	83.2	98.2
ATF1 (%)	95.4a	99.5	92.5
ATW2 (%)	79.9c	84.3	76.4
P-value	<0.001

Pigs were classified as at the feeder (ATF) when head of an individual pig was determined to be within the annotated location of the feeder (Figure 3a).

Pigs were classified as at the waterer (ATW) when the head of an individual pig was determined to be within the annotated location of the waterer (Figure 3a).

Denotes differences (P < 0.05) within columns between behavior classifications.

During the first 22 d, the greatest (P ≤ 0.001) number of errors for classification of behaviors were associated with walking and standing, while the fewest (P < 0.001) errors were associated with ATW (Table 6). During the final 19 d, more (P ≤ 0.001) errors occurred when classifying pigs as ATF and the least for walking. For the behaviors of lying, standing, and walking, the greatest number of errors identified were during days 1–22. However, for the behavior of ATF the greatest rate of errors occurred during the final 19 d. The errors for the behaviors of lying, standing, walking, and ATF were less (P < 0.001) during the final 19 d of the nursery period. There was no difference (P = 0.46) for the behavior of ATW (Table 6).

Breakdown of the NUtrack systems errors in classification of behaviors compared to the behaviors determined by the trained evaluators reported in Table 7. Overall, during the entire 42-d nursery period, a total of 87 errors were related to classification of lying occurred. Of these lying errors, most (P < 0.001) occurred when trained evaluators classified pigs as either ATF or ATW. As time increased in the nursery period, the number of errors related to error in classification decreased (P ≤ 0.02) for standing, and ATW, there was no difference (P ≥0.40) for walking and ATW. During the 42-d period, a total of 132 errors related to standing occurred. Of these errors, the majority (P ≥ 0.001) were a misclassification when pigs were ATF. During the first 22 d, there was a greater (P < 0.04) for lying, ATF, and ATW. There was no (P = 0.16) change over time for standing errors associated with walking. A total of 108 errors were identified for the classification of walking. The majority (P ≤ 0.001) of walking classification errors were related to the NUtrack system classifying a pig as walking while the behaviors were ATF. A total of 65 errors associated with the NUtrack system incorrectly classified the behavior of ATF. The majority (P ≤ 0.001) of these errors were the result of the NUtrack system classifying the behavior as ATF when the correct behavior was lying. Unlike the previous behaviors of lying and standing, most errors for classification of ATF occurred during the final 19 d of the nursery period, there was no difference for errors related to walking (two errors during the first 22 d and two errors in the final 19 d). Overall, a total of 32 errors were identified where the NUtrack system incorrectly classified the behavior of ATW. Of these errors, most (P < 0.03) occurred when pigs were standing. In terms of differences between the two time periods (first 22 d and last 19 d), there was no differences (P > 0.16) in the occurrence of ATW errors across the two time periods. Of note is that there were no errors of classification for ATF and ATW. This lack of any errors is due to the requirements for classification of both ATF and ATW requiring a pig to be located at the polygonal shaped zone that defined the feeder and waterer.

Table 7.

Breakdown of NUtrack errors for classification of behavior compared to the behavior determined by the trained evaluators of 28 newly weaned group-housed nursery pigs over 42-d period

NUtrack1/ Observed2	Days 1–42	Days 1–22	Days 23–42	P-value
	n	n	n
Lying Errors	87	76	11	≤ 0.001
Lying/ Standing	12b	10c	2	0.02
Lying/ Walk	1b	1c	0	0.52
Lying/ ATF3	48a	42a	6	≤ 0.001
Lying/ ATW4	26a	23b	3	0.40
P-value	≤ 0.001	≤ 0.001	≥ 0.57
Standing Errors	132	101	31	≤ 0.001
Standing/ Lying	10b	1c	9	0.04
Standing/ Walk	6b	2c	4	0.16
Standing/ ATF	86a	76a	10	≤ 0.001
Standing/ ATW	30b	22b	8	0.01
P- value	≤ 0.001	≤ 0.001	≥ 0.23
Walking Errors	108	104	4	≤ 0.001
Walk/ Lying	3c	2c	1	0.56
Walk/ Standing	5c	5c	0	0.58
Walk/ ATF	71a	68a	3	≤ 0.001
Walk/ ATW	29b	29b	0	0.006
P-value	≤ 0.001	≤ 0.001	≥ 0.49
ATF Errors	65	7	58	≤ 0.001
ATF/ Lying	49a	3	46a	≤ 0.0001
ATF/ Standing	12b	2	10b	0.004
ATF/ Walk	4b	2	2b	1.0
ATF/ ATW5	–	–	–	–
P- value	≤ 0.001	0.72	≥ 0.003
ATW Errors	32	12	20	≤ 0.001
ATW/ Lying	8b	0b	8	0.16
ATW/ Standing	22a	10a	12	0.23
ATW/ Walk	2b	2b	0	0.16
ATW/ ATF5	–	–	–	–
P- value	≤ 0.03	0.01	≥ 0.79

Classification of behavior determined by NUtrack.

Classification of behavior determined by trained evaluators.

Pigs were classified as at the feeder (ATF) when head of an individual pig was determined to be within the annotated location of the feeder (Figure 3a).

Pigs were classified as at the waterer (ATW) when the head of an individual pig was determined to be within the annotated location of the waterer (Figure 3a).

The lack of errors associated with ATF/ATW and ATW/ATF is due to classification of behavior reliant upon pig being within a specific area within the camera view.

Denotes differences (P < 0.05) within columns between behavior classifications.

Time Associated with Behaviors

Overall percent of time associated with behaviors and distance traveled (meters/d) determined by the NUtrack system are reported in Table 8. Overall, during the 42-d period nursery period, pigs spent 77.6% ± 1.6% of the time lying, 8.5% ± 1.1% of the time standing, 2.9% ± 0.4% of time walking and pigs traveled 939.3 ± 106.1 m/d. For ATF, on average pigs spent 9.9% ± 1.7% of time ATF, averaging 89.5 ± 6.7 visits/d with a mean of 95.2 ± 14.4 s/visit. Of the times pigs were determined to be ATF, 49.9% of visits were greater than 1 min and 38.5% were less than 30 . For ATW, on average, pigs spent 1.0% ± 0.3% of time ATW, with an average of 59.1 ± 10.2 visits/d and spending 13.5 ± 2.3 s/visit. Of the times pigs were determined to be ATW, 52.4% of visits ranged from 5 to 30 s and 11.6% were less than 1 s.

Table 8.

Overall percentage of time associated with the behavior of lying, standing, at the feeder and at the water, in addition the number of visits, and duration of visits to the feeder and waterer of 28 individual nursery pigs during a 42 d of the nursery phase, as classified by the NUtrack system

	Lying	Standing	Walking		ATF3			ATW6
	% Time1	% Time	% Time	m/d2	% Time	visits/d4	s/v5	% Time	visits/d	s/v
Ave.	77.7	8.5	2.9	939.9	9.9	89.5	95.2	1.0	59.1	13.5
Min	74.4	6.5	2.1	692.2	1.1	71.9	70.8	0.3	38.9	8.7
Max	80.5	10.4	3.7	1,140.6	17.4	98.9	132.0	2.2	78.5	17.9
SD	1.6	1.1	0.4	106.1	1.7	6.7	14.4	0.3	10.2	2.3
Weeks
1–7 d	72.3a	15.7a	3.7a	1,195.8a	7.3a	84.2a	74.2a	1.1	56.8a	18.8a
8–14 d	77.2b	10.2b	3.0a	990.9b	8.7b	80.7a	94.3b	0.9	56.3a	18.5a
15–21 d	78.1c	8.4c	3.6a	1,219.6a	9.0b	92.3b	84.7c	1.0	67.2b	14.4b
22–28 d	78.4c	5.8d	2.6b	819.5c	11.8c	114.3c	105.2d	1.1	77.2c	12.9c
29–35 d	80.0c	5.3d	2.3c	722.2d	11.8c	97.2b	105.3d	0.9	61.6b	13.8b,c
36–42 d	80.4c	5.6d	2.1c	691.5d	11.3c	95.2b	99.6b	0.8	54.2a	15.8d
P-value	≤ 0.001	≤ 0.001	≤ 0.001	≤ 0.001	≤ 0.001	≤ 0.007	≤ 0.01	0.11	≤ 0.04	≤ 0.002
SEM	0.56	0.32	0.09	141.0	0.48	2.72	3.84	0.06	2.84	1.19
Gender
Barrows (n = 12)	77.2	9.0	2.9	954.5	9.8	92.0	94.4	1.0a	63.2	15.0
Gilts (n = 16)	77.8	8.3	2.9	934.5	10.0	96.0	93.5	0.9b	61.5	16.2
P-value	0.52	0.13	0.83	0.98	0.53	0.35	0.88	0.04	0.74	0.63
SEM	0.47	0.32	0.11	223.2	0.49	2.94	4.26	0.06	3.74	1.78

Percent of time (% time) associated with lying, standing, walking, at the feeder and at the waterer as determined by the NUtrack system during the 42-d nursery period.

Meters traveled per day (m/d) as determined by the NUtrack system.

Pigs were classified as at the feeder (ATF) when head of an individual pig was determined to be within the annotated location of the feeder (Figure 3a).

Number of visits per day (visits/d) to the feeder or waterer determined by the NUtrack system.

Mean duration in seconds/visit (s/v) to the feeder or waterer determined by the NUtrack system.

Pigs were classified as at the waterer (ATW) when the head of an individual pig was determined to be within the annotated location of the waterer (Figure 3a).

Denote differences (P < 0.05) within columns between variable of interest.

Changes in percent time associated with behaviors across the 6 wk of the nursery period are reported in Table 8. Aside from the time ATW, time associated with lying, standing, walking, and ATF changed (P ≤ 0.001) over time. For time spent lying, the percent time lying increased (P ≤ 0.001) as days in the nursery increased. Pigs spent the least (P ≤ 0.001) amount of time lying (72.3%) during the first week of the nursery period, when compared to all remaining weeks. From week 3 through week 6, the percent of time lying was similar (P = 0.36). An inverse relation to time lying and time standing transpired. Pigs spent the greatest (P ≤ 0.001) amount of time standing during the first week, when compared to week two through six. Percent of time walking was greatest (P ≤ 0.001) during the first 3 wk, when compared to the last 3 wk. While the percent of time associated with walking was similar (P = 0.44) for the first 3 wk, the distance traveled during the 3 wk was variable. The average distance walked during the first (1,195.8 m/d) and third week (1,219.6 m/d) was similar (P = 0.69), but greater (P ≤ 0.001) than the distance traveled during the second week (990.9 m/d) and the last 3 wk (22–42 d). The distance traveled was least (P ≤ 0.001) during the last 2 wk (29 – 42 d) of the nursery period.

Changes in percent time, number of visits/d, and duration of visits (s/visit) to the feeder and waterer are reported in Table 8. Pigs spent the least (P ≤ 0.001) amount of time ATF during the first week (1–7 d) and the greatest (P ≤ 0.001) amount of time during the final 3 wk (22–42 d); week 2 through 3 were intermediate. On average, pigs visited the feeder 89.5 visits/d and the duration of each visit was 95.2 s/visit. The fewest visits to the feeder occurred during the first two weeks; 84.2 and 80.7 visits/d, respectively. The greatest (P ≤ 0.001) number of visits/d to the feeder (114.3 visits/d) occurred during the fourth week (22–28 d), when compared to all other weeks during the nursery period.

As expected, there was an increase in duration of time ATF (s/visit) as the days in the nursery period increased. The durations of visits (s/visit) were the shortest (P ≤ 0.001) during the first week (74.2 s/visit) and greatest during weeks 4 and 5 (105. 2 and 105.5 s/visit, respectively), when compared to weeks 2, 3, and 6. In terms of percent of time ATW, there was no difference (P ≥ 0.11) in percent time across the 6-wk nursery period. The number of visits/d at the water fluctuated across the six weeks of the nursery period. The number of visits/d was least for weeks 1, 2, and 6 (P ≥ 0.001), and greatest (P ≥ 0.04) during the fourth week (77.2 visits/d). While weeks one and two had the fewest visits/d, the duration (s/visits) ATW was greatest for weeks 1 and 2 (18.8 and 18.5 s/visits, respectively), when compared to weeks 3 through 6.

The average time associated with lying, standing, walking, distance walked, percent time, visits/d, and s/visit ATF and ATW compared sex as determined by the NUtrack system reported in Table 8. There was no difference (P ≥ 0.11) between sex associated with all general behaviors (lying, standing, walking, and distance walked). The percent of time ATW was greater (P = 0.04) for barrows (1.0%) when compared to gilts (0.9%).

DISCUSSION

The results of this study indicate that the NUtrack system is capable of automatically detecting individual pigs within a group-housed environment. This ability of detection is in line with that reported by Zhu et al. (2015). Using a Kinect v2 camera and 3D point cloud data capture, Zhu et al. (2015) reported the ability to automatically detect pigs in a group-housed environment. Zhu et al. (2015) reported an overall accuracy of detection of 97.5%. In the current trial, we were able to achieve an accuracy of 98.8% for detection (based upon only lost label identification errors). One key difference between the current study and Zhu et al. (2015) is the size of pigs used. Zhu et al. (2015) utilized pigs within two weight groups, 25 kg and 60 kg. In the current study newly weaned nursery pigs were utilized with an average weight of 6.6 kg. Pigs entered the nursery at 6.6 kg and were tracked across the entire 42-d nursery period. In both the current study and Zhu et al., (2015) accuracy for detection increased as the weight of the pigs increased. In addition to the capabilities of accurately detecting individual pigs, results also suggest that the NUtrack system can determine the individual ID of each pig and retaining the individual ID. Of the 10,655 individual observations, 95.6% of the observations for identification by the NUtrack system were in line with the visual ID (ear tags) determined by trained evaluators. Overall, the reported results indicate that the NUtrack system can detect individual pigs and accurately determine individual ID of nursery pigs within a group-housed environment. In addition, searching of previous research indicates that this is the first report of a form of PLT with the capability to identify individuals and maintain the identity of the individual.

To the best of our knowledge, this is the first report detailing the capabilities of a computer vision platform that can autonomously classify multiple behaviors (lying, standing, walking, ATF, and ATW) of individual pigs within a group-housed environment. Based upon the results of this study, the NUtrack systems is highly capable of accurately classifying the behaviors of lying, standing, walking, and ATF and ATW behaviors of group-housed nursery pigs. The accuracy achieved for the behavior of lying of the current study is comparable to the 95.8% reported by Nasirahmadi et al. (2015). Regarding pigs walking, results of the current study (88.5%) were slightly less than the 98.9% reported by Kashiha et al. (2014). In terms of classification of ATW, results of the current study (79.5%) were less than the 90.7% reported by Zhu et al. (2017). This difference in accuracy is most likely due to the selection of frames for analysis of drinking time. Zhu et al. (2017) utilized 140 3-min segmented video clips selected from 35 h of video collected over a period of 5 d. These 140 segments Zhu et al. (2017) selected were based upon containing images with an individual pig drinking (based upon HO). For the current study, there was no selection of frames based upon containing a specific behavior and accuracy for the classification of ATW was based upon 800 frames across the entire nursery period with 10,655 individual pig observations.

For the duration of a 42-d nursery period (1,008 h), the NUtrack system was able to collect data related to the amount of time nursery pigs budgeted towards the behaviors of lying, standing, walking, ATF, and ATW. Results of the current study are in line with data reported by Wang et al. (2012). The time nursery pigs spent in various behaviors of the current study are similar to those reported by Wang et al. (2012). Wang et al. (2012) reported 76.8% of time lying (77.7% in current study), 12.6% of time standing (8.5% in current study), 7.6% of time eating (9.9% in current study), and 0.7% of time drinking (1.0% in current study). While overall time budgeted is similar, there are two primary differences between Wang et al. (2012) and the current study. Wang et al. (2012) utilized HO of video collected over a period of 96 h. This 96-h segment was from the first 72 h and last 24 h in the nursery and only accounted for 2.3% of the pig’s time in the nursery. For the current study, data generated to determine time budgeted was comprised of 1,008 h of video (the entire 42-d nursery period). Using the same time as that of Wang et al. (2012), the NUtrack data indicated that pigs spent 70.0%, 18.1%, 5.8%, and 1.3% of time of the time lying, standing, ATF, and ATW, respectively.

The ability of the NUtrack system to continuously monitor pigs’ behavior also provides the capabilities to determine and evaluate changes in time budgets to individual behaviors over time. In the current study, changes in the time pigs allocated to lying, standing, walking, ATF, and ATW were based upon the entire 6-wk nursery phase. These changes in time budgeted indicated an increase in time lying and ATF while a decrease in time standing and walking. The changing of time ATF during the nursery period agrees with Gonyou et al. (1998) in that there was a change over time. However, the direction of this change does not align. Gonyou et al. (1998) indicated that as time in the nursery period increased, time allocated to eating decreased from 13.2% during the first week compared to 8.7% during the sixth week. For the current study, during the first week, pigs spent 7.3% of time ATF and increased to 11.3% during the sixth week. This difference between Gonyou et al. (1998) and the current study is most likely due to the time frames used for determining time ATF. Gonyou et al. (1998) collected behavior data between 0900–1100 and 1500–1700 hours. Data from the current study would indicate that 77.2% of the time pigs spent ATF occurred outside of these two time frames. A similar difference in time ATW was reported by Davis et al. (2006). The percentage of time pigs spent ATW in the present trial (0.95%) is less than the percentage of time (2.5%) reported by (Davis et al. 2006). This difference may be the result of Davis et al. (2006) utilizing two 1-h time periods (0900–1000 and 1400–1500 hours) on days 0, 7, 14, 27, 35, 38, 44, and 65 of the study. Using this methodology, Davis et al. (2006) only evaluates 1% of the total time in the nursery period. In addition, data from the current study indicated that only 13% of the time pigs spent ATW was during the specific time periods evaluated by Davis et al. (2006). When evaluating changes in behavior, most studies evaluating time budgeted to general behaviors have only been able to evaluate a snapshot in time of specific behaviors. To determine time alloted to behaviors, Ott et al. (2014) evaluated 41% of the total study time, He et al. (2018) evaluated 14% of the total study time, Gonyou et al. (1998) evaluated 7.1% of the total study time, and Davis et al. (2006) evaluated 1.2% of the total study time. Data from the current study evaluated 100% of the total study time (1,008 h evaluated) which is a significant step forward in understanding the changes in behavior of nursery pigs.

CONCLUSIONS

Results of this study suggest that the NUtrack system can accurately detect group-housed pigs, identify, maintain identification, and continuously track individual nursery pigs within a group-housed environment. The NUtrack system is also capable of accurately classify and continuously tracking the behaviors of nursery pigs lying, standing, walking, at the feeder, and at the waterer for an indefinite period. The NUtrack system is also capable of determining the distance traveled per day, the number of times pigs visit feeder/waterer, and the duration of time spent at the feeder and waterer. To our knowledge, this is the first reporting of a computer vision platform that can accurately detect individual pigs, maintain ID, and track the behaviors of nursery pigs. To our knowledge, this study also represents the longest duration of continuously monitoring the behaviors of nursery pig. Data generated from the continuous monitoring of the behaviors can potentially provide the ability to identify changes in behaviors that may be indicative of alterations to homeostasis of nursery pigs. While these results are encouraging, future research is still needed. A major aspect of needed research is to identify a cost efficient 2D camera system that provides a larger field of view and is capable of surviving the challenging environment of a swine facility. Current research by our team is focused utilizing on Power-Over-Ethernet security camera systems to fulfill this need. Future research also includes means by which to enhance the capabilities of the NUtrack system to classify additional behaviors such as tail-biting that can impact the health and welfare of pigs. Future research is still needed, but the status of the current system may provide a tool to researchers to evaluate the impact of a wide array of factors on the behavior of nursery pigs. In addition, with continued research, the NUtrack system may have the potential to serve as a viable tool for assisting human observers (not replacing HO) in identifying behavioral changes that may indicate potential health and welfare issues.