Are Baseball Statistics an Appropriate Tool for Assessing Return to Play in Injured Pitchers? Analysis of Statistical Variability in Healthy Players.

BACKGROUND: Basic pitcher statistics have been used to assess performance in pitchers after injury or surgery without being validated. Even among healthy pitchers, the normal variability of these parameters has not yet been established.
PURPOSE: To determine (1) the normal variability of basic and advanced pitcher statistics in healthy professional baseball pitchers and (2) the minimum pitches needed to predict these parameters. STUDY
DESIGN: Cross-sectional study; Level of evidence, 3.
METHODS: Publicly available data from the MLB Statcast and PITCHf/x databases were used to analyze MLB pitchers during the 2015 and 2016 seasons who recorded a minimum of 100 innings without injury. Basic and advanced baseball pitcher statistics were analyzed. The variability of each parameter was assessed by computing the coefficient of variation (CV) between individual pitchers and across all pitchers. A CV <10 was indicative of a relatively constant parameter, and parameters with a CV >10 were generally considered inconsistent and unreliable. The minimum number of pitches needed to be followed for each variable was also analyzed.
RESULTS: A total of 118 pitchers, 55 baseball-specific statistical metrics (38 basic and 17 advanced), and 7.5 million pitches were included and analyzed. Of the 38 basic pitcher statistics, only fastball velocity demonstrated a CV <10 (CV = 1.5), while 6 of 17 (35%) advanced metrics demonstrated acceptable consistency (CV <10). Release position from plate and velocity from the plate were the 2 most consistent advanced parameters. When separated by pitch type, these 2 parameters were the most constant (lowest CV) across every pitch type.
CONCLUSION: We recommend against utilizing nonvalidated statistical measures to assess performance after injury, as they demonstrated unacceptably high variability even among healthy, noninjured professional baseball pitchers. It is our hope that this study will serve as the foundation for the identification and implementation of validated pitcher-dependent statistical measures that can be used to assess return-to-play performance after injury in the future.

A recent study has demonstrated high injury rates in Major League Baseball (MLB) and Minor League Baseball players, with approximately 50,000 injuries identified between 2011 and 2016 using the MLB Health and Injury Tracking System. Of these injuries, 39% were sustained in pitchers, resulting in 722,176 days missed during 6 seasons—a significant burden for the players, teams, fans, and professional baseball. Even with modern injury prevention strategies, rehabilitation protocols, and advanced surgical techniques, the annual number of injuries in professional baseball players has not decreased between 2011 and 2016.

Historically, treatment efficacy of professional baseball injuries has hinged on return-to-play (RTP) criteria using patient-reported functional outcome scores and variable RTP protocols. Outcome scores such as the American Shoulder and Elbow Surgeons score are of limited value for high-level athletes owing to the ceiling effect. Because these basic patient-reported outcome measures utilized in orthopaedic surgery may not be suitable for professional athletes, basic performance-based metrics are increasingly being used (eg, earned run average [ERA], strikeout percentage, walks + hits per inning pitched [WHIP]). However, these basic metrics are not entirely pitcher dependent. All these variables are largely influenced by myriad factors outside the pitcher’s control, such as skill of the batter, umpire, quality of the defense in the field, size of the ball park, pitching role (starter vs reliever), league of play, weather, and timing of game (day vs night), to name a few. For this reason, current studies have turned to advanced metrics that are more pitcher dependent (eg, release position, spin rate, spin axis, horizontal/vertical movement); however, these have not yet been validated.

In a recent study, van der List et al reviewed outcomes reporting in professional baseball using an analysis of 54 studies. The authors noted great variability in the definition and duration of follow-up (games vs seasons, months) and performance parameters utilized, and no study using baseball-specific performance measures “validated these performance-based statistics, determined minimal time of follow-up needed, or assessed the baseline variability in these statistics among non-injured players.”

Without high-quality validated data-measuring tools, it is difficult to determine whether players, in this case pitchers, are ready to RTP. Development of new pitcher-dependent statistics or validation of currently utilized statistics may help in dictating whether pitchers are returning to their preinjury or preoperative levels of performance. Before utilizing existing parameters to assess the performance of injured players, we must first understand the normal variability of these statistics in noninjured athletes. Similarly, the minimum follow-up (ie, number of pitches, innings, and games) for these parameters that is required to accurately determine their predictive abilities has not been determined.

To address this critical void, the purposes of this study were to determine (1) the normal variability of basic and advanced pitcher statistics in healthy professional baseball pitchers and (2) the minimum pitches needed to predict these parameters. It was our hypothesis that the basic metrics would demonstrate unacceptably high levels of variability while the advanced parameters would reflect more acceptable variability and may be more suitable for use as performance surrogates.

Methods

Publicly available data from the online databases MLB Statcast (http://baseballsavant.mlb.com/statcast_search) and the PITCHf/x (http://www.brooksbaseball.net/pfxVB/pfx.phpandhttp://library.fangraphs.com/misc/pitch-fx) were used to analyze the performance of MLB pitchers during the 2015 and 2016 seasons who pitched >100 innings without spending any time on the disabled list. PITCHf/x is a tracking system created by Sportvision that has been utilized in every MLB stadium since 2006 to track advanced parameters, including but not limited to pitch velocity, movement, release point, spin, and location. The MLB Statcast system was built on the previous PITCHf/x system by utilizing Doppler radars and higher-definition cameras, allowing additional precision and collection of baseball movement at 20,000 frames per second. This system also allowed player tracking of variables such as running speed, distance, and direction.

The PITCHf/x and MLB Statcast systems log a variety of denoted “basic” and “advanced” pitcher statistics. A total of 38 basic statistics and 17 advanced statistics were available and analyzed (Table 1). Advanced statistics are typically defined as statistics used at the advent of sabermetrics to objectively identify and define baseball player characteristics based on in-game data collection.

Table 1

Basic and Advanced Pitcher Statistics and Their Definitions

Parameter	Definition	Parameter	Definition
Basic statistics		Basic statistics (continued)
FBv	Fastball velocity	HR	Home run
FB%	Fastball %	HLD	No. of holds
LOB%	Left on base %	HR/9	Home runs per 9 innings
TBF	Total batters faced	W	Win
Pitches	No. of pitches	L	Loss
Strikes	No. of strikes	HBP	Hit by pitch
Balls	No. of balls	BS	Blown save
K/9	Strikeouts per 9 innings	CG	Complete game
xFIP-	Expected fielding-independent pitching (adjusted for park/league)	SV	No. of saves
xFIP	Expected pitcher’s ERA (adjusted for league)	Advanced statistics
SIERA	Estimate of pitcher’s ERA (adjusted for strikeouts, walks, home runs, batted balls)	release_pos_y	Release position from catcher’s perspective
K%	Strikeout %	vy0	Velocity in y-dimension
K/BB	Strikeouts to walks ratio	release_speed	Release speed
BAA	Batting average against	effective_speed	Speed based on pitcher’s extension
K	Strikeout	release_pos_z	Vertical release position
BABIP	Batting average on balls in play	release_extension	Release extension of pitch based on pitcher
H	Hit	ay	Acceleration in y-dimension
WHIP	Walks plus hits per inning pitched	release_spin_rate	Spin rate
FIP-	Fielding-independent pitching (adjusted for park/league)	az	Vertical acceleration
FIP	Fielding-independent pitching	release_pos_x	Horizontal release position
BB%	Walk %	plate_z	Vertical position of ball at plate
GS	Games started	pfx_x	Horizontal movement from catcher’s perspective
K-BB%	Strikeouts to walks %	pfx_z	Vertical movement from catcher’s perspective
BB	Walk	ax	Acceleration in horizontal dimension
BB/9	Walks per 9 innings	vx0	Horizontal velocity
R	Runs allowed	vz0	Vertical velocity
ER	Earned run	plate_x	Horizontal position of ball at plate
ERA-	Earned run average (adjusted for park/league)
ERA	Earned run average

A search algorithm from publicly available repositories was utilized to obtain all the aforementioned data on noninjured pitchers (those who did not spend any time on the disabled list) who pitched a minimum of 100 innings between 2015 and 2016. Basic and advanced baseball pitcher statistics were analyzed as follows.

Statistical Analysis

The variability of each parameter in this study was assessed by computing the coefficient of variation (CV) between individual pitchers and averaging across all pitchers. A CV is a standardized measure of dispersion distribution and is often used in fields such as engineering and biomedical research for determining the volatility of variables.

In the current study, a CV <10 (representing 10%) was considered an indication of a relatively constant parameter, and parameters with a CV >10 were considered highly variable and unreliable for prediction. Median CV and interquartile range are reported. This methodology was used to not only calculate the CV for all the parameters assessed but also determine the minimum number of pitchers needed per variable.

Calculation of Minimum Number of Pitches

The minimum number of pitches needed for adequate predictive ability was estimated as the number of repeats (pitches) necessary to detect values within 5% of the difference from the mean of all values for a selected parameter. For these calculations, the probability of a type I error (alpha) and a type II error (beta) was assumed to be 0.05 (5%) and 0.2 (20%), respectively. Values within 1% and 10% of the difference from the mean were also explored with the caveat that one would require a smaller sample size to predict a variable with a larger allowable error. The minimum number of pitches was calculated using the formula in Figure 1. Statistical software R (Version 3.5.1; R Foundation) was utilized for data analysis.

Figure 1.

Formula for calculating the minimum number of pitches for adequate predictive ability, estimated as number of repeats (pitches) that is necessary to detect values within 5% of the difference from the mean of all values for a selected parameter. The probability of a type I error (alpha) and a type II error (beta) was assumed to be 0.05 (5%) and 0.1 (10%), respectively.

Results

The study resulted in the inclusion of 118 noninjured pitchers. A total of 55 parameters (38 basic and 17 advanced) were included, and out of 7.5 million available pitches, approximately 380,000 were analyzed.

Parameter Variability

Of the 38 examined basic parameters, only 1 (3%) demonstrated an acceptable CV value (fastball velocity, CV = 1.5). All other basic parameters had CV values significantly >10 (Table 2), and 22 (58%) had CV values >100.

Table 2

CVs for Basic Statistics in Noninjured Major League Baseball Pitchers

Rank b	Parameter	CV c	IQR Range Magnitude	Minimum No. of Pitches
Statistics with acceptable variability (CV <10)
1	Fastball velocity	1.5	0.3	1.0
Statistics with unacceptable variability (CV >10)
2	Fastball %	23.1	11.8	1152.0
3	Left on base %	32.0	8.2	875.0
4	Total batters faced	54.5	26.0	210.9
5	Pitches	56.4	32.1	NA
6	Strikes	56.5	30.2	88.3
7	Balls	67.4	40.6	200.5
8	Strikeouts/9 innings	77.9	49.1	212.9
9	Expected fielding-independent pitching (adjusted for park/league)	81.3	53.9	21.9
10	Expected pitcher’s ERA (adjusted for league)	81.4	54.2	478.4
11	Estimate of pitcher’s ERA (adjusted for K, walks, HR, batted balls)	83.4	66.0	541.7
12	Strikeout %	83.5	53.4	7999.6
13	Strikeouts to walks ratio	88.8	28.6	1764.1
14	Batting average against	90.3	67.1	10,002.3
15	Strikeout	94.2	51.9	1708.2
16	Batting average on balls in play	95.2	77.8	8749.0
17	Hits	106.6	78.7	2292.1
18	Walks plus hits/inning pitched	109.0	64.9	1803.2
19	Fielding-independent pitching (adjusted for park/league)	133.2	90.3	32.4
20	Fielding-independent pitching	133.3	89.9	777.9
21	Walk %	150.2	107.0	51,219.0
22	Games started	151.0	242.6	149,029.0
23	Strikeouts-walks %	153.8	90.8	24,195.6
24	Walks	154.3	96.7	8903.6
25	Walks/9 innings	172.3	117.1	1249.0
26	Runs allowed	177.6	133.4	8859.1
27	Earned runs	183.9	135.2	10,129.1
28	ERA (adjusted for league/park)	220.0	122.3	57.8
29	ERA	221.9	108.8	1243.5
30	Home runs	279.0	216.2	74,061.1
31	No. of holds	285.5	132.7	198,317.5
32	Home runs/9 innings	306.8	236.4	8782.2
33	Wins	353.9	328.8	167,342.6
34	Losses	452.1	182.0	474,285.6
35	Hit by pitch	476.7	361.1	410,514.9
36	Blown saves	528.0	218.2	458,544.0
37	Complete games	553.4	419.8	563,391.2
38	No. of saves	567.2	633.6	908,022.5

CV, coefficient of variation; ERA, earned run average; HR, home run; IQR, interquartile range; K, strikeouts; NA, not applicable.

Least to most variable.

Ascending order.

Of the 17 advanced parameters, 6 (35%) had acceptable CV values (Table 3): release position from catcher’s perspective, velocity in the y-dimension, release speed, speed based on pitcher’s extension, vertical release position, and release extension of pitch based on pitcher. Of these, the lowest CV belonged to release position from catcher’s perspective (CV = 0.5). Of these 6 parameters, 3 were related to release position, and 3 were related to speed/velocity. Two additional parameters had values slightly >10 for CV: acceleration in the y-dimension (CV = 10.9) and spin rate (CV = 11.0).

Table 3

CVs for Advanced Statistics in Noninjured Major League Baseball Pitchers

Rank b	Parameter	CV c	IQR Range Magnitude	Minimum No. of Pitches
Statistics with acceptable variability (CV <10)
1	Release position from catcher’s perspective	0.5	0.1	1.0
2	Velocity in y-dimension	2.6	1.7	1.5
3	Release speed	2.6	1.7	2.0
4	Speed based on pitcher’s extension	2.9	1.8	2.1
5	Vertical release position	3.3	1.3	16.1
6	Release extension of pitch based on pitcher	4.5	1.2	20.4
Statistics with unacceptable variability (CV >10)
7	Acceleration in y-dimension	10.9	3.0	12.5
8	Spin rate	11.0	8.8	1.0
9	Vertical acceleration	14.7	9.2	37.8
10	Horizontal release position	23.0	13.6	417.2
11	Vertical position of ball at plate	38.8	19.4	422.3
12	Horizontal movement from catcher’s perspective	107.3	46.2	19,107.0
13	Vertical movement from catcher’s perspective	108.6	35.0	2492.0
14	Acceleration in horizontal dimension	166.1	45.1	62,089.8
15	Horizontal velocity	177.5	42.2	375.2
16	Vertical velocity	253.6	59.9	1.5
17	Horizontal position of ball at plate	1202.9	477.3	10,279,886.2

CV, coefficient of variation; IQR, interquartile range.

Least to most variable.

Ascending order.

Minimum Number of Pitches

In the basic parameters, the minimum number of pitches needed to predict fastball velocity was 1.0. In the advanced parameters, the minimum number of pitches needed to predict parameters with a CV <10 was <30 for all parameters (Tables 1 and 2). Overall, the variable “release_extension” (release extension of pitch based on pitcher) demonstrated the largest number of pitches (minimum, 20.4) that needed to be followed to obtain adequate predictive ability. Some parameters with acceptable CVs (ie, CV <10) still had a large number of pitches necessary for prediction owing to the small sample size relative to the whole cohort.

Pitch Type

The MLB Statcast system allows for subanalysis via pitch type, and in this study, 11 distinct pitch types were available in the included pitches: changeup, cutter, 4-seam fastball, 2-seam fastball, slider, knuckle-curve, curveball, sinker, splitter, knuckleball, and screwball. In this subanalysis by pitch type, 5 variables had a CV <10: speed based on pitcher’s extension, release extension of pitch based on pitcher, release position from catcher’s perspective, vertical release position, and velocity in the y-dimension. Three other parameters—acceleration in the y-dimension, acceleration in the vertical dimension, and spin rate—had CV values that were close to or <10 for some pitch types while being >10 for others (Table 4).

Table 4

Heat Map Revealing CVs of Advanced Parameters by Pitch Type

All blue shading indicates CVs <10, a relatively constant parameter; red shading indicates increasing variation. In this heat map, the following parameters had acceptable CVs across all pitch types assessed: speed based on pitcher’s extension, release extension of pitch based on pitcher, release position from catcher’s perspective, vertical release position, and velocity in the y-dimension. CH, changeup; CU, curveball; CV, coefficient of variation; FC, cutter; FF, 4-seam fastball; FS, splitter; FT, 2-seam fastball; KC, knuckle-curve; KN, knuckleball; SC, screwball; SI, sinker; SL, slider.

As a demonstration of pitcher-specific parameters, the selected basic and advanced pitcher statistics for pitcher 8 are shown in Figure 2.

Figure 2.

(A) Conventional and (B) advanced statistics of pitcher 8, with smoothing splines to represent data distribution and moving averages. ERA, earned run average; FBv, fastball velocity; K%, strikeout percentage; release_pos_z, vertical release position.

Discussion

Previous studies assessing RTP in professional baseball players, specifically pitchers, have utilized nonvalidated outcomes with variable follow-up. Since athletes would benefit from validated RTP measures, the goals of this study were to report normal variability of basic and advanced parameters in noninjured MLB pitchers and determine the minimum number of pitches needed to predict these validated measures accurately. In the current study, only 1 basic parameter had acceptable variability (fastball velocity), while 6 of the 35 advanced parameters had acceptable reliability. Last, this study revealed that all 7 of these parameters required <50 pitches for prediction of parameters with an error ≤5%.

In a 2017 systematic review on outcomes after shoulder and elbow injury in baseball players, Makhni et al determined that most studies inadequately reported on RTP with regard to time to RTP, and there was significant variability in subjective and patient-reported outcome scores that may “undermine the ability of clinicians” to assess results in professional athletes. van der List et al reviewed outcomes reporting in professional baseball and called for increasing “validation and consistency,” as none of the 54 studies in the review attempted to validate these performance-based parameters or determined the minimum follow-up needed. In the studies in that work, the most common statistical outcome measures reported were ERA, WHIP, strikeouts per 9 innings, walks per 9 innings, games, pitch velocity, total pitches, fastball percentage, win percentage, strikeouts to walks ratio, and losses. Based on the current study, fastball velocity is the only metric with acceptable baseline variability, and it was reported in just 5 of 54 studies, with none able to indicate how many pitches were followed. In 2014 and 2016, multiple authors utilized fastball velocity as an analysis tool in MLB pitchers undergoing medial ulnar collateral ligament (UCL) reconstructions. Jiang and Leland, in the former study, stated that there was no difference in fastball velocity between pitchers who returned to play and those who did not, though the study had a small sample size (n = 38). In the latter study, Chalmers et al found that pitchers with higher velocity were predisposed to medial UCL reconstructions, but the authors did not assess whether return to previous velocity was predictive of RTP, though these data would not be available to pitchers who did not return to sport. In addition, this study did not specify the number of pitches included for analysis.

Of the 17 advanced pitcher parameters in the current study, 6 (35%) had acceptable variability. Specifically, these 6 parameters could be divided into 3 relating to release position and 3 to velocity/speed. This indicates that advanced parameters may be more consistent predictors of performance in pitchers. Only 1 study assessed release position in pitchers and noted that pitchers may have a more lateral (horizontal) release position when compared with controls who did not require UCL reconstruction; however, this study did not assess release position after UCL reconstruction and whether it returned to baseline or was a reliable predictor of RTP. In the current study, horizontal release position had a CV >10 (13.6), which suggested that it may be relatively unreliable. No other studies in the literature assessed any of the other advanced parameters found to be of acceptable variability in this study.

It is important to note the statistical tools and future implications of this study. The CV is one of many ways to assess the variability of parameters. Still, the minimum number of pitches needed to predict each parameter with an acceptable CV is done using investigator-chosen alpha and beta values. In this study, we have presented data using an alpha of 5% since that is the most commonly accepted error rate for most statistical analysis; however, plots for an alpha of 10% are provided in Appendix Figure A1. As further steps are taken to validate these statistical measures in injured athletes, we may discover that some of these variables are so consistent that they do not change even in the setting of an injury, which would make them inappropriate tools for assessing RTP. Additionally, there may be some variables where the 5% error rate is acceptable, while a larger or smaller error rate is more appropriate for others. It is also critical to keep in mind that as variability increases, the number of pitches needed to follow many of these metrics is just too large to be practical. Last, in future work, we may find that certain parameters are significantly affected by certain types of injuries but not by others. For example, it may be that vertical release position is significantly affected by an injury to the UCL of a pitcher’s arm but it is minimally affected if the pitcher sustains a hip injury.

Figure A1.

Minimum number of pitches necessary to predict (A) basic and (B) advanced parameters with the probability of a type I error (alpha) set at 10% Dark line, median; gray box, interquartile range; whiskers, minimum and maximum values. See Table 1 for definitions of parameters. LOB% and FB% are denoted as LOB. and FB.

In this manner, future work is needed to determine (1) which parameters in the current study are actually affected by injury (as opposed to those that are so stable that they do not even change in the injured state), (2) which metrics are most appropriate for each injury (ie, what measures best reflect an elbow injury vs a hip injury), and (3) what is the minimum clinically important difference in each parameter so that it is suggestive of actual change in performance. With the foundation laid in this study, it is our hope that future research will answer these next questions and create a new standard for RTP assessment for each injury. Doing so would set the framework for this methodology to be extended to positional players, batters, lower levels of play, and other sports.

Strengths and Limitations

This study is not without limitations. Each pitcher has different morphological parameters (height, arm length, etc), which may allow for variation in data. Similarly, pitchers tend to pitch certain pitches more frequently than others; therefore, some of the variability is limited by subtype evaluation where rarer pitches may have increased variability owing to rare occurrence in a game and lower power in data analysis. Additionally, these variables have not been assessed or validated for pitchers at other levels (college or high school athletes). While the sample of pitchers in this study was confirmed to be noninjured during the study period, additional exploration is required to evaluate these parameters in injured players. Last, this data analysis is limited to a 2-season period, and though this assesses relatively modern data, it may not account for certain trends in pitching over a longer period.

It is our goal that this work will serve as a benchmark in establishing new methodology to evaluate high-level athletes with injuries in addition to guiding RTP after injuries. Additionally, similar methodology has the potential to be utilized for other sports and injuries to develop validated performance metrics that are more suitable for high-level athletes, for whom basic orthopaedic outcome measures intended for the general population are not appropriate.

Conclusion

In total, only 1 of 38 basic baseball metrics and 6 of 17 advanced metrics had acceptable consistency and reliability, and all required <50 pitches for accurate prediction. Accordingly, we recommend against utilizing nonvalidated statistical measures (eg, ERA, WHIP) to assess performance after injury since they demonstrated unacceptably high variability even among healthy, noninjured professional baseball pitchers. It is our hope that this work will serve as the foundation for the identification and implementation of validated pitcher-dependent statistical measures that can be used to assess RTP performance after injury in the future.