Effectiveness of pre-admission data and letters of recommendation to predict students who will need professional behavior intervention during clinical rotations in the United States.

The study aimed at finding the value of letters of recommendation in predicting professional behavior problems in the clinical portion of a Doctor of Physical Therapy program learning cohorts from 2009-2014 in the United States. De-identified records of 137 Doctor of Physical Therapy graduates were examined by the descriptive statistics and comparison analysis. Thirty letters of recommendation were investigated based on grounded theory from 10 student applications with 5 randomly selected students of interest and 5 non-students of interest. Critical thinking, organizational skills, and judgement were statistically significant and quantitative differentiating characteristics. Qualitatively, significant characteristics of the student of interest included effective communication and cultural competency. Meanwhile, those of nonstudents of interest included conflicting personality descriptor, commitment to learning, balance, teamwork skills, potential future success, compatible learning skills, effective leadership skills, and emotional intelligence. Emerged significant characteristics did not consistently match common non-professional behavior issues encountered in clinic. Pre-admission data and letters of recommendation appear of limited value in predicting professional behavior performance in clinic.

Almost 17,000 individuals were seeking admission to Doctor of Physical Therapy (DPT) programs in 2014-2015. Over 95,000 applications were processed by the Physical Therapy Centralized Application System (PTCAS) [1, 2], and ultimately, by DPT admission committees. These applications are being processed in a variety of ways, often involving lengthy and time consuming procedures to procure what they believe to be a cohort of students that will have success in both the academic and clinical portions of the program [3,4,5]. As with other healthcare admissions processes, there appears to be minimal evidence available that provides information related to the predictive value of much of the data/information that is requested of DPT applicants and which is subsequently processed by DPT programs during an admission cycle [6,7,8]. The purpose of this study was to examine the value of letters of recommendation (LORs) in predicting a professional behavior problem in the professional clinical portion of a DPT program. Since the LORs from PTCAS include insights regarding professional behaviors, we hypothesized that the ratings and qualitative language in LORs would predict students who required major interventions in the clinic.

In this mixed methods study, de-identified records of 137 DPT graduates of the University of Cincinnati were examined in this retrospective cohort study to determine if predictive quantitative and qualitative indicators of clinical professional behavior problems could be identified. Records of graduates of the classes of 2009-2014 revealed that 21 graduates had at least one major clinical intervention or two minor clinical interventions requiring an onsite visits and/or more than two in-depth phone communications and follow-up involving the Director of Clinical Education (DCE). These students were defined as cases (students of interest, SIs). In each of these 21 cases, all SIs had professional behavior issues and some additionally had clinical skills deficits. Considering Fig. 1, there is an illustration of the frequency of the professional behavior issues of these students in clinic. The remaining 116 student records served as controls (non-students of interest, NSIs).

Fig. 1.

Frequency of professional behavior issues demonstrated during clinical internships of 21 students of interest who were admitted to the Doctor of Physical Therapy program in 2009-2014 at the University of Cincinnati, OH, USA.

With respect to the quantitative aspect of the study, the background demographic status (age, gender, marital status, and responsibility for dependent children in the home), academic factors (undergraduate major, overall Grade Point Average (GPA), required pre-requisite science GPA, anatomy, physics, chemistry, and biology GPA’s, and Graduate Record Examination scores), pre-admission LOR’s and number of observation hours were studied. Additionally, rankings of professional behavior characteristics from LOR’s and the professional identification of the recommender (Physical Therapist (PT) or Non-PT) were collected. Once collected, investigation of the data was done by descriptive statistics to ascertain if demographic and academic differences existed. Wilcoxon Rank Sum Test was used to know if there were statistical differences with LORs between SIs and NSIs. All analyses were performed using R version 3.2.3 (2015-12-10) (R Foundation for Statistical Computing Platform). Statistical significance was determined by P-values < 0.05.

With respect to the qualitative aspect of the study, Strauss and Corbin’s grounded theory [9] was used to investigate 30 LORs. These LORs originated from randomly selected 5 SIs and 5 NSIs, all of which were de-identified. Themes emerged using open and then axial coding. The researchers selectively coded the data to create a qualitative data table, which consisted of SIs and NSIs characteristics. After constant comparison, the coders reduced the data table to determine significant and non-significant characteristics. Data were then reduced again to allow for comparison to determine if the LORs could predict SIs. Individual characteristics were defined as significant for the group of SIs if ≥ 4 or ≤ 1 students possessed the characteristic. This same process was used to identify significant characteristics for NSIs. However, if the characteristic was deemed significant for both the NSIs and SIs then the characteristic was deemed non-predictive. Only the characteristics that were deemed significant and were unique to the SIs were considered to be predictive. Prior to initiating the qualitative aspect of the study, the researchers sought to insure dependability which translates to reliability for quantitative studies. One trial LOR was reviewed and coded by both researchers simultaneously for coding training and confirming expectations and methodology. Both researchers independently open and axial coded all LORs. Analysis of 10% of the qualitative data set resulted in 87% coding agreement between researchers. With negotiation, the researchers came to 100% agreement.

Quantitatively, demographic differences were observed between SIs vs. NSIs in age at the start of the program (µ= 25.76 SIs, µ= 23.67 NSIs), gender (SIs= 87% female, 13% male; NSIs = 76% female, 24% male), the number of children at the start of the program (SIs 4.76%, NSIs 5.17%) and children during the program (SIs 9.5%,NSIs 5.2%). Academic differences in biology GPA (µ= 3.46 SIs, µ= 3.39 NSIs), physics GPA (µ= 3.48 SIs, µ= 3.39 NSIs), and verbal GRE percentile scores (µ= 59.6 SIs, µ= 53.7 NSIs) were noted. Other differences between the groups were negligible. On average, SIs were two years older, had more observation hours, have higher biology GPA (+0.07), higher physics GPA (+0.09), higher verbal GRE scores (+5.91%) and were less likely to take the GRE more than once in comparison to NSIs. The Wilcoxon rank sum test (Mann-Whitney u-test) was utilized in analyzing the differences in numeric ratings on LORs between the SIs and NSIs. Analysis of each of 11 professional behavior characteristics rated on LOR’s by PT’s showed three characteristics to be significantly different between the SIs and the NSIs; critical thinking (P= 0.0029), organizational skills (P = 0.0363) and judgement (P = 0.0001). None of the other professional behavior characteristics rated by PT’s in LORs were significant. PT ratings on LORs of communication (P= 0.0731) and professionalism (P= 0.0789) indicate marginal findings and may deserve further attention. Additionally, none of the professional behavior characteristics rated by Non-PTs in LOR’s were found to be significant.

Qualitatively, in order to establish the dependability of the study, the researchers individually, hypothesized which of the 10 students were SIs. Upon comparison of the researcher’s independent hypotheses, each had selected the same five students as SIs. However, only 2/5 hypothesized SIs were correct (40% accuracy). Using the previously stated criteria for significance, the following characteristics were determined to be significant for SIs: communicates effectively (4/5) and cultural competency (4/5), as seen in Table 1. NSIs characteristics included: conflicting personality descriptor (4/5), commitment to learning (5/5), balance (4/5), teamwork skills (0/5), potential future success (5/5), compatible learning skills (0/5), effective leadership skills (4/5), and emotional intelligence (4/5) as seen in Table 2. To clarify, an example of a conflicting descriptor is as following: One applicant being described by a recommender as “introverted” in 1 LOR; while, another LOR describes the applicant as “extroverted.” Quantitatively, we found significant differences in PT ratings on pre-admission LORs between SIs and NSIs for the characteristics of critical thinking, organizational skills, and judgement. However, we found no significant differences in pre-admission ratings by PTs in other professional behavior characteristics on LORs between SIs and NSIs. No significant differences were found in LOR ratings by Non-PTs in all professional behavior characteristics.

Table 1.

Significant characteristics of 5 randomly selected students of interest who were admitted to the Doctor of Physical Therapy program in 2009-2014 at the University of Cincinnati, OH, USA based on grounded theory

Student's No.	Communicates effectively	Cultural competency
1	X	X
2	X	X
3	X
4		X
5	X	X

Table 2.

Significant characteristics of 5 randomly selected non-students of interest who admitted to Doctor of Physical Therapy program in 2009-2014 at the University of Cincinnati, OH, USA based on grounded theory

Student's No.	Personality conflicting descriptors	Commitment to learning	Team work skills	Potential future success	Compatible learning skills	Effective leadership skills	Emotional intelligence
1	X	X		X			X	X
2	X	X		X			X
3	X	X		X			X	X
4		X		X				X
5	X	X		X			X	X

Therefore, our quantitative findings suggest our PT program may benefit from using PT ratings in pre-admission letters in the critical thinking, organizational skills, and judgement characteristics as predictors of future professional behaviors in the educational clinical setting. Use of LORs provided by Non-PTs did not appear to offer any predictive value in estimating a potential student’s professional behaviors in clinical education. Although critical thinking, judgement, and organizational skills were found to be significant predictors from LORs, our DCE spends the majority of time dealing with issues related to communication, clinical skills, confidence, commitment to learning and interpersonal skills which were not found to be significant predictors from either PTs or Non-PTs. As Table 3 illustrated, despite having followed a trustworthy qualitative method to analyze the data, the data did not yield any new insights to indicate that LORs could be used to predict clinical performance as it relates to professional behaviors. Thus, LORs appear to be of limited value in predicting professional behavior performance in clinic.

Table 3.

Communication predictor from letters of recommendation (LOR) and actual clinical outcome of 5 randomly selected students of interest who were admitted to Doctor of Physical Therapy program in 2009-2014 at the University of Cincinnati, OH, USA based on grounded theory

Student's No	LORs communication predictor	Clinical intervention
1	X
2	X	X
3	X	X
4		X
5	X

Limitations for this study included the sample size which was based upon one DPT program with a sample size of 137 students. Another limitation to our study was the skewed data from LORs and a ceiling effect on the rating scale for professional behaviors, as evidenced by overall positive scores on LOR’s and very few low ratings on a 5 point scale. Future research should include multiple DPT programs for a larger sample size. Additionally, an investigation of pre-admission characteristics for the development of a prediction rule for specific professional behaviors that relate to employability skills must occur as our profession goes forward [10].