Psychometric properties of the mini peer assessment tools (Mini-PAT) in emergency medicine residents.

Background: A few studies have been done regarding the validity and reliability of the Mini-Peer Assessment Tool across various specialties. This study was conducted to determine the reliability, content and construct validity of Mini-Peer Assessment Tool to assess the competency of emergency medicine residents.
Methods: This study was carried out to investigate the psychometric properties of the mini-PAT tool to evaluate the professional competencies of emergency medicine residents in educational hospitals affiliated to Tehran University of Medical Sciences. The initial Mini-Peer Assessment Tool was translated into Persian. After that, the content validity index and content validity ratio determined by consulting 12 professors of emergency medicine. The construct validity was determined with exploratory factor analysis and investigation of the correlation coefficient on 31 self and 248 peer assessment cases. The reliability of the mini peer assessment tool was determined by internal consistency and item deletion by using Cronbach's alpha coefficient. Reliability was also assessed by determining the agreement between the two tools of self-assessment and peer assessment by using the diagram Bland and Altman.
Results: The results showed content validity ratio (CVR) of the items ranged from 0.56 to 0.83, and the content validity index (CVI) of the items ranged from 0.72 to 0.90. The reliability of the self-assessment and peer-assessment tools were 0.83 and 0.95 respectively and there was a relative agreement between the self-assessment method and the peer assessment method. Finally, the tool underwent exploratory factor analysis resulting extraction into two factors namely 'clinical competencies' and 'human interactions' in the peer assessment tool. In the self-assessment tool, the factors of 'good practice' and 'technical competence' were extracted.
Conclusion: The results of the present study provided evidence of the adequacy of content validity, reliability of the contextually customized mini-peer assessment tool in assessing the competencies of emergency medicine residents.

↑ What is “already known” in this topic:

Mini-PAT, as a 360-degree assessment tool, enjoys appropriate validity and reliability in assessing medical students' competencies. As a result of the exploratory factor analysis of mini-Pat, previous studies of two dimensions ‘clinical competence’ and ‘human interactions’ have been reported.

→ What this article adds:

The mini-pat tool in self-assessment of emergency medicine residents, as well as peer assessment, has appropriate validity and reliability. The results of the exploratory factor analysis of the mini-pat tool led to the recognition of two dimensions ‘good practice’ and ‘technical competencies’ in self-assessment and two-dimensional ‘clinical competence ’and‘ human interactions ’in peer assessment.

Introduction

In recent decades, the importance of assessing competencies such as professional behavior, communication skills (1, 2), teamwork (3), interaction and managing patients in the clinical set ting (2, 4), communications with colleagues, and performance in hard clinical settings (5) caused the Peer Assessment Tool to be chosen from other workplace-based assessment tools. Peer-assessment tools have attracted increasing importance on several accounts including creating further and more active participation learners in the process of assessment (4, 6, 7), promotion of a sense of responsibility regarding peer learning (8), and improvement of self-awareness (9) by clinical trainers. Furthermore, the use of self-assessment tools has moved to the spotlight by the clinical trainers due to its several merits including preparing the grounds for promotion of learner’s critical understanding of knowledge, self-reflection, and promoting personal growth and promotion (10).

Following this development, various peer assessment tools were developed (4, 9, 11-16). Out of these Peer Assessment Tools (PATs), the Archer mini PAT instrument, a modified version of Sheffield mini PAT, was developed in 2008. Archer’s tool has 15 items in five areas of ‘clinical care’, ‘medical performance’, ‘communication with patients’, ‘teaching and training appraising skill’, and’ working with colleagues’. It was used for assessing medical students' competencies in England in 2008 (4) regarding its psychometric characteristics. It has been underscored as an assessment tool with appropriate reliability and validity to assess first and second-year medical students’ competency (4, 17, 18) Mini-PAT has been increasingly used as a formative assessment tool in clinical settings within which the learner anonymously receives collective feedback and assessment from their peers along with the self-assessment In addition, it is a 360-degree assessment tool to evaluate the learners’ competencies such as professional performance, teamwork skills, communication with patients, communication with other medical professions, which receives scant attention via other assessment tools such as DOPS and Mini-CEX. This has been highlighted by many studies (1, 4, 7, 15, 19). The authors, however, emphasized developing a compatible mini PAT with specific fields, specialties’ differences in performance, various learning settings and expected necessary competencies (19). Considering the fact that in the review studies, the psychometric characteristics of assessment tools need to be highlighted before further development (20). Since the application of mini PATs in different fields may vary, the reliability and validity need to be reexamined to ensure greater generalizability (17, 21). In addition, since no report is provided with regard to the results concerning the use of peer assessment and the self-assessment tool in the Archer’s paper, the present study aimed to determine the psychometric characteristics (reliability and validity) of Mini-PAT instrument to evaluate the competencies of emergency medicine residents.

Methods

Study setting

This study was conducted in two university-affiliated tertiary-care teaching hospitals, and the study population consisted of first- and second-year emergency medicine residents. Emergency medicine is a three-year task-based program with bedside and didactic training and direct clinical supervision by board-certified emergency physicians.

Participants

At the time of the study, there were 57 first- and second-year residents from which a random sample of 31 persons entered the study. Of these, 20 residents (64.5%) were affiliated to Imam Khomeini Hospital and the other 11 residents (35.5%) were affiliated to Shariati Hospital. Also, 18 (58%) were first-year residents and 13 (42%) were second-year emergency residents in educational hospitals affiliated to Tehran University of Medical Sciences.

Procedure

The Mini-PAT form, developed and validated by Archer et al., was adapted for the purpose of this study. Written permission was obtained from the developer (personal communication) and then the translated version that we will explain below was modified to match the unique characteristics of the emergency medicine and to make it suitable for self-evaluation as well as peer-assessment. Face, content, and construct validity, as well as reliability of the instrument, were investigated using the following stages according to Toolkit on Translating and Adapting Instruments the guidelines for instrumentation of the human sciences and research Institute of Cambridge University (22).

Translating the mini-PAT instrument: The instrument was translated by the researchers. The Persian version of the subject and concepts was reviewed by six people consisting of two Ph.D. candidates of medical education, two emergency medicine department faculty members, and two faculty members of the medical education department. Next, it was re-translated from Persian to English by two experts of the English language and compared to the original questionnaire. No incongruity was observed in terms of the concept between the new tool and the original one.

Face and content validity of mini-PAT instrument: A four-hour focus group discussion was conducted with the participation of 12 faculty members of the emergency medicine department. The participants were asked to discuss the significance, clarity, and practicality of the items, as well as their match with the setting of emergency medicine and the coverage of the required competencies. Based on the comments provided by the participants, items were confirmed, altered, eliminated, or added to the questionnaire. The participants were then asked to categorize the relevance and necessity of each item in one of the three classes: “it is essential”, “it is useful but not essential”, and “it is not necessary”. According to Lawshe formula, the content validity ratio (CVR) of each item was calculated and given the number of panelists, a value above 0.56 was considered as acceptable (23) Content validity index (CVI) was calculated for the entire instrument based on Waltz method (24).

Assessment by mini-PAT instrument: After receiving an informed consent form each resident participating in the study was given individual training about the objectives of the study and the assessment procedure. The 31 study participants selected, out of the list of senior residents with whom they had had clinical shifts over the past six months, at least ten residents served as their assessors. The investigators randomly selected eight residents to conduct peer assessment. Using the mini-PAT instrument, the participating residents assessed their own performance (self-assessment) and each resident was assessed by eight senior peers in four domains of competency that includes ‘clinical care’, ‘medical practice’, ‘communication with the patient’ and also ‘cooperation with the healthcare team’. The collection of the peer assessment forms was performed within one month. The mean scores of all items of all of eight assessors were reported as the peer assessment score.

Reliability of the mini-PAT instrument: The reliability of peer- and self-assessment results using the mini-PAT instrument was investigated by calculating Cronbach’s alpha. Furthermore, alpha was also calculated by eliminating every single item in the two instruments, and the agreement between self-assessment and peer-assessment was determined by using the diagram Bland and Altman.

Construct validity of mini-PAT instrument: We employed two methods to assess the construct validity: exploratory factor analysis and determining the correlation between the constructs and total score of the questionnaire.

Statistical analyses

In order to determine the mean score, standard deviation, and frequency of each resident, descriptive statistics were used. To compare the mean difference between the peer assessment and self-assessment, the independent t-test was used. The Pearson correlation coefficient was calculated to determine the correlation between the construct and the total score of the questionnaire. By determining the Kaiser-Meyer-Olkin (KMO) index and Bartlett test, the proportionality of data and correlation between the variables were investigated to perform the exploratory factor analysis. Thereafter, exploratory factor analysis was performed through determining competence areas and varimax rotation. Statistical data were then analyzed using SPSS version 22.

Results

The results showed 279 mini-PAT assessment forms were filled out and completed by the residents, including 31 self-assessments and 248 peer assessments. Demographic information of the participants showed the mean age of the emergency residents, participating in the study as the intervention group was 36.3±7.3. The mean age of first-year residents was 35.6 ±7.4. In the 2nd year residents, the mean age was 37.4±7.1.

Content validity: Based on the results of the content validity (two items, i.e., ability to assess the psychosocial aspects of the disease and willingness to train colleagues) were removed from the original Mini-PAT instrument based on the idea of the emergency medicine specialists. In addition, Two items (verbal and written communication with colleagues) were merged together, and three new items (the ability of multi-tasking simultaneously, the ability of suitable decision-making for the patients and the ability of documentation) were also added. Furthermore, based on the comments of experts to facilitate the scoring, a six-point Likert scale was changed into a four-point Likert scale. In this way, eventually, the contextually customized instrument was finalized with 16 items, where one of its items, as with the original instrument, is used for the general assessment of the competence of emergency medicine residents. The results of the CVR given the number of the participants (12) and its minimum threshold (0.56) indicated out of all the items, two items namely, ‘the ability to assess the psychosocial aspects of the disease and willingness to train colleagues’ failed to meet the requirements based on Lawshe table. In other words, their CVR was lower than 0.56. The results showed that the content validity index (CVI) of the items ranged from 0.72 to 0.90.

Residents’ scores: The results of 279 cases assessed by mini-PAT instrument (including 31 self and 248 peer assessment) indicated that the mean±2SD of residents’ self-assessment scores (47.16±4.73) was significantly higher than their peer assessment scores (43.86±8.50) (p<0.001). The comparison of mean±2SD for self and peer-assessment of residents showed significant differences in the areas of clinical care, medical performance and communication with patient’s competencies. However, the result did not show a significant relationship between self and peer assessment in cooperation with the healthcare team (p=0.188) (Table 1).

Table 1

Mean score and standard deviations of each Domain of competency on the modified Mini-PAT used as self-assessed and peer-assessed instrument

Domain of competency	Self-Assessment(n=31)M±SD	Peer Assessment(n=248)M±SD	T	p
Clinical Care	12.42±1.41	11.48±2.59	3.13	0.001
Medical Practice	12.23±1.75	11.24±2.75	2.75	0.019
Communication with the patient	9.87±1.41	9.12±1.71	2.35	0.021
Coop eration with the healthcare team	12.65±1.72	12.03±2.51	1.32	0.188
Total Score	47.16±4.73	43.86±8.50	3.28	0.001

Reliability: The Cronbach’s alpha coefficient of self-assessment and peer-assessment mini-PAT was 0.83 and 0.95, respectively. The Cronbach alpha coefficient for each individual competency, namely clinical care provision, medical performance, communication with patients, and cooperation with the treatment team area has been provided in Table 2. Furthermore, results indicated that the deletion of items did not bring about any significant change in the value of Cronbach alpha. The results showed a relative agreement between the two methods of self-assessments and peer-assessments. The mean difference between the two methods by using the Bland and Altman chart is near zero (0.18) (Fig. 1).

Table 2

Cronbach's alpha coefficients of the subdomains of Mini-PAT Questionnaire

Subdomains	Cronbach's alpha coefficients of Mini-PAT Questionnaire
	Peer Assessment Tool	Self-Assessment Tool
Clinical Care	0.88	0.59
Medical Practice	0.89	0.66
Communication with the patient	0.84	0.73
Cooperation with the healthcare team	0.90	0.77
Total score	0.95	0.83

Graph. 1

Construct Validity: The analysis of the construct validity determined by the correlation between the domains of the competencies with the total score in the self-assessment tool and peer assessment tool score, exploratory factor analysis revealed the following results:

The results of the construct validity of each domain of the self-assessment instrument with the total score indicated a strong positive correlation between clinical care and the total score (r=0.839), communication with the patients (r=0.673), medical performance and the total score (0.722), and eventually, a positive and relatively strong correlation between cooperation with the healthcare team and the total score (r=0.774) was observed (Table 3). The construct validity resulting from the peer-assessment tool indicated a strong positive correlation between the domains of the competencies with the total score compared with the self-assessment tool. The correlation coefficient in the domain of medical care with the total score of the peer assessment tool was (0.923), communication with the patients and the total score was (0.785) and the cooperation with the treatment team and the total score (0.897). (Table 3).

Table 3

Inter-correlations between the Self Mini-PAT subdomains and total score& Peer Mini-PAT subdomains and total score in the emergency medicine residents of Tehran University Medical Sciences

	Subdomains	Medical Care	Clinical Practice	Communication with the patient	Cooperation with the healthcare team	Total score
Self Mini-PAT	Medical Care	1.000				0.839**
	Clinical Practice	0.543**	1.000			0.722**
	Communication with the patient	0.479**	0.229	1.000		0.676**
	Cooperation with the healthcare team	0.542**	0.338	0.420*	1.000	0.774**
	Total score	0.839**	0.722**	0.676**	0.774**	1.000
Peer Mini-PAT	Medical Care	1.000				0.923**
	Clinical Practice	0.832**	1.000			0.905**
	Communication with the patient	0.614**	0.577**	1.000		0.785**
	Cooperation with the healthcare team	0.752**	0.706**	0.707**	1. 000	0.897**
	Total score	0.923**	0.905**	0.785**	0.897**	1. 000

**. Correlation is significant at the 0.01 level (2-tailed).

The results of the exploratory factor analysis of the mini PAT tools with the KMO value of 0.90 indicated that all the items were loaded on two factors. In addition, the scree plot graph shows the variation of the eigenvalues with respect to the factors and the number of agents, and it is evident that the two factors having the eigenvalue greater than one were analyzed as two important factors. In a way, the results of exploratory factor analysis showed that 46.6% of the total variance in the self-assessment tool and 69.4% in the peer-evaluation tool were identified by two factors. However, the sequence and positioning of these factors varied across these two instruments (Table 4).

Table 4

Total Variance Explained and Factor loading of the items of the Peer Mini-PAT& self-mini-PAT by the Component Matrix Rotation

Questions	Component Peer mini-PAT		Component self-mini-PAT
	1	2	1	2
Ability to interpret patients’ problems	0.811	0.264	0.564	0.330
Ability to decision making such as providing management plan	0.825	0.251	0.073	0.627
Awareness of his/her limitations such as knowledge, procedural or technical skills	0.711	0.361	0.363	0.447
Efficient use of resources such as Para clinical tests	0.697	0.406	0.577	0.319
Ability to Proper management of time and prioritization	0.775	0.290	0.056	0.830
Ability to do multitasking simultaneously	0.796	0.281	0.078	0.608
Appropriate management of the patient	0.825	0.310	0.228	0.656
Application of technical skills included in the residency program	0.741	0.297	0.236	0.526
Appropriate communication with the patients	0.283	0.792	0.569	0.083
Communicating with the patients’ families	0.253	0.841	o.845	0.111
Respect for patients and their right to	0.182	0.749	0.666	0.329
Verbal and written communication with the colleagues	0.371	0.744	0.806	0.119
Documentation (such as writing a clear and organized history taking)	0.592	0.565	0.636	0.201
valuing participation of the health care team and perception of its importance	0.567	0.622	0.294	0.364
Accessibility and reliability	0.499	0.676	0.480	0.601
Total Variance explained by each factor Before rotation	60.367	9.067	33.152	13.508
Total Variance explained by each factor After rotation	40.181	29.253	25.017	21.643
Number of items for each factor	9	6	7	8
Cronbach’s for each factor	0.94	0.90	0.766	0.738

Relative agreement between the two methods of self-assessment and evaluation by peers to assess the ability of emergency medicine residency.

In the self-assessment tool, the first identified factor was 25% of the total variance and included 7 items out of 15 related to “medical care, clinical performance, patient relationship and collaboration with the health team” of domains of the competencies and the second factor explained 21.64% of the total variance and included eight items. One item from “medical-care” and three items from clinical performance and one item from “cooperation with health team” were placed/loaded on the second factor (Table 4). The identified factors in the self-assessment tool were termed “good practice” and “technical competency.”

In the peer assessment instrument, the first recognized factor claimed 40.2% of the total variance and included 9 out of 15 items related to “medical care, clinical performance”. The second item contributed to the 29.25 of the total variance and included 6 items related to “communication with the patient and corporation with the healthcare team” (Table 4). The identified factors in the peer assessment tool were termed “clinical competence” and “human interactions”. The results showed the two extracted factors in self- and peer-assessment tools have appropriate reliability.

Discussion

In the present study, the psychometric characteristics of a customized version of mini-PAT to assess emergency resident’s competencies were confirmed through validity and reliability. The results of the exploratory factor analysis of 279 mini-PATs including 248 peer-assessment and 31 self-assessment identified two factors in every two tools, ‘good practice’ and ‘technical competency’ in the self-assessment questionnaire, and two factors of ‘clinical competence’ and ‘human interactions’ recognized for the peer assessment tool. One of the findings of the present study was that mean of self-assessment of competency was more significant than the mean of peer assessment in three domains of competency.

Studying the content validity of the mini-PAT instrument was carried out along with modifying some items in the process of contextually customizing it. In addition, regarding the discrepancies in performance existing between the specialty of emergency medicine and other fields, two items related to ‘psychological and the social’ aspects of the disease and ‘residents’ willingness to teach’ were eliminated, and three new items namely ‘multitasking’, ‘patient-management and decision making’, and ‘clearly documenting activities’ were added to the instrument. Thus, given the emphasis on the development of a specialized mini-PAT instrument based on functional differences of each specialty and expectable learning consequences (17), making changes in the items of this instrument seems to be essential to evaluate the competence of emergency medicine residents. Regarding this, AZ Abdullah, in an analytical article, considered mini-PAT as an assessment tool that benefits from appropriate validity and reliability yet not fully evaluated in various areas and needs to be further investigated (18). Norcini reports that peer-assessment can yield valid and reliable information; however, this validity depends upon many factors including investigating the reliability of the assessment tool too (7).

The result of the present study indicated that the mean scores obtained from self-assessments were significantly different from their scores when assessed by their peers. The finding showed self-assessment in three domains of competencies, including clinical care, medical performance and communication with patients was significantly higher in comparison with peer-assessment. However, in working with the health team, there was not any significant difference in self-assessment score and average peer evaluation score. Regarding this Donnan et al. have reported self-assessment of professional competencies was significantly lower in comparison with peer assessment. The results of the present study are not in line with a similar study carried out in Calgary University in Canada (25). Besides, our results were not in line with LaMantia’s study which reported the self-assessment score of emergency medicine assistants was lower than other evaluators. In other words, these assistants have evaluated their competencies in communication and interpersonal skills as lower compared with other evaluators (26). On the other hand, according to a study by Karakaya on medical students by themselves, their professors, and peers reported self-assessment scores of medical students to be higher than those of peers and teachers (27).

The reliability coefficient of the Persian and customized mini-PAT instrument was (0.83) and (0.95) for self and peer assessment, respectively. This was indicative of the coherence of the items. In line with this study, Archer’s study indicated that the Cronbach alpha coefficient of the mini-PAT instrument carried out on 554 medical students in the British Foundation Program to be 0.98 (4). Archer’s study has failed to report the Cronbach alpha for different domains of competency and separately for self and peer assessment mini-PAT tools (4). The results of the present study are also in line with a similar study carried out in Calgary University in Canada regarding the reliability of self and peer assessment mini-PAT tools (0.85 and 0.91) respectively to evaluate the professional competencies of medical students (25). In the present study, the internal consistency of the items of the instrument showed that the deletion of each /every item did not significantly influence the Cronbach alpha coefficient confirming the fact that all items equally contributed to the total score. The items of the mini-PAT instrument enjoyed appropriate consistency and none of the contextually customized items were modified or eliminated. The results study showed a relative agreement between the two methods of self-assessments and peer-assessments by using the Bland and Altman diagram, this result was not reported in any other related studies including Archer and Donnan but in both studies, the reliability was studied and reported using G theory (16, 25).

The construct validity of the correlation coefficient of each domain of competency with all of the items in the peer-assessment tool (between 0.61 and 0.89) and in the self-assessment tool (between 0.67 and 0.83) (p<0.001), were in line with Archer’s study claiming to have a correlation coefficient between 0.65 and 0.93 (4). Archer’s study has failed to report the correlation coefficient separately for self and peer assessment mini-PAT tools (4).

The result of the exploratory factor analysis indicated that the rotation of variances led to retaining two factors, namely ‘clinical competency’ and ‘human interactions’ in the peer assessment, which was in line with Archer’s study (4). The highest variance percentages in the first factor were items from the subcategory of the provision of medical care and medical performance. The first factor was entitled clinical competency. The second factor that included items in ‘communication skills with health team’ and ‘communication skills with patients’ was entitled ‘human interactions’ which was in line with Archer’s study (4). Exploratory factor analysis identified two factors in self-assessment, but since the pattern of the items was not similar to those of peer-assessment tools, they were named “good practice” and “technical competency”. Archer’s study has failed to report Factors extracted separately for self and peer assessment tools. However, these results are in line with the results of the Dunoon study, which reported differences in the pattern of item placement in self-assessment and peer assessment tools (25). Also, Thomas et al. at Johns Hopkins University assessed the ability of interns last month by assistants and faculty members, the results of the exploratory factor analysis of the mentioned study led to the formation of two factors of technical skill and interpersonal communication skills, in this respect, the present study is somewhat similar to that study. However, due to a good correlation between self-assessment of interns, self-assessment of colleagues and faculty members, the ability of interns in different factors has been reported (28) In this respect, it is not in line with the present study, because in our study, faculty members did not assess using mini pat tools. Also, Sang Chul Kim et al. believed that self-survey competence assessment could be a good example of formative and summative forms of assessment. In addition, their report has been shown that two characteristics were consistently identified in the results of factor analysis of medical students, residents, and physicians' viewpoints, The first factor was related to medical knowledge or technical competence and the second factor included the interpersonal or patient communication skills (29) which is in line with the results of the present study.

Although in the factor analysis of the present study, the pattern of items in the peer assessment tool and the self-assessment tool differed, the Cronbach's alpha values of each factor after factor analysis in the peer assessment tool (0.94, 0.90) and self-assessment (0.76, 0.73) indicated acceptable reliability of the mini-PAT tool and the fit of the items to the relevant factors (15). In this regard, Archer’s study only Cronbach's alpha of the first factor (0.98), which accounted for the largest percentage of variances, was reported (16).

Limitation and delimitations

As the scope of the present investigation was to evaluate the validity, reliability and the factor analysis of mini-PAT instrument to assess the competencies of emergency medicine residents in the referral hospitals of Tehran University of Medical Sciences, no heed was paid to the issue of feedback as one of the pivotal dimensions of mini-PAT. Carrying out psychometric analyses of mini-PAT in other specialties of medicine is recommended. In addition, a feasibility study aiming at evaluating its development and completion in general medicine can be another area for research in which an investigation can answer the questions.

Conclusion

The present study confirmed the reliability and validity of the mini-PAT instrument as a contextually customized instrument to assess the competencies of performance in the workplace of emergency medicine residents regarding their clinical competence, communication skills, teamwork, and professional behavior by themselves or by their peers.

Acknowledgment

This paper is the result of part of the PhD thesis approved by Tehran University of Medical Sciences (Thesis code: 13). Thus, the authors of the paper highly appreciate the financial supports of the vice presidency for research affairs, professors, and emergency medicine residents and all people who cooperated in this research.

Ethical confirmation

The present study and its conforming to the ethical principles of research were confirmed by the Ethics Committee of Tehran University of Medical Sciences with the ethics code of 1397-4911.

Conflict of Interests

The authors declare that they have no competing interests.