Development and preliminary validation of the competency model for rural general practitioner in China.

BACKGROUND: There is a shortage of rural general practitioners (GPs) in China. Training programs have been initiated to meet the needs of GP workforce in the rural areas; however, there is an absence of validated tool to assess their competencies.
OBJECTIVE: This study aimed to develop a competency model for rural general practitioner (CMRGP) after training in China and to examine its validity and reliability.
METHODS: A multistage process was adopted to develop the CMRGP comprised literature review, panel discussion and expert consultation, and the initial version of CMRGP was reduced from 10 domains and 77 items to 7 domains and 54 items. A pilot study was conducted among 202 rural GPs for the psychometric evaluation and application of the initial version of CMRGP, in which a questionnaire on the importance of items and self-evaluation was completed by the GPs.
RESULTS: In the pilot study, 132 completed questionnaires (65.3%) were returned. Acceptability and face validity of the CMRGP were supported by high importance scores of the items, in which 52 out of 54 items achieved score higher than 4.00 (possible score from 0 to 5). Factor analysis supported the construct validity. After the modification, the final version of CMRGP contained 6 domains and 47 items. Good reliability was supported by internal consistency reliability (Cronbach's α was 0.98) and split-half reliability (Spearman-Brown coefficient was 0.99).
CONCLUSIONS: The CMRGP demonstrated good reliability and validity. Pilot study showed its potential for application in the rural general practice and training program.

The CMRGP is a tool to assess the competencies of rural GPs after training.

The CMRGP achieved good validity and reliability.

This tool can be used in outcome evaluation of the rural residency training.

Introduction

General practitioners (GPs) provide first contact of care for patients, i.e. basic medical care, preventive care, health management and referral to specialists when necessary (1). There are three GP training programs in China: (i) the standardized residency training program (3-year residency training for graduates of 5-year medical school study), (ii) the on-job training program (1-year training for doctors who want to register as GP) and (iii) the ‘3 + 2’ rural GP residency training program (2-year residency training for graduates of 3-year junior college study) (2). The trainee will be able to register as a GP upon completion of the standardized residency training program or the on-job training program. The trainee of the ‘3 + 2’ training program will register as an assistant GP, who will work as a rural GP in China. There were only 0.38 rural GPs for one thousand rural populations till 2018 (3), which is far from the goal of at least one GP per thousand rural populations (4). The ‘3 + 2’ training program is a transitional strategy to meet the needs of rural GPs in China.

As the competency-based education model was embraced, it was asserted that competency in practice was fundamentally contextual. Professional competency was defined as the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values and reflection in daily practice for the benefit of the individual and community being served (5). As more challenges might confront the rural GPs in providing medical services due to isolation, limited staff and resources, differences in health patterns and the unique socio-cultural environment in rural communities (6), there might be competencies particularly important for GPs in the rural areas. A study from Australia in 2004 found that rural physicians, being isolated geographically, with few available resources and limited access to specialists, needed to be more decisive, self-reliant and independent (7).

Over the past two decades, work has been done to define the competencies of rural GPs (8–15). A study in the US revised competencies of GPs into rural context, including 8 domains and 35 competencies (9). Another study in Japan reported 14 competencies of GPs in non-urban areas, in which 9 competencies were identical with the competencies of GPs in urban areas (10). Specifically, national rural medicine groups in Australia and Canada also defined competencies for rural physicians (8,11,12). In comparison with international experience, only a few studies were found in this area in China (13–15), and the lack of psychometric evaluation and practical application were often the limitations in these studies (14,15). Therefore, the aim of this study was to develop and validate a framework of competencies to be used in the assessment of rural GPs, which also can be used in outcome evaluation of the rural residency training in China.

Methods

A multistage approach was adopted in the study to ensure the reliability and validity of the instrument (16,17), which consisted of three steps: (i) develop the framework based on literature review and panel discussion; (ii) identify domains and items by expert consultation; (iii) conduct pilot study and psychometric evaluation. The development process of the competency model for rural general practitioner (CMRGP) is shown in Figure 1.

Figure 1.

Development process of the CMRGP (2017).

Framework development

We carried out literature searches in PubMed, EMBASE, Google-Scholar and three Chinese databases (China National Knowledge Infrastructure, Wanfang Data, VIP Chinese Periodical Services) with terms commonly used to describe GP (e.g. GP, family physician, family doctor and community health worker) and competency (e.g. competency, competencies and core competencies) in the rural areas. Five articles describing detailed competencies of rural GPs were identified from the literature review (8,9,13–15). We also performed a targeted internet search for competency model or competencies from five international general practice organizations: the World Organization of Family Doctors (WONCA) (18), the College of Family Physicians of Canada (CFPC) (19), the Accreditation Council for Graduate Medical Education (ACGME) (20), the Royal College of General Practitioners (RCGP) (21) and the Royal Australian College of General Practitioners (RACGP) (22). In addition, the National Health Commission of China was searched for policy documents of rural GP training (23).

Through the content analyses of the literatures, 10 domains and 82 competencies were identified based on the framework from ‘Construction and application of Chinese doctors’ competency model’ proposed by Sun (24). Panel discussion was carried out to refine the framework of competencies. Nineteen rural GPs were recruited by purposive sampling, based on the following criteria: (i) work experience as a rural GP for more than 5 years; (ii) experience as a trainee in the ‘3 + 2’ training program; (iii) willingness to participate in the study. The rural GPs were randomly allocated to two panels and the discussion focussed on the following topics: whether the items extracted were suitable for competency measurement of rural GPs after training in China, whether there were any new competencies which should be added into the CMRGP, and whether there were any modifications needed for the description of the items and domains. After the panel discussion, six competencies were deleted, one competency was added and six competencies were modified. The initial version of the CMRGP was established, which contained 10 domains and 77 items.

Item identification and instrument refinement

An expert consultation was convened in summer 2017 for item identification and refinement of the CMRGP. Ten experts were invited purposively based on the following criteria: (i) extensive experience in general practice service or education; (ii) familiarity with characteristics of rural general practice; (iii) willingness to participate in this research. The domains and competencies in the initial version of CMRGP were rated for importance by each expert using a 5-point Likert scale (1 = not important; 2 = slightly important; 3 = moderately important; 4 = important; 5 = very important). After the first round of rating, domains and items with significant discrepancy in the rating were discussed and a second round of rating was held. Competencies deemed as ‘important’ or ‘very important’ (mean score ≥4.00) were included, while competencies considered as ‘not important’ or ‘slightly important’ (mean score <3.00) were excluded (25). Competencies deemed as ‘moderately important’ (mean score ranged between 3.00 and 3.99) would be further discussed by the experts.

After the expert consultation, no competency was rated as ‘not important’ or ‘slightly important’. There are revisions on six domains, including ‘Patient-centred Care’ was merged to ‘Communication’; ‘Practice Based on Health Care Systems’ was merged to ‘Information utilization and integrated management capabilities’; ‘Medical Knowledge and Research Capabilities’ and ‘Practice-based Self-learning and Improvement’ were merged as ‘Knowledge Application and Self-improvement’. Twenty-two competencies were suggested to be merged due to similar meaning, two new competencies were added and three competencies were removed for irrelevancy to the rural GPs’ work (importance scores of items and the experts’ opinion are available in Supplementary Material 1). The initial version of the CMRGP was merged into 7 domains and 54 items.

Psychometric evaluation

In December 2017, a pilot study was conducted to test the feasibility and measurement properties of the initial version of the CMRGP among 202 rural GPs selected by purposive sampling, based on the following criteria: (i) work experience within 1 year upon completion of the ‘3 + 2’ rural training program; (ii) working location in rural general practice; (iii) willingness to participate in the study. The questionnaires were answered via online survey, in which self-evaluation using the CMRGP was completed by the rural GPs. Self-evaluation was based on the GPs’ ratings for their own performance using a 5-point Likert scale (1 = poor; 2 = slightly poor; 3 = fair; 4 = good; 5 = excellent). The importance of items in the CMRGP were also rated in pilot study. And the information was used as an additional round of expert consultation, in which items scored <3.00 were excluded.

The acceptability and face validity were evaluated by the response rate and importance rating of each item. If responses were missing for ≥50% of items in a questionnaire, the questionnaire was considered as missing. Exploratory factor analysis was performed to analyse the internal structure of CMRGP and to determine if the items within the domains formed a distinct construct, separate from each other. The principal component method with varimax rotation was used to extract factors with eigenvalues ≥1.0 (26). Factor loading >0.50 was used as the criterion for item selection (27).

The reliability of the CMRGP was analysed by internal consistency reliability (Cronbach’s α coefficient ≥0.70 is acceptable) (28) and split-half reliability. Split-half reliability was estimated using the Spearman–Brown coefficient. Correlations between the subscales were analysed by Spearman’s correlation coefficient as the data were not normally distributed (29). In general, a split-half reliability coefficient >0.75 was excellent and <0.40 was poor (30).

Descriptive statistics, including mean [with standard deviation (SD)] for quantitative variables and frequency distribution for qualitative variables, were used for presenting demographic characteristics of the participants. Mean (with SD) was used for presenting the CMRGP scores. Statistical analyses were conducted using SPSS 22.0.

Results

Demographics of the participants

The demographics of the participants in this study are shown in Table 1. Nineteen experienced rural GPs participated the panel discussion, and the mean age was 35.11 ± 5.18 years. There were 15 females and 4 males, and the working years ranged from 5 to 20 years. Ten experts participated the expert consultation including eight senior-level GPs and two professors, and the mean age was 46.20 ± 6.80 years. There were six were females and four males and the working years of the experts ranged from 10 to 41 years. In the pilot study, 132 rural GPs filled out the questionnaire, with a response rate of 65.3% (132/202). The mean age of the rural GPs was 25.64 ± 1.55 years and 63.6% were female. All of the rural GPs had just completed the ‘3 + 2’ training program and participated in the work for less than half a year with junior titles.

Table 1.

Demographic characteristics of the participants (2017)

Characteristic	Panel discussion	Expert consultation	Pilot test
	n (%) or mean ± SD	n (%) or mean ± SD	n (%) or mean ± SD
N	n 1 = 19	n 2 = 10	n 3 = 132
Age (years)	35.11 ± 5.18 years	46.20 ± 6.80 years	25.64 ± 1.55 years
Gender
Male	4 (21.1)	4 (40)	48 (36.4)
Female	15 (78.9)	6 (60)	84 (63.6)
Working years	9.47 ± 4.01 years	24.90 ± 7.60 years	Half a year
Professional positions
Junior-level title	5 (26.3)	0	132 (100)
Intermediate title	14 (73.7)	0	—
Senior-level title	0	10 (100)	—

Self-evaluation by rural GPs in pilot study

All the 54 items were with missing values <10% except 2 items i.e. ‘2.7 Know how to end of life care’ and ‘2.10 Know about the planned immunization’ (10.6% and 12.1% missing, respectively). The mean score of each item of CMRGP in self-evaluation was higher than 4.00, please see Table 2. The highest score was 4.82 ± 0.43 for item ‘4.5 Protect the privacy of patients’. The lowest scores were 4.15 ± 0.75 and 4.15 ± 0.78 for item ‘5.2 Have ability to search and apply the medical literature’ and item ‘5.4 Be familiar with the basics of statistics’, respectively.

Table 2.

Self-evaluation and importance score of the trial version of CMRGP by 132 rural GPs in Beijing (2017)

	Self-evaluation score	Importance score (X ± S)
1. Basic medical services
1.1 Ask about medical history completely	4.46 ± 0.61	4.81 ± 0.53
1.2 Give standardized physical examination	4.38 ± 0.64	4.86 ± 0.41
1.3 Select the laboratory and imaging examination properly	4.50 ± 0.61	4.84 ± 0.43
1.4 Be capable of clinical diagnosis and differential diagnosis	4.27 ± 0.68	4.80 ± 0.45
1.5 Be proficient in clinical skills	4.31 ± 0.66	4.80 ± 0.52
1.6 Treat the patients with diseases at early stages	4.25 ± 0.72	4.73 ± 0.52
1.7 Treat the patients with common diseases/symptoms	4.47 ± 0.64	4.83 ± 0.52
1.8 Identify and treat the patients at acute, severe and dangerous stages	4.16 ± 0.67	4.70 ± 0.58
1.9 Develop a suitable treatment plan for patients	4.27 ± 0.69	4.77 ± 0.49
1.10 Provide individualized advice of health promotion and disease prevention to patients	4.47 ± 0.62	4.77 ± 0.49
1.11 Arrange referrals and consultations for patients when necessary	4.45 ± 0.61	4.76 ± 0.53
1.12 Implement the home services when necessary	4.29 ± 0.68	4.68 ± 0.57
2. Basic public health services
2.1 Manage the chronic diseases	4.51 ± 0.60	4.83 ± 0.39
2.2 Conduct the community diagnosis and develop plans based on the main health problems in the community	4.39 ± 0.71	4.73 ± 0.58
2.3 Establish and manage the health files of residents	4.50 ± 0.66	4.72 ± 0.57
2.4 Organize community health education	4.44 ± 0.66	4.73 ± 0.54
2.5 Provide advice and guidance of rehabilitation	4.39 ± 0.68	4.78 ± 0.47
2.6 Manage the special population in the community (elderly, women, children, disabled, patients with mental illness)	4.37 ± 0.68	4.76 ± 0.55
2.7 Know how to end of life care	4.30 ± 0.67	4.65 ± 0.65
2.8 Assist in the disposal of public health emergencies	4.44 ± 0.66	4.79 ± 0.50
2.9 Discover and report infectious diseases in a timely manner	4.59 ± 0.61	4.89 ± 0.36
2.10 Know about the planned immunization	4.33 ± 0.64	4.73 ± 0.56
3. Communication
3.1 Cooperate with patients, family and community members to know about the patient’s medical background	4.42 ± 0.64	4.86 ± 0.37
3.2 Establish a good doctor–patient relationship with the patient	4.55 ± 0.67	4.82 ± 0.44
3.3 Understand, trust and respect the patients and their families	4.57 ± 0.62	4.86 ± 0.37
3.4 Consider the impact of family factors on the patient’s disease	4.51 ± 0.66	4.81 ± 0.41
3.5 Know about and deal with the psychological problems of patient	4.44 ± 0.65	4.80 ± 0.44
3.6 Discuss the plan with patients and family members	4.53 ± 0.60	4.86 ± 0.37
3.7 Communicate with patients and family members actively, and explain the diagnosis and treatment process	4.48 ± 0.64	4.81 ± 0.43
3.8 Prevent and resolve the conflict with patients actively	4.55 ± 0.59	4.86 ± 0.37
4. Professionalism
4.1 Have the sense of responsibility	4.63 ± 0.54	4.85 ± 0.38
4.2 Abide the medical rules and regulations strictly	4.76 ± 0.46	4.81 ± 0.36
4.3 Improve the quality of care and ensure patient safety	4.60 ± 0.55	4.83 ± 0.39
4.4 Adhere to core values (honesty, integrity, altruism, etc.)	4.68 ± 0.53	4.84 ± 0.39
4.5 Protect the privacy of patients	4.82 ± 0.43	4.85 ± 0.38
4.6 Maintain the patient’s right to know	4.78 ± 0.47	4.87 ± 0.38
4.7 Have humanitarian and cultural literacy (compassion, love, patience, etc.)	4.73 ± 0.48	4.86 ± 0.39
4.8 Can adjust psychological state under pressure	4.53 ± 0.64	4.83 ± 0.39
5. Knowledge application and self-improvement
5.1 Have a strong medical knowledge	4.29 ± 0.70	4.86 ± 0.39
5.2 Have ability to search and apply the medical literature	4.15 ± 0.75	3.77 ± 0.30
5.3 Be familiar with the general principles and methods of scientific research	4.26 ± 0.70	3.83 ± 0.40
5.4 Be familiar with the basics of statistics	4.15 ± 0.78	4.66 ± 0.65
5.5 Follow the principles of evidence-based medicine	4.24 ± 0.71	4.75 ± 0.48
5.6 Update knowledge and professional skills in medical practice actively	4.37 ± 0.67	4.85 ± 0.40
5.7 Know about your own strengths	4.50 ± 0.65	4.85 ± 0.40
5.8 Have a career plan or goal	4.38 ± 0.72	4.82 ± 0.43
6. Teamwork
6.1 Collaborate and communicate effectively with colleagues	4.63 ± 0.59	4.84 ± 0.41
6.2 Understand your responsibilities of work	4.68 ± 0.55	4.86 ± 0.37
6.3 Cooperate with health system managers	4.62 ± 0.60	4.82 ± 0.46
6.4 Establish a good relationship with doctors in higher-level hospitals	4.57 ± 0.65	4.84 ± 0.47
7. Information utilization and integrated capabilities
7.1 Effectively use the information technology	4.47 ± 0.66	4.86 ± 0.37
7.2 Reasonably control the medical expenses	4.60 ± 0.62	4.80 ± 0.49
7.3 Have the ability of cooperation and leadership	4.28 ± 0.71	4.77 ± 0.46
7.4 Use the medical service resources reasonably	4.50 ± 0.61	4.83 ± 0.40

Acceptability and face validity

The median item response rate was 98.5% (ranged from 87.9% to 99.2%). When items were ranked upon non-response rate, all items in the lowest quartile pertained to the ‘Communication’ construct. Besides, there was a strong consensus among participants on the importance of the items as most (45/47) of the scores were ≥4.00 (also shown in Table 2). The highest importance score was 4.89 ± 0.36 for item ‘2.9 Discover and report infectious diseases in a timely manner’. The lowest importance score was 3.77 ± 0.30 for item ‘5.2 Have ability to search and apply the medical literature’. Item ‘5.3 Be familiar with the general principles and methods of scientific research’ (3.83 ± 0.40) also scored below 4.00.

Construct validity

Exploratory factor analysis on 132 questionnaires with valid CMRGP scores yielded six common factors. Overall, the rotated sum of square loading showed that 74.65% of the variance was explained by the six factors. The descriptions of most domains were kept except the sixth factor which was rephrased as ‘teamwork and information utilization’. Seven items with low (<0.50) factor loadings were deleted. As a result, there were 6 domains and 47 items in the final version of the CMRGP (items of the final version of the CMRGP are shown in Supplementary Material 2). The number of retained items of each domain is shown in Table 3. Factor loadings of the retained items ranged from 0.50 to 0.90.

Table 3.

Results of exploratory factor analysis and internal consistency analysis of the CMRGP (n = 132) (2017)

	Number of retained items/original items	Factor loadings on the component	Item-total correlations	Cronbach’s α
Basic medical services	11/12	0.50–0.78	0.70–0.86	0.95
Basic public health services	7/10	0.53–0.73	0.64–0.82	0.92
Communication	8/8	0.51–0.66	0.70–0.87	0.95
Professionalism	6/8	0.63–0.90	0.70–0.85	0.91
Knowledge application and self-improvement	9/8	0.51–0.83	0.67–0.84	0.93
Teamwork and information utilization	6/8	0.50–0.79	0.73–0.81	0.92
Total	47			0.980

Reliability

In terms of internal reliability and homogeneity of the CMRGP, the Cronbach’s α was 0.98 (Table 3). High inter-item correlation was found for all new domains (Cronbach’s α ranged from 0.91 to 0.95). The Spearman–Brown coefficient in split-half reliability analysis was 0.99.

Discussion

We developed the CMRGP in this study which can be used to assess the competency of rural GPs in China. This tool was reduced from 10 domains and 77 items in the initial version to 6 domains and 47 items in the final version. There were basic medical services (11 items); basic public health services (7 items); communication (8 items); professionalism (6 items); knowledge application and self-improvement (9 items); teamwork and information utilization (6 items).

In comparison with other competency assessment instruments for GPs in urban areas (31,32), besides the skills related to the daily work of general practice, the CMRGP emphasized comprehensive medical service competencies, communication and self-improvement. As shown in a study conducted in Beijing, GPs in rural areas delivered care for patients of all age groups and the reasons for encounters were distributed in almost all organs systems (33). Therefore, the comprehensiveness of medical services and the improvement of clinical knowledge are essential for rural GPs due to the isolation and limited resources in rural communities (6). Communication skill plays a very important role in medical care (34,35). In the rural areas, the education level of patients is lower than in urban areas, which posed a great challenge to communication. Therefore, rural GPs should pay more attention to communication skill, which can help to foster better relationship with patients and influence individual health outcomes (36).

There are differences between the CMRGP and other competency assessment tools for rural GPs. A study in the US revisited eight dimensions of capabilities for rural and underserved care, including adaptability, agency and courage, resilience, etc. (9). Another study conducted in Japan described 14 competencies of GPs in non-urban areas (10). The CMRGP highlighted additional skills beyond basic medical skills such as public health relevant knowledge and skills. In China, GPs in primary care settings also need to undertake public health tasks besides basic medical services (23). In comparison with other instruments for rural GPs in China (15,16), standard psychometric evaluation methods were used to evaluate the CMRGP, and good validity and reliability were shown in this study. Furthermore, nowadays competency-based medical education has become the prevailing curricular framework, which is an outcome-based approach to the design, assessment and evaluation of a training program using predefined competencies as a framework (37). In this study, the CMRGP was tested to be used in rural GP training program for outcome evaluation, as it was developed in accordance with the content of the ‘3 + 2’ training program in China (23).

Strengths and limitations

This study presents a number of methodological strengths. Evidence of the content and face validity of CMRGP is supported by the development of the conceptual model, systematic review of instruments and documents, two panels of discussion and an iterative process of expert consultation. Views from experts, who were working as managers, educators and GPs in rural general practice, provided comprehensive insights on the competencies for rural GPs, and strengthened deeper elaboration of this certain issue.

Our study has some limitations. The test–retest reliability is an important part of psychometric evaluation, which is not examined due to the difficulty to ask the same group of rural GPs to fill out the questionnaire again. However, the internal consistency reliability and split-half reliability of the instrument were measured, and the results proved its good reliability. Although participants from different areas (managers, educators and rural GPs) provided their views on the competencies of rural GPs, the opinions from patients were not involved in the study. According to Donabedian’s declaration for incorporating patient perception into quality assessment, health care providers should incorporate patient-centred care as an essential component of health care (38). Therefore, the addition of patient opinions may make the competency model more comprehensive. The competencies to distinguish rural GPs with outstanding performance were not identified in the CMRGP. As in the ‘iceberg model’ proposed by Spencer in 1993, competence consists of seven elements: knowledge, skills, behaviour, traits, values, self-concept and motivation. Among them, knowledge, skills and behaviour are the external parts of the ‘iceberg model’, which are necessary parts of competence and easy to master through learning and training. Traits, values, self-concept and motivation are the internal parts, which are important parts to determine the work performance (39). In this study, the CMRGP was used to identify whether the rural GPs are competent for their work, which was the external parts of the ‘iceberg model’. Further study can concentrate on the exploration of the internal parts of rural GPs’ competence and give feedback of outstanding performance.

Implications

The CMRGP was developed to measure the competencies of rural GPs after training and work in rural primary care, which can be used as a tool for competency assessment during work and give direct feedback on their strengths and weaknesses. Furthermore, the CMRGP can also be used for outcome evaluation of rural GP training program and provides a resource to develop training curricula for the next generation of health professionals, as well as refining their continuing education programs.

Conclusion

In this research, the CMRGP has been developed with good reliability and validity, which contains 6 domains with 47 items. It was found to be a useful instrument for measurement of rural GPs’ competencies and outcome evaluation of rural GP residency training program in China.

Click here for additional data file.

Click here for additional data file.