Evaluations of refraction competencies of ophthalmic technicians in Mozambique.

PURPOSE: Ophthalmic technicians (OT) work at health facilities in Mozambique and are trained to provide primary and secondary eye care services including basic refraction. This study was designed to assess OT competence and confidence in refraction, and investigate whether an upskilling programme is effective in developing their competence and confidence at refraction.
METHODS: Thirty-one trainee OTs and 16 qualified OTs were recruited to the study. A background questionnaire was administered to determine the demographic profile of the OTs. A confidence levels questionnaire explored their self-reported skills. Clinical competencies were assessed in relation to knowledge (theory exam) and clinical skills (patient exams). 11 OTs were upskilled and the clinical evaluations carried out post training.
RESULTS: Initial evaluations demonstrated that confidence and competence levels varied depending on the OTs training (location and duration), and their location of work (clinical load, availability of equipment and other eye care personnel). The qualified OTs were more competent than trainee OTs in most of the evaluations. Post upskilling results demonstrated significant positive impact on confidence and competence levels.
CONCLUSION: These evaluations identified factors affecting the refraction competencies of the OTs and demonstrated that upskilling is effective in improving confidence and competence levels for refraction. They demonstrate the need for a refraction competency framework. The overarching aim of this research was to inform the development of a nationwide programme of OT mentoring, upskilling and leading to the establishment of clinical competency standards for the new OT curricula, relevant to the professional demands.

Introduction

Africa is characterised by an extreme paucity of eye care personnel and facilities. Currently, eye care in Africa is provided by a range of professional cadres including ophthalmic nurses, ophthalmic assistants, refractionists, ophthalmic technicians (OT), optometrists and ophthalmologists. VISION 2020 targets suggest each country should aim to achieve a ratio of one eye care personnel who can perform refractions per 50,000 people by 2020. Based on the recommended VISION 2020 figures, Mozambique would need at least 500 trained eye care personnel to serve its population of 25 million by 2020. In Mozambique, in 2011 when this study commenced, there were 13 ophthalmologists and 34 qualified ophthalmic technicians (OT).

The scarcity of health professionals with sufficient training motivated a Ministry of Health (MISAU) decision to start an 18-month competency-based OT specialist-training programme in order to educate additional OTs. 31 students working as nurses or health agents (18-month training) were enrolled on the course. Curricula to train mid-level eye care (MLEP) vary significantly across countries based on local need. The emphasis is on a shorter training duration due to a lack of other trained personnel. The curriculum was adapted from the Institute of Health Science (IHS) OT curriculum from 1996.

The course outcomes are broad with no defined list of refraction competencies. They state that the OT should be competent in managing refractive errors and prescribing spectacles. The course syllabus for refractive error includes a module on refractive error (diagnosing myopia, hyperopia, astigmatism, aphakia, anisometropia and presbopia; clinical refraction including retinoscopy and subjective refraction; and prescribing glasses for children) and anomalies of accommodation (accommodation insufficiency, paralysis, spasm, convergence/accommodation ratio and effects of refractive error on accommodation and convergence). The refraction component consisted of 40 h of theory and 40 h for practical sessions however this was condensed to a three-week period (two weeks theory, one week practical) within the overall curriculum. There was no known validation of the programme. The students completed a portfolio stating they had completed five refractions to make them eligible for the final clinical evaluation. However, competency in refractive error was not assessed during their training. There was no framework in place to evaluate refraction skills. This was the first known evaluation of OTs refraction competencies.

The development of competency-based education for MLEP has been identified as an important component in the solution to avoidable blindness and vision impairment. A competency is defined as the ability to perform the activities within an occupation to the standard expected in employment. Competencies are the skills, attitudes and knowledge required of particular professions.

The functions of MLEP have not been definitively outlined, and the competencies they require to provide high quality comprehensive eye care have not yet been established. To our knowledge, only one study has evaluated and reported on the refraction capabilities of MLEP even though a major part of their responsibility involves refractive care. It appears, therefore, that the refractive competencies of MLEP remain largely undetermined.

A desk review was conducted with reports and direct communication from non-governmental organisations (NGOs) and personnel working in eye care and in training OTs in Mozambique.9, 10 Electronic searches of Google Scholar, ERIC database, HINARI and PubMed were made using the following search terms qualified with “Africa” “sub-Saharan Africa” and “Mozambique”: ophthalmic technician, ophthalmic clinical officer, MLEP and refraction. Other terms searched were optometry and ophthalmic technician in Cuba and competencies and competency-based education in eye health.6, 11 Overall there is little evidence-based information directly related to mid-level eye care personnel in low income countries and inadequate documentation about the refraction skill level of the OTs in Mozambique.

Refraction, even though it is only a minor element of the existing OT curriculum comprises a very significant component of their daily workload. In the Central Hospital of Beira in 2011, out-patient conditions seen by the OTs in order of prevalence, were refractive errors, conjunctivitis, cataract, ocular trauma and glaucoma. Hence, the question arises on the relevance and length of the refraction component in the context of conditions they treat in their day-to-day work. The study is designed based on the fact that in 2011 OTs were the only cadre providing a refraction service. The evaluation was done not in relation to just what they were trained on, but in relation to their practice, where they are expected to cater to refraction needs.

The knowledge and level of refraction skills of the existing OTs in Mozambique is uncertain because the duration, location and specifications of their training are known to be varied. The OTs that are working at present appear unlikely to have uniform skills, and are thought to be deployed in situations with different characteristics, especially with regards to equipment availability. Assessment of their competencies using a competency framework which was validated for MLEP carrying out refraction was deemed appropriate and necessary.Identification of the quantity, distribution and role of OTs in Mozambique, and determination of their refraction skills are important steps to understand gaps in their existing knowledge, and necessary to inform training and professional education.

This study was designed to assess the confidence and competence in refraction of trainee and qualified OTs in Mozambique, and investigates whether an upskilling programme is effective in developing their confidence and competence at refraction. The overarching aim of this research was to inform the development of a nationwide programme of OT mentoring, upskilling and leading to the establishment of clinical competency standards, for the new OT curricula, relevant to the professional demands.

Methods

All 31 trainee and 34 qualified OTs who work at health centers and hospitals providing primary eye care to the populace were invited to participate in the study, which was conducted in collaboration with the Ministry of Health (MISAU) and Universidade de Lúrio in Mozambique. The evaluations took place in ten health facilities (two central, two provincial, one rural hospital and five regional health centres) in the four provinces of Nampula, Sofala, Manica and Inhambane in Mozambique between January and October 2011. All 16 OTs who worked in the study sites and all 31 trainees agreed to participate and were enrolled into the first part of the study evaluating the existing competence and confidence levels of the cadre. This represented a 100% participation rate in the study sites. The assessment of the trainees was conducted in the second semester of their internship, following completion of their theoretical and practical refraction training, within the respective provinces.

The upskilling training concentrated on theory (two days) and practice (eight days) of objective (retinoscopy) and subjective refraction. It was carried out in two provinces for two weeks each in January and June 2013, on 11 of the original 47 participants, by three experienced optometry lecturers from Unilurio. All the OTs working in the two provinces where the upskilling was carried out were invited with a 100% participation rate in Inhambane (4 OTs) and 78% in Sofala (seven of the nine OTs). The numbers were limited to allow for each optometrist trainer to supervise 3–4 OTs. Retinoscopes, trial lens cases with trial frames and cross-cylinders were obtained for all the participants. Post-upskilling evaluations were carried out on completion of the training period. Informed consent was obtained from each participant, and the research was conducted in accordance with the tenets of the Declaration of Helsinki.

A mixed methods approach was used to assess OT confidence and competence, including questionnaires to assess participant background and confidence in refractive techniques, practical assessments to evaluate clinical competency, and theory examinations to determine underlying theoretical knowledge. They were all administered for the initial evaluations and post upskilling training.

Background questionnaire

OTs were asked to complete a background questionnaire designed to elicit information about their demographic profile. It included their educational background, training history, professional rank, years of working experience within eye care, and workload to get an indication of how their refraction skills vary depending on their training background, location of work and experience.

Confidence levels questionnaire

The confidence levels questionnaire was adapted from a previously piloted and validated questionnaire used in evaluations of MLEP in India. The OTs were asked to rate their confidence in performing 13 different refraction tasks. A five-point Likert scale was used to assess confidence (1 – never confident, 2 – sometimes confident, 3 – not sure, 4 – almost always confident and 5 – always confident).

Practical assessment

Methods of assessing performance were reviewed in order to identify the performance indicators and develop the criteria required for the assessment. Two frameworks of evaluations have been used in this study:

A competency-based theoretical framework: The refraction competencies for this study were based on a curriculum designed to train nurses or those with a suitable non-nursing tertiary education, into MLEP. This competency framework was validated prior to implementation as being appropriate to meeting educational standards of MLEP in Papua New Guinea.

An expert practitioner approach: Various qualitative and quantitative methods have been reported in the literature for assessing competence in clinical practice that include direct observation of patient–provider interactions, record abstraction, clinical vignettes, standardised patients and interviews with patients and providers. Direct observation by an expert optometrist using actual clinic patients was preferred for this as the refraction competencies of OTs were entirely unknown. To assess refraction competence it was essential to have a competent clinician from a highly trained eye care cadre. In the absence of any local experts, the selection criteria for the expert were clinical optometry experience, ability to communicate in Portuguese, familiarity of the health context and availability for placement in Mozambique. The researcher with 10 years clinical and public health experience in optometry, and experience in the training and evaluation of pre-registration optometry students in the UK, met the criteria to carry out the evaluation of the OTs.

Practical competency assessments were carried out as each OT performed refractions on two clinic patients. Retinoscopes and cross-cylinders were available for all the evaluations. To establish a consistent standard and to maintain fair and accurate practical assessments in certifying the OTs and trainees as competent in refraction, a practical exam guide and checklist was used. The checklist was designed to uniformly assess the student on each aspect of refraction including; case history, measuring visual acuity (VA), retinoscopy, best vision sphere, cross cylinder refraction, +1 blur test, binocular balance, near refraction, and determining final prescription. Competency for near refraction was assessed on the OT's ability to calculate the best addition in order to see N8 as per the patients distance mean spherical equivalent, age and visual symptoms.

For each procedure the knowledge, set-up, communication, procedure steps, record keeping and accuracy were graded. Accuracy for both retinoscopy and sphero-cylindrical refraction was examined. The Dreyfus and Dreyfus model of skill acquisition was selected as it could provide suitable anchors for the skill scale and relate to the changes expressed by the OTs. In this model the practitioner moves through five stages: 1 – novice; 2 – advanced beginner; 3 – competent; 4 – proficient; 5 – expert. In order to obtain an overall pass for the practical assessment, OTs were required to demonstrate satisfactory competence (skill was well developed, they can work unsupervised and address complications) in each component of the refraction procedure. For this evaluation the focus was on whether the OT could demonstrate competence rather than proficiency, and whether upskilling moved the OTs from novice to advanced beginners or competent. Hence only grades 1, 2 and 3 were used.

Knowledge of refraction exam

A theory exam was conducted for each participant to assess their refraction-related knowledge. The exam was set by external examiners working in the refraction training of MLEP and comprised technical questions on the theoretical aspects of refraction, patient management and specific refraction tasks. Each section comprised a checklist of items that covered the competencies, with marks associated with each item. Participants were graded by the assessor as competent or not based on a pass mark of 50% for each section.

The competency in practical assessment was combined with the information gathered in the knowledge of refraction exam to inform a judgement as to participant ability to apply their knowledge in the workplace. Only those OTs who demonstrated competence in both were graded as competent overall in that discipline.

Poorly equipped eye departments can account for a lack of confidence amongst eye care personnel in performing ophthalmic assessment. As a means to explain potential confidence and competency deficiencies amongst participants in different clinical settings, an analysis of equipment available at each health facility, including an inventory and an assessment of the functionality of the equipment was also performed.

Data was inputted into SPSS (Version 21.0). Reliability of the confidence levels survey and competence assessment was measured by Cronbach's alpha. Because of the small sample size, non-parametric tests were used. Kruskal–Wallis test was used to investigate relationships between the confidence and competence levels and the different training institutions and OTs location of work. The Mann–Whitney U test was used to compare results between the trainees and qualified OTs pre-training. Wilcoxon signed ranks test was applied to compare the results of confidence and competence levels pre- and post training. A p-value of <0.05 was considered statistically significant.

Results

OT background

The qualified OTs were trained at different institutions across three different countries. Seven (43.8%) studied in Cuba on a three-year training programme, two in Malawi (12.5%) on a one-year training programme, and seven (43.8%) in Mozambique. Of those trained in Mozambique, two completed (12.5%) an 18-month programme, and five (31.3%) completed a two year programme. The refraction component varied across the training programmes, from theory only (Malawi and Mozambique 18 month course) to theoretical and practical training (Mozambique two year and Cuba three year course).

The OTs ranged in age from 37 to 49 years, with a mean ± standard deviation of 43 (±5) years. Their clinical experience averaged 13 (±6.5) years. The average number of daily refractions conducted routinely by participants was 15 (±8), from an average daily total of 25 (±8) patients examined daily in an eye clinic. OTs at central and provincial hospitals conducted more daily refractions (18 ± 8) relative to those at rural hospitals (15) or local health centres (8 ± 2).

The 31 trainee OT participants originated from nine (out of 11) provinces across Mozambique, ranging in age from 26 to 48 years. The mean age was 33.4 (±5.79) years. All trainees were previously qualified as nurses (21) or health agents (10) in Mozambique. The range of years worked within the health system was between 1 and 16 years, with a mean of 7.23 (±3.99) years.

The results of the equipment audit and the eye care personnel working in refraction are presented in Table 1.

Table 1

Situational analysis of refraction equipment and eye care personnel providing refraction at 10 eye clinics.

Location	Number of
	OTs	Ophthalmologists	VA Charts	Trial cases and frames	Retinoscopes	Auto-refractors	Cross-cylinders	Lensmeters
Central and Provincial hospitals
HCN	3	2	6	5	1	1a	0	1a
HCB	2	2	7	3	2	1a	1	1a
HPC	2	1	4	2b	1	1a	0	0
HPI	2	0	2	2	1	1a	0	1a
Rural Hospitals
HRC	2	0	3	2	0	0	0	0
Health Centres
CSN	1	0	2	1	0	0	0	0
CSD	1	0	1	1	0	0	0	0
CSMunhava	1	0	1	1	0	0	0	0
CSMuhala	1	0	1	1	0	0	0	0
CS25	1	0	1	1b	0	0	0	0

Abbreviations: HCN: Central Hospital of Nampula; HCB: Central Hospital of Beira; HPC: Provincial Hospital of Chimoio; HPI: Provincial Hospital of Inhambane; HRC: Rural Hospital of Chicuque; CSN: Centro de Saude (Health Center) Nhaconjo; CSD: Centro de Saude Dondo; CSMunhava: Centro de Saude Munhava; CSMuhala: Centro de Saude Muhala; CS25: Centro de Saude 25 de Septiembre; VA, visual acuity; OT Ophthalmic technician.

Not calibrated.

One broken trial frame.

Confidence levels survey and competency evaluations

All participants reported they were confident with and were graded competent at history taking and visual acuity at distance and near, measuring the pupillary distance, using a pinhole, using the trial lens set, and referring patients. Low confidence levels were reported for retinoscopy, sphero-cylindrical refraction, binocular balancing and +1.00 blur test. All the qualified OTs and 27 trainees were graded as competent in correcting presbyopia. However, none of the participants were graded competent at retinoscopy, sphero-cylindrical refraction, +1.00 test and binocular balance and determining the final prescription. Only one qualified OT was graded as an advanced beginner (grade 2) at retinoscopy and cross-cylindrical refraction and six as advanced beginners for best sphere and determining the final result. Theory of refractive error, retinoscopy, subjective (sphero-cylindrical refraction and lack of awareness of the concept of accommodation and binocular balancing) emerged as the principal gaps in the theoretical knowledge possessed, by all participants.

The detailed findings of the confidence levels survey and practical competency assessment comparing trainee to qualified OT, their location of work and training are presented in Table 2. Confidence levels are graded from 1 to 5 and competence levels from 1 to 3. The results of the theory exam to determine underlying refraction knowledge are presented in Table 3. The reliability of the confidence levels survey is 0.7 and the practical and theoretical competence assessment 0.76 (Cronbach alpha).

Table 2

Mean self-reported confidence levels (maximum score = 5.00) and competence levels (maximum grade = 3.00) for both OTs and trainees and significance to experience, location of work and training.

Skill	Total n = 47Mean (SD)		Qualified OT n = 16Mean (SD)		Trainee OT n = 31Mean (SD)		Difference between qualified and trainee†p-value		Location of work‡p-value		Location of training‡p-value
	Self-reported confidence	Level of competence	Self-reported confidence	Level of competence	Self-reported confidence	Level of competence	Self-reported confidence	Level of competence	Self-reported confidence	Level of competence	Self-reported confidence	Level of competence
Performing retinoscopy	1.16 (0.38)	1.02 (0.15)	1.44 (0.51)	1.06 (0.25)	1.03 (0.18)	1 (0)	0.00	0.16	0.00	0.24	0.00	0.13
Performing best spherical refraction	3.11 (1.52)	1.13 (0.40)	3.69 (1.66)	1.38 (0.50)	2.81 (1.34)	1 (0)	0.55	0.00	0.08	0.00	0.15	0.00
Performing sphero-cylindrical refraction	1.85 (1.02)	1.02 (0.15)	2.38 (1.5)	1.06 (0.25)	1.58 (0.50)	1 (0)	0.16	0.16	0.05	0.24	0.18	0.13
+1.00 test & binocular balance	1 (0)	1 (0)	1 (0)	1 (0)	1 (0)	1 (0)	1	1	1	1	1	1
Near refraction	5 (0)	2.91 (0.28)	5 (0)	3 (0)	5 (0)	2.87 (0.34)	1	0.14	1	0.53	1	0.61
Determining the final prescription	3.4 (1.6)	1.13 (0.34)	4.31 (1.35)	1.38 (0.50)	2.94 (1.52)	1 (0)	0.00	0.00	0.00	0.00	0.09	0.00

Abbreviations: OT, ophthalmic technician; SD, standard deviation.

Mann–Whitney test.

Kruskal–Wallis test.

p < 0.05.

Table 3

Mean theory exam score OT's and trainees (maximum score = 100%) and significance to experience, location of work and training.

Theory exam	Mean total markn = 47Mean (%) (SD)	Qualified OT: mean total markn = 16Mean (%) (SD)	TraineeOT: mean total markn = 31Mean (%) (SD)	Trainee compared to qualified OT†p-value	Location of work‡p-value	Location of training‡p-value
Refractive errorx/4	86%(19)	86%(16)	85%(22)	0.73	0.12	0.51
Retinoscopyx/3	23%(23)	17%(17)	26%(26)	0.24	0.04	0.55
Subjectivex/6	53%(17)	57%(9)	51%(20)	0.54	0.14	0.54

Abbreviations: OT, ophthalmic technician; SD, standard deviation.

Mann–Whitney test.

Kruskal–Wallis test.

p < 0.05.

Trainee OTs compared to qualified

The qualified OTs expressed greater statistical confidence with performing retinoscopy (p = 0.00) and determining the final prescription (p = 0.00) compared to the trainees. They demonstrated greater statistical competence with best sphere (p = 0.00) and determining the final prescription (p = 0.00). There was no significant difference in knowledge of refraction between the two groups.

Location of work with respect to clinical load, referral structures and equipment

Those who worked in central and provincial hospitals examined a greater number of patients and expressed greater statistically significant confidence for retinoscopy (p = 0.00), sphero-cylindrical refraction (p = 0.05) and determining the final prescription (p = 0.00) than those who worked in rural hospitals or local health centres. They demonstrated greater statistically significant competence for best sphere (p = 0.00), determining the final prescription (p = 0.00), and knowledge in the theory of retinoscopy (theory exam p = 0.04).

Location of training with respect to length and refraction content of course

Those who trained in Cuba stated greater statistical significant confidence levels in retinoscopy (p = 0.00) compared to those who trained in Mozambique and Malawi. They demonstrated statistically significant competence levels in best sphere (p = 0.00) and determining the final prescription (p = 0.00). There was no significant difference in the theory exam.

Upskilling

11 qualified practicing OTs were upskilled. Three of the OTs had been trainees during the first part of these evaluations and had since been working a year in practice. The age range of the 22 patients refracted during the evaluations was between four to 62 years (mean ± SD, 40.8 ± 18.7 years). Refractive error measurements ranged from −7.00 D to +6.00 D (mean ± SD, −0.26 ± 3.32 D).

Post-upskilling, all demonstrated a statistically significant improvement in both the confidence levels and competence for performing retinoscopy (p = 0.00), sphero-cylindrical refraction (p = 0.01 for confidence and p = 0.00 for competence) and +1.00 blur test and binocular balance (p = 0.00) and higher competence levels in best vision sphere (p = 0.00) and determining the final prescription (p = 0.00).

In the competency of ‘determining the final prescription’, pre-training mean confidence level was high but mean competency level low. Post-training, a significant improvement in the mean competency level was observed, indicating a good response to the practical training. Theoretical knowledge of refractive error was good pre-training and did not improve, but the knowledge of the skills required to correct it, retinoscopy (p = 0.00) and subjective refraction (p = 0.01), did improve significantly. There was no significant relationship between the location of training and location of work with the post upskilling confidence and competence level scores.

Overall, post training, all of the participants were competent at history taking, measurement of VA and correcting presbyopia. Three OTs were graded as competent at retinoscopy, four at best vision sphere and determining the final prescription, and one at cross-cylindrical refraction and binocular balancing. Only one OT failed the theory exam for retinoscopy. All the others passed for all the modules. The results of the evaluation post upskilling are as demonstrated in Table 4.

Table 4

Mean score qualified OTs pre- and post upskilling and significance of upskilling.

Skill	Mean (SD)Pre-trainingn = 47	Mean (SD)Post-trainingn = 11	p-value†
Confidence levels questionnaire (maximum value 5)
Performing retinoscopy	1.16 (0.38)	2.27 (0.91)	0.00
Performing best spherical refraction	3.11 (1.52)	3.91 (1.3)	0.12
Performing sphero-cylindrical refraction	1.85 (1.02)	2.82 (1.17)	0.01
+1.00 test and binocular balance	1 (0)	2.27 (0.91)	0.00
Near addition	5 (0)	5 (0)	1.0
Determining the final prescription	3.4 (1.6)	3.64 (1.36)	0.66
Practical competency assessment (maximum grade 3)
Retinoscopy	1.02 (0.15)	2.09 (0.71)	0.00
Best vision sphere	1.13 (0.40)	2.27 (0.65)	0.00
Sphero-cylindrical refraction	1.02 (0.15)	1.82 (0.60)	0.00
+1.00 test and binocular balance	1 (0)	1.82 (0.60)	0.00
Near refraction	2.91 (0.28)	3 (0)	0.32
Determining the final prescription	1.13 (0.34)	2.18 (0.75)	0.00
Theory exam (Maximum value 100%)
Refractive error	86% (19)	88% (16)	0.99
Retinoscopy	23% (23)	80% (25)	0.00
Subjective	53% (17)	71% (17)	0.01

Abbreviations: OT, ophthalmic technician; SD, standard deviation.

Wilcoxon signed ranks test.

p < 0.05.

Discussion

This is the first study to report on the refraction capabilities of MLEP in Africa, even though a major part of their responsibilities involves refractive care. Overall, the existing level of performance of the OTs is not sufficient for provision of adequate refraction services. A clinical evaluation of refractive error comprises of two key criteria, objective refraction (which requires minimal participation from the patient) and subjective refraction (based on the patient's feedback on different trial lenses). None of the participants in the original evaluations reached competency standards in objective (retinoscopy) or subjective (cross-cylindrical refraction and binocular balance) refraction and determining the final prescription and therefore needed upskilling. The upskilling demonstrates that the extra two week training was effective in developing OTs confidence and competence in both objective and subjective refraction.

The skills of both the qualified and trainee OTs appear limited to the most basic components of spectacle refraction, and fail to extend beyond the simplest of spherical or presbyopic errors. This presents a significant restriction on the capacity of the current refractive error service providers to detect and manage everyday refractive problems such as astigmatism. The lack of awareness of methods to control accommodation during the refractive examination, further erode the possibility that OTs might determine an accurate prescription and thereby appropriately manage presenting refractive complaints. The lack of competency in objective methods represents another important limitation, which is likely to pose difficulties in basic cases, and especially in more complex refractions, such as those involving paediatric and low vision cases, or in those with communication issues.

The results can be explained by a number of critical factors including inadequate refraction training and practical experience, and location of work with regards to poor equipment availability, peer support and clinical load. The relatively small sample size, however, limits the generalisability of these factors. Further research is necessary for them to be representative of all OTs.

Cuban trained OTs displayed a more comprehensive refraction skill set compared to those who trained in Mozambique and Malawi. This could be accounted for the longer training time of their course (3 years) compared to the others and a greater focus on aspects of refraction compared to the Malawi (Kaphle D, written communication, November 24th 2014) and Mozambique training programmes.

Confidence levels and competence are also related to the location of work with regards to clinical load, supervision and referral structures and the range of equipment available. Mozambique has three central hospitals in Maputo, Beira and Nampula. Every province also has a main referral provincial hospital. The background questionnaire and situational analysis of refraction equipment and eye care personnel revealed that those working in local health centres worked on their own without any peer support or supervision from other OTs or ophthalmologists, and with very little refraction equipment beyond the basic acuity chart and trial frame and case. They saw a lower number of patients due to lower catchment areas where they worked. The lack of experience, existing support structures, refraction equipment and reduced capacity to use the equipment is reflected in their poorer performance in competence in the majority of refraction related tasks. If the minimum desirable equipment are not available in the hospital or health centre, then competencies achieved in training will quickly be lost, and the effectiveness of training eroded over time. It is essential; therefore, that equipment availability is extended to include at least the equipment mentioned in Table 1 (visual acuity chart, trial case and frame, retinoscope, cross-cylinders and lensmeter) for training purposes, and in the health centres and hospitals to which OTs are posted for clinical practice.

For current OT trainees the three week training programme is insufficient to develop competence in anything apart from managing presbyopia. Qualified OTs demonstrated greater degree of competence in the clinical skills compared to the trainees due to their increased practical experience. A mid-level course training nurses to be competent in refraction in a similar context in the Gambia is of six month duration.

On completion of the two week upskilling course, the overall results demonstrated a significant improvement in confidence levels and competence levels of the clinical skills that were practiced, irrespective of the original training and location of work of the OT. Only 11 OTs completed the training, hence the reduced sample size limits the weight of the evidence. It was recognised that a two week upskilling course was brief and there still remains a significant gap between the confidence and competence of the OTs and that required for a fully competent MLEP providing refraction services.

The combination of self assessment (confidence levels) and objective assessment provided a comprehensive methodology for evaluation of the OTs. Reliability analysis showed good reliability of the confidence level survey (Cronbach's α is 0.7), practical and theoretical competency assessment (Cronbach's α is 0.76). However, there are several factors that need to be considered in assessing the implications of our study.

Direct observation of practice, by an optometrist, in the assessment of refraction performance of MLEP has been documented. However, this methodology is increasingly challenged on the grounds of authenticity and unreliability due to examiner related bias. This relates to variations in expert observer judgement where one person's judgement in the assessment of clinical skills, without explicit criteria and training, is likely to be subjectively biased. Providing the examiner with a standardised checklist increased the reliability. The other factors to take into account with direct observation are patient variance and the ‘Hawthorne effect’ where practitioners will behave differently when being observed. The only way of controlling the Hawthorne effect is using unannounced standardised patients for clinical competency evaluations. In this study using standardised patients could not be justified due to high costs of training to ensure reproducibility and consistency of scenarios. Further research on what the most practical, cost-effective and acceptable options for competency assessment are in a low-resource environment is required.

In this study, the setting of practical competency standards was from a literature review. The expert optometrist's refraction results are considered as the benchmark. A variance of ±0.50 D spherical equivalent was set as the limit of acceptability for retinoscopy and subjective refraction, according to accepted norms. The theory exam was marked in terms of content and difficulty. Absolute standards are most appropriate for tests of competence, where the purpose is to establish that the examinees know enough for a particular purpose. However, in the long run a more systemic approach to standard setting needs to be adapted. Standards should be consistent with the purpose of the test and based on expert judgement informed by data about examinee performance.

In the absence of an accurate objective refraction, the competency for near vision was assessed to the nearest 0.25 dioptre in order to see N8. In rural areas of low-income regions, the prevalence of presbyopia is high and the most common reason for requiring spectacles. Due to the lack of personnel refracting in Mozambique, the level of accuracy for presbyopia competence for this study has been redefined. Moreover, dispensing skills were not assessed as the spectacle supply system is limited apart from in the larger central and provincial hospitals.

Confidence in clinical skills has been used as a subjective indicator of clinical competence. However due to the mixed scales used in this study there was inadequate information to examine a relationship between perceived (confidence level) and actual (competence) refraction skill. For this study, the confidence level questionnaire was principally designed to help the OTs analyse their work practice and identify their upskilling needs. Further research is needed using best practice templates for studying self-assessment and its relationship to competence for OTs.

The other limitation to this study was the delay of over a year between the original evaluations and upskilling due to time limitations. This raises the question to what extent upskilling actually improved competence and confidence. The results from the original evaluations were used as pre-training ratings. There is evidence that confidence and competence improve with increased experience as the OT is able to practice and become more comfortable with the skill. Moreover the improved competence post-upskilling could be due to memory retention and recall bias rather than improved understanding. A continuing education, monitoring and evaluation framework has been recommended to ensure that OTs become competent and maintain competence in performing accurate refractions leading to better eye care.

This evaluation and training arose out of a perceived need to provide a better refraction service in Mozambique because of the absence of any other eye care personnel. The study met its objectives of assessing the confidence and competence in refraction of trainee and qualified OTs in Mozambique and identified factors affecting their ability to provide a comprehensive refraction service. The goal of eliminating or reducing the global and regional burden of visual impairment can be assisted by adequately trained MLEP, but only by those who are competent, to fulfil the eye care needs of the population. Future training for OTs needs to be robust and standardised. The results from these evaluations demonstrate the need for a defined list of competencies for refraction to inform the design of new curriculum and upskill existing OTs. Only when OTs become competent in the provision of refractive services can they address the significant issue of avoidable visual impairment as a consequence of uncorrected refractive error in Mozambique.

Conflicts of interest

The authors have no conflicts of interest to declare.