Glucose point-of-care meter operators competency: An assessment checklist.

BACKGROUND AND OBJECTIVES: Glucose point-of-care testing meters are essential technology ubiquitous in hospitals. They are operated by non-specialized staff who are assessed through an auto-recertification process that is dependent on operators successfully producing expected outcomes. Alternatively, we suggest that operator practices be directly observed using a competency assessment checklist.
METHOD: We designed a checklist based on literature and manufacturers' instructions and tested it by observing 30 operators at two sites (three hospitals) over two months in 2018.
RESULTS: Despite all operators being auto-recertified, the checklist revealed that only 20% met the 80% threshold of compliance to standards. Moreover, the site with a POCT coordinator had a compliance rate of 82% versus 67% for the site that did not. DISCUSSION: The checklist is more reliable than auto-recertification in assessing operators' competence. It also highlights areas for process improvement and provides an opportunity to give personalized feedback to operators.

Introduction

Hospitalized diabetic patients who experience episodes of hypoglycemia and/or hyperglycemia have poorer health outcomes [1], increased mortality [2] and increased lengths of stay [3]. Early identification, effective treatment and glucose monitoring of diabetic patients can help reduce these outcomes [1]. Failure to provide immediate results impedes physician decision making, negatively impacts patient quality of care, and results in greater healthcare costs associated with the management of hypo- or hyperglycemic episodes [4]. Fast and accurate results are especially critical for diabetic patients requiring blood glucose control within narrow therapeutic ranges [5]. Plasma glucose testing traditionally performed in the central laboratory has a turnaround time of 30–60 ​min; from the time of a clinician’s order to the review of a result [6].

As an alternative, glucose point-of-care testing (POCT) allows for testing to be performed at or near the patient’s bedside, providing an immediate result [4]. POCT is also beneficial in that testing only requires a small volume of blood from the patient; only 0.3–1 ​μL of blood compared to a volume of 1–3 ​mL for the central laboratory glucose test [4], which is especially important when diabetic patients require multiple blood draws across their hospitalization.

However, one challenge of POCT is that results generated by meter operators are typically less accurate than the central laboratory method because of errors in the pre-analytical and analytical phases of testing, which corresponds to POCT maintenance, Quality Control (QC) testing and the patient testing process [6]. Consequently, the accuracy and precision of a result is sensitive to the variability in skill level of a POCT operator [[7], [8], [9], [10]].

To address these issues, accreditation programs require operators to complete training before being authorized to perform POCT; this commonly involves hands-on training and learning the relevant principles of the test, the importance of quality assurance, how to properly interpret results, and the limitations of the test [10]. Following this initial certification, user competency must be re-assessed periodically to ensure satisfactory levels of competence; with the most common period for recertification being annually [11]. Some accreditation standards also require a quality assurance program to monitor the performance and compliance of operators with policies and procedures [12], but they do not provide specific guidelines on how to meet this standard. For that purpose, many institutions rely on the auto-recertification feature available in most POCT data management systems. To be auto-recertified, meter operators must successfully perform a specified number of QC runs and/or patient tests within a defined period of time [8]. As such, certification does not expire for operators who regularly perform POCT as they can easily meet the minimum criteria for auto-recertification [7]. It remains unclear, however, whether these are adequate measures of compliance with standards of operations.

As an answer to the limitations of auto-recertification, some accreditation programs suggest complementing them by using direct observation of routine patient testing to assess operator competency [13]. The methods for doing so, however, remain fledgling. A study by Tongtoyai, Tientadakul, & Chinswangwatanakul in Bangkok, Thailand described the development and use of two forms, one for on-site inspection and one for staff competency assessment to directly evaluate the implementation of ISO 22870:2006 standards [14]. The observations helped to identify common non-compliance issues, knowledge gaps and areas for improvement. However, no comparable attempt in a North American setting has been published to our knowledge and it remains unclear how this approach compares to other reassessment approaches such as auto-recertification.

Another intervention to ensure effective compliance is to assign a senior level laboratory technologist as the POCT coordinator. The POCT coordinator is responsible for resolving commonly encountered problems associated with instrument maintenance, QC, procedural issues, reporting results, training and they also provide guidance to POCT operators [4]. In one US study, 59% of institutions had a full-time POCT program coordinator [8], suggesting popularity but not unanimity over the value of such a role. It remains unclear whether the guidance provided by a dedicated POCT coordinator has a significant impact on compliance with recognized standard practices versus the alternative of spreading the responsibility among several laboratory technologists trained in POCT.

In line with these uncertainties, this study has two objectives. First, to develop a Competency Assessment Checklist (CAC) to assess compliance in POCT operation procedures with recognized standards, pilot it in a North American context and assess its results in comparison to auto-recertification. Second, to identify organizational factors that may contribute to compliance rates, such as the presence of a dedicated POCT coordinator.

Material and methods

This study was carried out at two hospital organizations using the Abbott FreeStyle Precision Pro® glucose meter. Site A has 318 beds and adopted the glucose meter in October 2017, three months before the study. Site B is composed of two hospitals, forming a single organization with a combined total of 510 beds and has been using the same meter for several years. Although the same equipment and auto-certification feature are used at both sites, the practices, policies and standard operating procedures vary between the two sites.

Site A has a dedicated Senior POCT technologist acting as POCT coordinator responsible for the daily operations of the POCT program. She also conducts informal mini-audits of POCT operators as a complement to the auto-recertification feature of the POCT data management system. She also develops POCT educational materials and provides training at nursing orientations. Site A also has dedicated nursing staff members called Clinical Practice Leaders (CPLs) in charge of education and training for their respective nursing units. The nursing staff at Site A also appeared to have had an existing positive relationship with the POCT technologist that developed from interactions during informal mini-audits in the past.

In contrast to Site A, laboratory charge technologists from Site B are collectively responsible for the daily operations of the POCT program in addition to the daily operations of the clinical laboratory. The POCT program at Site B also did not include random auditing of nursing staff and relied solely on auto-recertification to evaluate operators’ competency. Site B also had dedicated nursing staff in charge of education and training, called Clinical Resource Leaders (CRLs). Table 1 summarizes the features of each site as it relates to POCT procedures.

Table 1

Comparison of POCT processes and structures between sites.

	Site A	Site B
Number of beds	318	510
Glucose POCT technology	Abbott FreeStyle Precision Pro® glucose meter
Nurses in charge of POCT education and training	✓	✓
Auto-recertification of meter operators	✓	✓
Dedicated POCT technologist	✓	X
Audits of meter operator	✓	X

Glucose POCT results are less accurate compared to the central laboratory method which can be attributed to using different test methods and/or interfering substances from sampling whole blood versus plasma, however literature suggests that accuracy and precision errors can also be attributed to meter operators’ skills [[7], [8], [9], [10]]. To measure these, the Competency Assessment Checklist (CAC) was modelled after the one designed and used in Tongtoyai, Tientadakul, & Chinswangwatanakul [14] but adapted to the organization based on the current standard operating procedures and policies of the organization, in combination with the Operator’s Manual for the Abbott FreeStyle Precision Pro® glucose meter (2012), the package inserts for the Abbott FreeStyle Precision Pro® test strips (2012) and MediSense control solutions, and the 2017 IQMH ISO 151289 Plus™ Point-of-Care Testing Accreditation Requirements.

The items included in the CAC assessed the three phases of the glucose POCT process; pre-examination, examination and post-examination using the following three categories: (1) Patient Identification; (2) Examination, broken down into: a. Preparation, b. Maintenance, c. Quality Control, d. Patient Testing, e. Safety, d. Critical Values, e. Performance, and f. Interfering Substances and Precautions.; and (3) Data Uploading.

The checklist items developed were reviewed by a subject matter expert for content validity and pre-tested on three glucose meter operators. The subject matter expert used in this study was the POCT coordinator from Site A. She has over 10 years of experience working in POCT, and is involved in nurse orientation and training, the development of educational materials, and has an in-depth understanding of the accreditation standards and requirements associated with POCT. The subject matter expert also acted as the assessor conducting the observations.

A pre-test was conducted, involving the evaluation of three glucose meter operators’ competency with the first draft of the CAC and observing how the data collection process unfolded. This led to modifications to the CAC. Some items were never observed during the pre-test and were thus replaced by standardized questions assessing the operators’ knowledge. The pre-test also helped refine the CAC delivery process, as the same assessor was used to ask all of the questions, eliminating any prompting from the assessor to facilitate the meter operator into providing the correct answer, and by streamlining the assessment time from 40-60 ​min to 20–30 ​min.

After these revisions, the final CAC contained 61 assessment items; 44 items of which involved direct observation and 17 items were linked to the operator’s knowledge. See Appendix A for the complete Competency Assessment Checklist.

To select the sample of meter operators for assessment, glucose test volumes from December 2017 were retrieved at the two sites and used to identify nursing units with high-volume POCT activity. For an equal comparison across sites, nursing units with similar patient acuity levels were targeted. All glucose meter operators encountered on the nursing units at the time of the assessor’s visit were included in the sample for assessment; participation was voluntary. In total, 10 certified glucose meter operators (all nurses) were observed at Site A and 20 were observed at Site B (10 from each of the two hospitals) from January to March 2018. One assessor, the subject matter expert, conducted the assessments of meter operators at all sites, with the first author of the paper present during these observations. All glucose meter operators provided verbal consent to be observed and were assessed using the CAC on a one-on-one basis as they performed glucose POCT on a patient. Patients were informed of the study, but consent was not required, as patient information was not collected.

Actual behaviors and responses were not documented, only whether or not the meter operators complied with standards or knew the correct response. The achievement of a competency assessment item was represented as a “yes” answer percentage in a table format in the final analysis, where “yes” indicated staff compliance. To determine the average staff compliance level by site, the “yes” answer percentages from the assessments performed at each site were averaged.

Results

During the pre-test, it was noted that meter operators could not by-pass or perform certain steps out of sequence to what the meter manufacturer recommends, thereby producing mandatory sequential responses from meter operators. For example, the test strips are designed to allow blood to be applied in a top-fill or end-fill manner and will not allow the meter operator to begin testing until sufficient blood is applied (assessment item 39). Another example of this inability to by-pass steps is linked to scanning or entering the meter operator ID number (assessment item 9 and 25); a failure to do this step would lock-out the meter operator. Such steps were necessary in the process of glucose meter testing for the Abbott FreeStyle Precision Pro® glucose meter and make the checklist easier for the assessor to follow. In total, 17 CAC items were mandatory and have been codified with an asterisk (∗).

Table 2 displays the competency assessment items and achievement levels by hospital site. An assessment item was considered non-conformant or deficient in practice if less than 80% of operators achieved that assessment item and are marked in red in the table [14]. Overall, operators met the 80% compliance threshold on 28 items (including the 17 mandatory items marked with an ∗) but failed to meet the threshold on 33 items.

Table 2

Competency assessment item achievement levels.

When we look at the differences between sites, glucose meter operators from Site A achieved an average score of 82% on all items, while operators from Site B achieved 67%.

On an individual basis, only 6 out of 30 (20%) glucose meter operators met the 80% competency threshold. All six meter operators were from Site A (See Table 3).

Table 3

Staff achievement levels.

Staff meeting the 80% threshold (%)
Site A (10 operators)	Site B (20 operators)	Total
60%	0%	20%

The process of observing and assessing POCT operators also led to the identification of process issues in the organization. First, item 2 helped identify that many operators had difficulty scanning the barcoded armbands and resorted to using the admission labels from the patient’s chart. The organization’s policy allows this practice provided that the meter operator uses the admission label to confirm the information on the armband. In the end, operators checked for two patient identifiers (Table 2, Assessment Item 2) only in 75% of cases. The difficulty in scanning barcoded armbands identified during the observations prompted further investigation as to why the meters cannot scan the barcoded armbands.

Second, items 59, 60 and 61 showed issues of double charting. Glucose meters at Site A are not connected with the Electronic Medical Record (EMR) system and results are not automatically uploaded to the patient’s EMR. Meter operators maintained paper charts but also manually added the result to patients’ EMR. Despite not having a direct electronic connection to the patient’s EMR, meter operators knew to dock the glucose meters to transmit QC results to the POCT system (100%), to check the patient’s EMR for the manually entered result (86%) and who to contact in the event a result discrepancy was found (100%) (Table 2, Assessment Items 59, 60 and 61, respectively).

In contrast, at Site B, POCT results are automatically uploaded into the patient’s EMR but meter operators do not regularly check the EMR for the proper documentation of POCT results (75%) and do not know who to contact in the event that a discrepancy is encountered (67%) (Table 2, Assessment Items 60 and 61, respectively). Upon further investigation and discussion with the laboratory director, it was found that many nursing units at Site B still performed paper charting, so the practice of checking a patient’s EMR for proper documentation is not common or mandatory.

Third, users were found to be non-compliant with maintenance and quality control tasks. Basic maintenance of the glucose meter, where the meter operator is expected to clean the surface of the meter with an approved disinfectant was performed poorly at both sites; 60% and 30% at sites A and B, respectively (Table 2, Assessment Item 6). Meter operators at both sites failed to perform the majority of the manual steps involved in QC testing such as checking the QC expiry dates, mixing the solutions, inverting and tapping the bottle to expel air bubbles, and discarding the first drop of QC material before use (Table 2, Assessment Items 10, 16, 17, 18, respectively).

Finally, results in the Critical Value (33%) and Performance (35%) knowledge questions were particularly low, as many nurses failed to know the critical values for glucose and the performance range of the meter. It was found that nurses considered critical ranges to be different according to which doctor he/she reported to or what the patient’s medical condition was, rather than referring to standard practices.

Discussion

Our first objective was to develop and pilot a Competency Assessment Checklist (CAC) to assess compliance with POCT operating procedures using an observational approach and compare the results to auto-recertification. The results support the CAC as an appropriate measure of compliance among POCT operators revealing that POCT operators failed to meet the competency assessment threshold on 33 items. On the contrary, the findings from this study using the CAC suggest that the standard auto-recertification approach to competency assessment does not necessarily confirm a meter operator’s actual competency skills. All meter operators in this study were certified operators according to the auto-recertification feature of the POCT system, but only 20% met the 80% threshold upon observation using the CAC. This points to several limitations associated with using the standard auto-recertification process as a measure of POCT competency.

Auto-recertification does not assess the process of POCT, it only assesses the outcome, whether or not a meter operator is able to generate a patient or a QC result. In contrast, a standardized approach to compliance auditing can provide direct feedback about the actual processes and procedures being practiced by meter operators. Auditing using a standardized site-specific assessment checklist can help identify inadequate processes in an organization, knowledge gaps amongst operators and specific opportunities for improvement in a POCT program, in addition to non-conformances/deficiencies, that auto-recertification fails to reveal.

There is also a formative potential in direct observational assessment of meter operators; this study found that auditing meter operators in real-time provided an opportunity for direct one-on-one feedback. Nurses in general were receptive to the assessments, and this approach generated informal training and education sessions at the nursing stations. To sustain these benefits, it may be worthwhile for healthcare institutions to consider augmenting their current POCT training strategies with small scale direct observational auditing at a frequency that suits their assessment needs.

Our second objective was to identify organizational factors that may contribute to compliance rates. Results show significant differences in compliance rates between sites and between items.

Site A out-performed Site B overall, with a compliance rate of 82% compared to 67% (Table 3). Several reasons may account for the better compliance at Site A. First, the glucose meter had been adopted within the year and staff had received training approximately three months before the observations took place at Site A. With the training being more recent at Site A, operators may have been more likely to remember standard procedures than at Site B, where it had been in place for years.

Second, Site A had a dedicated POCT coordinator to communicate with the POCT operators. The POCT coordinator also performed random auditing of nursing staff. The occasional face-to-face interaction between nursing staff members and the POCT coordinator may have contributed to the development of a strong rapport and understanding about the importance of adhering to the testing steps. During a site visit, the dedicated Senior POCT technologist was observed to be working with the CPLs and operators to create new e-learning materials, performing linearity studies to troubleshoot problematic meters, reviewing POCT data and sending personal emails to each meter operator who failed to properly document meter cleaning (Items #20 and #23). Compliance rates on these items were 90% and 80% versus 70% and 50% at Site B (Table 2, Assessment Items 20 and 23). In contrast, the laboratory charge technologists from Site B were responsible for the daily operations of the POCT program in additional to their clinical laboratory duties, which likely meant that they did have enough time to dedicated to the POCT program. For instance, they relied solely on the auto-recertification feature of the POCT management system to evaluate operators’ competency on an annual basis. The presence of laboratory charge technologists at Site B does not seem to have acted as an equally efficient substitute to the POCT coordinator. The long-term success of a glucose POCT monitoring program requires continued communication and cooperation between physicians and nurses at the patient’s bedside and the clinical biochemists and technologists behind the scenes, in the laboratory [4]. The results of this study suggest that a dedicated POCT technologist acting in the role of a POCT coordinator may be an effective way to support this communication. It may also point to the necessity of high level support for POCT coordinators, since more than half of the respondents in a survey conducted by the POCT interest group of the Canadian Society of Clinical Chemists (CSCC) cited a lack of staff to support their POCT program [8].

Beyond the contribution of a dedicated Senior POCT technologist, using a standardized checklist helped identify several areas for improvements that contributed to some low compliance rates.

As observed in other settings [7], POCT operators may fail to confirm two patient identifiers (Item #2), leading to risks of measuring the wrong patient. Issues with the barcode on the armbands are the likely cause for these issues.

Double charting and using paper charts also emerged as problems (Items #59, #60 and #61). While it is not uncommon for healthcare institutions to use a combination of paper and electronic documentation methods, accreditation standards require all results to be documented in the patient’s medical record if they are to be used for patient management decisions [15]. Paper-based charting can result in 30% of POCT glucose results being transcribed incorrectly [16]. The CAC may help identify problematic and informal practices such as paper charts being maintained instead of electronic charts.

Maintenance and Quality Control issues may generate delays and inefficiencies. First, improperly trained POCT operators may send out devices for repair that do not actually require a repair [9]. Second, devices may automatically lock out operators if scheduled maintenance and QC is not performed (Abbott, 2012), which can lead to a delay in patient testing. Failure to clean the device is a key maintenance issue and it may be widespread; as 25% of meter operators did not follow manufacturers’ instructions with appropriate sanitization prior testing [8], and a study on blood gas POCT devices suggests that cleaning was not performed in 75% of POCT blood gas tests and that operators were unwilling to perform maintenance [17]. Items 6, 10, 16, 17, 18 in the CAC may help proactively identify such operators and fix these problems ahead of time.

Knowledge questions in Critical Value and Performance revealed knowledge gaps. Accreditation standards do not specify that meter operators need to know the critical values of a POCT test because such values are often unreliable and should be retested for confirmation [18].

Item 4 and 5 under the Preparation category, identified another non-conformance: the lack of hand hygiene and donning disposable gloves prior to patient testing. Basic hand hygiene compliance among healthcare workers is a universal problem within many healthcare facilities [19]. Direct observation studies found that hand hygiene compliance in inpatient units and emergency departments prior to patient contact occurs slightly over 80% of the time [20], which compares similarly to the findings of this study where meter operators would perform hand hygiene 70% and 79% of the time at Site A and B, respectively. Similar results for hand hygiene were observed in the Safety category (Table 1, Assessment Item 44), 80% and 68% for Sites A and B, respectively. Direct observation studies found that hand hygiene compliance after patient contact occurs approximately 80% of the time, and after removing gloves, slightly over 95% of the time [20]. An opportunity exists for the organization to improve infection control at the point of care and perhaps during other opportunities of patient interaction. Regular hand hygiene requires “behavioural change, culture change, and training that may take many years and require ongoing actions across all organization levels”[21] (p262).

A possible limitation to this pilot study was the inability to assess the competency skills of meter operators on the night shift. QC runs at the sites were typically performed by night shift nursing staff. It was unclear if nursing staff assessed in this pilot study participated in regular shift rotations and had the opportunity to perform QC testing. Despite this possible limitation, it is still imperative that all meter operators are competent in QC testing as 20% of QC tests are performed by day shift nurses because testing was missed by the night shift nursing staff [9].

Another limitation is the use of the 80% threshold. It is a useful heuristic but it appears arbitrary in the study by Tongtoyai et al. It is recommended that healthcare institutions set their own pass criterion based on performance targets, industry benchmarks research and manufacturer recommendations for optimal meter performance. The Angoff method is commonly used in competency skill testing to set a pass rate, where the pass rate of each assessment differs according to the difficulty level of the test [22]. Given that multi-site healthcare institutions may use a variety of glucose meters, subject matter experts such as clinical biochemists from each institution should review their site-specific assessment checklist and identify mandatory items for competency. Items that are not mandatory for competency may be included to help identify inadequate processes or knowledge gaps.

Also, these study results were only tested on two sites. Nevertheless, results, such as those in the “Interfering Substances and Precautions” category are similar to the results in the study conducted by Tongtoyai et al., 2012, suggesting reliability of the CAC across organizations. The checklist should be adapted and tested in other settings to further confirm both the reliability of the CAC and to assess if the compliance issues identified in the current study are generalizable to other sites. Future studies should consider the validity of each items. For instance, items that cannot be bypassed could be removed to keep the assessment tool lean. This partly depends on the technology used and suggests that compliance could be improved by making more steps required by the POCT system. Moreover, emphasis could be put on items with great performance variability between users such as disinfecting the meter, mixing QC solutions, or wiping away the first drop of blood.

Conclusion

The glucose meter Competency Assessment Checklist developed in this study proved to be a useful tool in the assessment of POCT operators’ competences. It revealed that auto-recertification using a POCT data management system failed to reliably assess meter operator competency. It also helped identify areas for improvement in organizational POCT practices and made it possible to provide useful personalized feedback to POCT operators. The findings of the checklist also suggest that a dedicated POCT coordinator, as well as random auditing of users may help lead to better compliance among operators.

Practical application of direct observation using a CAC is challenging especially in a large facility. However, this tool can be used to target compliance issues at the micro and macro level. POCT programs can use this tool to audit meter operators on a schedule that suits their daily operation and organizational needs. Audits can be performed once a week or as resources permit. Organizations may choose to target specific wards or units with suspected performance issues or more generally by randomly selecting operators from all areas of their institution. On a micro level, the individual assessments will allow meter operators to receive direct one-on-one feedback. Organizations auditing specific units may wish to address common compliance issues collectively with in-person re-training sessions or with online re-training modules. It is recommended that user specific issues be addressed on a one-on-one basis under the discretion of the POCT program administrator. In this study, there was once instance in which we encountered a meter operator that used a co-worker’s access code to perform glucose POCT on patients under his care. The POCT coordinator assessed the meter operator and then followed up with the user by email. The user was provided his own access code after the POCT coordinator was able to locate records of his initial training and confirm that he obtained a passing score on his initial competency test. On a macro level, the cumulative assessment data can provide an overall sense of how well a unit or ward is performing and what areas of the POCT process needs to be modified or studied further. For example, barcode scanning and double charting emerged as problems specific to these two sites which will require further examination. In addition, comparing cumulative data from year-to-year can be used to evaluate a POCT program before and after a major intervention or to measure year-to-year performance or improvement of a POCT program.

Declaration of competing interest

There are no known conflicts of interest associated with this publication and no financial support program for this work could have influenced its outcome.