Recommendation and protocol compliance: "Yes, I do" may not be true; the complexity of measuring provider adherence.

The current COVID-19 pandemic might instigate researchers to evaluate healthcare professionals’ adherence to institutional, regional, or international guidelines (Jansson et al., 2020). Others might be interested in the extent to which guidelines are followed in their unit or hospital, or on a broader scale.

Mapping provider adherence is indeed necessary and important. Together with provider competence and treatment differentiation, provider adherence reflects treatment integrity, i.e. the degree to which a treatment is implemented as intended (Perepletchikova and Kazdin, 2005). As such, adherence rates considerably contribute to quantifying the quality of patient care (Milchak et al., 2004). In certain contexts, such as adherence to personal protective equipment use and sharps handling guidelines, they also partially reflect the extent to which healthcare professionals work safely. Besides, adherence rates provide baseline data for setting up tailored improvement projects and educational programmes in line with detected needs.

Obtaining correct and reliable data on provider adherence is however no easy task, and several methodological challenges can cross the researcher’s path. Besides the World Health Organisation’s (WHO) recommendations for measuring hand hygiene compliance (World Health Organisation, 2009b), utmost few resources are however available that provide specific methodological guidance. The aim of this manuscript is therefore to provide a non-exhaustive overview of methodological considerations for all nurses interested in measuring guideline adherence.

The term ‘provider adherence' refers to the extent to which healthcare professionals follow evidence based recommendations for patient treatment and care (Beidas et al., 2013). The corresponding Medical Subject Heading ‘guideline adherence’ was introduced in 1998 as ‘conformity in fulfilling or following official, recognized, or institutional requirements, guidelines, recommendations, protocols, pathways, or other standards’ (U.S. National Library of Medicine, 1998). In the literature on patient adherence with prescribed therapies the term ‘compliance’ has been abandoned because of its paternalistic connotation (Perepletchikova and Kazdin, 2005). This is however less true in the area of provider adherence where compliance and adherence are used as synonyms.

Sampling considerations

As in any research project, a first step in measuring adherence is to clearly define the population of interest and to determine whether probability or non-probability sampling techniques will be used. Next, careful consideration should be given to the inclusion criteria. Temporary workers, members of the in-hospital mobile team, those who newly joined the unit, and those who have recently returned from maternity or long-term sick leave might not, not yet, or no longer be fully familiar with recommendations of interest. The researcher might choose to exclude these individuals, or contrarily, include all staff and statistically analyse adherence rates according to specific demographic characteristics. This also implies that the participants’ characteristics that will be gathered will need to be considered thoughtfully and all exclusion criteria motivated.

Reducing the risk of bias

Irrespective of the sampling and data collection methods (see below) that are chosen, a number of conditions must be met prior to the actual measurement in order to reduce the risk of obtaining biased results. These conditions are valid for any research, but need specific attention in case of conducting multi-unit or multicenter trials that include settings of which the researcher is not fully familiar with the organisational context and the available resources:

(a) all resources needed to adhere to the recommendation of interest should be checked for availability. Stockouts or stock shortages may temporary impede adherence, and in under-resourced countries certain materials may just be permanently unavailable;

(b) recommendations of interest should be checked for conformity with institutional or local recommendations. A typical example of contradicting recommendations is that the WHO recommends the use of examination gloves for open endotracheal suctioning (World Health Organisation, 2009a) while various institutional protocols instruct to use sterile gloves (Day et al., 2002);

(c) recommendations of interest should be checked for being the subject of an ongoing or recent study or quality improvement project. Temporary increased attention to a guideline can result in temporary increased adherence, and as such not reflect usual practice.

Risks of bias induced by the above or other potentially confounding factors should be taken into account when interpreting the results, and acknowledged in any report resulting from the research.

Operationalising adherence

Adherence is rarely a dichotomy. Although full guideline adherence represents the ideal situation, it seldom represents reality. Healthcare professionals’ behaviour can deviate from the recommendations in various ways: they can never follow a recommendation (e.g. because they do not agree with it); follow a recommendation for a specific category of patients only (e.g. only for immunocompromised patients) or only in certain circumstances (e.g. in case of adequate staffing levels only). Besides, they can adhere only partially (e.g. wearing sterile gowns, sterile gloves, caps and masks covering both mouth and nose, but not using full-body patient drapes when inserting central venous catheters), and this again in certain patient populations or circumstances only. To deal with this complexity, adherence is preferable expressed as a percentage that can be calculated using the formula: (observed adherence /number of opportunities to adhere)*100 (World Health Organisation, 2009b).

This preferred way of operationalising implies that data collection relying on self-reporting questionnaires requires a measurement instrument that allows reporting percentages. For example, Likert scale categories should not only be expressed as worded frequencies but should also mention the corresponding percentages, e.g. always (100%) – mostly (80% to 99%). The rationale for the specific numerical translation of the verbal frequencies should be mentioned.

Lastly, the question remains which percentage of adherence is clinically relevant and/or represents appropriate patient care. As an example in the field of adhering to hand hygiene guidelines, even mathematical modelling did not succeed in determining the level of compliance that is necessary to halt transmission of healthcare-associated pathogens (Bonten et al., 2001). In the literature on patient adherence, a 80% threshold was set in the 1980 s (Haynes et al., 1980) and has not been rejected nor confirmed to date (Baumgartner et al., 2018). Although related, patient adherence and provider compliance are however different areas of research, so it would be unwise to simply extrapolate findings from one domain to the other. Researchers eager to setting thresholds will need to determine which cut-off they consider relevant for their specific field of investigation and motivate their choice.

Measuring methods

Various methods can be used to measure provider adherence, of which each has specific advantages and disadvantages, and underlying assumptions. These methods are described below and summarised in Table 1 . It is however important to realise that behaviour may change over time, and that each of the methods presented below will result in adherence rates that are valid for the research period only.

Table 1

Methods for measuring provider adherence, advantages, disadvantages and assumptions.

Method	Advantages	Disadvantages	Assumptions
Direct observation	Gold standard	Time- and resources-consuming	Observers are well-trained
	Most accurate data	Prone to observer and selection bias, and to observation bias if not concealed	The opportunities for adherence and actions required to meet adherence criteria are clearly predefined
		Short-period observations may be biased by random confounding	Ethical clearance explicitly states permission to use concealment
Self-Report	Inexpensive	Subjective	The measuring instrument is reliable and validated
	Easy to quickly collect a large sample	Prone to social desirability bias / overestimation of true compliance	The measuring instrument allows to calculate adherence rates
			Negative consequences of reporting nonadherence are guaranteed to be absent
Electronic Monitoring	Objective data	Expensive in purchase, installation,and maintenance	An appropriate washout period for novelty effect is observed if the device is recently installed
	Allows to monitor changes over time	Prone to failure	Potential Hawthorne-like effect is accounted for
			All ethical and legal requirements re tracking of individual activity are met
Documentation review	Easy and inexpensive	Indirect data	All ethical and legal requirements re consulting confidential documents are met
	Can be done independently	Not able to control the quality of the documents
		May be time-consuming
Proxies	Easy and inexpensive	Indirect data	There is a direct link between use or application of the proxy and adherence
		Proxy use may not reflect adherence
		May be time-consuming

Direct observation

Direct observation generates the most accurate findings and is considered the gold standard. The main drawback is that this method is extremely costly, and time- and labour-intensive (Polit and Beck, 2004, World Health Organisation, 2009b). Direct observation requires well-trained observers, able to correctly count the occurring opportunities for adherence and the proportion of opportunities being met. Clearly predefining these opportunities is an important prerequisite, as is defining all actions needed to be taken to be considered compliant.

Direct observation is prone to inducing observer and selection bias (World Health Organisation, 2009b), of which the risk may be reduced by thorough observer training and instructions. Observation bias, in turn, refers to the tendency of people to behave differently when they know that they are being observed (Polit and Beck, 2004). This so-called Hawthorne effect can be countered by using observation methods in which researcher identity is concealed. Concealed observation has however long been rousing debate on ethics in terms of deceit and the lack of informed consent from participants (Bulmer, 1982). The researcher will need to explicitly motivate the choice for this methodology when applying for approval from any ethics or institutional review board. This definitely also applies to observation of participants’ behaviour by means of concealed video recording equipment, which additionally requires compliance with all legal requirements, including privacy legislation (Asan and Montague, 2014).

Finally, one-time or short-period observation may provide results that are biased by random confounding factors, such as extremely low staffing levels or extremely high patient turn-over on the single observation day. Therefore, it can be recommended to observe over an extended time period or to repeat the observations at intervals. The frequency of opportunities for adhering to the recommendation of interest will need to be taken into account when determining the length of the observation period.

Self-reporting techniques

Self-reported adherence rates can be obtained by questionnaire or interview, which are inexpensive techniques. Questionnaires moreover allow to be distributed easily and quickly to a large amount of individuals. However, these methods generate subjective data and have long been recognised as being prone to result in overestimated adherence rates (Adams et al., 1999). Participants may report in a socially desirable manner, whether or not out of distrust of the purpose of the study or out of fear that individual results might be reported to their superiors. To minimise the risk of such bias, the researcher is advised to communicate that truthful reporting is of key value to the research project, as well as to assure that no individual or team results will be shared with management staff and that full participant anonymity is guaranteed. Besides, the instrument to measure adherence should be reliable and validated, and is preferably tested in a pilot study. Techniques for appropriate survey and interview design and methods are beyond the focus of this manuscript and extensively described elsewhere (Oppenheim, 1992, Polit and Beck, 2004).

Electronic monitoring

Evolving technologies have led to the development of automated devices to monitor adherence (DeMellow and Kim, 2018), particularly to hand hygiene recommendations. These devices electronically monitor sink, and soap or handrub dispenser use, resulting in insight into patterns of hand hygiene frequency and changes in adherence over time (World Health Organisation, 2009b). As major drawbacks, these systems are expensive in terms of purchase, installation and maintenance, and may be prone to malfunction. If participants are aware that the equipment monitors adherence, bias may be induced through a Hawthorne-like effect. Additionally, a novelty effect may temporarily increase compliance. To avoid this effect, researchers have observed a three-week washout period after activating such device before starting data collection (Morgan et al., 2012). In the literature on promoting patient adherence to medication regimes, the novelty effect of electronic monitoring devices has however been suggested to be 35 to 40 days (Williams et al., 2013).

If the device uses concealed monitoring and/or video recording of the participant, the same ethical and legal requirements are valid as described above for concealed and video-based observation.

Documentation review

Patient charts, medical files, administrative data, and bundle compliance sheets can be retrospectively checked to extract adherence rates. Documents may however be incomplete or flawed by both underreporting and overreporting. In spite of its ease of use, this method produces indirect data that rarely can be checked for accuracy and reliability. As these resources hold confidential information, it is moreover paramount to respect the applicable legislation on patient privacy in addition to the standard consideration of research ethics when considering their use (Saranto and Kinnunen, 2009).

Proxies for provider adherence

Indirect data can also be gathered from proxies for adherence, e.g. the number of gloves used during a procedure, of kits for oral care utilised during 24 h, or of patients installed in semi-fowler position at the end of a shift. Opening a package or disposing of products however does not directly imply that these have actually been used. Materials can be disposed of unused due to breaches in integrity of the package or for a variety of other reasons. Therefore, caution is warranted when using material proxies. Although knowledge about the recommendation of interest is a primordial prerequisite for adherence, it can unfortunately not be considered a proxy to date. The gap between knowledge and daily practice still appears to be wide, moreover taking into account that it has usually been measured by self-reports and as such prone to overestimation (Jansson et al., 2018, Madhuvu et al., 2020).

Combining methods

Combining two or more of the methods presented above has advantages. Method #2 might uncover information that was missed by method #1, or, method #2 may act as a reliability check for the data generated by using method #1, and vice versa. This is clearly illustrated by Al-Wazzan and colleagues, who assessed nurses’ hand hygiene practices simultaneously by direct observation and self-report, and found significantly different adherence rates when comparing the results (direct observation: 33.4% versus self-report: 73.8%; p < 0.001) (Al-Wazzan et al., 2011).

In conclusion, measuring guideline adherence is an important but complex activity for which little specific methodological guidance is available. Various measuring methods can be used that all have inherent advantages and disadvantages. Researchers interested in mapping adherence are advised to thoroughly consider the method that is most appropriate for their specific area of research and in accordance with their availability of resources. They are advised to motivate the chosen approach and will need to report the potential risks for bias.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.