Developing a medication adherence technologies repository: proposed structure and protocol for an online real-time Delphi study.

INTRODUCTION: An online interactive repository of available medication adherence technologies may facilitate their selection and adoption by different stakeholders. Developing a repository is among the main objectives of the European Network to Advance Best practices and technoLogy on medication adherencE (ENABLE) COST Action (CA19132). However, meeting the needs of diverse stakeholders requires careful consideration of the repository structure. METHODS AND ANALYSIS: A real-time online Delphi study by stakeholders from 39 countries with research, practice, policy, patient representation and technology development backgrounds will be conducted. Eleven ENABLE members from 9 European countries formed an interdisciplinary steering committee to develop the repository structure, prepare study protocol and perform it. Definitions of medication adherence technologies and their attributes were developed iteratively through literature review, discussions within the steering committee and ENABLE Action members, following ontology development recommendations. Three domains (product and provider information (D1), medication adherence descriptors (D2) and evaluation and implementation (D3)) branching in 13 attribute groups are proposed: product and provider information, target use scenarios, target health conditions, medication regimen, medication adherence management components, monitoring/measurement methods and targets, intervention modes of delivery, target behaviour determinants, behaviour change techniques, intervention providers, intervention settings, quality indicators and implementation indicators. Stakeholders will evaluate the proposed definition and attributes' relevance, clarity and completeness and have multiple opportunities to reconsider their evaluations based on aggregated feedback in real-time. Data collection will stop when the predetermined response rate will be achieved. We will quantify agreement and perform analyses of process indicators on the whole sample and per stakeholder group. ETHICS AND DISSEMINATION: Ethical approval for the COST ENABLE activities was granted by the Malaga Regional Research Ethics Committee. The Delphi protocol was considered compliant regarding data protection and security by the Data Protection Officer from University of Basel. Findings from the Delphi study will form the basis for the ENABLE repository structure and related activities. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The diverse expertise and geographical spread of the European Network to Advance Best practices and technoLogy on medication adherencE COST Action members (39 European countries) and their wider professional network represents a unique and timely opportunity to develop a repository of medication adherence technologies that meets the needs of a diverse audience.

The scope and content of the Delphi survey represent the work of extensive literature review combined with multidisciplinary expertise of the steering committee.

The real-time Delphi approach provides improved efficiency of the process, shortens the time of study completion and is particularly suitable for managing larger groups and including people from different geographic locations.

The Delphi study will use state-of-the-art methodology to measure agreement and predetermine agreement/consensus criteria as well as stability of responses.

The real-time approach requires specialised software, which limits the range of possible survey configurations and raw data availability for detailed process analyses and requires relatively elaborate instructions for participants, which may increase participation burden.

Introduction

Taking medication as prescribed often proves difficult for people when managing their health, particularly in the long term.1 Medication adherence is suboptimal in numerous chronic conditions2 3 and has a negative impact on chronic disease management, patient’s general health status, quality of life, working ability and healthcare costs.2 4 5 Research on medication adherence has expanded and contributed to raised awareness of the prevalence of suboptimal adherence and how it affects health outcomes. Digital technologies have increasingly gained interest as new interventions for supporting medication adherence have been developed. A diversity of technologies has been proposed, from electronic monitoring devices to mobile applications, to support medication adherence measurements and empower patients with their disease management. However, the rapidly expanding offer of medication adherence technologies (MATech) makes it increasingly difficult to access, evaluate and compare different technologies to make informed decisions and select appropriate tools for specific clinical or research needs. In a 2018 review by Ahmed et al,6 5881 medication adherence apps were identified on Google Play and Apple App Stores. However, most of them lacked evidence of effectiveness and did not involve healthcare professionals (HCPs) during their development.6 Lack of collaboration between stakeholders results in a limited number of developed MATech actually being implemented into the healthcare systems and used daily by HCPs and/or patients.7 Furthermore, due to differences in healthcare systems across countries, healthcare organisations and reimbursement processes, harmonisation of implementation strategies are lagging behind, which further delays adoption of best practices across countries.4 7

The ENABLE COST Action (‘European Network to Advance Best practices and technoLogy on medication adherencE’, CA19132)8 was initiated by experts in medication adherence and digital technologies to fill these gaps regarding evidence and implementation of MATech within healthcare systems. ENABLE aims to raise awareness of available technologies, expand multidisciplinary knowledge on medication adherence at multiple levels, accelerate knowledge translation to clinical practice and collaborate towards economically viable implementation of best practices and technologies across European healthcare systems. These objectives are being pursued within a 4-year period (2020–2023), by three distinct and inter-related working groups (WGs) that map best practices available (WG1), identify and showcase adherence technologies (WG2) and identify suitable reimbursement strategies for implementation in healthcare systems (WG3), supported transversally by a WG4 coordinating communication and dissemination. At present, the ENABLE Action includes a large interdisciplinary network of experts in medication adherence from 39 European countries.8

Effective implementation of technology-supported healthcare has been facilitated by centralisation of information in public repositories or ‘solution showrooms’, where users can search for technologies that meet their specific requirements.9 Several such repositories already exist in the field of digital health, including medication adherence (eg., NHS app Library,10 MyHealthApps,11 InterventieNet,12 GGD AppStore,13 DIGA,14 Weisse Liste15), but are limited to single countries or types of technology and none represents a comprehensive resource to facilitate adoption of appropriate MATech across health systems. Therefore, ENABLE sets out to develop and maintain a public online repository of MATech where patients, HCPs, researchers and healthcare managers would be able to access and select technologies for adoption in their adherence management activities.8 For example, a patient may be interested more in the practical benefits of using a MATech in their daily lives, while a researcher may be keen to examine in detail the methodology theory and evidence base behind the MATech development. To meet this goal, the ENABLE repository would need to represent a flexible knowledge management system that would include information relevant to the needs of different stakeholders in a user-friendly format. In medical informatics, knowledge management relies on standardised terminologies, classifications and ontologies to record, share and use data on healthcare research and practice. These standards specify the types of information to encode in the form of distinct ‘entities’ representing objects or phenomena in the real world and their properties (‘attributes’), thus enabling knowledge generation through inference and learning.16 Adoption of evidence-based health innovations is also facilitated by these common standards, as new technologies need to interact with existing ecosystems in terms of both data interoperability and communicating with potential users in appropriate domain-specific language.17

The field of medication adherence is highly interdisciplinary, therefore a useful repository would cross multiple knowledge domains and align with several standards, whether medical (eg., WHO International Classification of Disease18), behavioural (eg., the Behaviour Change Intervention Ontology (BCIO)19 20) or technical (eg., WHO Classification of Digital Health Interventions21). Stakeholder involvement would need to be at the core of this development process, to ensure its content is relevant, clear and complete, and meets community needs.22 The diverse and geographically spread ENABLE membership and their wider professional network represents a unique and timely opportunity to conduct this work. Considering these quality standards and following methodological recommendations,22–24 the initial version of the repository structure was prepared. A stakeholder consultation process is proposed to explore their views and level of agreement on the relevance, clarity and completeness of the initial version.22 23 The resulting improved version would represent the structure of the ENABLE repository, which will be tested and populated in subsequent steps with users and developers of available technologies.

The present manuscript describes two elements:

The proposed structure for the repository.

The protocol of the real-time Delphi study to explore stakeholder views on this structure.

Methods and analysis

Steering committee

A steering committee (SC) was established within the COST ENABLE WG2 to coordinate and perform the work. The committee includes 11 ENABLE members from 9 countries in the following areas of expertise: adherence research and education, clinical practice, policy making and technology development. Members are responsible for: (i) determination of the repository scope and framework of attributes defining repository structure, (ii) preparation of the Delphi protocol, (iii) configuration and piloting the Delphi survey, (iv) selection and invitation of stakeholders to participate in the study, (v) moderating study performance via the online tool and (vi) analysis and interpretation of results.

Determining the repository scope and framework of attributes defining its structure

The determination of scope and development of the attributes’ labels with definitions aimed to align with ontology development procedures as described by Wright et al 24 and follow a stakeholder engagement methodology as described by Norris et al 22 and Khodyakov et al.25 The principles of ontology development, actions taken when generating the framework of attributes and examples of how these principles are applied in the ENABLE project are presented in table 1. The stakeholder engagement is primarily achieved through the proposed real-time Delphi study, which is described in more detail in the next sections.

Table 1

Principles of ontology development after Wright et al 24 and actions taken in the ENABLE project

Principles	How they have been applied in the ENABLE project
Have specified scope and scientifically sound and relevant content	Selection of established definitions for delimiting the scope, consultation of stakeholders, piloting for data input and platform search.
Meet the needs of community of users	Consultation of stakeholders, steering committee and Action members sampled from the user community and including diverse areas of expertise.
Enabling users to understand the meaning of entities	Naming examples of existing ontologies, piloting Delphi survey, technology description form, user form and platform use.
Be logically consistent	Using the methodology recommended for attribute description, checking consistency via Ontology Web Language.
Be interoperable with existing ontologies	Adopting attributes and labels available in existing ontologies and classifications, expert input on additional attributes and recommendations for interoperability.
Reflect changes in scientific consensus and remain accurate over time	Repository in open access, sustainability plan developed with Action members and stakeholders.

ENABLE, European Network to Advance Best practices and technoLogy on medication adherencE.

Scope and definition of MATech

Four established definitions were used to define the scope of repository and set the framework of attributes: (i) WHO definition of health technologies 26; (ii) the ABC definition of medication adherence 1; (iii) the WHO definition of adherence to long-term therapies 2 to highlight the importance of shared decision-making between the patient and the healthcare team and (iv) the definition of best practice in healthcare proposed by the European Commission to guide improvements in European health systems.27 The information in this definition denotes evidence on safety, efficacy, effectiveness, cost-effectiveness, appropriateness, social and ethical values and quality of the healthcare interventions.

Therefore, we propose to define MATech as devices, procedures or systems developed based on evidence to support patients to take their medications as agreed with healthcare providers (ie, to initiate, implement and persist with the medication regimen).

Devices, procedures or systems emphasise the inclusion of all technologies, irrespective of their mode of delivery (whether based on electronic or printed supports, delivered through human interaction or a combination of these), with the aim to construct a comprehensive repository in which users can identify diverse technologies to fit their potentially diverse needs.

Developed based on evidence encompass the requirement of evidence/research that supports at least a potential contribution to either measurement or intervention on medication adherence (eg, validation or pilot studies). Thus, technologies that are not (yet) supported by evidence (eg., development and testing stages), or clinical practice protocols without an evidence base on at least one aspect (safety, efficacy, effectiveness, cost-effectiveness, appropriateness, social and ethical values or quality), will not be (yet) included in the repository until such evidence is produced and reported.

Support patients to take their medications as agreed with the healthcare providers (ie, to initiate, implement and persist with the medication regimen) encompass the contribution of the technology to medication adherence management—either directly in patients’ self-management, or by supporting professionals to offer such services to patients through all phases of medication adherence. Thus, technologies that focus on other medication management goals, but do not target adherence specifically would be out of scope for this repository.

Furthermore, the technologies included would need to be described in terms of their technical characteristics and validation, their behaviour change content, format and context, as well as the characteristics facilitating appropriate implementation in care processes. Hence, evidence from behaviour,19 28 implementation29 30 and computer sciences18 21 31 32 informed the initial scope and attributes framework to ensure key features, such as user-centeredness, trustworthiness/credibility, accuracy and relevance of the presented information, tailoring to the needs of different users and interoperability with existing evidence and other sources of information on healthcare technologies.

Framework of attributes

An initial list of attributes was developed based on a literature review and knowledge from the ENABLE members activities such as (i) an ongoing systematic review of e-health interventions on medication adherence for chronic conditions,33 (ii) a checklist of e-health quality criteria under development,34 (iii) Interventienet.nl—platform showcasing evidence-based medication adherence interventions in the Netherlands12 and (iv) the ABC taxonomy—consensus-based terminology and definitions of medication adherence.1

The initial list was presented to the SC and discussed via several videoconferences to generate a more detailed list of attributes grouped on several themes. Each theme was further elaborated by a subgroup of two SC members following a standard format including labels and adherence-related definitions. We adopted the approach from BCIO,19 where related attributes were searched in topic relevant ontologies/taxonomies/classifications and original definitions and codes were added. The reasons for the choice of certain attributes and labels were detailed for each attribute group. The proposed framework of attributes is graphically presented in figure 1 and, while rationale and sources used to define the labels for the MATech repository are presented in table 2 and online supplemental file 2.

Figure 1

The interactive graph showing the framework of attributes for medication adherence technologies (MATech) (‘the MATech tree’). The MATech tree is available as interactive feature in the.

Table 2

The proposed framework of attributes used in the MATech repository

Domain and attribute group	Core question	Rationale	Existing ontology/taxonomy/classification used and adapted
D1 (D1.1) Product and provider information	What product does the entry refer to, who provides it, who entered its description in the repository and when?	Each entry in the ENABLE repository will refer to a unique product, which will be identified with a unique ID, provided by a unique organisation (manufacturer, developer) with its own unique ID and related metadata (eg, date of entry, verification process, etc) to present the identity of the described MATech and its provider.	Ontology for medical technology innovation in healthcare centres by ITEMAS58—only concepts referring to products and their providers were used and adapted.
D2.1 Target use scenario	What use scenarios and types of users is the technology intended for?	We can distinguish two general categories of users and their characteristics that might influence the choice of technology: (i) self-management use (patients and caregivers)—labels describing patients’ characteristics or their condition (age, functional status, (health) literacy, etc); (ii) adherence support use by healthcare or social care providers and health system managers, who can initiate a search for MATech to integrate in their practice. The provider and the setting are also the focus of separate attribute groups.	Systematised Nomenclature of Medicine, Clinical Terms (SNOMED-CT).32 WHO International Classification of Functioning, Disability and Health.59 The WHO DHI.21 ABC Taxonomy.1
D2.2 Target health conditions	Which health conditions could the technology be used for as part of adherence support?	MATech are usually developed and validated to be used in one or several clinical domains and potential users may search for technologies applicable to the health condition(s) they aim to manage. Since our stakeholders also include lay individuals, special focus was put on using simplified language to avoid misunderstandings and knowledge gaps.	The International Classification of Disease 11th revision.18 The Health Research Classification System from the UK clinical research association.60
D2.3 Medication regimen	What type of medication regimen(s) is the technology intended for?	Medication regimen can take different schematic forms and be of varying complexity, which may influence the complexity and extent of medication adherence. MATech may be developed for medications with different characteristics, hence the repository users should be able to indicate the type of regimen to find a MATech that fits its specific characteristics.	SNOMED-CT.32 National Cancer Institute Thesaurus.61 Medical Subject Headings.62
D2.4 Medication adherence management components	What adherence management types and phases does the technology target?	Management of adherence entails two management types, for example, monitoring/measurement (D2.4.2.A) and support/intervention (D2.4.2.B) by any stakeholder, including the patient himself. Both elements may require different approaches depending on the targeted phase of adherence (D2.4.1).	ABC Taxonomy.1
D2.4.2.A Monitoring/Measurement methods and targets	If measurement is a component, what measurement methods does the technology use and what do they measure?	A broad range of measurement methods for adherence are available. In addition to adherence behaviours, measurement can also target adherence determinants, other self-management behaviours and outcome measures (eg, HRQoL). Therefore, we have selected a range of measurement models as well as a selection of self-management behaviours to offer the possibility to describe technologies from a measurement perspective.	SNOMED-CT.32 Extensive existing literature2 3 63 and own (steering committee’s) methodological know-how. Train4Health (T4H) behaviour change competency framework.64 BCIO.19
D2.4.2.B.1 Intervention modes of delivery	If intervention is a component, how is it delivered to its users?	Mode of delivery is ‘physical or informational medium through which a given behaviour change intervention is provided’,19 can affect intervention effectiveness. Although digitalisation has entered in all aspects of everyday life, the analogue mode is still very relevant. This is especially true within the elderly, who on one hand require more support in medication adherence65 and are on the other hand less digitally literate.66 Hence, the repository should encompass all modes.	BCIO.19 specifically a taxonomy of modes of delivery of BCI.67
D2.4.2.B.2 Target behaviour determinants	If intervention is a component, what reasons for non-adherence can the technology help address?	The MATech can address different reasons for non-adherence, defined as determinants of behaviour, which can be non-modifiable or modifiable.2 19 68 Individual-level and modifiable determinants are encompassed as capability (psychological and physical), opportunity (social and physical) and motivation (reflective and automatic), also known as the COM-B model.69	COM-B model and Behaviour Change Wheel.69 Theoretical Domains Framework.70 BCIO,19 specifically The Mechanisms of Action Ontology.71 72 International Classification of Health Interventions.31
D2.4.2.B.3 Behaviour change techniques	If intervention is a component, what are the ‘active ingredients’ present in the technology that may trigger change in the reasons for non-adherence targeted?	To trigger/support change in a health behaviour, interventions act by generating change in determinants of the targeted behaviour. The ‘active ingredients’ in these interventions are labelled BCTs. We included only user-level BCTs (ie, BCTs that provide support to medication users) and mapped them according to the COM-B model and across domains.70 If considered relevant, HCPs level or system-level BCT can be included in the future.	BCT taxonomy.28 73 T4H behaviour change competency framework.64 Cards for Change.74 75
D2.4.2.B.4 Intervention providers	If intervention is a component, who delivers the intervention to users?	The provider of intervention is a role played by a person, population or organisation that provides/delivers an intervention. This includes their occupational role and type of relatedness. In medication adherence, the provider is often HCP, hence the quality of the HCP-patient relationships (communication skills, collaborative decision-making, trust in the HCP, HCPs’ cultural competences) correlate with patients’ adherence.76	BCIO,19 specifically Intervention Source Ontology.77 Gender, Sex and Sexual Orientation ontology.78
D2.4.2.B.5 Intervention settings	If intervention is a component, where is the service for improving adherence delivered?	Setting is the social and physical environment in which the technology is used to manage medication adherence. Implementation29 and behavioural19 science emphasise the importance of understanding and describing the environment in which a certain intervention is delivered as it can significantly influence its outcomes. In addition, not every intervention is applicable or transferable to every setting. We can distinguish between physical and virtual settings as well as the possibility of applying the intervention in any setting.	BCIO,19 specifically Intervention Setting Ontology.79 Consolidated framework for advancing implementation research (CFIR).29
D3.1 Quality indicators	How does the technology meet key quality indicators from different perspectives?	QIs are standardised, evidence-based and measurable items for monitoring and evaluating the quality of healthcare performance.80 They describe the structure, process and outcomes of care35 and based on them the standards and review criteria are developed. The target audience of the repository is very diverse and with specific individual needs related to MATech. Thus, we decided to group quality indicators according to their different purposes of use (eg, general, research, decision making, use).	A checklist of e-health quality criteria (under development).34 Mobile Application Rating Scale.81 Consort-EHEALTH guideline.82 Health Technology Assessment Core Model, V.3.0.83 O’Rourke et al.84
D3.2 Implementation outcomes and strategies	What implementation outcomes and strategies are needed and available for adopting this technology in the intended setting?	Implementation sciences provide knowledge on how to facilitate the adoption and use of technologies in real-world settings. The development of MATech often starts without considering the actual use in real-world settings, which prevents successful adoption and scaling up into clinical care.85 Three implementation outcomes were selected for ENABLE repository: acceptability; feasibility and sustainability to target early, mid and late implementation phases. In addition, eight implementation strategies were selected and adapted to present information on training users for working with MATech, availability of education materials, expertise needed to use the MATech previous implementation experiences, financial, accreditation and other legal aspects of the use.	Proctor et al.86 CFIR.29 The Expert Recommendations for Implementing Change.87 Interventienet.nl.12

Each group is presented with the core question it is addressing, rationale and sources used to create labels within the group. A detailed description of all attribute groups with labels and definitions is also available in the online supplemental file 2.

BCIO, Behaviour Change Intervention Ontology; BCT, Behaviour Change Techniques; COM-B, Capability, Opportunity, Motivation and Behaviour; HCP, Healthcare Professional; HRQoL, Health-related Quality of Life; MATech, Medication Adherence Technologies; QI, Quality Indicator; WHO DHI, WHO Classification of Digital Health Intervention.

The final proposed framework consists of three domains: (i) product and provider information (D1), (ii) medication adherence descriptors (D2) and (iii) evaluation and implementation (D3) aligning with the three elements of the Donabedian healthcare model (i) structure, (ii) process and (iii) outcomes.35 The domains branch in 13 attributes groups, which then branch further to up to four sublevels of attributes. Each attribute is described with a label and related definition.

Choice and description of the study design

We will perform an online real-time Delphi (RT-Delphi) survey to explore the level of agreement on the MATech definition and relevance, clarity and completeness of the proposed framework of attributes defining the repository structure and gain a deeper insight into stakeholders’ distinct needs and requirements. The Delphi process is a flexible iterative process to consult and/or reach consensus among a group of people on a particular topic.36 37 The key characteristics of a Delphi study are anonymity, iteration, controlled feedback and statistical description of group response.38 The RT-Delphi approach was developed by Gordon and Pease to improve efficiency of the process and shorten the time of performance.39 Since then, several online tools have been developed to facilitate the RT-Delphi design40 and literature describing the use of RT-Delphi and comparison with the traditional multiround Delphi approach is growing.23 41–44 In contrast to the traditional Delphi, the real-time approach is round-less and offers a constant iteration by providing immediate (real-time) individual and aggregated feedback. Based on new information participants can rethink and modify their answers, which could lead to reconciliation of opinions and eventually to consensus. Participants are encouraged to revisit and engage in the survey several times during the study period.39 40 42 44 In comparison with the traditional approach, the real-time approach encompasses all key Delphi features43 and is similar from all key perspectives.23 41 43 44 Furthermore, the real-time approach is particularly suitable for managing larger groups, decreases moderators’ workload, simplifies inclusion of people from different geographic locations and can be leaner in costs.23 39 44 On the other hand, the approach requires specific software, which can sometimes be rigid in terms of survey configuration and analysis, contributes to increases study costs and requires specific instructions for participants.40 44 Acknowledging the potential challenges, the advantages of the approach outweighed them and supported a decision to adopt the real-time approach for our Delphi study.

Sampling and sample size

We aim to include stakeholders from all 39 countries, participating in the COST ENABLE, covering five different backgrounds per country: (i) adherence and eHealth research (measurement, intervention development, implementation science, health economics), (ii) clinical care (specialist and primary care practitioners providing medication adherence support), (iii) patient representation (age >18 years, active representative in patient associations or healthcare facilities), (iv) policy making and (v) technology development. Hence, the targeted sample size is at least 195 panellists to be invited in the study (39 countries×5 stakeholders).

Purposive sampling will be applied to identify potential panellists. First, requests will be sent through the ENABLE Cost Action membership list to representatives of all 39 countries, requesting them to identify suitable panellists from all five backgrounds. ENABLE members will provide the SC the name, background and email for every potential panellist. Participants’ emails will be entered in the online platform (eDelphi.org—Delphi method software45), which will enable anonymity in further steps, that is, individual’s activity and or/answers will not be linked to personal data. All communication with the panellists (invitation, reminders, etc) will be performed through the platform. If more candidates from the same background and country will be suggested, we will invite all candidates to increase the likelihood of achieving the planned sample size. If the expressed interest exceeds the planned sample size, purposeful sampling will be performed to ensure variation in expertise, country and balance other characteristics (eg, years of expertise, gender). To reach simple size and variation in sample characteristics, key international organisations from the field (eg., The International Society for Medication Adherence (ESPACOMP), Pharmaceutical Care Network Europe (PCNE), European Medicines Agency (EMA), European Patient Forum (EPF), etc.) will be contacted to fill any missing gaps, if needed.

Patient and public involvement

The goal of this Delphi consultation is to involve stakeholders (patient representatives among them) in decisions regarding the development of ENABLE repository and is part of the broader approach to patient and public involvement followed in the ENABLE Action. Results will be communicated to all stakeholders, and they will be listed and acknowledged among ENABLE collaborators.

Data collection

We will use an online platform, eDelphi.org (Metodix, Helsinki, Finland45) for data collection. All survey activities—distribution, reminders, communication with and between the panellists and interim analysis of the process will be performed through the tool. The survey will be conducted from 1 October 2021 to 15 January 2022 in three stages:

Pilot stage: at least 10 members of the COST ENABLE Action, specifically members of the WG2, will be asked to test the survey (including instructions for participants) and to provide feedback on face validity as well as user experience.

First stage phase: invitation of 20 purposefully selected stakeholders (aiming for variation in expertise, geographical location and gender) to create initial aggregated feedback of the RT-Delphi.

Full-scale RT-Delphi: all remaining stakeholders will be invited to participate in the study.

Stakeholders will receive an email invitation via the eDelphi platform with a personalised link to the survey. Detailed instructions describing survey aims, rules of engagement and how to use the platform will be available on the platform.

At the beginning of the survey, participants will be encouraged to think of a hypothetical situation in which they would search for MATech applicable to their own setting/role and to assess the proposed attributes from this perspective throughout the survey. First, panellists will be asked to familiarise with the proposed structure and provide general feedback on the completeness. Furthermore, they will be asked to rate agreement with and clarity of the MATech definition and relevance and clarity of each proposed attribute group on a 9-point Likert scale, where 1 represents extremely irrelevant/unclear and 9 represents extremely relevant/clear. We will use the Live 2D format,45 where each outcome represents one of the two dimensions, that is, the x-axis stands for relevance and the y-axis stands for clarity. Additionally, an open-text field will be provided for panellists to comment on completeness of each attribute group, that is, proposing additional attributes or revising definitions. We will moderate the discussion in the following ways: (i) address technical issues with the platform by responding to the comment when the issues will be solved or provide instructions how to manage the issue and (ii) outline the progress of the study and the most commented questions in bulletins send through the platform once a week. We considered these strategies to encourage panellists to participate, taking into account the length of the survey and the complexity of the concepts they are rating. Delphi survey materials (online supplemental file 3, online supplemental file 4 and online supplemental file 5), including all attributes labels and definitions as well as participant instructions (online supplemental file 6), are shown in the online supplemental materials.

For sample description purposes, participants will be requested to provide information on their expertise (profession, years of experience, relevant professional experiences) and demographic characteristics (age, gender, country of practice). This information will also be used to examine differences in participants’ ratings and comments depending on their background and location. These data will be presented in aggregated form and not linked to the individual’s activity or answers. Revisiting and rerating will be encouraged by weekly reminders.

Data collection will be stopped on reaching adequate sample size and characteristics to achieve sufficient representability and generalisability of the opinions gathered. Therefore, we propose stopping the Delphi when three criteria will be met: (i) the total response rate to the survey is ≥30% (number of participants completing the survey, of the total number of stakeholders invited)46; (ii) a minimum of 10 panellists in each stakeholder group completed the survey; (iii) a minimum of two stakeholders from at least 2/3 of the COST ENABLE countries has completed the survey. We will operationalise survey completion as providing background data and answering at least 75% of the repository structure questions.

Data analysis

Descriptive statistics will be used to characterise the sample of panellists and each stakeholder subgroup regarding profession, years of experience, age, gender and country.

Several measures can be used to determine when consensus is reached, with the percentage of agreement being the most common.47 Prespecification of the consensus measure and criteria for consensus increases trustworthiness of findings.48

Level of agreement on relevance, clarity and completeness

Stakeholder agreement on the proposed definition and attributes will guide decisions on the repository structure. Therefore, we selected a set of criteria representing different levels of agreement and consequently carrying different weights in these decisions. The level of agreement on every attribute for both outcomes (eg., relevance and clarity) will be quantified using the interpercentile range adjusted for symmetry (IPRAS) analysis technique from the RAND/UCLA Appropriateness Method (RAM).49 First, the disagreement index (DI) will be calculated as a ratio between the interpercentile range (IPR) and IPRAS. A DI >1 (ie, IPR >IPRAS) indicates disagreement exists. IPR is calculated using the 30th–70th percentile. IPRAS for the 9-point Likert scale is calculated according to the formula presented in the RAM User Manual.49

Second, the median and DI will define different levels of agreement and steer the decisions about the repository structure. For the relevance:

items with the median of 7–9 and no disagreement will be considered as relevant and mandatory;

items with the median of 4–6 or disagreement will be considered as optional;

items with the median of 1–3 and no disagreement will be considered not relevant and candidates for exclusion.

For an even number of participants, median ratings of, for example, 6.5 or 3.5 will be assigned to the higher level.49 Stakeholders’ responses per question will be summarised using descriptive statistics.

For clarity ratings, the above criteria will be applied as (i) sufficiently clear to remain unchanged; (ii) optional changes and (iii) candidates for rephrasing.

Panellist comments in the open-text fields will be analysed qualitatively, using content analysis. Findings will be used to rephrase and improve clarity of certain attributes or to add additional attributes proposed by stakeholders.

Subgroup analysis

Following the primary analysis on the whole sample, a subgroup analysis per stakeholder group will be conducted to examine variation in opinions and potential differences among subgroups. The same agreement criteria will be applied and descriptive statistics will be stratified by stakeholder group. In addition, we will determine the reliability of ratings per question within stakeholder group by calculating the intraclass correlation coefficient (ICC). The ICC calculation is based on the two-way random model, considering type (average measures) and definition of relationship (consistency) and is presented in Equation 1. ICC >0.70 will indicate moderate-to-good reliability.50 51

Equation 1. Calculation of the ICC expressed in %. MSR stands for mean square for rows and MSE stands for mean square for error.

Analysis of process indicators

By analysing process data from the online tool, we will describe in more detail how stakeholders’ responses evolved through iterations and how consensus or certain level of agreement has formed.25 52

Stability of response presents the consistency of responses within the study period and between respondent group stability, which is considered a necessary precondition for determining the level of agreement or if consensus was achieved.53–55 Different measures of dispersion (eg., median, IQR) and statistical approaches (eg., descriptive, inferential) can be used44 55 to measure stability, which can be calculated between rounds (traditional Delphi) or at the end of the study (RT-Delphi).41 44

We will use the coefficient of quartile variation (CQV) as a descriptive measure of response stability. CQV will be calculated over all participants (CQVtotal) and within the same stakeholder group (CQVsub) to account for expected higher variation in response between different stakeholder groups. A CQVtotal <30% and CQVsub <15% will be considered as stable response. CQV calculation is shown in Equation 2.54 56

Equation 2. Calculation of the CQV, expressed in %. Q3 stands for value of the third quartile and Q1 for first quartile.

Final repository structure

After conducting the analyses described above (planned to be finalised at the end of April 2022), results suggesting modifications to the proposed structure will be considered for adoption by the SC in a subsequent version, which will represent the final structure of the ENABLE repository implemented on the initial ENABLE repository version. Further work will be considered to address results that might suggest ongoing debates in the field about certain attribute groups or the need for more in-depth consultation and evidence generation. This work will accompany the iterative improvement of the repository during the ENABLE Action.

Ethics and dissemination

Ethical considerations and consent to publish

The study is designed to ensure participants’ anonymity and to manage personal data in line with EU regulation. Before starting the survey, every participant will provide an informed consent electronically on the study entry page. Participants will be asked to carefully read through the statement regarding the study aim and nature as well as the data handling procedures and to mark their understanding and agreement. The results will only be published in an aggregated form and no personal details will be revealed.

An ethical approval for the activities of the COST ENABLE Action, including this Delphi study, was granted by the Malaga Regional Research Ethics Committee (‘Comite de Etica de la Investigacion Provincial de Malaga’) on 29 April 2021 (online supplemental file 7). In addition, a data protection assessment was carried out by the Data Protection Officer at the University of Basel. According to this instance, the Delphi study protocol was determined as compliant regarding data protection and security (online supplemental file 8).

Future implications and challenges

The proposed scope and framework of attributes together with findings from this Delphi study will represent the first steps on the pathway to create an evidence-based, interoperable and user-friendly MATech repository. Following the Delphi consultation and integration of the repository module on the ENABLE website,57 providers of MATech (public or private) would be invited to upload information on their products via a MATech description form based on the final repository structure. The accuracy of the information would be verified by an independent review panel through a procedure yet to be established. Important challenges lay ahead, such as how to select MATech for inclusion in the repository given the broad scope of the definitions proposed, how to ensure accurate information about the technologies included, how to provide the information in other languages than English and in non-technical language accessible for all and how to maintain a representative and varied offer of technologies in the long term. Nevertheless, the ENABLE repository promises to bring together stakeholders from different backgrounds to build a common language which can have an important positive impact on medication adherence research and practice.

Dissemination

The repository will be publicly accessible for interested parties. The use of the repository will be promoted and supported by dissemination meetings, workshops and training schools. The findings of the study will be presented via publications (reports and manuscripts in open access peer-reviewed journals) and oral presentations to different stakeholders in conferences and meetings. The spirit of COST Actions is networking and dissemination of ideas; hence, the action is open for anybody who would wish to join or would like to be informed about its activities.