MAPPinfo, mapping quality of health information: study protocol for a validation study of an assessment instrument.

INTRODUCTION: Health information is a prerequisite of informed decision-making. Criteria for development, content and presentation have recently been published in a corresponding guideline. Within a systematic search, 27 relevant checklists were identified, none of them, however, complying with the guideline or providing reasonably operationalised measurement items. Therefore, a draft of a checklist with 19 criteria was drafted. The current study aims at developing and validating this measure of quality. METHODS AND ANALYSIS: The validation design consists of five single studies to be conducted at the University of Halle-Wittenberg/Germany and Graz/Austria. (1) Achieving content validity through expert reviews of the first draft, (2) achieving feasibility using 'think aloud' in piloting with untrained users, (3) pretesting the instrument applied to health information materials without use of secondary sources: determining inter-rater reliability and criterion validity, (4) determining construct validity using information on proceedings and methods in the development process provided by the developers and (5) determining divergent validity in comparison with the Ensuring Quality Information for Patients (EQUIP) (expanded) Scale. The substudies will use varying samples of experts, students and developers and will apply the instrument to materials of various domains. Sample sizes will be adjusted to the particular research designs and questions. Analyses will employ qualitative methods, such as content analyses and discourse within the expert panel, and correlation-based methods both for determining inter-rater reliability and validity. ETHICS AND DISSEMINATION: The project is approved by the ethics committee of the Martin Luther University Halle-Wittenberg (approval number: 2019 115). Results will be published, and the instrument made accessible on public health platforms. It is meant to become a certification standard. MAPPinfo can be used as a screening instrument without training or secondary sources. Although developed in the German language, the instrument will be applicable also in other languages. TRIAL REGISTRATION NUMBER: AsPredected22546; date of registration: 24 July 2019. PROTOCOL VERSION: July 2020. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The study is strong as it builds on a systematic search and evidence update.

The study provides the first approach to measuring quality of health information using evidence-based criteria.

The new instrument passes a sound multistep validation procedure.

Quality assessment is limited to evidence-based criteria, implying that potentially important aspects are not considered.

The new instrument works like a quality screening and does not consider potentially revealing background information.

Introduction

The study comprises the development and validation of an instrument that operationalises the ‘guideline evidence-based health information (EBHI)’,1 2 which has been developed in a cooperation project between the University of Hamburg, Health Science and Education and the German Network of Evidence-Based Medicine (DNEbM), section ‘patient information and participation’.

The provision of science-based and easily understandable health information—so-called EBHI—is called for both nationally and internationally.3 EBHI is a prerequisite for informed choice, assuming that those concerned have sufficient knowledge and are in line with individual preferences.4 Most citizens want more information and in particular more reliable information as well as greater participation in the decision-making process.5 In Germany, the patients’ right to EBHI is based on the Patient Rights Act and an ethical claim.6 7 In 2015, the German National Cancer Plan developed a ‘Roadmap—informed and shared decision-making till 2020’. The DNEbM’s section ‘patient information and participation’ published the second version of the ‘Good Practice Health Information’ in 2016, which refers to the need for standards.8

In Austria, the Health Target Number 3—to enhance health literacy in the population—refers to the need for ‘facilitating access to objective, easily comprehensible information of assured quality’.9 The Austrian Target-Based Governance Agreement 2017–2021 at federal level was agreed for the years 2017–2021. Within the framework of this agreement, numerous measures have been agreed to improve health and strengthen health literacy, respectively, for example, implementation of standards for good-quality health information.10 In 2018, the Austrian quality criteria for good health information were published by the Federal Ministry of Labor, Social Affairs, Health and Consumer Protection and the Austrian Health Literacy Alliance.9

Although the criteria for EBHI have been defined by national and international working groups for years with the aim of enabling informed decisions,11 12 the implementation into practice is currently not successful.13 14

The ‘guideline EBHI’1 is also an international novelty. Representatives from 22 institutions and associations as well as three patient representatives worked on developing the guideline, which in the long run shall contribute to ensuring the quality of health information and thus support informed decision-making by patients. In healthcare, guidelines are important tools for everyday care and can improve the quality of care if successfully implemented.15 In addition, training programmes are recommended to facilitate the implementation of clinical guidelines.16

The efficacy of the ‘guideline EBHI’ combined with a training programme is currently under evaluation in a randomised controlled trial.2 Compliance to the guideline will be determined by assessing the quality of newly developed health information material using the instrument to be developed according to the present study protocol.

The current protocol describes the process of developing and validating MAPPinfo to assess the quality of health information as an outcome measure. Until now, there is no instrument available that is capable of assessing the quality of health information according to the guideline criteria. We carried out a systematic research for such instruments in the Pubmed, Psyndex and Psycinfo databases. Search terms are provided in box 1.

Group 1: assessment (apprais* OR assess* OR measur* OR instrument* OR checklist OR rating OR rate OR scale* OR test*), Group 2: quality criteria (qualit* OR criteria OR standard*), Group 3: health information (“Patient Education Handout”(Publication Type) OR “Consumer Health Information”(Mesh) OR “Pamphlets”(Mesh) OR “health information” OR “patient information” OR “decision aid” OR “decision aids” OR “information material” OR “information materials” OR ((patient OR health) AND (brochure OR leaflet OR pamphlet OR flyer OR booklet))).

Among a total of 5544 matches, two independently working project members (JL, JK) identified a pool of 40 potentially relevant instruments in a total of 256 publications. Measures focusing purely on readability or specific medical domains or published in other than English or German language were excluded. Table 1 outlines the characteristics of the 27 included instruments.

Table 1

Characteristics of included instruments

Checklist		Author	N of items (response format)	Validated	Evidence-based criteria complete	Non evidence-based criteria avoided	Operationalisation provided
LIDA	Minervation validation instrument for healthcare websites	Carlsson (2015/2007)	6 (5 point Likert)	Inter-rater reliability	No	No	None
EMORY	Health-related website evaluation form emory questionnaire	Bastos (2011), Teach (2009)	36 (dichotom.)	Not reported	No	No	Blurredly
AHQA	Andalusian health quality agency	Agencia de Calidad Sanitaria de Andalucı’a. (2012)	17 (dichotom.)	Not reported	No	No	None
GCS	German cancer information scale	Liebl (2015)	23 (3 point Likert)	Not reported	No	No	Unclear
DOHHS	Website information reliability evaluation instrument	Devine (2016)	35 (4 point Likert)	Inter-rater reliability Kappa	No	No	Manual
AFGIS	Aktion Gesundheitsinformationssystem	Afgis (2016)	10	Not reported	No		Blurredly
GPGI	Good practice guidelines for health information	Workinggroup GPGI	15	Evidence update expert validation	Yes	Yes	None, theoretically
Medisuch	Institute for quality and transparency of health information	Pauer (2015)	Unclear	Not reported	No	No	Not reported
eEurope 2002	Quality criteria for health-related websites	Commission of the European Comm.	12	Not reported	No	Yes	No instrument
NLM	National library of medicine guideline	US National Library of Medicine	–	Not reported	Unclear	Yes	Tutorial
WebMedQual	Instrument assessing quality of health related websites	Provost (2006)	95+3	Content validation expert panel review	No	No	None
Checklist AWCHI	Checklist for assessing written consumer health information	Currie (2000)	40 (3 point Likert)	Not reported	Yes	Yes	Not reported
HON	Health on the net foundation	Selby (1996)	Eight principles	Expert consensus	No	Yes	Yes
JAMA	Journal of the American Medical Association: Standards for Health Information on the Internet	Silberg (1997)	Unclear	Not reported	No	Yes	Not reported
DISCERN	Quality criteria for consumer health information	Charnock (1999)	16 (5-point Likert)	Inter-rater agreement	No	Yes	Manual with examples
WebEval	University of michigan website evaluation checklist	Anderson (1999)	43 (dichotom)	Not reported	No	No	Not reported
HITI	Criteria for assessing the quality of health information on the internet	Ambre (1997)	127 items	Not reported	No	No	Manual
SAM	Suitability assessment of materials (SAM)	Doak (1996)	22 (3 point Likert)	Not reported	No	No	Manual
Trevena principles	Guiding principles for presentation of risks in Das	Trevena (2012)	–	Evidence update	No	Yes	None
PEMAT	The patient education materials assessment tool	Shoemaker (2014)	26 (dichotom)	Inter-rater reliability kappa	Yes	No	Manual with examples
IPDASi	International patient decision aid standards instrument	Elwyn (2009)	Checklist: 74 Instrum.: 47 Minimal: 44	Intra-class correlations and Cronbach’s alpha	Yes	No	None
Kellogg	Evaluation of health information on the internet	University of Dalhousie	Dichotom	Not reported	No longer available
EBPI	Evidence-based patient information quality criteria	Bunge (2010)	–	Evidence update and ethical	Yes	Yes	Theoretical
NCCAM	10 things to know about evaluating medical resources on the web	National Centre for Complem-entary & Inte-grative Health	10	Not reported	No	Yes	Unclear
EQIP (expanded)	Ensuring quality information for patients	Charvet-Berard (2008)	36	Reliable, and proved useful	No	No	Unclear
C-H-i-Q	Centre for health information quality	Shaddock (2002) www.chiq.org	–	None	No	No
CHECKED	Austrian quality approval system	Beunders, Fokus Patient	19–22	Referring to DISCERN	No	No	Rater training

Based on the guideline’s quality concept,1 none of these was considered sufficiently suitable for registering the quality of health information. The following shortcomings have been identified:

Incompleteness of the quality criteria.

Inadequate or lacking operationalisation of individual criteria.

Use of criteria not covered by the present state of knowledge (ethics or scientific evidence).

Lack of validation.

Against this background, the development and validation of an assessment tool for health information are considered necessary.

Drafting the measuring instrument

Through discussion in the research group, an instrument has been drafted as a checklist that can be used without special training or secondary sources. The concept of quality, operationalised in the instrument, is not a new invention but corresponds to the criteria agreed on in the guideline. The draft includes criteria and items, a category structure, a corresponding simple manual and a measurement concept. The instrument is called MAPPinfo.

Objective

In the current study, MAPPinfo is to be tested for its quality parameters. The validation steps planned here are intended in particular to examine the extent to which consistent quality assessments of independently working assessors are generated (reliability) and whether limiting the assessment to the immediately observable features permits a sufficiently accurate quality assessment (validity).

After its validation, MAPPinfo will be published in English and made available via the project page of the guideline in order to achieve the greatest possible transparency for the quality concept and user-friendliness of quality evaluations. Depending on the results of the validation studies, additional assessment procedures will be developed to allow for more complete and comprehensive assessments and evaluations where appropriate.

Methods and analysis

The development process consists of the following steps (hereafter called substudies):

Expert review/content validity

The draft of the MAPPinfo instrument will be reviewed by relevant experts not previously involved in test development. Comments and objections will be reexamined during the discussion and used as a basis for an initial revision.

Pilot testing (trial application)/feasibility study

Members of the target group will then carry out a pilot test with MAPPinfo. This process will be accompanied by a qualitative study, which uses interviews to collect data about the usability and comprehensibility. The findings will be included in a new revision.

Pretest/determining the (inter-rater) reliability

MAPPinfo will then be used extensively by the user target group to assess health information from a test sample. The aim of this study is the determination of inter-rater reliability values under field conditions. In case inter-rater agreement is not sufficient, substudies 2 and 3 have to be repeated until the current version has proven reliable.

Comparison with self-declarations by information developers/construct validity

In this substudy, the assessment quality of MAPPinfo will be compared with the judgement of an expert that is based on more extensive background information provided by the developers by means of self-declaration. This method provides, on the one hand, a quantitative indicator for estimating the construct validity. On the other hand, the construct validity can also be described qualitatively (correlation design/survey).

Comparison with alternative instrument/divergent validity

In this substudy, MAPPinfo evaluations will be compared with quality evaluations ascertained with another established quality measuring instrument. Discriminative validity would be meaningful to determine the relation to several existing instruments. For the first attempt, EQIP (EQIP 36 items17) will be used, which is an instrument that addresses not only some evidence-based criteria but also other reasonable criteria (correlation design/survey).

Samples

The validation of the new assessment tool over five substudies will involve four different samples, HI test samples (materials), experts of EBHI, representatives of the targeted user groups and developers.

For validation purposes, the use of several domains for homogenous health information samples seems reasonable. Health information samples in the domain of contraception, gonarthrosis and endometriosis will be generated by systematic searches. HI for substudy 4, however, will be selected from mixed domains striving to maximise heterogeneity in this validation step. A screening process is employed to identify the health information material suitable for MAPPinfo assessment among a bigger pool of various HIs. The following inclusion criteria will be used:

The information has patients or medically non-professional people as its target group.

As a minimum precondition of choice, the information deals with different management alternatives.

The part of the source relevant for the assessment is identifiable: for example, chapter, excerpt or subpage.

In the substudies, the targeted user group will be represented by students of Health and Nursing Science at the Martin Luther University Halle-Wittenberg, who have not been directly involved in the present implementation project of the guideline EBHI. These students will pilot and pretest the assessments of the given HI examples.

Experts of EBHI will be involved at several stages in the project. First, researchers in the field not previously involved in the development will be recruited from the environment of the DNEbM network and asked to review the draft as a content validation. In addition, members of the project group will act as the reference standard in several substudies and will also appraise additional information collected from HI developers (substudy 4). Those experts will always work as a panel.

Substudy 4 requires recruitment of developers as an additional information source for a more comprehensive appraisal of HI. Developers will be strategically recruited from six distinct developer groups: health insurances, practitioners, patient organisations, commercial providers, developers from the state sector and hospitals.

Strategies of recruitment and suitable size of the respective samples vary between the substudies. Table 2 gives an overview of the sampling within the entire validation project.

Table 2

Sampling overview for the entire validation project

Substudy	Sample HI: sampling strategy and size	Sample assessors: sampling strategy and size
1. Expert review of the draft	–	Convenient
		Two experts in the field
2. Pilot testing	Convenient	Convenient
	5 HI/ contraceptives	Three raters: health scientists
	5 HI/ gonarthrosis
3. Pretest	Random	Convenient
	25 HI/contraceptives	Five raters: students of health sciences and expert panel (project group)
	25 HI/gonarthrosis
4. Construct validation	Strategic	Strategic
	50 HI	50 developers of HI, from five developer groups
		Expert panel
5. Divergent validation	Full survey (x)	Convenient
	Endometriosis HI	2×2 raters: health scientists

The table presents two types of samples: materials and assessors, the people either reviewing or applying the instrument.

HI, health information materials.

Data collection

Subsequent substudies in this validation plan will not be initiated until the respective previous steps have achieved their aim. New revisions will be elaborated and the conduct of each validation (substudy) will be repeated until proof of validity has been provided to a satisfying extent. The history of the revisions will be documented in a separate log including causes of revisions and corresponding discourse within the research panel.

Description of the draft: MAPPinfo

MAPPinfo has been developed on the basis of valid quality criteria, the good practice guidelines for health information,18 the criteria of evidence-based patient information11 and the ‘guideline EBHI’.1 However, the selection and distribution of items as well as decisions about the particular operationalisation and answering formats have been inspired by a fused total pool of items from all instruments existing so far. In particular, items were defined based on guideline recommendations. The total pool of items from the review was then screened for additional evidence-based criteria not covered by the guideline. Moreover, operationalisations from different versions of assessing the same criterion were compared and discussed before a new item was constructed. MAPPinfo is designed to assess materials intended to inform medical decisions. Medical decisions are defined as decisions on prevention and health promotion, early detection, diagnostics, treatment, palliation, rehabilitation, nursing, after-care and coping with diseases. The manual distinguishes itself from other types of health information, by explaining, for example, the implementation of a specific measure, or informing about the health system or giving general health tips.

The instrument is intended for evaluating those quality criteria that can be observed in the respective health information or directly linked sources. This means that, if required, transparency can be shown by means of a methods’ report or documentation on health information. Quality criteria in the categories definitions, transparency, content and presentation are regarded as verifiable. However, with MAPPinfo, it will not be possible to check the correctness of the content of the information and the appropriateness of the methods used in the development process.

MAPPinfo is intended to provide a structure to the assessment of compliance with guideline recommendations, as justified either by the ethical guidelines or by the current state of research. Thus, the criteria listed in the taxonomy are considered essential but not sufficient to promote an informed decision. Many of the characteristics likely to be important for the quality of health information are not part of the examination, as a particular recommendation is not yet justifiable. On the other hand, non-compliance with any of the criteria in MAPPinfo is already seen as a barrier to informed decision-making.

MAPPinfo is conceptualised so that the instrument can be used by people with basic knowledge of evidence-based medicine but without any training as an evaluator.

Ad 1: expert review/content validity

Two independently working experts in the field of EBHI will review the draft with regards to content validity of the particular operationalisation of the criteria. Written comments will be collected and in addition interviews will be used to secure understanding. A revision of the instrument will be created after discourse in the research panel.

Ad 2: pilot testing (trial application)/feasibility study

The MAPPinfo will then pass a number of consecutive test applications with three different raters from the target group and two different kinds of health information. The test applications will be accompanied by participating observation (by JK) using the ‘think aloud’ method. The interviewer will encourage the participants repeatedly to express any ideas, doubts or concerns. If participants indicate a problem, the interviewer will explore its nature and importance and invite the participants to discuss possible solutions. All the problems will be documented for later analysis in the research group. Findings from this pilot testing will be fed continuously into the revision of the instrument and new pilots will be conducted based on a revised version. This kind of investigation will be continued until saturation with regards to the feasibility and comprehension of the MAPPinfo is reached.

Ad 3: pretest/determining the (inter-rater) reliability

The proven feasible version of the new instrument will then be exposed to a pretest, comprising appraisal of a selection of 50 health information websites by five members of the target group and additionally by an expert panel (AS, JK) representing a reference standard. Each piece of information will be independently rated two times using the MAPPinfo. The expert panel will make a general judgement on a 0–10-point Likert-type scale. All the raters will be blinded to each other’s judgements.

Ad 4: comparison with self-declarations by information developer/construct validity

In this step, additional information relevant to quality of the health information websites will be collected from the developers. For this purpose, a questionnaire will be created in an online format (MAPPinfo-Develop), which developers of the aforementioned information materials are asked to complete. In particular, the questionnaire shall provide clues for extended quality appraisal, which beyond the categories definitions, transparency, content and presentation also leans on the categories development methods and development process. The latter are not directly observable within the health information but are to a far extent accessible based on the developers’ self-declaration. The questionnaire will use free text answering formats and will be piloted before the initiation of the survey.

The following particular topics will be contained in the self-declaration questionnaire. The specific formulation and design of the items is subject of the respective study step 4:

Justification of the search strategies for the evidence update.

Proceedings for inclusion of studies.

Appraisal of the existing evidence.

Exhaustiveness of the set of patient-relevant outcome parameters.

Evaluation (stage of evaluation, methods used at different stages of piloting, where population was studied, results of the evaluation steps, conclusions drawn from results of evaluation and revision done…).

Developers will be identified via HI published in the internet and invited to fill in the online questionnaire focusing on one particular HI they have developed. There will be no need to provide personal data, whereby consent is given to proceed as described.

The health information examples chosen by the developers will be rated with the MAPPinfo checklist by health students. In addition, they will pass an expert appraisal by using the collected free text answers with regards to an additional set of quality criteria. The experts will be blinded towards the results of the checklist, and the students will be blinded to the experts’ appraisal. The experts’ appraisal will be transformed into a quantitative supplemental score (MAPPinfo-extended) to be added to the corresponding MAPPinfo score.

Ad 5: comparison with alternative instrument/divergent validity

Provided that the inter-rater reliability at step 3 was proven satisfactory, a new set of information materials will be assessed using MAPPinfo and EQIP36.17 Ratings with the two different instruments will be done independently by different raters. Both rater groups will be drawn from the same population (health sciences studies) and need comparable levels of preknowledge.

Analyses

The substudies 1 and 2 will use discourse within the research group as qualitative methods of problem analysis and management. Each documented problem needs to be fully understood and appraised with regards to the risk this problem includes. Then an inductive process will be initiated where solutions are drafted and respective risks are appraised before a decision is made about whether to keep the original version or to refine the instrument in a specific manner. Decisions based on discourse in this study will be unanimous.

Substudy 3 will calculate T-coefficients and percentage agreement between two data columns representing pairwise judgements of each item.19 T has been chosen as a suitable coefficient for ordinal scaled scores, while—contrary to Cohen’s kappa—expected frequencies are not drawn from the empiric data but from the theoretical marginal distribution. This decision is based on the assumption that a rater using the MAPPinfo should in his or her judgements not be influenced by the specific distribution in the test sample. Four items address both the quality of content and of presentation and will enter the analysis with a double weight. T-coefficients will be considered as moderate between 0.40 and 0.60, as strong higher than 0.60 and as excellent higher than 0.80.20 Spearman correlation coefficients will be calculated to compare expert panel ratings on a 0–10 scale with the MAPPinfo mean score. This correlation is an estimator for criterion validity.

Substudy 4 will calculate Spearman correlation coefficients between MAPPinfo mean scores and MAPPinfo plus MAPPinfo-extended mean scores. The resulting correlation will be an estimate of the MAPPinfo checklist’s construct validity. In a second step, discrepancies between judgements based on MAPPinfo only or with the expansion will be analysed on a qualitative level. The aim is to identify eventual constellations where judgement without insight into the development process might be of low validity or even misleading.

Substudy 5 will calculate Spearman correlation coefficients between quality appraisal made by using MAPPinfo and the EQIP36. Therefore, the mean scores of the two instruments will be used. The correlation is expected to achieve significance level and will be used as an estimate for a divergent validity.

Patient and public involvement

No patients are involved.

Discussion

We expect the new instrument to prove capable of checking criteria in the categories definitions, transparency, completeness of contents and appropriate presentation. The instrument can be used without training and will be applicable for information about all kinds of health problems, if more than one option exists. The quality of the instrument will be described in terms of its psychometric properties. There are, however, a couple of limitations and barriers. First of all, we have to bear in mind that not all features of health information will be tested, and therefore recommendations can be provided only for some of them. Moreover, MAPPinfo will only check a selection of these evidence-based criteria, omitting those not appraisable without access to background materials and procedures. MAPPinfo will therefore be acting on the level of a screening instrument. We will provide data to quantify the quality of the proxy with regards to an in-depth review of health information. The instrument can be considered suboptimal with regards to some of the recommendations from the classical test theory. Operationalising a rigorous ethical approach, all the items are constructed with low item difficulty and mostly dichotomously. Although a more graduated assessment format and higher difficulty would make it easier to generate normally distributed data on information quality, this distribution would not reflect the current reality in the field of health information from the users’ point of view.

To summarise, the new instrument, MAPPinfo is urgently needed. The concept of quality is thoroughly anchored in the synthesis of the best available evidence. The validation studies will provide information about the reliability, accuracy and validity of the checklist.

Ethics and dissemination

The protocol has been approved by the ethics committee of the Martin Luther University Halle-Wittenberg (approval number: 2019 115) and the Ethical Principles for Medical Research Involving Human Subjects (World Medical Association Declaration of Helsinki) will be respected in this project.21 All the participants will provide informed consent.

The new measure is intended to become a certification standard for public health information in Germany. Operationalisation of criteria of evidence-based HI has repeatedly been demanded by the DNEbM. The results will immediately be published in open access, peer-reviewed journals and translations of the checklist provided in the Norwegian and English languages. In addition, MAPPinfo will be available, for example, on the websites of the DNEbM and the guideline EBHI.1 The criteria assessed by MAPPinfo are applicable regardless of language, country or health system. However, the use of the instrument in other languages will require some additional validation works.