Usage and Scientific Properties of the ADVOCATE Oral Health Care Measures.

BACKGROUND: Feedback information about the quality of oral health care is important for reflective learning by oral health care professionals and the wider health system. To this end, a list of 48 topics describing oral health and oral health care was recently agreed as part of the EU H2020 ADVOCATE project.
OBJECTIVE: This article reports on the formulation of measures based on the ADVOCATE topics and provides information on usage, reporting, validity, and reliability of the measures.
METHODS: The AIRE instrument was used to guide the methodological approach adopted. The appropriateness of the measures was tested among 39 general dental practitioners (GDPs) in Denmark, Germany, and the Netherlands. Data were collected from a convenience sample of patients via a questionnaire deployed in an online application in their practice. Feasibility, acceptability, and usefulness of the measures were evaluated through focus group interviews with GDPs. Face validity and test-retest reliability of the measures were assessed.
RESULTS: For 46 of the 48 topics, a measure could be defined by constructing a definition and a numerator and denominator. Data collection for all 46 measures was feasible and acceptable for patients using the online questionnaire. The practicalities of using claims data for the purpose of giving feedback to individual and groups of GDPs proved to be challenging in terms of timely access of such data, the granularity of the data, and matching the content of the data with the consented items on quality of oral health care. Face validity was considered appropriate, as the patients found the questionnaire easy to understand. Test-retest reliability was found to be acceptable for 36 of 46 measures.
CONCLUSION: The broad range of the ADVOCATE oral health care measures could make a useful contribution to a more transparent, evidence-based, and patient-centered oral health care system. KNOWLEDGE TRANSFER STATEMENT: This study shows the usage, reliability, and validity of 46 oral health care measures. The measures, which include patient experience and health behaviors, were found to be useful to stimulate discussions about clinical practice. The measures can provide essential information for quality improvement strategies and useful and relevant feedback information for GDPs.

Introduction

Society no longer accepts that health professions alone have the authority to judge the quality of their own work (Berwick 2016). There is an increasing recognition that dentists and the health systems in which they work need to be explicitly accountable. To fulfill the demand for transparency in oral health care, measures need to be available to provide information on the processes and outcomes of care delivery, unexplained variation in dental practice, and efficient use of resources (Institute of Medicine 2001). Measures should go beyond counting treatments; they should also provide information on patient-relevant aspects, such as patient-reported outcomes, experiences, satisfaction, involvement in decision making, and individual risk factors for oral disease (Institute of Medicine 2001; Baâdoudi et al. 2016; Berwick 2016).

In 2011, the Institute of Medicine stated that there was a lack of measures in the field of dentistry, creating a barrier to quality improvement in oral health care. Since then, the pressure to develop comprehensive measures has increased. A recent systematic review identified 21 publications from the last 15 y describing the development of 215 measures for oral health care (Righolt et al. 2019). The majority of these measures narrowly assess the processes of oral health care, including the provision of oral treatment and preventive services, clinical periodontal and dental disease outcomes, or access to care. Few measures focus on patient safety, satisfaction, behaviors, or affordability of care (Byrne et al. 2018; Righolt et al. 2019).

While the development process of most of these published measures has been comprehensive, only 4 studies reported on the additional scientific properties of the measures (e.g., validity, reliability) or tested the utility of measures in practice (Herndon, Crall, et al. 2015; Bhardwaj et al. 2016; Hummel et al. 2017; Neumann et al. 2017). Robust measures that can be applied to routinely collected data would contribute to a more transparent, evidence-informed, and patient-centered oral health care system (Baâdoudi et al. 2017). The issue now may not be a lack of measures but rather the limited evidence that the measures developed so far have been explicitly demonstrated to be fit for purpose.

If measures are to be meaningful, generalizable, and interpretable, they need to be developed systematically and with scientific rigor. The minimum prerequisite for a measure is that it be based on scientific evidence, accepted by experts in the field, and measured with reliable data sources (McGlynn and Asch 1998). The AIRE instrument has been developed as a validated critical appraisal tool to assess the methodological quality of health care measures (de Koning et al. 2007). It provides a standardized approach for the assessment of scientific soundness and applicability of measures. It consists of 20 criteria grouped into 4 domains: 1) the purpose, relevance, and organizational context; 2) stakeholder involvement; 3) the scientific evidence; and 4) the formulation and usage of the measures.

In May 2015, the ADVOCATE project (Added Value for Oral Care) was launched, funded by the EU Commission’s Horizon 2020 program (Leggett et al. 2016). Part of this project was to establish measures of oral health for transparent and explicit reporting of routine data to facilitate more patient-centered and prevention-oriented oral health care. To accomplish this, an intermediate objective was to develop a comprehensive list of topics that were perceived as being valid, important, and relevant for describing oral health and oral health care by a range of stakeholders—notably, patients, policy makers, researchers, and oral health care professionals. The 48 identified candidate topics were divided into 6 themes: access to dental care, symptoms and diagnosis, health behaviors, oral treatments, oral prevention, and oral perception. The development of these topics has been reported (Baâdoudi et al. 2017), and this relates to the first 3 domains of the AIRE instrument. This study describes the formulation of measures based on the topics and provides additional information on usage, reporting, validity, and reliability of the measures as represented by the fourth domain of the AIRE instrument.

This study was approved by the VU Medical Ethical Committee in the Netherlands, the Heidelberg Ethics Committee in Germany, the Copenhagen Videnskabsetiske Komiteer in Denmark, and the Danish Data Protection Agency.

Methods

The methods of this study are reported according to the fourth domain of the AIRE instrument, which provides a structured approach to describe the formulation and usage of measures. The fourth domain consist of 9 items:

Definition of the target population

Efforts needed for data collection

Description of numerators and denominators

Instructions for presenting and interpreting the measures

Assessment of the measures in practice

Validity of the measures

Reliability of the measures

Discriminative power of the measures

Description of a strategy for risk adjustment

The last 2 themes were not assessed because they were not applicable for the purpose of the ADVOCATE measures. However, these themes are addressed in the Discussion section of this study.

Definition of the Target Population

The aim of the ADVOCATE measures was to provide feedback information on oral health and oral health care delivery of general dental practitioners (GDPs) in Europe, irrespective of the practitioners’ age or sex. Feedback information was obtained from adults and children attending general dental practice who were able to provide consent.

Efforts Needed for Data Collection

Prior to the construction of measures, various potential data sources were considered for each topic for adults and children—for example, in-practice administrative data, patients files, insurance claims data, and patient-reported data. Data sources suited for cross-country comparisons within the ADVOCATE project were insurance claims data and patient-reported data.

Insurance claims data are routinely collected for purposes of processing reimbursement claims by GDPs from the health insurer or health system. It has the theoretical advantage of covering the majority of GDPs and, upon implementation in state-of-the-art information systems, keeps the promise of swift data processing. However, such data provide information only on treatment procedures or activities for which claims are required to be made. Methods for the acquisition of claims data required for this study are described elsewhere (Haux et al. 2017).

The ADVOCATE list of topics included a large number of items for which data could not be obtained from readily available administrative data—for example, patient behaviors and perceptions. A questionnaire was therefore constructed to obtain additional patient-reported data on each of the 48 previously identified candidate topics. The questionnaire was developed for adults and for caregivers filling in the questionnaire for a child. Questions were obtained where possible from earlier validated surveys and literature; if not available, questions for a topic were constructed by the research team (Gabel et al. 2017). Face validity of the questionnaire was tested among GDPs and patients in 2 general dental practices. After amendments, the questionnaire was translated and back-translated to Danish, Dutch, and German. The questionnaire was made digitally available in an online application (Fig. 1) and could be accessed through a tablet in dental practice. The online application contained a consent page before the participants were presented with the questions. The questionnaire is available in an earlier publication (Baâdoudi et al. 2019).

Figure 1.

Screenshot of the questionnaire in the online application.

Description of Numerator and Denominator

For 46 of the 48 previously defined topics, a measure was created by constructing a definition and a numerator and denominator. No measure could be created for the topic “risk assessment for tailored prevention,” since data could not be obtained from claims or patient-reported data, and 1 combined measure was constructed for the 2 topics “patient satisfaction with received treatment” and “patients’ perception of dental care.” For the patient-reported data, the denominator was the number of patients responding to the related question in the online application. The numerator was the number of patients who were part of the denominator who responded with a specific answer to the question. For the claims insurance data, the denominator was the number of claimed services per year from a sample of insured patients. The numerator was the unduplicated number of claimed services with a specific restorative or preventive service performed. Examples of the measures created for the topic “fissure sealants” are presented in Table 1. The full list of definitions, numerators, and denominators of measures is reported in the Appendix. A description of the numerators and denominators was published as part of the additional information of the ADVOCATE Field Studies protocol (Baâdoudi et al. 2019).

Table 1.

Definition, Numerator, and Denominator of Measure on Fissure Sealants.

Patient-reported data
Definition	Percentage of adults who respond yes to the question “In the last 24 months, have you had a protective layer put onto your teeth (sealant)?”
Numerator	Number of adults responding yes
Denominator	Total number of adults responding to the question—survey responses: yes, no, don’t know
Claims data
Definition	Percentage of adults with fissure sealants claimed per year
Numerator	Number of adults with at least 1 fissure sealant claimed per year
Denominator	Total number of adults with at least 1 service claimed per year

Instructions for Presenting and Interpreting the Measures

Patient-reported data and claims data were aggregated and presented in an electronic dashboard (Figs. 2 and 3). In the ADVOCATE project, data from participating individual GDPs were presented anonymously in the dashboard, with each GDP having a coded identifier. According to the preferences of the data viewer, a variety of visualizations of the data was possible—graphs, tables, or text—including percentages, ranges, medians and quartiles where appropriate. Data viewers could see their own data or compare themselves with other GDPs. In the ADVOCATE project, a trained facilitator supported GDPs with interpretation of data in the dashboard.

Figure 2.

Screenshot of the patient-reported data presented in the dashboard.

Figure 3.

Screenshot of the claims data presented in the dashboard.

Testing the Measures in Practice

The measures were tested among 39 GDPs in Denmark (n = 17), Germany (n = 12), and the Netherlands (n = 10) as part of the ADVOCATE project. In this study, academic detailing (a defined form of educational outreach) and feedback data were used to stimulate discussions among groups of GDPs and, where appropriate, to initiate changes in clinical practice (Baâdoudi et al. 2019). Patient-reported data were collected in the GDPs’ dental practices from a convenience sample of patients using the online application on a tablet. In addition, aggregated summaries of claims data were available in the dashboard at national and regional levels (Gabel et al. 2017).

GDPs participating in the study had 3 group meetings supported by the trained facilitator in which the feedback data were discussed. During the last meetings, a semistructured focus group interview was conducted in each group of GDPs to evaluate the entire ADVOCATE approach. The methodology, data management, analysis, and interpretation of focus group interviews are described in the ADVOCATE design paper (Baâdoudi et al. 2019). Codes and themes relating to the usage and interpretation of data derived from the measures were extracted with conventional content analysis (Hsieh and Shannon 2005).

Validity of the Measures

Face validity refers to the extent to which a measure is subjectively viewed as covering the concept that it purports to measure. For the ADVOCATE measures, face validity was verified through the extensive 4-stage approach in the development of the ADVOCATE list of topics. This included stakeholder and expert meetings, a 2-stage online Delphi process, and a world café meeting to obtain consensus among stakeholders on topics that were valid, important, and relevant in oral health and oral health care (Baâdoudi et al. 2017). In addition, face validity was broadly appraised by patients completing the questionnaire. At the end of the questionnaire, patients responded to “I found this questionnaire easy to understand” with the following options: strongly agree, agree, neither agree nor disagree, disagree, and strongly disagree.

Reliability of the Measures

The reliability of each measure was evaluated by the within-subject percentage agreement in a test-retest study design. Patients attending a group practice in the Netherlands completed a copy of the patient questionnaire application on a tablet. Those who indicated a willingness to complete the questionnaire again after a period of 3 weeks received an email with the request to fill in the questionnaire once more. A reminder was sent a week later to those who had not yet filled in the questionnaire. For the within-subject percentage agreement, the health status of the participants was assumed stable over the defined period (Streiner et al. 1994).

Results

Usage of the Measures

Obtaining Data

The acquisition of patient-reported data resulted in 3,611 questionnaires completed by patients in the study. Experiences of data collection with the online application varied among the GDPs. Collecting data potentially involved additional time, effort, and a change in daily routine. This often meant practice staff interacting with patients rather than the dentist giving the patient the tablet. Some practices seemed to find this change in routine easier than others. GDPs considered that alternative approaches to the online application were worth considering—for example, having the patient fill in the questionnaire online at home or having the tablet placed on a pillar in the dental practice with only a selection from the questions.

Patients who completed the questionnaire were largely prepared to spend time providing data that covered the broader range of measures: the majority (91%) stated that the time that it took to fill in the questionnaire was reasonable, and the majority (93%) also reported finding the questionnaire easy to understand. GDPs reported that patients needed about 10 min to fill in the questionnaire. However, most GDPs thought that the questionnaire was too long, and a few GDPs also reported that some of the questions were unclear and difficult to complete and would benefit from some revision.

The acquisition of claims data was not straightforward. The barriers included the establishment of strict data protection and privacy measures, as well as complex data acquisition processes that needed to be bespoke for each data owner. Data contents and qualities differed among data sets due to the underlying heterogeneous regulations in different health systems. Eventually these barriers were overcome, but the data could not be aggregated at the level of individual GDPs, only at regional and national levels.

Usefulness as Perceived by GDPs

The GDPs in the study reported during the focus group interviews that the collated data based on the ADVOCATE measures stimulated discussions about their clinical practice. They found the data derived from the online application more informative than the regional claims data because 1) the data set used a larger and more broadly based set of measures and 2) data were reported at the individual GDP level. In contrast, the claims data were considered not very useful because they covered a smaller number of measures, mainly process, and the level of reporting was regional or national. The consequence of a data set comprising a large number of measures meant a requirement for the group facilitator to not only review it but also help the participating GDPs with its interpretation.

Data Derived from the Measures

Table 2 presents the aggregated data collected with the online application by country for each group of measures. The data show that it is feasible to collect patient-reported data on all 46 measures. Overall, more data were collected in the countries with more participating GDPs. These data were represented in the dashboard at the level of individual GDPs (Fig. 2). Figure 3 shows how aggregated claims data were reported in the dashboard.

Table 2.

Descriptive Results from Patient-Reported Data and Test-Retest Reliability of the Measures.

	Patients, n (%) or Mean ± SD
	Denmark(n = 1,858)	Germany(n = 1,204)	Netherlands(n = 549)	Test-Retest Reliability,[a] %
Group 1: Access to dental care
Reason for dental visit				67
New symptoms	173 (9)	136 (11)	37 (7)
Planned check-up	1,225 (66)	608 (51)	393 (72)
Planned treatment	343 (19)	296 (25)	108 (20)
Planned preventive treatment	291 (16)	314 (26)	62 (11)
Trauma/emergency treatment	72 (4)	47 (4)	5 (1)
None of the above	60 (3)	46 (4)	0 (0)
Funding of the dental care provided				67
Completely by public insurance	236 (13)	510 (42)	0 (0)
Partially by public insurance	1,076 (58)	247 (21)	0 (0)
Completely by myself	0 (0)	115 (10)	116 (21)
Fully by private insurance	0 (0)	125 (10)	177 (32)
Don’t know	81 (4)	75 (6)	81 (15)
Partially by private insurance	465 (25)	67 (6)	175 (32)
Partially by private insurance and partially by public insurance	0 (0)	65 (5)	0 (0)
Interval of dental check-ups				100
Every 3 mo	268 (14)	59 (5)	12 (2)
Every 6 mo	883 (48)	722 (60)	355 (65)
Every 9 mo	129 (7)	60 (5)	47 (9)
Every 12 mo	411 (22)	252 (21)	85 (16)
Irregularly	167 (9)	104 (9)	46 (8)
Never	0 (0)	7 (1)	4 (1)
Referrals				67
Hygienist	90 (5)	18 (2)	249 (45)
Periodontist	35 (2)	3 (0)	26 (5)
Root canal specialist	86 (5)	17 (1)	33 (6)
Orthodontist	73 (4)	190 (16)	101 (18)
Other	202 (11)	90 (8)	48 (9)
None of the above	1,413 (76)	904 (75)	200 (36)
Decision not to proceed with recommended dental care
Solely due to costs: yes	306 (17)	117 (10)	95 (17)	92
For other reasons than costs: yes	209 (11)	95 (8)	65 (12)	92
Access to dental care: yes	44 (2)	49 (4)	47 (9)	92
Group 2: Symptoms and diagnosis
Current symptoms				67
Toothache or pain in the mouth	175 (9)	147 (12)	57 (10)
Discomfort in the mouth	132 (7)	104 (9)	48 (9)
Bleeding gums	130 (7)	97 (8)	68 (12)
Dry mouth	53 (3)	26 (2)	16 (3)
Bad taste	30 (2)	47 (4)	14 (3)
Bad breath	62 (3)	31 (3)	21 (4)
Problems with jaw, muscle or joints	36 (2)	72 (6)	10 (2)
None of the above	1,376 (74)	827 (69)	372 (68)
Communication on
Current problems and treatment preferences: yes	1,364 (73)	902 (75)	420 (77)	67
Medical history and medicines: yes	1,042 (56)	741 (62)	221 (40)	83
Social history: yes	738 (40)	449 (37)	168 (31)	92
No. of teeth	25 ± 7.9	25 ± 8.5	26 ± 7.4	58
X-rays taken in the last 24 mo				83
Yes	1,462 (79)	852 (71)	409 (75)
No	302 (16)	298 (25)	98 (18)
Don’t know	94 (5)	54 (5)	42 (8)
Periodontal examination				92
Yes	1,627 (88)	1,061 (88)	504 (92)
No	32 (2)	20 (2)	4 (1)
Don’t know	72 (4)	123 (10)	41 (8)
Oral cancer screening				83
Yes	817 (44)	183 (15)	40 (7)
No	150 (8)	332 (28)	190 (35)
Don’t know	891 (48)	689 (57)	319 (58)
Examination of new caries lesions				83
Yes	1,778 (96)	1,105 (92)	525 (96)
No	20 (1)	19 (2)	7 (1)
Don’t know	60 (3)	80 (7)	17 (3)
Group 3: Health behaviors
Toothbrushing				83
Irregularly	18 (1)	6 (1)	33 (1)
Less than once a day	11 (1)	4 (0)	21 (1)
Once a day	192 (10)	198 (16)	485 (13)
Twice a day	1,417 (77)	881 (73)	2,712 (75)
More than twice a day	220 (12)	115 (10)	360 (10)
Use of fluoride toothpaste				75
Yes	1,569 (84)	686 (57)	346 (63)
No	87 (5)	197 (16)	62 (11)
Don’t know	202 (11)	321 (27)	141 (26)
Interdental cleaning				75
Never	89 (5)	112 (9)	52 (10)
Sometimes	414 (22)	354 (29)	123 (22)
Several times a month	139 (8)	132 (11)	71 (13)
Several times a week	347 (19)	263 (22)	140 (26)
Daily	869 (47)	343 (29)	163 (30)
Smoking				100
Yes, I smoke	277 (15)	217 (18)	79 (14)
No, but I have been a smoker in the past	692 (37)	386 (32)	194 (35)
No, I have never been a smoker	889 (48)	601 (50)	276 (50)
Alcohol consumption				67
Never	114 (6)	138 (12)	91 (17)
Once a month	347 (19)	289 (24)	54 (10)
2 to 4 times a month	521 (28)	395 (33)	141 (26)
2 to 3 times a week	524 (28)	262 (22)	168 (31)
4 to 5 times a week	215 (12)	73 (6)	66 (12)
More than 5 times a week	137 (7)	47 (4)	29 (5)
Diet: consumption of sugary foods and drinks				50
Never	132 (7)	72 (6)	61 (11)
Less than once a day	941 (51)	476 (40)	196 (36)
1 to 2 times a day	637 (34)	536 (45)	220 (40)
3 to 5 times a day	113 (6)	95 (8)	64 (12)
More than 5 times a day	35 (2)	25 (2)	8 (2)
Group 4: Oral treatments
Fillings in the last 24 mo				92
Yes	824 (44)	515 (43)	240 (44)
No	947 (51)	627 (52)	283 (52)
Don’t know	87 (5)	62 (5)	26 (5)
Root canal treatment in the last 24 mo				92
Yes	234 (13)	178 (15)	49 (9)
No	1,587 (85)	995 (83)	489 (89)
Don’t know	37 (2)	31 (3)	11 (2)
Crowns and bridges in the last 24 mo				100
Yes	344 (19)	199 (17)	24 (4)
No	1,494 (80)	984 (82)	517 (94)
Don’t know	20 (1)	21 (2)	8 (2)
Retreatment in the last 24 mo				83
Yes	90 (5)	29 (2)	17 (3)
No	1,713 (92)	1,139 (95)	505 (92)
Don’t know	24 (1)	17 (1)	8 (2)
Not applicable	31 (2)	19 (2)	19 (4)
Periodontal treatment				75
Yes	412 (22)	144 (12)	92 (17)
No	1,374 (74)	1,028 (85)	430 (78)
Don’t know	72 (4)	32 (3)	27 (5)
Extraction in the last 24 mo				100
Yes	294 (16)	207 (17)	75 (14)
No	1,552 (84)	990 (82)	473 (86)
Don’t know	12 (1)	7 (1)	1 (0)
Partial removable denture in the last 24 mo				100
Yes	64 (3)	46 (4)	14 (3)
No	1,772 (95)	1,150 (96)	527 (96)
Don’t know	22 (1)	8 (1)	8 (2)
Full removable denture in the last 24 mo				100
Yes	22 (1)	42 (4)	10 (2)
No	1,810 (97)	1,152 (96)	535 (97)
Don’t know	26 (1)	10 (1)	4 (1)
Dental implants in the last 24 mo				92
Yes	86 (5)	75 (6)	12 (2)
No	1,750 (94)	1,121 (93)	531 (97)
Don’t know	22 (1)	8 (1)	6 (1)
Aesthetic treatment in the last 24 mo				83
Yes	74 (4)	106 (9)	56 (10)
No	1,771 (95)	1,081 (90)	480 (87)
Don’t know	13 (1)	17 (1)	13 (2)
Complication as a result of treatment in the last 24 mo				75
Yes	90 (5)	29 (2)	17 (3)
No	1,713 (92)	1,139 (95)	505 (92)
Don’t know	24 (1)	17 (1)	8 (2)
Not applicable	31 (2)	19 (2)	19 (4)
Antibiotics in the last 24 mo				92
Yes	156 (8)	123 (10)	19 (4)
No	1,666 (90)	1,059 (88)	525 (96)
Don’t know	36 (2)	22 (2)	5 (1)
Group 5: Oral prevention
Fissure sealants in the last 24 mo				83
Yes	202 (11)	223 (19)	38 (7)
No	1,508 (81)	829 (69)	460 (84)
Don’t know	148 (8)	152 (13)	51 (9)
Fluoride application in the last 24 mo				92
Yes	476 (26)	533 (44)	67 (12)
No	1,155 (62)	404 (34)	401 (73)
Don’t know	227 (12)	267 (22)	81 (15)
Professional cleaning in the last 24 mo				50
Yes	1,735 (93)	836 (408)	408 (74)
No	106 (6)	334 (28)	105 (19)
Don’t know	17 (1)	34 (3)	36 (7)
Receipt of oral hygiene advice				50
Yes	1,538 (83)	1,088 (90)	453 (83)
Receipt of dietary advice				83
Yes	624 (34)	355 (30)	170 (31)
Receipt of smoking advice				67
Yes	232 (13)	195 (16)	83 (15)
No	364 (20)	472 (39)	158 (29)
Not applicable	1,262 (68)	537 (45)	308 (56)
Having caries under surveillance				75
Yes	650 (35)	412 (34)	201 (37)
No	1,115 (60)	657 (55)	317 (58)
Don’t know	93 (5)	135 (11)	31 (6)
Group 6: Patient perception
Satisfied with oral function (chewing, speaking, etc.)				67
Strongly agree	1,244 (67)	894 (74)	285 (52)
Agree	542 (29)	241 (20)	232 (42)
Disagree	61 (3)	58 (5)	29 (5)
Strongly disagree	11 (1)	11 (1)	3 (1)
Satisfied with the appearance of teeth				58
Strongly agree	530 (29)	432 (36)	93 (17)
Agree	1,024 (55)	562 (47)	339 (62)
Disagree	263 (14)	179 (15)	110 (20)
Strongly disagree	41 (2)	31 (3)	7 (1)
Having dental anxiety				67
Strongly agree	373 (20)	83 (7)	16 (3)
Agree	261 (14)	113 (9)	52 (10)
Disagree	343 (19)	92 (8)	96 (18)
Strongly disagree	322 (17)	254 (21)	158 (29)
Involvement in (shared) decision making				42
Always	1,686 (91)	1,036 (86)	452 (82)
Sometimes	140 (8)	145 (12)	84 (15)
Seldom	12 (1)	16 (1)	9 (2)
Never	20 (1)	7 (1)	4 (1)
Satisfaction with dental treatment / perception on dental care[b]	9 ± 1.5	9 ± 2.3	8 ± 1.5	58

Agreement.

Rating: 1 = very poor, 10 = excellent.

Validity and Reliability of the Measures

Face validity was considered appropriate, based on the findings of the patients who completed the questionnaire: 93% strongly agreed or agreed that the questionnaire was easy to understand, as opposed to 4% who neither agreed nor disagreed and 3% who disagreed or strongly disagreed.

Table 2 includes the test-retest reliability of the measures. Of 30 patients who completed the initial questionnaire, 12 returned a second questionnaire. The percentage within-person agreement for the 46 measures ranged between 42% and 100%. Of 46 measures, 36 had an acceptable percentage agreement ≥70%, meaning that for 36 measures, at least 70% of the 12 patients gave the same response to a question the first and second time that they completed it. Test-retest was particularly high for measures related to oral treatments; reliability was low for consumption of sugary foods and drinks, oral hygiene advice, professional cleaning in the last 24 mo, and shared decision making. The percentage agreement was also associated with the type of response scale, with higher agreement for measures with fewer response options.

Discussion

The ADVOCATE oral health care measures were successfully tested in general dental practice. It is feasible to collect data relevant to all 46 measures with an online application for patients; accessing claims data and matching the content with the identified measures proved to be challenging. The ADVOCATE oral health care measures have inherent face validity, and the majority have adequate test-retest reliability with patient-reported data.

This is one of the first studies that has further evaluated the usage and scientific properties of measures in oral health care (Righolt et al. 2019). There are now a number of sets of oral health care measures developed, but the number of measures and the rigor of the development process vary widely; only a minority of the measures have been evaluated by using them in practice. The Dental Quality Alliance (2016, 2018) developed a broad set of oral health care measures, with extensive stakeholder involvement and further evaluation (Herndon, Crall, et al. 2015; Herndon, Tomar, et al. 2015; Hunt and Ojha 2017). However, 23 of 24 measures are process measures, and only 3 of them have been assessed in practice. Process measures are more easily developed and evaluated from some data sets of administrative claims. However, they lack information on patient outcomes and often do not include information on diagnosis, demographics, or data about patients’ experiences and their health behaviors (Institute of Medicine 2011; Kalenderian et al. 2011; Herndon, Crall, et al. 2015). These are important limitations of process measures in quality-of-care evaluation. In contrast, the ADVOCATE project has demonstrated the successful development and testing of a broader set of oral health care measures.

Strengths of the ADVOCATE measures are the robust approach to their development, including the involvement of a wide range of relevant stakeholders. In addition to process measures, they include patient experiences and outcomes alongside health behaviors. This study provides additional information on their usage and scientific properties. Another strength is the use of qualitative and quantitative data in this further evaluation of the ADVOCATE measures.

Limitations of this study include the barriers identified with data collection in the dental practices by the GDPs. In addition, the response rate of patients recruited for the test-retest reliability was low. However, the number of patients was sufficient to distinguish measures with low, medium, and high retest reliability. Nevertheless, more data might have strengthened the evaluation of the reliability of the ADVOCATE measures.

Another limitation is the lack of assessment of the discriminative power and risk adjustment of the measures. In the ADVOCATE project, the measures were used in the context of a proof-of-concept study that eschewed a judgmental, normative approach to define “quality.” In addition, the use of a convenience sample and the relatively short time frame of the study did not allow assessment of trends over time.

Regarding risk adjustment, the online application included a number of questions on patient demographics (age, sex, and education level following the International Standard Classification of Education; Organisation for Economic Co-operation and Development 1999) that could theoretically be used to control for case mix and other covariables. Such control was not used in this study because it would have been inappropriate to do so given that a convenience sample was used. It would be appropriate to use such demographic variables in a further study with a representative sample of patients in dental practices.

The ADVOCATE oral health care measures were used in the ADVOCATE project to provide feedback data intended to stimulate reflection and group discussions about clinical practice with the intention of intrinsically motivating dentists toward improving quality of care or reducing unwarranted variation (Baâdoudi et al. 2019). These measures have the potential to be useful in the development of health policies aimed at improving the quality of health care and to routinely describe the care delivered by health systems.

Further research should focus on evaluation of the validity and reliability of obtaining oral health care data from patients by using a representative sample. It seems unlikely that a representative sample of data could be obtained with a modified online application completed by patients while at their dental practice. There are challenges in routinely collecting data for a wide set of measures. Alternative and modified approaches would be required, including potential incentives (e.g., value-based payments) created within the health system.

Conclusion

A total of 46 measures have been successfully developed from 48 predefined topics describing oral health and oral health care from the ADVOCATE project. The usage, reliability, and validity of oral health care measures have been tested in general dental practice and assessed according to the AIRE instrument. A broad range of measures, including patient experience and health behaviors, has been used and found to be useful to stimulate discussions about clinical practice. The measures have the potential to be useful in the future at all levels within a health system.

Author Contributions

F. Baâdoudi, contributed to design, data acquisition, analysis, and interpretation, drafted and critically revised the manuscript; N. Maskrey, contributed to conception, design, data interpretation, drafted and critically revised the manuscript; S. Listl, G.J.M.G. van der Heijden, contributed to conception, critically revised the manuscript; D. Duijster, contributed to design and data interpretation, drafted and critically revised the manuscript. All authors gave final approval and agree to be accountable for all aspects of the work.

Click here for additional data file.

Supplemental material, DS_10.1177_2380084420902464 for Usage and Scientific Properties of the ADVOCATE Oral Health Care Measures by F. Baâdoudi, N. Maskrey, S. Listl, G.J.M.G. van der Heijden and D. Duijster in JDR Clinical & Translational Research