Developing a Preliminary Conceptual Framework for Guidelines on Inclusion of Patient Reported-Outcome Measures (PROMs) in Clinical Quality Registries.

PURPOSE: Patient-centred and value-based health-care organisations are increasingly recognising the importance of the patient perspective in the measurement and evaluation of health outcomes. This has been primarily implemented using patient-reported outcome measures (PROMs). Clinical quality registries (CQRs) are specifically designed to improve direct clinical care, benchmark health-care provision and inform health service planning and policy. Despite CQRs having incorporated the patient perspective to support the evaluation of health-care provision, no evidence-based guidelines for inclusion of PROMs in CQRs exist. This has led to substantial heterogeneity in capturing and reporting PROMs within this setting. This publication is the first in a series describing the development of evidence-informed guidelines for PROMs inclusion within CQRs in Australia.
METHODS: This study consisted of three components: 1) a literature review of existing evidence of guidelines, enablers, barriers, and lessons learnt of PROMs use within the CQRs setting; 2) a survey of Australian CQRs to determine current practices for PROMs use and reporting; and 3) development of a preliminary conceptual framework for PROMs inclusion in CQRs.
RESULTS: Content analysis of the literature review and survey of 66 Australian registries elicited eight categories for the conceptual framework. The framework covers eight components: rationale, setting, ethics, selection of PROMs, administration, data management, statistical methods, feedback, and reporting.
CONCLUSION: We developed a preliminary conceptual framework, which classified findings, from both the literature and the survey, into broad categories ranging from initial development to outcome dissemination providing the structure for development of guidelines in the next phase of this project, engaging national and international leaders in health-related quality of life research, clinicians, researchers, patient advocates and consumers.

Introduction

Patient-reported outcome measures (PROMs) are standardized, validated questionnaires designed to assess patients’ perceptions of their own physical and mental status and well-being.1 There are numerous PROMs, differing in the constructs they assess, the wording and nature of the questions asked, the number of questions asked, and how the answers are scored.2,3 Many such measures were originally designed for assessing treatment effectiveness in the context of clinical trials, but are now being used more widely to assess patient-reported outcomes (PROs) in clinical practice.

Clinical quality registries (CQRs) are organizations, which systematically monitor the quality of health care within specific clinical domains by routinely collecting, analyzing and reporting health-related information.4,5 They use predefined indicators designed to assess variation across structural, process and outcome measures to benchmark quality of care. CQRs have received increasing attention as a means of improving quality and reducing the cost of health and medical care, through identifying variations in clinical practice and care, and assessing the uptake of effective treatment.6

CQRs have existed for many years, but have expanded significantly in the last decade to monitor clinical outcomes and facilitate evidence-based clinical practice. Emilsson et al4 recently reviewed 103 CQRs and concluded that Swedish quality registries contain comprehensive clinical data that provide an important source for assessment and development of quality of care and research. The Dutch Institute for Clinical Auditing in the Netherlands, established in 2011, facilitates the development and maintenance of national outcomes registries around various medical conditions.7 Acute stroke registries have been established in Sweden, Germany, Canada, Australia, and many other countries.8–11

In Australia, the number of known CQRs collecting clinical data has been growing and the scope of coverage has increased.12

PROMs are increasingly being introduced into CQRs in Australia, providing a personal perspective on the expectations and impact of treatment.5 For example, the Victorian Orthopaedic Trauma Outcomes Registry13 and the Prostate Cancer Outcomes Registry – Victoria14 both collect and report PROs at a time of clinical stability. The Australian Arthroplasty Clinical Outcomes Registry reports pre- and post-operative PROs for primary and revision procedures15 and the Australian Breast Device Registry16 captures PROs to monitor the long-term performance of breast devices, collects PROs at 1, 2, 5 and 10 years after procedure.

Including PROMs in CQRs offers numerous advantages.17 First, incorporation of the patient voice helps keep outcome measurements of care patient-centered. Further, symptom burden, quality of life (QoL), and satisfaction with care are dynamic variables that cannot be recreated accurately through retrospection; they are essentially lost if not captured “in the moment.” For this reason, routine, systematic, and longitudinal collection of PROs is recommended and should be a standard of clinical practice.18,19 Serial collection of PROs in CQRs can improve understanding of the trajectory of an individual patient’s symptom burden and QoL over the course of disease or treatment, to inform clinicians of the variability between patients and to provide information on the value that the individual patient places on their health status.17

Although some registries have included the collection of PROs as part of their current practice, widespread adoption of PROs as a key component in CQRs is yet to occur. Successful PROM implementation in CQRs includes many challenges and, therefore, requires clinical, operational, and analytic resources and expertise.20 Thus, evidence-based guidelines to guide successful PROM implementation are critically important to assure meaningful PROs data collection, use, interpretation, and reporting. Presently, evidence-based guidelines for the inclusion of PROs do not exist and anecdotal evidence suggests substantial heterogeneity in the capture and reporting of PROMs in this setting. Clear guidelines are needed to support ethical, effective, and transparent use of PRO data collected across all CQRs.21 In particular, guidelines should address when PRO data must be shared promptly with the treating clinician, as in the case of a safety signal, and ensure that adequate training is provided to physicians who are reviewing the PRO data.

This publication is the first in a series describing the development of evidence-informed guidelines for PROs inclusion within CQRs in Australia. The aims of this study were to 1) conduct a literature review of publications on existing guidelines, enablers, barriers and lessons learnt for PROMs inclusion in CQRs; 2) conduct a survey of Australian CQRs to understand current practices for PROMs inclusion and reporting; and 3) summarize the results as a preliminary framework which will ultimately inform the development of the guidelines.

Materials and Methods

Literature Review

An extensive literature review was conducted in MEDLINE and Embase databases as they are the most widely searched databases for health-related topics, and have comprehensive coverage and capture of the content of health services research. Further documents were identified via Google and Google Scholar. The search was limited to the English language, and included full peer-reviewed abstracts from 2000 to 2018. Studies were included if they met at least one of the following criteria: 1) they clearly described guidelines or recommendations for PROMs inclusion and reporting in the registry setting; 2) they presented challenges and lessons regarding PROMs collection in the registry setting; 3) they demonstrated the impact of PROMs collection on patient outcomes in the registry setting. Research publications arising from registry and/or PROMs data analyses were excluded from this review. The returned search results were screened for duplicates, irrelevant articles, conference abstracts, and commentaries.

Survey of Australian Registries

A cross-sectional 32-question online survey was distributed to registry managers, coordinators and data custodians of identified CQRs in Australia. Survey questions were based on the results of the literature review and discussions with registry managers at Monash University. The survey consisted of the following three areas: 1) information on type, coverage and size of the registry; 2) information on collection of PROs, rationale, instrument type, mode and frequency of administration, rate of completeness and storage of the PRO data; and 3) analysis, reporting and impact of PROs. A copy of the survey was delivered to each registry via Qualtrics Survey Software between July 2018 and September 2018.

A list of registries and contact details of their managers, coordinators and data custodians was generated through a web-based search of the grey literature using the following key words: registries, clinical, Australia, Victoria, Queensland, Western Australia, South Australia, New South Wales, Tasmania and Northern Territory. All clinical quality, device, procedure and drug registries were eligible for inclusion in the study.

An implied consent process was utilized to recruit potential survey participants. Each potential participant was provided an invitation letter with an explanation of the study, instructions on how to complete the survey and a unique hyperlink of the survey. Follow-up e-mails and telephone calls were made to respondents to clarify responses where necessary.

Conceptual Framework

According to Miles and Huberman,22 a conceptual framework is a visual or written product that “explains, either graphically or in narrative form, the main things to be studied-the key factors, concepts, or variables and the presumed relationships among them”. Although there is no standard methodology for constructing a conceptual framework, most researchers rely on an approach that combines multiple sources of data.23 Content analysis of the literature review and survey has been used to elicit the categories for our conceptual framework. The framework will be used as the supporting structure to facilitate both the grouping of attributes to be considered when developing guidelines for PROM use as well as ensure all aspects of PROM implementation in the registry setting are covered.

Data Analysis

Findings from the Literature

A data extraction form was developed to extract information from the identified articles relevant to the PROMs (study detail, rationale, setting, ethics, instruments, administration, data management, statistical methods, feedback, and reporting). Search results were summarized by themes to fulfil the aims of the review.

Survey

All clinical registries in Australia identified through the grey literature were invited to participate. A sample size calculation was deemed unnecessary given the study was primarily descriptive. Data were analyzed and reported using frequencies, means/medians and standard deviations (SD)/interquartile ranges (IQR) as appropriate. Data analysis was undertaken using the STATA 14 package. Where open-ended questions were used, response data were classified into key themes and subthemes for descriptive analysis and reporting.

Results

Review of the Literature

The search strategy yielded 3761 publications (Figure 1). Their titles and abstracts were screened according to the inclusion criteria, specifically focusing on lessons learnt, advantages and disadvantages, guidelines and recommendations for PROMs inclusion in CQRs. Of these, 3661 references were reviewed and full copies of 412 articles were retrieved. The screening of full texts resulted in retaining 10 papers.

Figure 1

Flow chart in PRISMA consort diagram of the screening and selection of studies.

The review found four papers on guidelines and recommendations for the collection and use of PROMs in registries. Breckenridge et al24 presented a summary of expert recommendations for PROs collection within renal registries in Europe. Rolfson et al,25 Franklin et al26, and Franklin et al20 presented a series of recommendations and developed a framework for selection, administration, and analysis of PROs by arthroplasty registries internationally.

There were six individual registry studies on PROs collection: three from the US,27–29 two from the UK,30,31 and one from Australia.32 There were two studies on cancer registries,30,32 a study on PROs in the management of chronic spinal pain,31 three studies on surgery registries: spine surgery,27 joint arthroplasty,29 and an unselected surgical patients registry.28

Rationale for Collecting PROs Data

In general, the rationale for PROs data collection reported in all individual registry studies and review/guidelines and recommendations reflects the definition of PROMs as tools to measure the patient’s perception of their own physical and mental status and well-being. Franklin et al20 in their framework to guide collection and use of PROMs in learning health-care systems, based on orthopedic registries, identified key stakeholders to whom PROMs provided valuable information; patients and clinicians, hospital leaders and clinicians, insurers and hospital leaders, as well as researchers, policy makers and funders (Table 1). The purposes of PROs collection focused on their value in tracking the patient recovery process to improve services and to develop interventions in particular clinical domains, such as surgery,27,28 spinal osteopathy services,31 cancer,30,32 and total joint arthroplasty.29

Table 1

Characteristics of Included Studies (N=10) on Capturing Patient-Reported Outcomes Measures in Clinical Registries

Study Details	Rationale	Setting	Ethics	Instrument(s)	Administration		DATA MANAGEMENT			STATISTICAL METHOD	Reporting
Author (Year)CountryAims	Rationale for collecting PROMs	PopulationSample Size (N)	Patient ConsentConfidentiality	New/ExistingGeneric/Specific	Frequency	Mode & Methods	Entry & Quality Check	Storage	Data Completeness	Data Analysis	Dissemination
Individual registry studies
Ashley et al (2011)30UK, cancer registryTo describe technical development of the electronic PROMs from cancer survivors system, for linking PROMs with clinical data in cancer registries	To improve understanding of cancer survivors’ experiences and outcomes to inform development and targeted provision of support services and interventions.	Non-metastatic breast, colorectal and prostate cancer patientsNS	Written informed consent	ExistingBoth	≤6, 9, and 15 months post-diagnosis	Online	Via unique ID and password to log in and complete PROMs	Within the registry	Extent of missing responses recorded in the electronic system	Summary statistics	NS
Azad et al (2016)27USA, spine surgery registryTo detail the methodology of constructing an integrated electronic health record system to collect QoL metrics and demonstrate the effect of data collection on routine clinical workflow.	To determine value of surgical care	All new patients after spine surgery791	NS	ExistingBoth	3, 6, and 12 months post-operative	Online, integrated electronic health record (EHR) system	Electronic entry	Within the registry	98%	NS	Via clinician-facing dashboard
Helsten et al (2016)28USA, unselected surgical patient registryTo describe methods of Services FollowingYearlong Surgical Outcomes Surveys (SATISFY-SOS) registry, which combines perioperative PROMs with granular electronic data to track patients’ postoperative trajectories.	To track patient recovery after surgery and to answer how the presence or absence of reported postoperative complications might impact QoL and functional recovery after surgery	All patients attending assessment center44,081	Consent	ExistingBoth	Baseline (enrolment), 30 days, and 1-year postoperatively	Online, mail, or telephone.If a patient did not respond to initial e-mail, reminders sent on day 2 and 4 after the first contact	Via unique ID	Aggregate data stored on a dedicated research database server	30-day survey (62% response rate), 1-year survey (71%)	NS	NS
Moloczij et al (2018)32Australia, lung cancer registryTo describe the design and structure of a study to develop and pilot test a PRO framework to systematically and longitudinally collect PRO data from a cohort of lung cancer patients at a comprehensive cancer centre.	To provide important information that have potential to translate to meaningful service improvement	Patients with lung cancer selected for the pilot30–60	Opt-in	ExistingDisease-specific	Baseline, 2, 6, and 12 months	Paper, online	Manually entered and checked	REDCap/Microsoft Access database	To be analyzed	Summary statistics	NS
Morris et al (2016)31UK, spinal osteopathy service registryTo report the feasibility of implementing routine registry data collection and sustaining a high rate of data capture within a secondary care spinal osteopathy service, using the Spine Tango Conservative registry data collection tool with multiple PROMs.	To provide important indicators of treatment effect	All patients349	Consent	ExistingBoth	Baseline, end of 1st treatment cycle, beginning of 2nd treatment cycle, discharge	Clinic. Paper	By data entry/analysis staff	Kept in spreadsheets and patients notes	55−90%	Summary statistics	NS
Rana (2016)29USA, joint arthroplasty registryTo describe one hospital’s experience in building and maintaining an electronic PROM database for a practice of 6 board-certified orthopedic surgeons.	To improve delivery of health care, including quality of care.	Primary and revision hip, knee, and shoulder arthroplasties>4500	Informed consent to allow use of data in future research	ExistingBoth	Preoperative (baseline), postoperative 6 weeks, 3 months, 1, 2 and 5 years	Electronic tablet + email	Database	Electronic PROM database	Baseline (91% response rate), post 6 weeks (60%), 12 months (70%)	Summary statistics	Patients; and surgeons: graphically present patients’ PROM data at their follow-up office visit.
Reviews, Guidelines and Recommendations
Breckenridge et al (2015)24Europe, renal registriesTo describe the main points learned from a consensus meeting on how to routinely collect PROMs in renal registries in Europe.	PROMs: to highlight relevant symptoms/changes in symptoms, promote patient engagement in treatment, improve patient outcomes, inform patients’ choice of treatment, and assess quality of care across different hospitals.PREMs: to improve quality in clinical services.	All patients on renal replacement therapyN/A	Consider ethics/consent/data protection	ExistingBoth	At least annually; preferably not during dialysis	Unassisted self-report; no clear preference for paper/web-based	NS	NS	NS	NS	NS
Rolfson et al (2016)25International, joint arthroplasty registriesTo describe the Working Group recommendations for selection, administration, and analysis of PROMs by arthroplasty registries.	To provide important information that matters to patients.	All – for quality improvement a sample – for research purposes	Informed consent	ExistingBoth	Pre- and post- joint replacement as determined	Paper-and electronic based. In clinic	Entered in an electronic form	Registry	Missing values should be reported per item	Propensity scores, regression, risk adjustment	Clinicians
Franklin et al (2013)33 [Review]International, total joint arthroplasty registriesTo review current PRO use by existing national registries, challenges to integrating PROsin national registries, lessons from national registries that have integrated PROs, and suggestions to guide future adoption of PROs.	To quantify optimal outcomes from total joint arthroplasty procedures as defined by the patient, as a measure of positive outcomes (eg, pain relief and improved function) and as a marker of risk for negative outcomes (eg, persistent pain or higher risk for implant failure).	All total joint arthroplasty patients or a representative subset of patients if 100% registry coverage	Consent	ExistingBoth	Pre-arranged timesAdvantages of direct-to-patient model independent of clinical visits	Electronic, computer-adapted measures are preferable.	Data infrastructure and data collection procedures must support a complete and accurate database.	Data infrastructure and data collection procedures must support a complete and accurate database.	Data infrastructure and data collection procedures must support a complete and accurate database.	NS	NS
Franklin et al (2017)20International, orthopedic registriesTo describe framework and steps that were identified by stakeholders as best practices to guide PROM capture and use in the arthroplasty registries	1. Patients and Clinicians/Individual patient care decisions: Individual patient-centered decisions to prioritize, treat, and monitor disease symptoms and health status.2. Hospital Leaders and Clinicians/Quality improvement: Monitor and improve aggregate patient outcomes as compared to national best practice and benchmarks3. Insurers and Hospital Leaders/Value-based payment: Measure outcomes as compared to costs and utilization to optimize health care value4. Researchers, Policy makers, and Funders/Population health and research: Generate new evidence for best clinical practices across patients to achieve optimal health status over time.	Clear definition of patient population is critical as the disease, symptoms, or procedure will inform the frequency and duration of PROM collection.	Consent	ExistingBothRecommended checklist for PROM selection: content, patient acceptance, cost/licensing, ease of clinic integration, and external considerations	Consistent time intervals when patients are likely to achieve peak outcomesAdvantages of direct-to-patient model independent of post-TJR clinical visits	Paper, web portals (incl. HER portals), tablets, personal computers, mobile phones, automated voice surveys, and scannable paper forms	PROM collection infrastructure must be flexible to meet demands of varied patients and diverse settings; depending on use of data, available resources and technology	Stored PROMs should include both the item level responses and the summary scores	Completeness of PROM collection is most likely if data collection is embedded in standard clinic workflow, supported by electronic means, and serves patient and clinician decisions	Scoring, risk-adjustment and clear visual displays	Appropriate clinician alerts; data aggregated at intervals to compare risk factors and outcomes; integration of clinical data and PROMs

Abbreviations: QoL, quality of life; PROMs, patient-reported outcome measures; PROs, patient-reported outcomes, NS, not stated.

Settings for PROs Data Collection

In general, studies included in the review either provided inclusion and exclusion criteria to identify suitable sub-groups of participating patients, or alternatively, PROs were collected or planned to be collected from all patients in the registry. Numbers of included patients ranged from 30 to 60 in a lung cancer registry32 to over 44,000 unselected surgical patients.28 Guidelines for PROs data collection in arthroplasty registries indicated that in some situations representative data may be obtained from using only a subsample of patients. When PROMs are used for quality improvement, all patients should be assessed. In contrast, a sample of patients may be adequate for research purposes when it is a random, unbiased sample that has a sufficient number of patients for analysis and drawing of conclusions.25 This is supported by other literature stating that PROs data may be collected from a representative sample of patients if a registry has 100% patient coverage33 and a random and unbiased sample could be selected for research purposes based on power-based sample size calculations.25 However, when assessing all patients in the registry, considerations need to be given to patient characteristics, such as their age, gender, socio-economic and mental health status, language, cultural barriers, health literacy, and other patient factors.34

Ethics and Confidentiality

All individual registry studies, except for one,27 reported using a written consent or opt-in procedure. This approach was also recommended by the guidelines for PROs data collection,20,25,33 while Breckenridge et al24 indicated the general “need to consider the legal, data protection and consent constraints of participating [in registry] countries” (p. 1612). An advantage of the written consent procedure is an increase in patient involvement and a better understanding of the advantages of PROs data collection.20

Instruments

All studies and guidelines indicated the use of existing PROMs, both generic and disease-specific tools. In some cases, e.g., total joint arthroplasty registries, use of several disease-specific PROMs measuring discrete outcomes, such as positive (e.g., pain relief, improved function) and negative outcomes (e.g., implant failure, persistent pain) was recommended.33 Franklin et al20 proposed a checklist for PROMs selection based on five criteria: 1) PROM content: items specific to single condition and/or treatment; or to assess global health status, 2) patient acceptance: language and cultural appropriateness for users; literacy level appropriate for users, 3) costs, licensing: proprietary with use fees or publicly available, 4) ease of clinic integration: number of items, administration time, and patient burden; valid across multiple modes of administration (electronic, paper or oral), and 5) external considerations: legacy or concurrent benchmarks available to guide interpretation; mandates for specific measures; legacy inclusion in disease-specific registries (p. 7).

Mode and Methods for PROMs Administration

The timing, frequency, and duration of PROs data collection depend on the individual disease, its symptoms and available treatments within the patient population. In general, registries collected data at pre-arranged times, which usually reflected critical times in the recovery process.20 The typical data collection points were before the procedure (e.g., arthroplasty25), at baseline (i.e., time of procedure or diagnosis), and follow-up (at least annually). PROs data collection at baseline was not always available due to the nature of condition or disease (e.g., trauma).

The registries reported using a wide range of methods of data collection: paper, web-based (via computer, tablet, mobile phone applications), and telephone. Some registries collected data using one method only, e.g., online27,30 or paper questionnaires.31

Other registries offered (or tested feasibility of) several data collection options to participants.28,32 In general, although there is a movement towards electronic data collection; it has been recommended that registries remain flexible and respond to the needs of particular groups of patients, as some populations may prefer specifically paper or web-based questionnaires.20 The available infrastructure and setting of the registry will also determine the mode and method of data collection.20

Direct-to-patient PROMs administration, i.e., collecting data directly from patients at their homes, was suggested as an alternative to collecting data during follow-up visits to clinicians’ offices or services.20,33 Timing of clinical visits often vary depending on need and doctor’s preference, and direct-to-patient data collection allows assessments at pre-determined times, which are independent of scheduled visits. In addition, this mode of data collection minimizes the burden to clinical staff.

Data Management, Storage and Completeness

Data were stored in electronic databases, such as REDCap,32 or in spreadsheet and patients notes.31 The reported data completeness varied across individual registry studies, over time, data collection methods, and administration time points. For instance, there was 98% QoL assessment completion via a computer survey in a spine surgery registry.27 The volume of data captured by a secondary care spinal osteopathy service registry using a paper PROM increased from 55% to 90% over 2011–2015.31 For unselected surgical patients, the 30-day PROM response rate was 62% (73% mail, 19% telephone and 8% e-mail) and 71% for the 1-year survey (78% mail, 8% telephone and 13% e-mail).28 Some registries used electronic options to monitor the completeness of the entered data.30 In a feasibility study of registry data collection, Morris et al31 identified common threats to the process of data collection and proposed measures to tackle the problem of incomplete data.

Statistical Methods and Data Reporting

Where reported, the majority of registries provided summary statistics to clinicians or in peer-review publications. Franklin et al20 stressed that meaningful interpretation of PROMs requires application of refined risk-adjustment methods.

Only two studies provided information on feedback and reporting: a surgical registry provided real-time access to patient data27 and an arthroplasty registry produced an annual report, including PROs data analyses and results, which is distributed to patients, primary care providers, and insurance carriers.29 A framework to guide PROs data collection and reporting stressed the usefulness of consistent measures scored in real-time and that trends over time for clinicians facing treatment decisions.20 It also suggested the use of appropriate clinician alerts based on registry PROs assessments (e.g., depression scores), aggregation of data at certain intervals, and integration of PROs and clinical data.

Survey Results

We identified 66 Australian registries through the grey literature review, internet, and through contact with the national Registry Special Interest Group, a peer-support group for researchers and clinicians managing or operating registries. Of these, 35 (52%) registries responded, 22 (33%) were followed-up by phone and confirmed they do not collect PROs, while we were unable to contact the remaining nine (14%) registries. Nineteen (29%) registries confirmed that they collected PROs (Table 2).

Table 2

Survey Responses for the Registries Who Collect PROMs in Australia (N=19)

Survey Question	N (%)
Registry Information
Type of the registry
Condition/disease	11 (57.9)
Procedure	6 (31.6)
Drug/device	2 (10.5)
Population coverage
<20%	8 (42.1)
20–39%	2 (10.5)
40–59%	1 (5.3)
60–79%	4 (21.1)
>80%	4 (20.1)
Mean (SD) years registries operating	8.9 (6.2)
Mean (SD) years registries collecting PROMs	7.4 (6.3)
PROMs COLLECTION
Rationalea
Quality of care and safety improvement	13 (68.4)
Monitoring outcomes of care	13 (68.4)
Population health studies	6 (31.6)
Measuring burden of disease	6 (31.6)
Shared decision making and patient-centered care	4 (21.1)
Other (clinical trials, measuring quality indicators)	6 (31.6)
Consent model
Opt-out model – part of the registry	14 (73.7)
Opt-in (written-informed) – separate from the registry	3 (15.8)
Opt-in (written-informed) – part of the registry	2 (10.5)
Instrument typea
Generic	13 (68.4)
Disease/condition specific	13 (68.4)
Performance/process measures (patient reported experience measures)	3 (5.8)
How the instrument were chosen?a
Practical/clinical utility	14 (73.7)
Reliability and validity of the instrument	12 (63.2)
Based on global standards	9 (47.4)
Availability and cost	8 (42.1)
Length of time required to complete	8 (42.1)
On recommendations of others	5 (26.4)
Style of the instrument	1 (5.3)
Developed a new instrument	1 (5.3)
Patient population
All	15 (78.9)
Specific	4 (21.1)
How routinely PROMs are captured?a
At multiple occasions	15 (78.9)
At baseline	12 (63.2)
At single time point	4 (21.1)
Mode of administrationa
Postal	11 (57.9)
Telephone call	10 (52.6)
On paper, given to the patient (e.g., in clinic)	6 (31.6)
Via email	4 (21.1)
SMS with/without a link to an email	1 (5.3)
How is the PROMs data collected?
Directly from electronic patient responses	9 (47.4)
By dedicated data collectors	7 (36.8)
By clinicians	3 (15.8)
Where is the PROMs data stored?
As a part of the main registry	15 (78.9)
On a stand-alone platform	2 (10.5)
Other	2 (10.5)
Completeness at baseline
<50%	5 (26.3)
>70%	5 (26.3)
Not described	9 (47.3)
Completeness at other time points
<50%	2 (10.5)
50–70%	2 (10.5)
70–90%	6 (31.6)
>90%	3 (15.8)
No known	6 (31.6)
Missing data handling
Proportion of missing data is unknown	9 (42.1)
Exclude missing data from the analysis	7 (36.9)
Use imputation models to replace missing data	3 (15.8)
Major challenges related to PROMs collectiona
Cost of data collection	10 (52.6)
Data completeness issues	8 (42.1)
Timing of administration	7 (36.8)
Integration with other outcome measures	4 (21.1)
Consent issues	2 (10.5)
Other	3 (15.8)
REPORTING AND IMPACT OF PROMs
How PROMs are reported?a
In aggregated form	15 (78.9)
Benchmarking reports (between sites)	8 (42.1)
Others –a for publications, or yet to report	6 (31.8)
Used for comparisons	2 (10.5)
At individual level	1 (5.3)
To whom PROMs are reported?a
Conferences and forums	10 (52.6)
Health services	9 (47.4)
Peer-reviewed publication and journals	8 (42.1)
Individual clinicians	6 (31.6)
Funders and industry	6 (31.6)
Government departments	5 (26.3)
Publicly available annual reports	1 (5.3)
Have not been decided	5 (26.3)
Available to clinicians	6 (31.6)
Has made an impact	2 (10.6)

Note: aMultiple response options were possible.

The most frequently reported reasons for those who do not collect PROs included budgetary restrictions and lack of resources for data collection. Some of the registries reported they would be considering the inclusion of PROs in the future.

Eleven (58%) registries that collected PROs were condition-related (e.g., arthritis, trauma, stroke, lupus, prostate cancer, multiple myeloma, various ophthalmic conditions, etc.). Another six (32%) were procedure-based registries (e.g., bone marrow transplant, cardiac outcomes, spine surgery, joint replacement) and the remaining two (11%) were device registries (breast device and bariatric surgery).

The average duration of data collection from registries who collect PROs was 8.9 (6.2) years, with eight (42%) covering more than 60% of the patient population. On average, the registries had collected PROs for 7.4 (6.3) years at the time of the survey.

In terms of rationale for PROs inclusion, monitoring outcomes and quality of care and safety improvement was mentioned by 13 (68%) of the registries, followed by measuring burden of the disease and population health studies. The opt-out approach as a part of the existing registry data collection was the most frequently used consent model in 14 (74%) registries.

Equal numbers of registries (13, 68%) used generic or disease-/condition-specific instruments. Only five (26%) registries collected generic instruments, while the other four (21%) collected disease/condition only specific instruments. The most frequently reported generic instruments included EuroQol‐5 Dimensions (EQ5D), Short Health Survey (SF12), Health Assessment Questionnaire (HAQ) and Patient Health Questionnaire (PHQ). Only one registry developed a new instrument.

The instruments were chosen based on their practical/clinical utility in 14 (74%) registries, followed by reliability and validity of the instrument as per published evidence in 12 (63%) registries. Other choices were based on global standards (9, 47%) (e.g., International Consortium for Health Outcomes Measurement) and availability and cost of the instrument in 8 (42%) registries.

The majority (15, 79%) of registries collected PROs from all the patients, and at multiple occasions. Twelve (63%) registries collected PROs at baseline. Eleven (58%) registries collected PROs responses via post, 10 (53%) – by telephone, and five (26%) employed multiple data collection modes. PROs were imported from electronic patient responses by nine (47%) registries. Data completeness varied between 70-90% for six (32%) of the registries. To maximize PROs data completeness rates, some registries made follow-up calls and regular mail-outs to patients or trained data collection staff.

The proportion of missing data was unknown in eight (42%) registries, whereas others excluded missing data from the analysis or used imputation models to replace missing data. The reason for the missing data was not reported. The cost of PROs data collection was the most frequently (10, 53%) reported challenge related to the data collection, followed by PROMs data completeness issues, timing of administration or integration with other outcome measures.

Most of the registries (15, 79%) reported their PROs data in aggregated form (e.g., annual reports or peer-reviewed publications), eight (42%) used the data as part of their benchmarking activities and only one registry reported the data back to patients. In terms of data analysis, PROs data were analyzed descriptively. Other methods included linear, logistic and survival analyses. Six (32%) registries stated that their data were available to clinicians, and only two (11%) said that PROs data had made an impact on patients by changing hospital practices, improved approaches to patient management, and development of an education program for specific groups.

Based on the findings from the literature and results from the survey results we developed a preliminary conceptual framework to guide development for PROs inclusion in CQRs (Figure 2).

Figure 2

Conceptual framework for patient-reported outcome inclusion in clinical registries.

The framework includes eight components (rationale, setting, ethics, instrument, administration, data management, statistical methods, and feedback and reporting) containing 24 sub-components. Rationale is divided into two sub-components: purpose of PROs data collection in the registry and stakeholders from whom data will be collected and/or reported (i.e., patients, clinicians, health-care services. Setting consists of three sub-components: eligibility of the sample, description of the population for which PROs will be captured and sample size (i.e., all patients in the registry vs a defined sub-group). Considerations taken into account will include language and cultural barriers, literacy and education, non-response rates, patient refusal, and other patient factors.

Ethics includes one sub-component covering patient consent-related issues (e.g., opt-in vs opt-out approach). Instrument(s) consists of three sub-components: patient involvement (e.g., development of a new or abbreviated measure, validity testing and evaluation of the measure), determining the instrument to be used for capturing PROs (i.e., whether a new instrument is needed), and whether the instrument should be generic or condition/disease specific.

PROMs administration covers the following three aspects: timing and frequency (e.g., determining a suitable baseline), mode and method (i.e., electronic, phone or pen and paper) and follow-up methodology.

Data managements consist of three sub-components: entry and quality check of the PROs data (i.e., determining data entry methods and algorithms for validity checks), storage (i.e., platform, portal, security, and data access) and data completeness.

Statistical methods are divided into six sub-components: descriptive analysis, inferential analysis, risk adjustment, benchmarking, time series and real-time analysis. These will overview potential data analyses and methodologies for the PRO data analyses.

The last component of the framework is called “feedback and reporting” and consists of the following three parts: dissemination of PROs results (i.e., alerts to clinicians, reports, and publications), data access policies (i.e., for research and industry) and time and frequency of regular reporting.

Discussion

Over the past eight years, there has been a 42% increase in the number of CQRs in Australia.12 New registries were developed to monitor surgical procedures, specific cancers, critical care services, and existing registries have increased patient and hospital coverage. While increasing number of CQRs capture PROs with outputs that are generally used for benchmarking purposes, assess quality assurance and undertake research, our study results suggest that in Australia the use of PROMs in registries is less mature, and may be supported by the development of a conceptual framework and evidence-based guidelines.3,35

Findings of our survey warrant further discussion. Initial registry information collected showed that although the surveyed registries had been operating for nearly 10 years, just over 50% had greater than 50% of all the eligible patients included in the registry, with only four of the 19 registries having >80% of the registry population captured. This highlights the barriers to collecting CQR information more broadly within the Australian setting relating to resource constraints, burdensome ethics and governance requirements, and lack of national leadership.36 The ongoing Commonwealth development of a draft National Strategy for CQRs, currently undergoing national consultation, is urgently needed.

The majority of the PROs collection for Australian registries has been for monitoring the quality and outcomes of patient care, with fewer using PROs to monitor the burden of disease and other population health studies. This highlights the role of PROs as a valuable source of information to support patient-centered care and continuity of care from one setting to the next.37 This focus on the clinical utility of routine collection of PROMs is also evidenced by the survey results showing PROs collection across all registry participants, and at multiple time points per patient. Routinely collected PROs may provide aggregate information regarding the quality of care for particular patient cohorts, particularly when the information is benchmarked against other health services and risk adjusted for patient factors, such as socio-economic status, demographic variables and comorbidities. PROM can also be used for comparative effectiveness analysis to evaluate an intervention.38

In the past PROMs were mainly collected for clinical trial studies.39 Compared to clinical trials, a major limitation to CQRs is their limited capacity to obtain baseline data, which is often hampered by recruitment timing and processes. For example, eligible patients are only identified post-event (e.g., trauma), outside the health-care setting (e.g., cancer where recruitment time frames might be reliant on the reporting time frames to cancer agencies), or post-procedure (e.g., non-elective emergency surgery).38 Baseline PROs are possible where registry recruitment processes fall within the health setting providing the initial consultation pre-procedure. This does assume that staff at the hospital are invested in the registry and take responsibility for the administration and collection, of at least the initial PROM. The best method of PROMs administration may be dependent on the characteristics of the population. For example, prostate cancer patients are generally older, and telephone administration remains a favored and effective method of administration, whereas young trauma patients prefer an electronic means of communication.14,40 Where there is high variability in characteristics within a registry population, e.g., broad age range, a number of different methods of administration may be required to obtain and maintain a high PROM response rate.

PRO data collection can be laborious and could be associated with high costs and patient burden.38 Clinical trials recruit to a finite sample population that is homogenous with limited follow-up time frames. In comparison, cost and administration burden is of particular importance to CQRs that generally aim to recruit whole patient populations on an ongoing basis. For example, the bariatric surgery registry recruits approximately 19,000 patients each year.41 Appropriateness of the PROMs may be dependent on whether the outcome measure of interest is relevant across the whole population or to a defined sub-group. CQRs inclusion criteria are often broader than a population required for a clinical trial. Unlike clinical trials, the purpose of registry data is not to provide proof of an intervention’s efficacy. Registry data describe the outcomes of proven therapies in non-experimental, clinical settings.31 Depending on the outcomes to be measured, the decision to include the whole registry population may affect the choice between generic and specific instruments, and ultimately, the PROs which will be measured.

Similar to population-level clinical data collection, there is a risk of poor PRO data completeness and missing data, which may affect the validity of the resulting analysis. New technologies are increasingly being deployed to increase data capture at minimal marginal costs; however, these are not without their own issues.42

The most significant finding from our study is the limited ways in which PROs are currently being analyzed, reported and used in Australia. Only one registry responded that they provided PROs feedback at the individual level, despite the evidence that PROs are most valuable when they are used to support individual clinical care interactions.43 Disappointingly, PROs are used for benchmarking of health service performance in less than 50% of the surveyed CQRs and are reported back to participating clinicians in 30% of registries. It is not surprising that only two of the 19 respondents stated that PRO data collection within a CQR has made an impact.

The development of a conceptual framework and subsequent guideline for PROMs in CQRs will be of great value in the Australian, and potentially international, context. Despite a relatively small number of registries collecting PROs in Australia, there is growing interest to capture PROs in both registry and clinical practice settings. However, the results from our survey suggest that their value and impact to clinicians, patients, and the broader health system could be improved. In particular, the implementation of PROMs at the aggregate level for system-level quality improvement and transparency is in its early stages of development. Therefore, an important next step is developing guidelines for including PROMs in registries to build efforts to describe how PROs would best be used in comparative effectiveness research.44,45 This work requires close engagement with clinicians and patients to understand the appropriate use of PROMs, treatment processes and outcomes.

Importantly, we hope that encouraging registry custodians to consider all elements of the framework before data collection commences will result in a clear rationale for PRO data collection, including analysis and reporting of the data. The guidelines for PROMs in CQRs may also lead to higher uptake and better use of PROMs. In addition, considering issues of data completeness and how these can be mitigated through utilizing multiple data collection methodologies should ensure high quality of the PROs data collected and their usefulness for analysis. Involving patients at the forefront of the development of PROs collection and throughout the process will ensure maximization of their important contribution to monitoring care and outcomes.46 On the other hand, certain types of instruments (e.g., SF-12 or EQ-5D) pose the question whether all the patient-administered measures truly reflect the patient voice due to limited patient involvement in their development process, and the generic nature of these instruments which may limit their appropriateness for all conditions.47,48

A main strength of this study was conducting a survey of Australian clinical registries which clearly identified common issues related to PROs data collection and reporting, which reflect issues reported in the literature. On the other hand, the study does have some limitations. Due to stringent inclusion criteria for the literature review, some papers might have been missed. Despite a range of registry studies existing that describe the implementation of PROMs, the literature review found a paucity of registry publications that focused on experiences distilled from PROM implementation, which could inform future practice.

Next steps will involve developing a set of guidelines that will be determined by an expert panel in a consensus process using a Delphi approach.49 A list of preliminary statements for the guidelines will be based on the findings from the literature review and the survey. The statements will be combined according to the conceptual framework themes shown in Figure 1. As part of the Delphi process, a series of online surveys will be conducted to consult key domestic and international stakeholders to achieve a consensus in what are considered guidelines and recommendations for PROs inclusion in CQRs.

Conclusion

We developed a preliminary conceptual framework, which classified findings, from both the literature and the survey, into broad categories ranging from initial development to outcome dissemination providing the structure for development of guidelines in the next phase of this project. The framework covers eight components: rationale, setting, ethics, selection of PROM, administration, data management, statistical methods, feedback, and reporting. The proposed framework will guide future PROM implementation efforts to assure that complete PROs are captured at the correct time, along with data on the associated risk factors. This will facilitate generating meaningful information to serve diverse registries. In addition, this framework will provide basis for the development of the guidelines engaging national and international leaders in QoL research, researchers, clinicians, patient advocates, and consumers.