Case Studies from the Clinic: Initiating and Implementing Patient-Reported Outcome Measures.

INTRODUCTION: Self-reporting by patients though the use of electronic patient-reported outcome (PRO) measures has been shown to use increase patient satisfaction with care, and improve patient-provider communication, symptom management, and health quality. Additionally, PROs are increasingly used in research to expand understanding regarding the relative risks, benefits, and burdens of interventions. While experience embedding patient-reported outcomes (PROs) into registries and clinical workflow is growing, there is little in the literature to guide those interested in incorporating PROs into routine clinical care and for use in research. CASE DESCRIPTIONS: The NIH Health Care Systems Research Collaboratory PRO Core interviewed investigators from seven programs to get their first-hand experiences on the incorporation of PROs for both care and research, and the investigators have contributed to this manuscript as authors.
FINDINGS: We use these case studies to present practical approaches to initiating and implementing PROS, including instrument selection, tips for integrating PRO collection systems into clinical workflow, considerations for user experience and data collection, and the methods to assess and monitor quality.
CONCLUSION: Because the decision to initiate and implement PRO collection impacts many different stakeholders, the solution requires collaboration among the involved parties, careful planning, and integration into clinical workflow.

Introduction

Longitudinal patient-reported outcome (PRO) collection is a growing priority for research and offers many potential benefits to clinical efficiency, symptom and quality of life monitoring, and patient care. There is abundant evidence in favor of self-reporting by patients: PRO use increases patient satisfaction with care, improves patient-provider communication and overall quality of life, improves symptom management and health quality, and can be used to inform clinical decisions, such as triaging or beginning supportive therapy [123456789]. As health systems shift toward clinical practice that continually learns—a learning health system-supporters are calling for the inclusion of PRO measures for both clinical care and research. Significant barriers to initiating and implementing PROs still remain, and practical information in initiating and implementing PROs for routine clinical care and for research is lacking [10]. To gather real-world experience on practical approaches for selecting, compiling, and curating PRO measures, the Patient-Reported Outcomes (PRO) Core of the National Institutes of Health (NIH) Health Care Systems Research Collaboratory (Collaboratory) visited, interviewed, and sourced information from stakeholders from several institutions.

In this manuscript, we present seven unique case studies of electronic PRO use as part of routine clinical care and for research. We use these case studies to present practical approaches to initiating and implementing PROs, including instrument selection, tips for integrating PRO collection systems into clinical workflow, considerations for user experience and data collection, and the methods used to assess and monitor quality.

Methods

The Collaboratory’s PRO Core holds monthly phone calls with the investigators of the Collaboratory Demonstration projects, and the information presented on the Collaborative Care for the Pain Program for Active Coping and Training (PPACT) and the Trauma Survivors Outcomes and Support (TSOS) Demonstration Projects was gathered during the course of the phone calls; additionally, the principal investigators (PIs) for both studies are authors on the paper In 2013, one- to two-day on-site interviews were conducted by members of the Core to Duke University’s Center for Learning Health Care, the University of Alabama Research and Informatics Center, and the University of Virginia Palliative Care Clinic. The PIs for the all studies are authors on this paper

Case Descriptions

The cases used are described below, and a summary of this information is shown in Table 1.

Table 1

PRO Collection

PROGRAM	WHAT IS MEASURED?	DATA COLLECTION/SETTING	DATA USE	INSTRUMENT
Duke University’s Center for Learning Health Care (CLHC)	Patient symptoms and distress at the Duke Cancer center	Data are collected on a tablet computer, which is given to patients in the waiting room or after they have been placed in a clinic room.	For routine clinical care. This summary report (Figure 2) highlights item responses that exceed a defined threshold (e.g., pain) and important patient issues.To trigger interventions, Such as referral to counseling or online educational modules. For example, distress screening with the Patient Care Monitor (PCM), and subsequent referral to psychosocial counseling, resulted in reduced distress scores in patients with metastatic breast cancer [12].	Patient Care Monitor (PCM) v2, an 80 item survey for males and 86 for females
Collaborative Care for Chronic Pain in Primary Care	Pain severity and pain-related functional interference	A Primary Care Physician PCP or medical assistant administers the Brief Pain Inventory Short Form (BPI-SF) verbally for chronic pain patients on long term opioid treatment and enters the information into the EHR. Collection is augmented by quarterly e-mails and phone calls.	For routine clinical care. Patients who demonstrate no improvement or worsening of function who are to be considered for opioid taper.For evaluation of interdisciplinary Pain Program for Active Coping and Training (PPACT) program.	Brief Pain Inventory – Short form (BPI-SF)
Carolinas Palliative Care Database Consortium	Quality of palliative care	Physicians collect data as a part of the routine clinical visit on various palliative care quality performance measures with the goal of improving conformance with these measures.	Support quality assessment and quality improvement activities.	Quality Data Collection Tool version 2.0 comprises 37 questions within 5 domains: demographics, symptom management, advance care planning, prognosis, and transition/discharge.
University of Alabama at Birmingham, Research and Informatics Service Center	Varies according to clinic need	The patient uses a tablet computer during “pockets of wait time,” during routine clinical care.	For routine clinical care, to trigger interventions, and research.	Any of the PROMIS Measures (http://www.nihpromis.org), which include a variety of domains for pediatrics and adults. In addition, instruments identified by sub-specialty clinics as measuring domains important to their care.
University of Virginia, Palliative Care Clinic at the Cancer Center	In the palliative care setting: anxiety, depression, fatigue, pain, physical function (all from PROMIS), and ECOG score and answer.	Nurses or nursing aids load questionnaire on an IPad or computer, where the patients complete the questionnaires with assistance from nurses, if needed.	For routine clinical care: data is available during the patient visit. For research: To assess symptoms and trigger interventions for cancer patients and to compare the functional status of cancer patients with and without palliative care.	The questionnaires mostly consist of NIH PROMIS items on pain, fatigue, depression, anxiety, and global quality of life, although the group created their own set of gastro-intestinal cancer modules.
Back pain Outcomes using Longitudinal Data (BOLD) Project	Patient reported pain characteristics, back-related disability, psychological distress, health-related quality of life, falls, and recovery expectation.	Baseline data were collected through in-person or telephone interviews. Follow-up questionnaires at 3, 6, and 12 months were self-administered using mailed hardcopy forms or were collected by a research coordinator over the telephone.	For research to change routine care: To test the hypothesis that imaging of the lumbar spine within 6 weeks of the index visit (early imaging) is associated with worse patient reported outcomes over time and increased health care utilization and costs.	1) 0-10 numerical rating scales (NRS) of average back and leg pain in past 7 days; 2) Brief Pain Inventory activity interference scale; 3) Roland Morris Disability Questionnaire, modified slightly to indicate disability due to back or leg pain (sciatica) 4) Patient Health Questionnaire (PHQ)-4 Depression and Anxiety screen; 5) the EQ-5D; and 6) Behavioral Risk Factor Surveillance System questions on falls. In addition, the duration of pain and recovery expectation (patients used a 0-10 NRS to rate their confidence that their pain will be completely gone or much better in 3 months) were assessed at baseline.
University of Washington Trauma Survivors Outcomes and Support Study (TSOS)	PTSD and depressive symptoms and alcohol use problems.	Clinicians at 24 level 1 trauma centers perform an initial PTSD risk evaluation and record findings in the EHR. Patients identified by the EHR evaluation as high risk for PTSD are formally assessed with the PTSD Checklist for study entry [9].	For research to change routine care: To test a collaborative care approach for patents who are identified as high risk for post-traumatic stress disorder (PTSD) and other comorbidities after experiencing a trauma.	The Posttraumatic Stress Disorder Symptom (PTSD) Checklist for the assessment of PTSD symptoms, The Nine-item Patient Health Questionnaire for the assessment of depressive symptoms, and the Alcohol Use Disorders Identification Test (AUDIT) for the assessment of alcohol use problems.

Duke University’s Center for Learning Health Care (CLHC) [111213] assessed patient symptoms and distress at the Duke Cancer Center using the Patient Care Monitor (PCM) v2, an 80 item survey for males and 86 for females. Data were collected on a tablet computer and a summary report (Figure 1) highlighted item responses that exceed a defined threshold (e.g., pain) and other important patient issues. This information was then used to trigger interventions, such as referral to counseling or online educational modules.

Figure 1

Example of a Summary Report from the Patient Care Monitor

Note: The report summarizes all responses, and highlights, via colors, areas of higher scores, as well as trends in scores over time, using colored arrows to the left of categories.

The Collaborative Care for Chronic Pain in Primary Care is a Collaboratory Demonstration Project evaluating the Pain Program for Active Coping and Training (PPACT) program. The PPACT program compares coordinating and integrating services to help patients adopt self-management skills against usual care for patients with chronic pain on long term opioid treatment [14]. It is a large mixed-methods, pragmatic, cluster-randomized clinical trial conducted in three regions of Kaiser Permanente health systems. The PEG (which assesses pain intensity [P], interference with enjoyment of life [E], and interference with general activity [G] PRO, [15] a 3-item version of the Brief Pain Inventory Short Form (BPI-SF), is administered verbally through a three-tiered system for chronic pain patients on long-term opioid treatment to assess pain severity and pain-related functional interference. PROs were not initially embedded in the electronic medical record (EMR) system in a way that would allow the investigators to extract the data, so the study team needed to build additional infrastructure to store it in the EMR for research use. Quarterly e-mails are initially sent through the patient health record with follow-up interactive voice response (IVR) calls if the patient didn’t respond online, and calls by medical assistants or similar staff if the patient didn’t respond to the IVR calls. Responses are automatically entered into EMR – Epic questionnaires while the project is working with IT to push PROs collected through the other modalities back into the EMR.

Carolinas Palliative Care Database Consortium measures the quality of palliative care to support quality assessment and quality improvement activities [16]. Physicians collect PRO data as a part of the routine clinical visit on various palliative care quality performance measures with the goal of improving conformance with these measures. The Quality Data Collection Tool version 2.0 comprises 37 questions within five domains: demographics, symptom management, advance care planning, prognosis, and transition/discharge.

University of Alabama at Birmingham, Research and Informatics Service Center created an interruptible web-based interface that works across multiple browsers and operating systems. The patient completes questionnaires on a tablet or wall mounted computers during identified “pockets of wait time” during routine clinical care to minimize impact on clinic workflow. The system serves multiple sub-specialty clinics (Hepatitis C, Palliative and Supportive Care, etc.), and any of the NIH Patient-Reported Outcomes Information System (PROMIS) Measures (http://www.nihpromis.org) can be used, which include a variety of domains for pediatrics and adults. In addition, instruments that are identified by sub-specialty clinics as measuring domains important to their care are also used. As an example, in the Center for Aids Research, PROs are collected in two phases: (1) a review of systems is done at each clinic visit, and (2) screening for depression, adherence to medications, sexual risk factors, substance abuse, tobacco use, and alcohol use are done on routine visits every four months. PROs are also used for research, such as investigating suicidal ideation in persons living with HIV [17] and the suitability of the PROMIS alcohol short form in people living with HIV [18].

The University of Virginia Palliative Care Clinic at the Cancer Center uses PROs in the palliative care setting to measure anxiety depression, fatigue, pain, physical function, and Eastern Cooperative Oncology Group (ECOG) score [19]. Nurses or nursing aids load the MyCourse questionnaires on a tablet or computer, where the patients complete the questions with assistance from nurses, if needed. MyCourse is a PRO outcomes informatics platform for palliative care used for PRO monitoring and clinical decision-making by clinicians. The application runs on the EPIC system and has the ability to chart the trajectory of PROs. The questionnaires mostly consist of NIH PROMIS items on pain, fatigue, depression, anxiety, and global quality of life, although the group created their own set of gastro-intestinal cancer modules. The PROs have been found to be reliable predictors of deteriorating health status leading towards end of life and are used to identify patients with declining symptoms for referral to palliative or more aggressive care [20].

Back pain Outcomes using Longitudinal Data (BOLD) Project was conducted at the University of Washington and 3 participating sites (Northern California Kaiser-Permanente, Henry Ford Health System [Detroit], and Harvard Vanguard Medical Associates [Boston]) [2122] The aim of the project was to determine if imaging of the lumbar spine within 6 weeks of the index visit (early imaging) was associated with worse patient-reported outcomes over time and increased health care utilization and costs. Researchers measured patient-reported pain characteristics (duration, location, severity, and interference with function, activity, and sleep); back-related disability; psychological distress; health-related quality of life; falls; and recovery expectation. Baseline data were collected through in-person or telephone interviews. Follow-up questionnaires at 3, 6, and 12 months were self-administered using mailed hardcopy forms or were collected by a research coordinator over the telephone. The tools used include 1) 0-10 numerical rating scales (NRS) of average back and leg pain in past 7 days; 2) Brief Pain Inventory activity interference scale; 3) Roland Morris Disability Questionnaire, modified slightly to indicate disability due to back or leg pain (sciatica); 4) Patient Health Questionnaire (PHQ)-4 Depression and Anxiety screen; 5) the EQ-5D; and 6) Behavioral Risk Factor Surveillance System questions on falls. In addition, the duration of pain and recovery expectation (patients used a 0-10 NRS to rate their confidence that their pain will be completely gone or much better in 3 months) were assessed at baseline.

The University of Washington Trauma Survivors Outcomes and Support Study (TSOS) is a NIH Collaboratory Demonstration Project conducted at 24 level 1 trauma centers across the US to test a collaborative care approach for patents who are identified as high risk for post-traumatic stress disorder (PTSD) and other comorbidities after experiencing a trauma. Because trauma care systems do not currently have the administrative databases that track patient outcomes after hospital discharge, multiple PROs are being used in the study to track key study outcomes at 3, 6 and 12-months after physical injury hospital discharge. PROs used in the study include: The Posttraumatic Stress Disorder Symptom (PTSD) Checklist for the assessment of PTSD symptoms, The Patient Health Questionnaire (PHQ-9) for the assessment of depressive symptoms, and the Alcohol Use Disorders Identification Test (AUDIT) for the assessment of alcohol use problems [23]. The study aims to influence the American College of Surgeon’s policy for PTSD and comorbidity screening and intervention.

A description of each program is shown in Table 1.

Findings

Initiating PRO Measurement

What to Measure and Selecting an Instrument

For most of the case studies, the first step in initiating their PRO collection system was choosing a PRO instrument. Defining what to measure and instrument selection begins with the considerations in Figure 2. In our examples above, constructs are shown in Table 1 and include (a) general assessment on topics such as health-related quality of life (e.g., the PCM at Duke Cancer Center, which measures symptoms and distress); (b) disease-specific assessments on specific areas (e.g., the program at the Center for Aids Research at the University of Alabama Birmingham); (c) symptom-specific assessments on concerns such as pain, breathlessness, or distress (e.g., the PPACT or BOLD projects); (d) functional status assessments on physical functioning, social functioning, or emotional functioning (e.g., the anxiety, depression, fatigue, pain, physical function measures at the University of Virginia Palliative Care Clinic at the Cancer Center); (e) satisfaction scores (e.g., the quality of palliative care at the Carolinas Palliative Care Database Consortium); and (f) other assessments that do not fit into these categories (e.g., adherence with therapy).

Figure 2

Considerations When Selecting a Patient-reported Outcomes Instrument

The PRO instrument may need to be specifically narrow to address the question at hand and meet regulatory requirements (e.g., the FDA) or broad and simple in order to assist with identification of uncovered concerns (e.g., clinical review of systems/symptom screening). Creating a measurement strategy may involve using an existing instrument (such as the PROMIS measures used in our case studies), combining previously developed and validated instruments, or developing a new instrument [24]. For example, although the University of Virginia Palliative Care Clinic uses mostly PROMIS measures, they have created their own set of gastrointestinal cancer modules to meet the needs of their patients. Instrument selection is based on the needs of the clinic and the outcomes under investigation.

How will the data be collected?

In our case examples, data were collected via a tablet computer the web, in person from the physician, nurse or clinician who is delegated the task, and over the phone (Table 1). Considerations included specifications for desirable instruments, reliable internet connectivity and respondent concerns, such as the amount of burden, maximum time expected to complete the instrument, preferred form of contact, internet access/email address, and literacy requirements. In the BOLD study, investigators initially planned on using electronic methods as the form of contact but found that the older adults in their cohort preferred to be contacted by phone.

How will the data be used?

Beyond the generation of a summary for use in routine clinical care, we found that the collection of PROs was used for both clinical care and research (Table 1) and added value in the following ways:

Improving efficiency of clinical documentation

At Duke University’s Center for Learning Health Care (CLHC), from the Duke and West Clinic (TN) experience, collection of PROs improved documentation of a clinical review of systems, documentation support for billing and coding, and triaging to psychosocial care providers; reduced dictation time; and made the reporting of treatment toxicity after chemotherapy more efficient.

Triggering referrals to another physician

At the University of Alabama at Birmingham Center for AIDS Research, one of the questions on the depression screen is about the frequency of suicidal thoughts during the last two weeks. If a patient responds over a certain threshold, an email is sent to the pager for the on-duty psychologist and to a team member in the clinic. The psychologist completes a full assessment, independent of the clinician and makes a recommendation to the clinician with a suicide response protocol. The program also screens for intimate partner violence and can trigger and intimate partner violence protocol; there is limited data on this in the literature, particularly in same sex couples. As another example, at the Duke Cancer Center, distress screening with the PCM and subsequent referral to psychosocial counseling resulted in reduced distress scores in patients with metastatic breast cancer [12].

Identifying patients for clinical trials

At the University of Alabama at Birmingham Center for AIDS Research, if PROs indicate that a patient is eligible for a clinical trial, an alert is sent to the nurse in charge of enrolling for the trial.

Triaging

At the University of Virginia Palliative Care Clinic at the Cancer Center, patients who report 7.5 or greater on anxiety depression, pain, or fatigue or score 5 or less on physical functioning, the response will trigger an alert when the clinician (MD only) opens the patient’s chart. When the system is triggered, an email goes to the Palliative Care Services group. The group will then determine if the patient is receiving palliative care services, and if not reach out to the treating physician. If the patient is currently receiving palliative care services, the group will discuss the patient’s case.

Triggering additional PROS

At the University of Alabama at Birmingham, PROs can be used to trigger additional PROs. For example, if there is an increase of pain from a previous visit to a current visit of more than 4 points, or if pain exceeds a specified threshold, the Brief Pain Inventory is automatically triggered.

Other Support

In the Back pain Outcomes using Longitudinal Data (BOLD) Project, to inform economic evaluations, they linked back-pain-related PROs with health utility measures, and additional methodological work is underway. PROs can also be used to trigger educational materials (PPACT) and to support billing.

Test the Instrument

Test for validity in context

In some instances, the tool validated for research was not appropriate for routine clinical use and needed to be evaluated in context. For example, the University of Alabama at Birmingham Research and Informatics Service Center for AIDS Research initially used two kiosks for PRO data collection in the waiting room. The instrument took over a year to develop, covered many research topics, and took 90 minutes to complete, leading to a bottleneck of patients. The program was so disruptive to clinic workflow, implementation only lasted for one day; the Clinic Director literally unplugged the computer. This experience highlights the importance of adjusting PRO collection to individual context.

Test for usability and feasibility

Usability testing was needed to ensure that respondents from the target population are able to use the software and the device appropriately Feasibility extends usability, establishing the practical implementation of the PRO collection system in the local setting (i.e., clinic, hospital, home). For example, in the University of Virginia Palliative Care Clinic at the Cancer Center, the group initially used PROMIS CAT However, the data and reports were difficult for the patients to understand, so they organized nursing staff to assist with the delivery of the assessments and developed a protocol for data collection. The Palliative Care group piloted the PRO system within their own clinic before moving into other clinics within the cancer center As another example, at Duke University’s Cancer Center physicians use the Patient Care Monitor v2 (PCM), an 80-item review of systems survey for males (86 items for females). A pilot study testing the acceptability and feasibility of using PCM on e-tablets found that, in addition to overall patient satisfaction with the tool, the e-tablet helped them remember symptoms to report to their physicians [11].

Implementation and Integration of PRO collection into clinical workflow

Based on our conversations with the Principal Investigators, we found that a PRO measure was more accepted if it added value to the clinician’s work, the patient’s care, and to other stakeholders, such as hospital administrators. For example, a review of systems, especially following chemotherapy, is an important tool oncologists use to screen for significant changes in cancer patients’ level of dysfunction and symptom severity. The Duke Cancer Center’s PCM assesses allergic/immunologic, respiratory, gastrointestinal, musculoskeletal, endocrine and psychiatric symptoms, among others. Reports from the survey are used to identify areas of concerns, or confirm symptom patterns for patients undergoing chemotherapy (Figure 1). Patients’ data can later be linked to other data within the electronic medical record (EMR) to support research activities [25].

Considerations for User Experience

In previous work at Duke University’s Cancer Center for Learning Health Care (CHLC), we found that patients prefer using electronic touchscreen interfaces like tablets or computers over a paper survey, as they often contain only one question per page and are easy to use [11]. At CLHC across sensitive domains, such as sexual dysfunction, patients were more comfortable sharing the information on an electronic PRO than face-to-face [26]. In cases where a patient feels too ill to complete the survey, a support person can assist by reading the questions.

The Patient Care Monitor asks 80-86 items at each visit (actual number depends upon gender); as long as patients understand the connection to their cancer care, they don’t find it burdensome. They report high levels of satisfaction using the instrument, [11] and over 80 percent of patient encounters result in fully completed instruments, i.e., no survey items were left unanswered. Further, over 95 percent of our patient encounters resulted in surveys with fewer than 10 percent of a survey’s items unanswered. This experience includes routine PRO collection for more than 7,700 unique patients with more than 24,000 clinical encounters. In order to explicitly connect responding to the PRO instrument, the PCM, with clinical care, several steps were undertaken: a color-coded report was printed as soon the patient finished answering the survey, doctors were taught to say “I looked at your report and…”, and nurses in the clinic provided tailored education based upon patient report. Patients who were only seen once at the clinic were the most likely to return a completely blank survey, but the rate of missing data decreased with repeated visits to the clinic, to <3 percent by the fourth visit, strongly suggesting that longitudinal use of the PRO data in the routine clinical care reinforces the message of its importance to the patient.

Data Quality

Given the paucity of information on assessing PRO data quality in the literature, we propose that routinely measuring and assessing the degree of missing PRO data as a first step towards defining data quality metrics. For example, at Duke University’s Center for Learning Health Care (CLHC), men being seen in the prostate cancer clinic have higher rates of missing data than women in the breast cancer clinic on the PCM item related to cooking for oneself. The team doubted that this was because therapy for prostate cancer, or the disease itself, impairs men’s ability to cook, and suspected that elderly men seen in this clinic don’t cook for themselves. Nurses made a point to clarify this issue with elderly men.

Limitations

The emphasis of this article is on practical considerations for clinical researchers (or clinical researchers in strong partnership with clinical delivery system operational and clinical leaders) when initiating and implementing a PRO system. We did not specifically interview patients, although we acknowledge that they are important stakeholders in the research enterprise. Further work should be done to gain their perspective. An important broader discussion that is beyond the scope of this article concerns the institutional incentives for initiating such a system and how features of that system might have to be designed to satisfy certain regulatory requirements, such as Meaningful Use, HIPPAA requirements, and other privacy concerns. A related discussion concerns how best to organize efforts early to ensure that the PRO system is designed from the start to simultaneously meet regulatory, clinical, and research needs. These topics are the focus of ongoing debate.

Discussion

Because the decision to initiate and implement electronic PRO collection impacts researchers, clinicians, nursing and other support staff, patients, caregivers, and administrators, the solution requires collaboration among the involved parties, careful planning, and integration into clinical workflow. It is crucial to involve clinicians in the entire implementation process and to generate interest and buy-in from providers, as patients take cues from providers regarding the value of the PROs. When deciding how data will be used, consider the PRO instrument as part of a larger system— for example, as a system for data collection and reuse and as a system that constantly learns, i.e., a learning health system. The more clearly defined and rationalized the system, the more likely it is that the PRO information will be put to good use and be valued and completed at the point of care. As the patient perspective grows in importance for both clinical care and comparative effectiveness research, initiating and implementing PRO measures will be essential.

Conclusion

Researchers who are helping to develop the elements of these systems must keep in mind that the instruments should be clinically feasible and relevant, fit into clinic workflows, and improve care for patients. These factors need not compromise the quality of data collected, so long as researchers and instrument developers are mindful of the requirements for learning health systems. We cannot sacrifice the utility and potential of PRO instruments due to an over-reliance on issues such as comprehensiveness.

Longitudinal collection of electronic PRO data has the capacity to transform clinical practice— improving efficiency and streamlining care, enhancing patient education, and supporting clinical decision-making. It can also serve as an important pillar for research within learning health care models, as the patient experience is critical to truly developing the ideal care model. The ultimate key to overcoming barriers to PRO collection is to collaborate with all the relevant stakeholders and make the data collected be relevant to the patient, the clinician, and the researcher.