Patient perceptions of receiving test results via online portals: a mixed-methods study.

Objective: Online portals provide patients with access to their test results, but it is unknown how patients use these tools to manage results and what information is available to promote understanding. We conducted a mixed-methods study to explore patients' experiences and preferences when accessing their test results via portals. Materials and
Methods: We conducted 95 interviews (13 semistructured and 82 structured) with adults who viewed a test result in their portal between April 2015 and September 2016 at 4 large outpatient clinics in Houston, Texas. Semistructured interviews were coded using content analysis and transformed into quantitative data and integrated with the structured interview data. Descriptive statistics were used to summarize the structured data.
Results: Nearly two-thirds (63%) did not receive any explanatory information or test result interpretation at the time they received the result, and 46% conducted online searches for further information about their result. Patients who received an abnormal result were more likely to experience negative emotions (56% vs 21%; P = .003) and more likely to call their physician (44% vs 15%; P = .002) compared with those who received normal results. Discussion: Study findings suggest that online portals are not currently designed to present test results to patients in a meaningful way. Patients experienced negative emotions often with abnormal results, but sometimes even with normal results. Simply providing access via portals is insufficient; additional strategies are needed to help patients interpret and manage their online test results.
Conclusion: Given the absence of national guidance, our findings could help strengthen policy and practice in this area and inform innovations that promote patient understanding of test results.

INTRODUCTION

Patients now have increased electronic access to their health information via portals., Much of this was stimulated by the implementation of national policies prioritizing patient access and growing patient demand by patients for access to their health information., This access, in line with basic tenets of patient-centered care, could lead to better patient engagement and greater involvement in medical decision-making. Initiatives such as OpenNotes have increased transparency by providing access to clinical notes, resulting in better patient understanding of health status and enhanced patient-clinician relationship. However, little is known about patients’ understanding of results received via the patient portal, and merely having access may not guarantee that patients know how to use the information. There is thus an emerging need to identify patients’ needs and preferences when viewing their test results to ensure that information is meaningful, useful, and actionable for patients.

Access to test results is an area of high interest to patients, and among the many portal functionalities, patients frequently cite it as most useful., Access to results provides an opportunity to foster patient involvement in care by preventing test results from being overlooked, a common patient safety concern. Further, portals that allow patients independent and unrestricted access may also help coordinate care among multiple physicians and avoid redundancy.,

Literature on patient access to test results through patient portal is sparse but growing. Prior studies have found that very few patients experienced negative emotions when viewing their test results on a portal.,, However, patients had trouble identifying when values are out of range in standard test result tables, especially patients with lower literacy and numeracy., It is not clear that patients are always able to interpret the meaning of their test results.,, This could, in part, be due to the test result report format and interface design, such as tables that are formatted for clinician interpretation., Creative visual displays could be used instead to increase patients’ understanding of test results.

Currently there are no national test result notification and interface design standards or guidelines for patient portals. Many portals may only provide test result values with a range or flag to indicate significance, similar to the format received by clinicians. Patient portals lack interoperability and consistency in features and design across providers, which can frustrate patients and limit their use – only 15%–30% of patients electronically access their health information., To inform future work and development of standards in this area, we conducted a mixed-methods study to explore patients’ experiences and preferences when accessing their test results via online portals.

METHODS

Study setting and population

We used multiple methods to recruit eligible participants (adults 18 years or older who viewed a test result in their portal) at 4 settings: an academically affiliated private general practice (n = 24), a primary care community clinic (n = 1), a network of private-practice physicians (n = 21), and an urban Veterans Affairs (VA) facility (n = 49). Three sites used MyChart from Epic Systems (Verona, WI, USA), a leading commercial provider of electronic health records (EHRs) in the United States, and the VA site used MyHeatlheVet. Together, users of the VA’s Computerized Patient Record System (CPRS) and Epic provide care for about 190 million patients in the United States.

We posted flyers at each site, and a research coordinator received permission to approach patients at 2 of the clinics. At one clinic, 3 physicians assisted with recruitment by handing out flyers to patients when the research coordinator was onsite. Finally, to boost sample size, we obtained a list of all portal users at one site and sent recruitment letters to a random sample of patients. The local institutional review board at each site approved this study.

Design

We used an exploratory mixed-methods design to explore the patient experience of receiving test results through the portal. We initially conducted 13 semistructured interviews recruited in person or with flyers in clinic waiting rooms. We asked participants to walk through the process of receiving a test result via the portal and probed them with questions about their understanding of the results, information needs, and emotional responses. These questions included: What does the test result mean? Did your physician talk to you about why the test was being done? Did you have any concerns when you checked the result in the portal? Interviews were conducted in person and by telephone, audio recorded, and transcribed verbatim. Based on our interview data and discussions with health information technology experts, we created a final structured interview. Structured interviews were conducted by telephone and in person at the clinics in a private room, and were not audio recorded. All participants were provided with a $25 gift card for their time.

Data collection

Data collection occurred between April 2015 and September 2016. To determine eligibility, we first asked potential respondents whether they had received a recent test result via their patient portal. If they responded with a yes, we asked whether it was normal or abnormal. Before the interview, respondents were asked to think about that test result and answer the questions based on that experience. Participants were approached in clinic waiting rooms, were informed of the study by their physician, or responded to posted flyers. This allowed us to recruit 43 participants. At one site, we obtained a list of all active portal users and selected a random sample of 450 users. We sent a recruitment letter with opt-out instructions and followed up by telephone 2 weeks later. Thirty-nine additional patients recruited this way agreed to participate. In total, 82 patients participated in structured interviews.

The structured interview assessed 4 domain areas: (1) participant characteristics, (2) physician and patient actions on test results, (3) patient perceptions of receiving results via the portal, and (4) portal concerns and suggestions for improvement. Participant characteristics we assessed included gender, age, race/ethnicity, chronic conditions, comfort with using the Internet, portal use frequency, a 10-item version of the Patient Activation Measure (PAM), and length of time as a portal user. Physician actions included whether the physician communicated a reason for the test, called to report the result, told the patient to check the portal, and explained the result in the portal. Patient actions included whether or not participants conducted online research, sent secure messages, called the physician following receipt of the result, and discussed the result with family/friends. Patient perceptions included how the patient knew the result was normal or abnormal, whether the test had been done before, why it was ordered, expectations about the result, whether the patient understood the result, and feelings upon seeing the result. We also included 3 open-ended questions to allow participants to discuss problems they experienced receiving test results via the portal and suggestions for improvement.

Data analysis

Two authors (TG and JB) coded the qualitative data independently using content analysis to identify emergent categories. Interviews (n = 13) were also transformed into quantitative data and included in the quantitative analysis. We reviewed the transcripts to identify instances where the structured interview question content was discussed. For instance, in the interviews we asked participants to discuss the first thing they remembered doing after seeing the result in the portal. If the participant said, “I called my doctor,” we would convert that to nominal data, 1 for yes, under “Did you contact your physician’s office?” If a participant did not mention contacting the doctor or was not specifically asked that question, we quantified that as missing data. Descriptive statistics were used to summarize the structured interview data. Fisher’s exact test was used for categorical variables (2-tailed) and independent t-tests for continuous variables using SAS.

RESULTS

We conducted 95 interviews (13 semistructured and 82 structured). Table 1 shows participant characteristics, which included 55.8% male, 65.3% white, average age 54.6 years (SD 15.6 years), and 62.1% with 1 or more chronic conditions. Almost three-fourths (71.6%) had been using the portal for 1 year or more, and 84.2% considered themselves comfortable with using the Internet. A majority of patients (85.3%) scored high on the PAM in levels 3 and 4, suggesting that overall they were more activated. Patients indicated that they underwent a variety of tests, including “annual blood work,” cholesterol, A1c, tumor-specific antigen, prostate-specific antigen, liver, kidney, Pap smear, urine analysis, electrocardiogram, and thyroid. While a majority of patients simply listed “bloodwork,” those who listed a specific test did not indicate its rationale or whether it was done for screening or diagnostic purposes.

Table 1.

Characteristics of participants who received normal and abnormal test results via their patient portal

	Test results
Participant Characteristics	Total (n = 95)	Abnormal (n = 43)	Normal (n = 52)	P Value
Age (years)
	54.6 (SD 15.6), (range 26–81)	57 (SD 14.7), (range 28–78)	52.7 (SD 16.2), (range 26–81)	0.18
Gender, n (%)
Male	53 (55.8)	23 (53.5)	30 (57.7)	0.84
Female	42 (44.2)	20 (46.5)	22 (42.3)
Race,an (%)
Caucasian	61 (64.2)	30 (69.8)	31 (59.6)	0.75
African American	25 (26.3)	9 (20.9)	16 (30.8)
Asian	2 (2.1)	1 (2.3)	1 (1.9)
Ethnicity,an (%)
Hispanic or Latino	11 (11.6)	6 (14.0)	5 (9.6)	0.59
Non-Hispanic or Latino	75 (78.9)	32 (74.4)	43 (82.7)
Chronic condition, n (%)
None	36 (37.9)	12 (27.9)	24 (46.2)	0.09
1 or more	59 (62.1)	31 (72.1)	28 (53.8)
Internet comfortable, n (%)
No	14 (14.7)	5 (11.6)	9 (17.3)	0.57
Yes	80 (84.2)	37 (86.0)	43 (82.7)
Portal use frequency,an (%)
Weekly	20 (21.1)	8 (19.0)	12 (23.1)	0.85
Monthly	25 (26.3)	13 (31.0)	12 (23.1)
Yearly	15 (15.8)	6 (14.3)	9 (17.3)
As necessary	33 (34.7)	15 (35.7)	18 (34.6)
PAM level, n (%)
1	7 (7.4)	3 (7.0)	4 (7.7)	0.98
2	7 (7.4)	3 (7.0)	4 (7.7)
3	53 (55.8)	25 (58.1)	28 (53.8)
4	28 (29.5)	12 (27.9)	16 (30.8)
Portal user length,a years, n (%)
0–1.99	26 (27.4)	10 (23.3)	16 (30.8)	0.90
2–3.99	33 (34.7)	14 (32.6)	19 (36.5)
4–6	22 (23.2)	10 (23.3)	12 (23.1)

aMissing data.

Table 2 shows physician and patient actions before and after receiving a test result via the patient portal. Most patients (89.5%) indicated that their physician explained why the test was being ordered. Half (50.5%) reported that their physician told them to check their portal for the result. However, 63.2% of the participants reported that their physician did not include a note or interpretation explaining the result.

Table 2.

Physician and patient actions before and after receiving a test result

	Test results
Physician and Patient Actions	Total (n = 95), n (%)	Abnormal (n = 43), n (%)	Normal (n = 52), n (%)	P Value
Physician actions
Did your physician tell you why s/he was ordering it?				0.75
Yes	85 (89.5)	38 (88.4)	47 (90.4)
No	10 (10.5)	5 (11.6)	5 (9.6)
Did your physician tell you if s/he would call if the result was abnormal/normal?a				0.51
Yes	47 (49.5)	23 (53.5)	24 (46.2)
No	46 (48.4)	20 (46.5)	26 (50.0)
Did your physician tell you to check your portal for the result?a				0.4284
Yes	48 (50.5)	20 (46.5)	29 (55.8)
No	43 (45.3)	20 (46.5)	23 (44.2)
Did the physician write a note to you explaining the result?a				0.9328
Yes	31 (32.6)	15 (34.9)	16 (30.8)
No	60 (63.2)	26 (60.5)	34 (65.4)
Patient actions
Did you go online to research?a				> 0.9999
Yes	44 (46.3)	20 (46.5)	24 (46.2)
No	44 (46.3)	20 (46.5)	24 (46.2)
Did you send a secure message?a				0.20
Yes	24 (25.3)	14 (32.6)	10 (19.2)
No	65 (68.4)	28 (65.1)	37 (71.2)
Did you call your physician’s office?a				0.002
Yes	27 (28.4)	19 (44.2)	8(15.4)
No	64 (67.4)	24 (55.8)	40 (76.9)
Did you talk to friends or family?a				0.05
Yes	49 (51.6)	27 (62.8)	22 (42.3)
No	42 (44.2)	16 (37.2)	26 (50.0)

aMissing data.

We found that participants who received an abnormal result were more likely to call their physician than those who received a normal result (44.2 and 15.4%, respectively, P = .002). Overall, a quarter of participants (25.3%) sent a secure message to their doctor regarding the test result, more commonly for participants with abnormal (32.6%) than normal (19.2%) test results. Participants who sent a secure message reported that they did so for further explanation of the test result or to determine next steps. “If I don’t [understand], if I see that something’s out, I can sit down and…e-mail my doctor and ask him to explain it.” (P 1001)

About half of all participants sought information about their result from sources other than their physician – 46.3% did online research and 51.6% discussed their result with friends or family. Participants explained that they conducted online research before or in addition to contacting their physician to look up unfamiliar terms or tests they encountered while viewing results, to look up potential diagnoses, and to avoid bothering their physician. “I’ll sit down and google it and see if I can figure it out that way before I bother [doctor name].… Those doctors are swamped.” (P 1001) “If there’s a certain condition that is referenced by a certain test, I might…just google it as a start…and learn more about whatever the condition is.” (P 1003) One participant indicated that the physician had included a “well explained” note but she still searched online for additional information, “I googled it and then clicked on some of the researchy [sic] stuff.… NIH has a little write-up on heart diseases, like a little pamphlet, which explains what to do.” (P 2004)

Table 3 shows patient perceptions related to receiving test results. A majority of participants indicated that they knew why the test was being ordered (95.8%) and had the test done before (84.2%), and 77.9% stated that they understood their result. In interviews, understanding the result meant different things. Some participants were able to discuss the meaning of the test: “This means that, for instance, with the cholesterol, that I have elevated indicators that can increase the risk of heart problems, strokes, and that I should just really watch them and to reduce them” (P1006); while others understood the results in terms of behavioral changes: “Basically…I would likely have to go on medication and change my diet at that point.” (P 2004)

Table 3.

Patient perceptions of using the portal to access test results

	Test results
Patient Perceptions	Total (n = 95)	Abnormal (n = 43)	Normal (n = 52)	P Value
At the time the test was done, did you know why it was ordered?a
Yes	92 (96.8)	41	51
		95.3%	98.1%
No	2 (2.1)	1	1	0.73
		2.3%	1.9%
Have you ever had this particular test done before?a
Yes	79 (83.2)	39	41
		90.7%	78.8%
No	14 (14.7)	4 (9.3)	10 (19.2)	0.25
Were you expecting the result to be normal?a
Yes	65 (68.4)	23 (53.5)	42 (80.8)
No	24 (25.3)	19 (44.2)	5 (9.6)	0.0002
How did you know the test result was abnormal or normal?a
Visual cue (range, flag, color, bold)	58 (61.1)	27 (62.8)	31 (59.6)
Clinician (physician or nurse)	16 (16.8)	8 (18.6)	8 (15.4)
Personal knowledge/experience	8 (8.4)	7 (16.3)	1 (1.9)
Did not know or looked it up	8 (8.4)	1 (2.3)	7 (13.5)	0.04
Did you understand the result?a
Yes	75 (78.9)	33 (76.7)	42 (80.8)
No	18 (18.9)	9 (20.9)	9 (17.3)	0.90
How did you feel when you saw the result?a
Negative (concerned, confused, anxious, scared, frustrated)	35 (36.8)	24 (55.8)	11 (21.2)
Positive (happy, curious, relieved, relaxed, satisfied)	17 (17.9)	4 (9.3)	13 (25.0)
Indifferent	40 (42.1)	14 (32.6)	26 (50.0)	0.002

aMissing data.

When asked about result expectations, patients with normal results were more likely to expect a normal outcome than those with abnormal results (80.8 and 53.5%, respectively, P = .0002). Participants indicated that they knew their result was normal or abnormal because they noticed a visual cue such as a flagged or bolded value (61.1%), a physician told them (16.8%), or they had personal medical knowledge (8.4%), and 8.4% did not know or looked up the test online.

Over half of participants with abnormal results (55.8%) experienced negative emotions, including confusion, concern, anxiety, fear, or frustration, when viewing the result. In interviews, participants expressed nuanced emotions related to their current health status, personal experiences, and uncertainty. Some participants with chronic conditions managed their expectations about their test results based on experience. “Well, I mean, I’ve been taking these blood tests for years and years and years.… There really wasn’t anything unexpected.” (P1005)

Other participants indicated that their anxiety and concern were related to the perceived new consequences of the test result. One indicated that while over time she has learned what her results mean, she still experiences anxiety about how it will impact her health. “So the first time it came back…I was a little bit concerned, but afterwards I kind of really learned to expect that. And with my cholesterol, I was a bit nervous…afraid that the doctor’s going to put me on some medications.” (P 1006) Another participant echoed this anxiety and needed reassurance from her doctor:

The initial [test], that was the only one where I was anxious and…I don’t want to say scared, but I was really concerned about what it could mean with kidney failure.… I think the second abnormal one I would’ve liked her to call, because it was more of a confirmation that “Oh that first one really was abnormal. It wasn’t just a fluke in the lab.” (P 2003)

While in both cases the participants understood their result, it was not clear what it meant for their future health, and they anticipated uncertain and potentially uncomfortable changes.

Almost a quarter of participants who received normal results (21.2%) also experienced negative emotions. One participant expressed concern because she was unsure how to interpret a positive test result:

I think some of them said negative and positive. But then I think for some of the tests that you’re supposed to be positive for an antibody. So if it says positive on it, you think positive means like bad, right?… It said positive and I freaked and then I went to talk to my doctor about it. (P 2005)

Half of the participants with normal results (50.0%) expressed indifference and indicated that they had existing personal medical knowledge about the test, a physician or nurse had called prior to viewing online results, or a normal result was simply not a concern.

Table 4 shows the number of patients reporting problems while interacting with the portal. More than half of participants (60.0%) indicated that they did not have trouble checking their test results on the portal. Patients commented on the time it took for results to post on the portal, display and usability issues, password issues, and lack of test result explanations and education. About half (52.6%) agreed that portals could be improved. Suggestions to improve portals included: improve the display and usability (eg, “Not good for computer ‘un-savvy’ vets, requires you to go through a maze. A SEARCH feature would be VERY helpful”), include timely test result notification with an explanation and follow-up instructions (eg, “An explanation of the reference range and what the result might mean to me”), and provide additional functionality (eg, “CT/MRI/imaging results on portal”).

Table 4.

Patient-reported portal concerns and suggestions for improvement

	Test results
Portal Concerns and Suggestions	Total (n = 95)	Abnormal (n = 43)	Normal (n = 52)	P Value
Have you ever had any trouble checking your test results on the portal?a
Yes	31 (32.6)	17 (39.5)	14 (26.9)
No	57 (60.0)	24 (55.8)	33 (63.5)	0.34
Is there anything that would make the portal better for you?
Yes	49 (51.6)	24 (55.8)	25 (48.1)
No	38 (40.0)	16 (37.2)	22 (42.3)	0.78

aMissing data.

DISCUSSION

While 94% of hospitals and 77% of health care providers routinely offer access to laboratory results, less than one-third of patients access this information., We conducted this study to better understand patients’ experiences accessing their test results through a patient portal. We found that more than half of the participants did not receive explanatory information or result interpretation in the portal at the time they received the result, and almost half conducted online searches about their result. While more than half of the participants did not call their physician following receipt of the result, participants who received abnormal results were more likely to call their physician than those who received normal results. Finally, participants who received an abnormal result were more likely to experience negative emotions than those who received a normal result.

Despite increased access to patient portals, there are no nationally recommended practices or guidance for test result notification via portals. Previous work shows that both patients and physicians see benefit in providing patients with an interpretation,, but little progress has been made to facilitate this. The typical table format may make it harder for patients to distinguish between results that are slightly out of normal range and clinically relevant findings. Only a third of participants received a note in their portal explaining the test result. Although three-fourths of participants indicated that they understood their test result, almost half reported searching online for additional information – sometimes before contacting their physician, in line with previous work.

Our work has several implications for policy and practice. Understanding test results is hard for most patients. Providing an interpretation along with the test result at the time of portal release should be considered best practice. Additionally, providing information about the test at the time of the order does not appear to mitigate patient online research. Several participants searched online for supplemental information even when they indicated they understood the result. Rather than discouraging patients from conducting online searches, they should be encouraged to use specific vetted websites and search options provided within the portal. The portal should provide easy access to such high-quality tools to support and educate patients.,,

While previous literature on patient access to health information has shown reduced or no effect on anxiety,, our participants experienced negative emotions when accessing their test results online. Surprisingly, this occurred with both abnormal and normal test results and included feeling concerned, confused, anxious, scared, and/or frustrated. Our findings suggest that some patients will likely experience uncertainty about their results regardless of the significance, underscoring the need for additional tools and resources to support them. Although the portal gives patients access to their health information, it might not be able to provide them with the appropriate information within the context of their health problems. Some patients might need additional personalized or contextual information, compassion, or reassurance – something current health information technology cannot provide. For instance, almost half the participants in our abnormal test results sample called their physician’s office, and about a third sent a secure message about the abnormal test result. Some tests results, especially sensitive results, may require additional support services or resources for patients. Patient navigators have been successful at reducing anxiety for patients who receive an abnormal mammography and are waiting for follow-up testing. Thus, ideal test result notification via portals should include information about the purpose of the test, the result in the context of the patient’s health, directions for next steps, and specific resources, including available support and educational services.,, Current health policies should support these strategies.

Based on our findings, we highlight key recommendations for improving the design and use of patient portals (Table 5). We mapped these recommendations to an 8-dimension sociotechnical model that we have found useful in our prior work. Briefly, this model posits that design, development, implementation, and use of various health information technology applications should be considered in the context of the larger complex adaptive health care system within which the technology is embedded. To make improvements, one needs to consider changes in multiple, and sometimes all, dimensions of the model.

Table 5.

Key recommendations based on study results aligned with 8 dimensions of the Sittig and Singh sociotechnical model

Dimension	Recommendations
Hardware and software	Ensure that the portal is available on both large-format computers and hand-held devices. Enable search functionality of the site.
Clinical content	Provide easy access to high-quality educational websites.
Human-computer interface	Provide users with access to an explanation of test results directly from results screen.
People	Ensure that patients have direct, easy access to “human” support services that include people, such as patient navigators, advocates, social workers, or others who work in related educational services.
Workflow and communication	Provide personalized or contextual information to help patients know what to do in light of the results (eg, make lifestyle changes, send secure messages to their providers, or make follow-up appointments).
Internal organizational policies, procedures, environment	Develop local policies and procedures to create standardized language guiding patients to a specific follow-up contact for any questions. Provide patients with educational content on portal-related support when they are having face-to-face visits.
External rules and regulations	Create national consensus and standards on timing and best practices for portal release of normal and abnormal test results, especially those with sensitive results (eg, HIV status or cancer diagnosis), and on proxy portal access, such as for older relatives.
Measurement and monitoring	Create mechanisms to evaluate patients’ experiences related to test result notification in portals and use this data to help developers improve portal usability and design innovations to promote patient understanding.

Our study has several limitations. First, we experienced challenges in recruiting eligible patients despite including 4 clinical settings. Consequently, we used multiple methods to bolster enrollment. At all our sites, we found that semistructured interviews took longer and were perceived as burdensome to patients. To boost sample size and recruitment, we used a shorter structured interview guide based on our semistructured interview data. While we made efforts to focus on just one test while conducting the interviews, some of our respondents had multiple chronic conditions and may have discussed multiple tests. We were not able to control for heterogeneity within subjects. Additionally, during recruitment we found that while many patients had signed up for and set up a portal account, they had never logged in to their account beyond set-up to view a test result. Finally, our results may not generalize to all types of patients. Our sample consisted of a large majority of patients who reported being comfortable with the Internet and predominantly scored on the higher levels of PAM: level 3, taking action, or level 4, maintaining behaviors.

In conclusion, our findings suggest that current patient portals are not designed to present information on test results in a meaningful way. Patients experienced negative emotions often with abnormal results, but sometimes even with normal results. While providing patients with access to their test results via portals is a good start, it is insufficient by itself to meet their needs. To facilitate patient engagement, this step should be accompanied by strategies to help patients interpret and manage their test results. Given the absence of national guidance, findings from this study could be useful for strengthening policy and practice in this area and inform innovations that promote patient understanding of test results.

Funding

This project was funded by Agency for Healthcare Research and Quality R21HS023602 and partially supported by the VA Health Services Research and Development–funded Center for Innovations in Quality, Effectiveness and Safety (CIN 13-413). HS is additionally supported by the VA Health Services Research and Development Service (CRE 12-033; Presidential Early Career Award for Scientists and Engineers USA 14-274), the VA National Center for Patient Safety, and the Agency for Health Care Research and Quality (RO1022087). The funding sources had no involvement in the study design; collection, analysis, or interpretation of the data; writing the report; or the decision to submit the manuscript for publication.

IRB Approval

This study was approved by the Baylor College of Medicine Institutional Review Board prior to data collection.