Patient and physician perspectives on the use and outcome measures of mHealth apps: Exploratory survey and focus group study.

Objective: Factors that physicians and patients consider when making decisions about using or recommending health apps are not well understood. We explored these factors to better assess how to support such decision making.
Methods: We conducted an exploratory cross-sectional study in Ontario using qualitative focus groups and quantitative surveys. 133 physicians and 94 community dwelling adults completed online surveys and we held two focus groups of nine community dwelling participants who had cardiovascular risk factors and an interest in using mHealth apps. Quantitative survey data was analyzed descriptively. Focus groups were audio-recorded and transcribed verbatim prior to inductive thematic content analysis. We integrated the results from the surveys and focus groups to understand factors that influence physicians' and patients' selection and use of such apps.
Results: Physicians recommend apps to patients but the level of evidence they prefer to use to guide selection did not align with what they were currently using. Patients trusted recommendations and reviews from medical organizations and healthcare professionals when selecting apps and were motivated to continue using apps when they supported goal setting and tracking, data sharing, decision making, and empowerment. Conclusions: The findings highlight the significance of evaluating mHealth apps based on metrics that patients and physicians value beyond usage and clinical outcome data. Patients engage with apps that support them in confidently managing their health. Increased training and awareness of apps and creating a more rigorous evidence base showing the value of apps to supporting health goals will support greater adoption and acceptance of mHealth apps.

Introduction

Mobile health (mHealth) technologies are rapidly emerging as tools to support the well-being of people and to diagnose, treat, mitigate or prevent conditions such as cardiovascular disease (CVD), diabetes, asthma and obesity. To maximize the positive effect of apps on health, effective and safe apps should be easy to identify and broadly available.

Both patients and their physicians can play a role in selecting apps; however, the ways these groups select apps are poorly understood. Only a small proportion of health apps have been evaluated through randomized controlled trials (RCTs) reporting clinical endpoints, and published trials are of short duration, with small sample sizes, and have high risk for bias. The paucity of evidence may explain why studies have shown that, although physicians are interested in apps for their patients, they are not likely to recommend them, particularly outside of their specialty. The UK physicians ranked stamp of approval by an accreditation body and published studies of safety and efficacy as important factors in their decision making about apps as their concerns of safety and clinical effectiveness must be alleviated before using apps in routine clinical practice.

The level of evidence for mHealth apps should be proportionate to any potential clinical, organizational or financial risk.[5,6] The increasing severity of consequences if apps do not perform as intended or the intent of a particular app to be used in conjunction with support from health care providers indicate that higher standards of evidence should be used to demonstrate effectiveness. In the absence of clinical evidence, various heuristic frameworks have been developed to assess mHealth apps. Common assessment criteria include clinical effectiveness, functionality, usability, privacy, security, accessibility, professional and expert endorsement and information quality.[7-10] Key organizations that provide these assessments and curate digital health libraries of apps include the Organisation for the Review of Care and Health Applications (ORCHA), DiGA, AppScript (IQVIA) and TherAPPx. They provide curated lists of mHealth apps that health care providers could use to navigate the ever-growing market of mHealth apps and digital health technology to direct patients to relevant tools for specific health conditions of concerns. Once an app is identified, much like a drug intervention, it must be used by the patient to be effective. Patients are most concerned with identifying and selecting apps of value to the maintenance of their health and safety but are reluctant to rely on these tools. Downloading and quickly abandoning apps or searching for replacements is a common behavioural pattern.[17,18] Continued use of mHealth apps is dependent on a user's assessment of the app's capabilities and their commitment to pursue their intended health goals; thus, negative assessments and low user persistence or motivation towards their health goals will cause one to abandon an app when it is not able to provide support features related to feedback, reminders and in-app peer/coach support and has technical difficulties.[17,19] Factors influencing decisions to continue app use include common user experience elements, such as design interface, navigation flow, notification alerts, goal management, depth of knowledge available, clarity of system rules, personalized actionable recommendations and fit between system features and user needs.[17,20] The expansion of technology-assisted delivery of healthcare must be informed by the perspectives and needs of patients to maximize their impact, particularly when considering that their acceptance of and engagement with apps hinges upon emotional factors like habit, social influence and trust.

When evaluating the effect of a mHealth app, it is relatively easy to select clinical variables to measure; treatment adherence, medication management, physical activity levels and parameter monitoring are a select few clinical variables that can be measured with mHealth apps. Parameter monitoring for a patient with CVD may include active minutes, burned calories, weight, heart rate, blood pressure, body temperature and early detection of atrial fibrillation, but given the focus on engagement for mHealth apps, patient-important outcomes should also be incorporated. There are numerous Patient-Reported Experience Measures (PREMS) and Patient-Reported Outcome Measures (PROMS), often for specific conditions. Understanding how people gauge the value of an app for managing their health could help identify a common PREM/PROM to employ in evaluations of mHealth apps that measure how patients assess their quality of life and physical and mental well-being. A greater focus on PREMS and PROMS elicited through methods like questionnaires and focus groups can provide information on how to improve quality of care, processes of care, how to better meet patient expectations and give clinicians closer insights into their patients’ clinical status, which lead to increased engagement with mHealth apps.[24,25]

Our objective was to first quantify how physicians recommend apps and how community-dwelling adults as patients use recommendations or other factors in selecting apps, and to integrate the results with a deeper understanding of how community-dwelling adults determine, heuristically, the value of an app for their health through focus groups. The findings will be used to inform the selection of patient-important outcomes in future studies of mHealth apps.

Methods

This exploratory cross-sectional survey study comprised two voluntary online surveys engaging physicians and community-dwelling adults as patients, separately, and focus groups of community-dwelling adults that were analysed thematically.

Recruitment

We used multiple methods of passive and active recruitment within multiple settings to gather a more representative sample of physicians. We contacted primary care and specialty practicing physicians to complete the anonymous survey via e-mail newsletters or invitations and at in-person McMaster University continuing medical education events. Email invitations and newsletters with information on the purpose of the research and a QR code and link were distributed by OntarioMD, McMaster Family Health Teams and the Hamilton Academy of Medicine. Continuing education events in Hamilton which comprised local physicians and physicians practicing in other parts of Ontario included McMaster Thrombosis 2018 Update, McMaster Cardiology 2018 Update and Technology & the Future of Health Care 2019 Conference in Hamilton. Participants attending the events were able to access the survey via their own Internet-connected device, or via a tablet set up by study personnel at a booth.

Patient recruitment also employed multiple methods to gather a representative sample of community-dwelling adults who own smartphones. We passively disseminated postcards that contained a link and QR code to the survey in the waiting rooms of primary care and specialty clinics in Hamilton, Ontario. We shared the survey link through personal networks on Facebook, Twitter and LinkedIn social media posts. We recruited participants through self-selecting sampling and snowballing methods, which included sharing the survey link through personal networks of the researchers and inviting participants to ask their networks to complete the survey. There were no other exclusion criteria.

The inclusion criteria for the face-to-face focus group study were community-dwelling adults with CVD or associated risk factors, with an interest in using mHealth apps to manage their health, the ability to comprehend and speak English and to travel to Hamilton to attend the sessions. For the focus groups, persons with CVD or risk factors for CVD were selected to inform outcome selection for future trials of mHealth apps designed to support this population of patients.

We recruited participants using a self-selecting sampling method on Facebook and RSearch, an online central hub for research study recruitment. Additionally, we distributed posters throughout the McMaster University campus, McMaster Institute for Research on Aging e-newsletter, local community newsletters and the local Coffee News newsletter. We passively disseminated postcards that contained a link and QR code to the survey in the waiting rooms of primary care and specialty clinics in Hamilton, Ontario and Brampton, Ontario. Surveys and survey administration

Survey questions were developed using previous research, professional medical organization guidebooks and websites that reviewed clinical apps.[1,9,10,26,27] The questions were based on criteria used to evaluate apps or health information technology including perceived clinical usefulness, usability, patient-reported outcomes, safety, clinical events,[1,9,10] behavioural intentions (attitudes and subjective norms) to use and recommend apps, and interest in or use of technology or apps with patients. The questions were informally tested within the research team and with clinical colleagues for clarity and comprehensiveness; they were not externally validated, as they were exploratory and devised to gain an initial understanding of app perspectives from physicians and patients.

The physician survey had 20 questions and was conducted from August 2018 to August 2019 to better understand why physicians may or may not recommend apps to patients, the information they use in their assessments and the levels of evidence they would prefer to have available before making recommendations (online Supplementary material). Similarly, the patient survey had 11 questions and was conducted from July 2019 to February 2020 and collected information on factors that influence the use and selection of mHealth apps (online Supplementary material). The questionnaire was intentionally brief to encourage quick and easy completion. Surveys were administered online using Qualtrics, a web application.

Focus groups

The primary aim of the focus group was to gain further understanding of measures or heuristic processes that patients use when assessing the value of an app. Two focus groups were conducted in a workplace boardroom in February 2020 and facilitated by the lead investigator (C.L.), a health science researcher with a background in critical appraisal of research studies and a primary research programme on the evidence of effectiveness of mHealth apps given the paucity of rigorous studies in the field. C.L. values the perspectives of patients and users and believes that traditional clinical outcome measures are not the only way to gauge the efficacy of apps. The lead investigator also has an interest in mHealth and 15 years of experience in the field of health informatics and eHealth, who has informal training in qualitative methods and has collaborated with experienced qualitative researchers on several studies using focus groups. An undergraduate and a Master's level student who analysed the focus group transcripts were present to take notes. There was no existing research relationship with the participants prior to the study. The sessions began with a short discussion primer of researcher introductions, and the participants were made aware of the reasons and goals of our research study. The sessions followed a semi-structured interview guide of 11 questions but also included probing and follow-up questions that allowed for discussion (online Supplementary material). Part of the interview guide was developed from the patient and physician survey questions and partly from features commonly available in apps and criteria used to evaluate apps such as information privacy and security, patient satisfaction, ease of use and endorsement by reputable medical organizations.[1,9] Participants were also asked exploratory questions on how they assessed or determined an app was of value to them, and about features of apps, both desirable and undesirable, that were important in their selection process. Focus group participants were asked to explain their rationale when choosing a mHealth app and describe how they determined when an app was useful for them. The focus group guide was reviewed internally by our research team comprised of clinicians, a research methodologist, students and a patient partner, for clarity and comprehensiveness. The participants provided written informed consent and completed a survey about demographics, cardiovascular risk factors and mHealth app usage. The moderator ensured all participants got an opportunity to speak and asked follow-up questions to probe for details and challenge views. To mitigate bias, the moderator avoided giving personal or professional opinions and used a neutral tone throughout. The two 90-minute sessions were audio recorded and transcribed verbatim. The participants were provided with a $25/hour honorarium.

Statistical analysis

Quantitative survey data were analysed using descriptive statistics. Two transcripts, one from each focus group session, were analysed using inductive thematic analysis in duplicate independently by the research students (S.B., R.M.) on NVivo 12 (QSR International). The analysis was independently reviewed by a third Master's level student (R.J.) and presented to the broader research team. S.B. had undergraduate level training in qualitative methods; R.M. and R.J. had informal training. During the open coding phase, we broadly identified preliminary themes based on the data provided in the transcripts and notes. Preliminary themes were reviewed and refined into more specific codes that were grouped into meaningful clusters. We then compared and discussed themes that were coded to determine the coherence of similar themes and full agreement. Themes were added, removed and refined to better capture the overarching themes. Both transcripts highlighted the same concepts, and no new themes were present in the second focus group transcript. As a validation step, focus group participants and our research patient partners received a summary of the identified themes to ensure they were accurate and meaningful and reflected assessments that they employ when making decisions about using apps. We present direct quotes based on the themes identified by the participants in this study.

As the study was exploratory, to better understand decisions about app selection and usage, we looked at the alignment between patient survey responses and the focus group findings. We integrated the quantitative results with the qualitative themes to allow for in-depth interpretation.

Results

Demographic characteristics

One-hundred thirty-three physicians and 94 community-dwelling adults as patients completed the surveys, with 15 and 11 partial responses, respectively. Nine community-dwelling adults as patients participated in the two focus groups of six and three individuals. The demographic characteristics for physician and patient participants are summarized in Table 1.

Table 1.

Survey and focus group participant demographics.

Sample	Category	N (%)
Physicians, n = 133 a
Age (n = 133)	25–34	24 (18)
	35–44	58 (43.6)
	45–54	27 (20.3)
	55–64	18 (13.5)
	65 +	6 (4.5)
Sex (n = 133)	Male	50 (37.6)
	Female	81 (60.9)
	Other	2 (1.5)
Practice type (n = 133)	General	67 (50.4)
	Specialist	66 (49.6)
Work setting (n = 133)	Community	87 (65.4)
	Academic	39 (29.3)
	Other	7 (5.3)
Comfort using technology (n = 129)	Expert	23 (17.8)
	Very comfortable	77 (59.7)
	Somewhat comfortable	28 (21.7)
	Not at all comfortable	1 (0.7)
Patients, n = 94 a
Age (n = 65)	25–34	7 (10.7)
	35–44	19 (29.2)
	45–54	19 (29.2)
	55–64	20 (30.7)
	≥65	0 (0)
Sex (n = 88)	Male	29 (33)
	Female	59 (67)
Living setting (n = 88)	Urban	49 (55.7)
	Suburban	35 (39.8)
	Rural	4 (4.5)
Comfort using technology (n = 88)	Expert	28 (31.8)
	Very comfortable	52 (59)
	Somewhat comfortable	7 (7.9)
	Not at all comfortable	1 (1.1)
Focus group, n = 9 a
Age (n = 8)	40–50	2 (25)
	51–60	1 (12.5)
	61–70	5 (62.5)
Sex (n = 7)	Male	3 (42.9)
	Female	4 (57.1)
Comfort using computers (n = 8)	Expert	2 (25)
	Very comfortable	4 (50)
	Somewhat comfortable	2 (25)
	Not at all comfortable	0
Comfort using smartphones (n = 8)	Expert	0
	Very comfortable	7 (87.5)
	Somewhat comfortable	1 (12.5)
	Not at all comfortable	0

Denominators to each question vary owing to missing data, and percentages might not add to 100 owing to rounding.

Surveys

Patients and physicians who responded to the surveys were mostly younger than 65 years of age, with approximately two-thirds identifying as female. Approximately 60% in each group considered themselves to be ‘Very comfortable’ using technology. The findings showed that among the 45% (n = 55) of physicians who have not prescribed or recommended apps, the most frequently cited reason was the lack of familiarity with apps (n = 38, 69%). This finding contrasts with their high level of comfort using technology (Table 1). Other reasons included the lack of time to review the evidence (n = 19, 34.5%), concerns about data security (n = 9, 16.3%), insufficient evidence (n = 8, 14.5%), concerns about medical-legal liability (n = 8, 14.5%) and other (n = 9, 16.4%). Examples of open text responses to ‘other’ reasons included not necessary, no demand, too many apps and elderly patients are not technologically savvy. The two most common forms of evidence used to support recommendations among the 55% of physicians who prescribed apps to patients were personal opinions (n = 47, 74.6%), followed by recommendations by patients (n = 23, 36.5%) (Table 2). Less common forms of evidence used by physicians were guideline recommendations, unbiased standardized consensus panel reviews by topic experts, RCTs, observational studies and app store reviews (Table 2). The levels of evidence physicians stated they would like to have available before recommending apps to patients were unbiased standardized consensus panel review by topic experts, guideline recommendations and RCTs.

Table 2.

Level of evidence used and preferred for mobile health (mHealth) app recommendation by physicians.

	What level of evidence did you use to support your recommendation? Check all that apply. (n = 63)	What level of evidence would you prefer to have available on health apps before recommending or prescribing them to patients? Check all that apply. (n = 117)
	N (%)
Unbiased standardized consensus panel review by topic experts	12 (19)	80 (68.4)
Guideline recommendations	15 (23.8)	72 (61.5)
Randomized controlled trials (RCTs) data	5 (7.9)	53 (45.3)
Patient recommendation	23 (36.5)	35 (29.9)
Observational study data	5 (7.9)	30 (25.6)
Personal opinion	47 (74.6)	23 (19.7)
App store reviews	5 (7.9)	13 (11.1)
Others	10 (15.9)	4 (3.4)

Contrastingly, patients predominantly considered recommendations from trusted medical organizations, their doctors or healthcare professionals, friends or social networks, reviews by clinical topic experts and reviews on app stores or other websites (Table 3). When asked if they would try an app if their doctor recommended it to them, 67% (55/82) of patients responded yes and 33% (27/82) responded maybe.

Table 3.

Sources that patients would consider if/when choosing an app, even if they were not currently using one. (n = 88).

	N (%)
Recommendations from a trusted medical organization	56 (63.6)
Recommendation from a doctor or other healthcare professional	56 (63.6)
Recommendation from friends or social networks	55 (62.5)
Reviews by clinical topic experts	42 (47.7)
Reviews by users on the app store or other websites	40 (45.5)
Guideline recommendations from a healthcare society	35 (39.8)
Data from clinical studies	25 (28.4)
Advertisements	10 (11.4)
Other	2 (2.3)

Focus group

Participants in the focus group were mostly (approximately 60%) between 61 and 70 years old. Several participants described desirable and undesirable app features they consider when selecting or deciding to continue using an app (Table 4), emphasizing their preference for customizability, ease of use and feedback for decision making. Four themes identified how patients gauge the benefit they get from using apps: supported decision making, goal setting, empowerment and data sharing. For decision making, apps that could provide sufficient feedback about certain behaviours were perceived to be more useful as participants described how mHealth apps helped them make decisions about their health and lifestyle. They could evaluate how they were doing, and adjust their behaviours, based on the feedback they received. One participant indicated that it ‘makes it easier to make decisions’ (F1, P4) while another mentioned that ‘[it] ultimately leads you toward like, oh, maybe I want to do more of this or less of that’ (F1, P4).

Table 4.

Desirable and undesirable app features identified by the focus group.

App features	Quote
Desirable features
Progress feedback	“What I like best about them is that you can usually get immediate feedback about how you’re doing and immediate metrics.” (F1, P3)
Data privacy and security	“…Big concerns about the stuff that goes on in the cloud when you download an app… the things that the fine print says they can now do with all that information, including personal identity information that they have about you. I’m not doing that for things unless I feel there's a really big value.” (F1, P1)
Low cost	“Cost is a factor. If I get something that doesn't cost a lot of money, I’ll do that.” (F2, P2)
Customizability	“I like the idea of the being integrated, easy to use, multifunctional, personalized within a reasonable degree.” (F2, P2)
Usability	“If it loads quickly and I can navigate to put the information in quickly that I want to put in and there aren't a lot of popping up ads.” (F1, P4)
Credibility	“If it was endorsed by a Canadian health agency, I would be much more likely to download it.” (F1, P3)
Integration	I’d love to be able to have one app or a way to integrate all these different apps or you know devices into one so I can interface with one and it could feed different programs.” (F2, TP)
Compatibility	“For me, it's the pairing of it to my watch.” (F2, P3)
Undesirable features
Unreliable technology	“Now it's an older phone…but I get a little anxious about the cost benefit ratio of somebody saying if you want to run on my app, go buy another five-hundred-dollar phone.” (F1, P1)
Learning curve	“Try it for fifty seconds and if it looks harder or if it doesn't look fun or if I’m going to have to look for that piece of data, it's gone.” (F1, P3)
Collection of personal information	“If this app wants me to register all that stuff before I can even look at it, I’m not doing it.” (F1, P1)
Distraction	“I don't end up keeping them long enough because I get too distracted by the information I don't want.” (F1, P4)
Features they were indifferent to
Data storage	“Not in today's generation of phone but the older ones.” (F1, P6)

With this information, participants were able to set their personal goals so they could track their wellness and achieve measurable activities such as calorie counts and hormone cycles. Some noticed a change in their behaviour as the app provided reinforcement after they set and reached their goals:

My goal is to change my behavior so by monitoring what I’m doing and tracking and doing all of that and getting the instant feedback. (F1, P3)

It notifies me when I meet my goals. It doesn't notify me when I don't meet my goals. So, there's a positive reinforcement. (F1, P5)

Participants reflected on their state of health. The apps provided validation, reassurance and awareness of engagement in positive health behaviours which made them feel empowered in managing their own health: ‘I like to have the information to be able to reflect on like where I’m at and why I am where I am’ (F1, P4). Apps that were valued inspired greater feelings of self-efficacy including being in control and taking care of oneself, as one participant reflected, ‘It validates that ok see, you were active, you were doing these things and staying active, because the one thing, though, I would really like is, the app to sort of help me advance’ (F2, P3).

Lastly, the potential for sharing health data quickly and easily with care providers as a facilitator for communication between patient and clinician was also a key theme for some of the participants: ‘I think being able to relate that information to my doctor, too, it's really important’ (F1, P2). Information in a shareable format could enhance two-way communication as one of the participants stated ‘…. If somebody had some very complex medical situations, it would be quite useful to have a tracking device to share with the doctor’ (F2, P2).

The focus group participants relied on similar sources to the patients in the survey to guide their app selections, such as app store reviews and recommendations from doctors, clinical experts or social networks. The focus group participants provided insight into other mechanisms that influenced their decision making about app choice and related thought processes.

I think it's a little bit word of mouth. There are a million people using [specific fitness app] … I found it because someone told me it was great. I looked at it and read the reviews. Basically, it's kind of trial and error. Aside from that, if I’m looking for an app that tracks a specific thing. Oh, go to the app store and look for it. (F1, P3)

I do look at a lot of the whatever user comments are in there and then review them. And I guess I primarily look at the ones that give a lousy score…When you go to the app store, there's a lot of like pictures of the different screens. (F2, P3)

Scientific research showing the app is effective was ranked ‘very low’ importance in the survey and was not significantly discussed in the focus group when asked about what elements draw them towards an app or choose one app over the other, contrasting the heavy physician preference for evidence-based apps. Similar to the focus group, the patients in the survey ranked cost and information privacy and security, followed by ease of use as the most important features to consider when they are choosing or using an app. The participants were concerned about the data being used by their apps and how they rationalize paying for apps or using free ones:

It allowed me to opt into studies, which I initially thought was really cool. And then I began to get a lot of really invasive questions from the app. And that made me really think about the data. And so I deleted it. (F1, T3)

Free ones can be annoying. But if there's something that's like five to ten-dollar range, you pay for it and then you have it and you know that you’re going to use it…In the absence of having conversations with your doctor about apps. (F1, P4)

Discussion

Our study suggests that what patients and physicians expect from apps slightly differs. Physicians themselves are looking for guideline recommendations or results from RCTs to recommend apps to patients. Patients consider how effective an app is by the functions it has and if they feel empowered, self-sufficient, validated and informed.

Processes to select and recommend apps

We found a disconnect between the information physicians are currently using to make recommendations versus what they would prefer to use to make these assessments. The level of evidence they prefer to see before recommending an app to a patient does not align with how they currently make such decisions. Most physicians in this study believed that apps are at least somewhat useful for patient care despite citing barriers to recommending them. These findings suggest that physicians see the value in mHealth apps for patient use but may be hesitant to adopt them in their practices. Currently, while some physicians are recommending apps to patients, there is a hesitancy to do so in a ‘prescriptive’ manner. The perspectives in our survey align with other studies showing lack of confidence in app prescribing habits due to poor knowledge or familiarity of apps with proven clinical outcomes. Conversely, the availability of published studies, health ministry stamps of approvals (e.g. National Health Service in the United Kingdom) and peer recommendations from other clinicians will facilitate the prescription of mHealth apps to patients. Similar sentiments exist as physicians have acknowledged the complementary nature of apps for patient management and medical education, but they lack knowledge of credible apps or the ability of apps to support patient outcomes. Increased training and awareness through online self-directed tutorials of apps could facilitate greater physician acceptance and adoption of mHealth.

The sources of mHealth app information for patients in our study are similar to other studies, highlighting the role of personal connections such as family members and friends as the most influential sources apart from direct contact with physicians.[32,33] However, the results from our focus group demonstrate that participants do not simply accept recommendations at face value as it may seem from the results of the patient survey. Rather, patients will seek these recommendations but go through their own selection process when deciding to use apps. They will heuristically evaluate the app based on the features and criteria important to them. A recent quantitative study in France found that general practitioners are more willing to prescribe mHealth apps or implement devices in their practice if they are validated through RCTs. Since participants suggested they would likely follow recommendations on apps from their physicians, the development of better evidence may drive better communication between patients and physicians regarding apps to use.

Patient- and physician- important factors

There is little understanding of the factors that encourage patients to continue using an app or knowing when the app is or is not benefitting them. The results from our surveys and focus groups show a level of uncertainty towards finding credible and trustworthy apps but they have a strong trust in their healthcare providers and organizations, which can facilitate the adoption of mHealth apps. A focus group study of students, health care providers and patients also highlight the value of the relationship with and endorsement of a healthcare provider in activating patients to seek health information related to their conditions using mobile electronic devices. Uncertainty may explain why patients and physicians will supplement recommendations with their own value judgements to determine an app's quality or efficacy, although they can spend unnecessary time and resources in exploring apps that they ultimately find unsatisfactory. This is not surprising as approximately half of the app users in the US national survey had stopped using some mHealth apps, citing high data entry burden, loss of interest, hidden costs, confusion and data being shared with their social networks.

Most patients in this study were currently or previously using an app to monitor or manage their health yet felt the cost of apps, concerns about privacy and security, ease of use, collection of personal information, a greater learning curve and distracting advertisements were barriers to their initial or continued use of apps. The focus groups valued features such as progress feedback, credibility, low cost, customizability, data privacy and security (Table 4). This is consistent with findings from a systematic review of qualitative studies that found motivational aspects, effective marketing, user-centred design, humanizing technology and accessibility were facilitators to app use, whereas non-conducive, invasive, disengaging content, inaccessibility and poor marketing were barriers to app use.

A key insight from the focus group participants was that they selected apps based on how well they perceived the app would support their health and enable them to be active and informed participants in managing their health. Participants in the focus groups found it important and empowering to have the ability to make informed decisions based on feedback they received; set goals as well as validate and reflect on their progress; make necessary changes to manage their health more effectively and share information with their healthcare team. Another study of features necessary for trustworthy mHealth apps using focus groups found similar factors affecting end user trust: information accuracy and understandability, organizational reputation, positive reviews, usability, privacy and empowerment.

Many mHealth apps have been created without the involvement of medical professionals. One of the most important areas of future research in mHealth is creating evidence of value, but researchers and app developers must consider the costs and complexities of running clinical trials to validate mHealth apps as the technology evolves more quickly than RCTs can be executed. The themes that emerged can help physicians and patients assess an app's suitability as physicians and patients do not value identical measures of efficacy. The focus group themes align with the Health Confidence Score which is a PROM comprising four items about how people feel about their level of health knowledge, self-management ability, care access and shared decision making. In addition to clinical outcomes, guidelines and trials that measure the efficacy of apps should include PROMS. Based on the focus group themes, we suggest that that the Health Confidence Score or similar PROMS be included to determine how well an app meets the needs of users. Of particular interest to measure are the emotional elements that patients consider when starting or continuing use of an app, such as fun, prior positive experiences and peer recommendations; these build behavioural intentions to use and accept certain apps. The generation of higher levels of evidence about clinical efficacy as established in the surveys (e.g. RCTs, cohort studies, expert panel reviews), patient-centred metrics discovered in the focus group and greater knowledge translation initiatives could instill confidence in physician-led app recommendations and sustained patient engagement or satisfaction with apps.

Strengths and limitations

Using both surveys and focus groups, we obtained multiple perspectives and richer data about values and attitudes towards mHealth apps. However, the survey questions were not validated externally and were completed by a small number of participants. Most respondents to both surveys were female and relatively few participants with advanced age or unfamiliarity with apps participated. Results may not be generalized to other groups, geographic locations or medical specialties. Due to the recruitment approaches taken, we do not have a count on the number of people who received the invitations to participate in the surveys. Missing data and the use of online surveys for data collection reduced the representativeness of the samples and the information was not verifiable. Duplicate entries were not prevented.

There were differences in the inclusion criteria for the surveys and the focus groups, including the general population for the survey and a subset of people with CVD risk factors for the focus groups. The self-selected sample size for the focus group was also small and may not be representative of the population with cardiovascular risk factors who use apps as the sample was subject to self-selection bias. There may be other important themes in the focus groups that we did not uncover that could have been discovered with more groups or participants. Group dynamics may have contributed to self-reporting and social desirability biases as some participants may have withheld views or given similar answers. We attempted to mitigate this limitation by having the moderator attentively engage with the group, encourage all participants to participate and use accessible language.

To increase reflexivity, the focus group questions were designed with the broader research team to elucidate themes and heuristics that inform how people make decisions about using apps. The data analysis was completed independently, with themes presented and discussed with the broader research team, which includes two patient partners. The themes were also shared back with the participants to ensure we captured them correctly. Although we had a research group with diverse backgrounds, we acknowledge that researcher subjectivity may have impacted the topics, generation of codes and interpretation of data.

Conclusion

This study investigated the current processes physicians and patients use to select, recommend and evaluate mHealth apps which have great potential to support improved health outcomes. While patients are eager to use mHealth apps, they rely heavily on recommendations from physicians and personal networks, but the evidence that physicians would prefer to have before recommending apps to patients does not align with how they are currently making their decisions, nor does it necessarily align with the metrics patients use to determine an app's quality or effectiveness. Our study findings are limited by sample size and response from people familiar with apps. Future research to evaluate mHealth apps should generate higher levels of evidence to enhance awareness, uptake and endorsement of these apps by physicians and patients, and should include patient-important outcomes, such as the Health Confidence Score. Efforts should be made to direct patients and physicians towards the most effective and safe apps, through enhanced health literacy, awareness, training and critical appraisal of potential apps, particularly with the inclusion of patient-defined metrics.

Click here for additional data file.

Supplemental material, sj-docx-2-dhj-10.1177_20552076221102773 for Patient and physician perspectives on the use and outcome measures of mHealth apps: Exploratory survey and focus group study by Rita Jezrawi, Sarmini Balakumar, Rafia Masud, Itzhak Gabizon, Vinai Bhagirath, Jobin Varughese, Michael Brown, Dan Trottier, JD Schwalm, Michael McGillion, Elizabeth Alvarez and Cynthia Lokker in Digital Health

Click here for additional data file.

Supplemental material, sj-docx-3-dhj-10.1177_20552076221102773 for Patient and physician perspectives on the use and outcome measures of mHealth apps: Exploratory survey and focus group study by Rita Jezrawi, Sarmini Balakumar, Rafia Masud, Itzhak Gabizon, Vinai Bhagirath, Jobin Varughese, Michael Brown, Dan Trottier, JD Schwalm, Michael McGillion, Elizabeth Alvarez and Cynthia Lokker in Digital Health

Click here for additional data file.

Supplemental material, sj-docx-4-dhj-10.1177_20552076221102773 for Patient and physician perspectives on the use and outcome measures of mHealth apps: Exploratory survey and focus group study by Rita Jezrawi, Sarmini Balakumar, Rafia Masud, Itzhak Gabizon, Vinai Bhagirath, Jobin Varughese, Michael Brown, Dan Trottier, JD Schwalm, Michael McGillion, Elizabeth Alvarez and Cynthia Lokker in Digital Health