Codesign approaches involving older adults in the development of electronic healthcare tools: a systematic review.

OBJECTIVE: The primary aim was to review and synthesise the current evidence of how older adults are involved in codesign approaches to develop electronic healthcare tools (EHTs). The secondary aim was to identify how the codesign approaches used mutual learning techniques to benefit older adult participants.
DESIGN: Systematic review following the Preferred Reporting Items for Systematic Reviews 2020 checklist. DATA SOURCES: PubMed, Embase and Scopus databases were searched for studies from January 2010 to March 2021. ELIGIBILITY CRITERIA: Inclusion criteria were studies employing codesign approaches to develop an EHTs, and the study population was aged 60 years and older. DATA EXTRACTION AND SYNTHESIS: Data were extracted for analysis and risk of bias. We evaluated the quality of studies using the Agency for Healthcare Research and Quality Evidence-based Practice Center approach.
RESULTS: Twenty-five studies met the inclusion criteria for this review. All studies used at least two involvement processes, with interviews and prototypes used most frequently. Through cross-classification, we found an increased utilisation of functional prototypes in studies reaching the 'empower' level of participation and found that studies which benefitted from mutual learning had a higher utilisation of specific involvement processes such as focus groups and functional prototyping.
CONCLUSIONS: We found gaps to support which involvement processes, participation levels and learning models should be employed when codesigning with older adults. This is important because higher levels of participation may increase the user's knowledge of technology, enhance learning and empower participants. To ensure studies optimise participation and learning of older adults when developing EHTs, there is a need to place more emphasis on the approaches promoting mutual learning. PROSPERO REGISTRATION NUMBER: CRD42021240013. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This review provided a comprehensive assessment of the utilisation of codesign approaches in studies that involve older adults in developing electronic healthcare tools.

We developed a broad search strategy to ensure all codesign or ‘collaborative’ approaches were captured in this review.

We used widely reported classification methods to categorise the included studies.

We limited our analysis to the overall involvement processes and levels of participation at the study level; cross-classification of these variables at the design stage level for each study was not conducted.

This review was limited to initial studies and did not evaluate whether subsequent papers by research groups provided a broader analysis of user experience.

Introduction

Codesign approaches aim to elicit ideas and foster a non-hierarchical environment for stakeholders, in which their concepts, tacit knowledge and lived experiences can be applied to develop tools that meet their needs, improve usability and impact outcomes from use of the system.1 2 Codesign is also likened to participatory design, each defined as focusing on the engagement and creativity of stakeholders, or those untrained in design efforts, but through collaborative efforts bring mutual learning into the development process.3 4 Despite the underlying similarity of these approaches, there is heterogeneity in the terminology of codesign approaches, which leads to variation in the evaluation measures employed, involvement processes, levels of participation and learning outcomes.3 4 For codesign to work effectively, all stakeholders must be provided opportunities to equally engage and learn, and their values and culture must be incorporated into the development process. Failing to optimise codesign approaches across the development process inhibits the representation, learning and empowerment of stakeholders.2

Barki and Hartwick suggested that we need to rethink what it means to involve someone versus having someone participate. Involvement can be described as the extent to which a system reflects what is important and relevant to the stakeholder, versus participation can be described as the extent to which specific actions or behaviours are employed by the stakeholder during system development.5 The term involvement has many connotations ranging from soliciting end user insight into involving them as codesigners.

Determining the value of the stages and levels of participation is further complicated when we define this based on the target population for this systematic review of those aged 60 or older. The WHO estimates that by 2050, the world population of those aged 60 and older will increase to 2 billion, or nearly double what it was in 2015.6 As the number of people within this age group increases, aligning systems that meet their needs is imperative. While user needs and experience with technology may vary at any age, system designers have focused on proven areas that address the accessibility needs when designing for the ageing population,7 8 but less has been done to evaluate the learning outcomes associated with involving older adults as well as the levels of participation during the development of electronic healthcare tools (EHTs). Involvement and participation occur at varying levels, and when users are more fully engaged, they feel a stronger sense of participation, which has the potential to impact their learning outcomes.2 While several factors make up the stages of involvement and participation, further evaluation as to the level of involvement at each stage is necessary, as little is known about which approaches are the most effective in involving older adults.9 The intent to involve stakeholders can be difficult as the framework to conduct these efforts is not prolific.3 This is further exacerbated by the hierarchical challenges that may arise between designers, healthcare providers and older adults, which can lead to selective inclusion of ideas.7 10–12 However, by addressing this challenge, the opportunity for mutual learning and knowledge transfer among stakeholders can occur.2 13–15

EHTs were selected for this review because the evidence supports the significant impact these tools have on improving health outcomes, including increasing knowledge, improvement in risk perception and improvement in communication between patients and their clinicians.16 We are using the term EHT to encapsulate all terms classified and defined by the WHO Digital Health Interventions.17 EHTs are interactive web applications, mobile applications or wearables that track or provide guidance on relevant healthcare issues to improve well-being, for instance, monitoring health risks (ie, risk of falling, risk of heart failure) or personal health information management (ie, patient decision aids, access to health records and educational materials). While evidence supports the value in involving older adults in the design process, and positive impacts have been reported in the utilisation of EHTs, we need to be cognizant that inconsistencies exist within these approaches; therefore, evaluating the involvement process further is necessary to ensure that the expended efforts of older adults result in a mutual exchange of knowledge and improvements to the development process without detriment to their well-being.

In a previous systematic review by Fischer et al, evidence supported involving older users in technology design, as indicated in the positive outcomes of involvement, specifically regarding learning, adjusted design and an increased sense of participation.18 Further supported in the review are the comparison examples between outcomes of designs that involved older adults during the development of the tool versus those that did not, and of the results of those who involved older adults, the developed tool was determined to better fit the needs of the intended population.18 Fischer et al explored the purposes and consequences of involving older adults in technology design, such as involvement, sense of participation and learning. While we review similar consequences, we take this a step further and expand on the involvement processes and levels of participation. We also analyse learning as an outcome, including who benefits from the learning as well as the user testing measures employed. A previous systematic review published by Lancaster et al focused on electronic health tools but synthesised the usability and effects19 of the tools, rather than codesign approaches. Additionally, their target population was not defined as older adults. Another systematic review was published with similar methods and target populations as our review;1 however, they focused on assistive technology, which promotes independence of living rather than on the development of EHTs. While similarities exist between our review and these, we have enough distinction from these reviews that our search will draw from a different group of studies, therefore will provide for a robust review of the current state of codesign approaches used to involve older adults in the development of EHTs.

Aim

Primary research question

What is the current state of utilising co-design approaches with older adults in the development of electronic healthcare tools?

What approaches were used for involving older adults in developing EHTs and how were they defined?

What theoretical frameworks and design principles were used to develop EHTs for older adults?

Was iterative development used? If so, how many phases were conducted?

What were the subjective and objective measures used for study endpoints, and were the measures validated?

What involvement processes were employed?

What were the level of participation for older adults?

Did participants reflect on testing measures? (ie, reflect on why an error was made)

Was there a bidirectional flow of knowledge between participants and researchers?

Methods

Patient and public involvement

No patient involved.

Systematic literature search strategy

We conducted a systematic review, following the Preferred Reporting Items for Systematic Reviews (PRISMA) 2020 checklist.20 Consultations with an experienced research librarian (Jennifer Bissram) were held to develop our search strategies. In March 2021, searches were developed for PubMed, Embase and Scopus. A broad search strategy was used to ensure all codesign approaches that involve end users in the design of EHTs were captured in this review as depicted in online supplemental appendix A—search results).

The search results were screened according to the criteria defined in table 1.

Table 1

Inclusion and exclusion criteria

Selection criteria	Inclusion criteria	Exclusion criteria
Participants	Older adults aged 60 or older, without regard to gender, race or ethnicity	If the study population included adults of all ages, including greater than 60 years of age, but evidence could not be extracted from those greater than 60 years of age, the study was excluded
Study setting	All study settings were included
Study design	All study designs were included
Methods and tools	Studies published that employed co-design approaches in the development of electronic healthcare tools	Studies that used non-electronic healthcare tools
Language		Studies in which publications are not in the English language were excluded

Data extraction and synthesis

Data extraction (selection and coding)

All records were uploaded to Covidence for screening purposes.21 The authors (ACC, KA) independently screened titles and abstracts that were identified in the search results from the database and excluded those which did not meet the selected inclusion criteria. A third reviewer (LM) resolved conflicts. After acquiring the full-text article, each author (ACC and KA) independently reviewed and assessed each article to determine if it met the inclusion criteria. The final list was reviewed by the two authors to confirm which articles should remain in the review. A third reviewer (LM) resolved conflicts.

Study data were extracted for analysis and included study setting, population characteristics (age and health status), theoretical frameworks and principles used, terms used to define codesign approaches, types of EHTs, user testing measures, types of codesign involvement processes (workshops, focus groups, interviews, prototyping, think-aloud, observation, contextual inquiry, usability testing, etc), levels of participation and learning outcomes.

Risk of bias (quality) assessment

While our systematic review was not focused on reported outcomes and may not have required a quality assessment, as suggested in similar systematic reviews related to involvement processes, we wanted to provide qualifications as to how we assessed each study.18 22 We evaluated the quality of studies included in this review, using the Agency for Healthcare Research and Quality Evidence-based Practice Center approach.23 Per this approach, two authors (ACC and KA) rated studies on directness, consistency, limitations and reporting bias, and a consensus was reached through a discussion of conflicts.23

Strategy for data synthesis

The data from each study were combined to provide a narrative or descriptive overview. We developed a data extraction form in Excel using a random sample of 2–3, including full texts of articles and revised it iteratively. The main characteristics extracted from the included studies are outlined in online supplemental appendix B—study classification methods). A comprehensive assessment of all study results has been conducted, highlighting similarities and differences in each study regarding the types of codesign approaches used. This comparison was necessary, as there was anticipated heterogeneity in the terminology and approaches. We used widely reported classification methods to categorise the included studies on the types of healthcare tools,17 types of involvement processes,24 levels of participation3 and levels of learning.25 26

To determine whether the tool described in the study met the inclusion criteria, each tool was classified using the WHO Classification of Digital Health Interventions, which is a framework that focuses on objectives within health sectors and categorises the digital technology that fits within each sector.17 The classification scheme focuses on interventions for four primary users, including clients, healthcare providers, health system or resource managers and data services. Our target population is older adults; therefore, they fit within the client framework, but it should be noted that caregivers also fit within this grouping. The WHO Classification framework was developed to synthesise evidence for digital health interventions, with the terminologies and definitions refined through public feedback, therefore is an appropriate framework for categorising the EHTs in this review.17

The classification tool for clients consists of seven categories, each further broken down into subcategories. The tools which were developed or redesigned within each included study have been classified within five of the seven overarching WHO categories, including Targeted Client Communication, Client to Client Communication, Personal Health Tracking, Citizen Based Reporting and On-Demand Information Services to Clients.

Involvement processes were extracted from the studies and classified by the codesign framework originally developed by Leinonen and since further refined, representing four distinct phases, including contextual inquiry, participatory design, product design and software prototype as hypothesis (functional prototypes).24 27 Levels of participation were extracted and classified based on Vaughn’s framework, with levels ranging from being informed, consulted, involved, in collaboration as a coleader, to empowering oneself and others.3 Table 2 provides Vaughn’s definitions for levels of participation. Leinonen and Vaughn’s framework were combined to evaluate how the core involvement processes that span across the various codesign approaches relate to the five levels of participation, as depicted in figure 1.

Table 2

Levels of participation

Levels of participation	Definitions-Vaughn et al3
Inform	Information is shared with community members and could become more participatory if members ask for information relative to their interests.
Consult	Participants provide feedback to the researchers, as related to a specific decision point.
Involve	Participants provide feedback to researchers during the entire process.
Collaborate	Participant input is valued equally, as a co-leader in the development process.
Empower	Participants take a key role in leading the initiative to get others engaged.

Figure 1

Extracted from Vaughn’s involvement framework and Leinonen’s process framework and modified to reflect the bi-directional aspect of involvement as a process in collaboration with the stages of the co-design involvement processes.

Within Leinonen’s framework, participatory design is defined as the stage at which stakeholders have the opportunity to provide input.27 For this literature review, this subcategory encompasses all involvement processes extracted from the studies in which stakeholder insight was represented. Other involvement processes were defined but did not fit within the cataloguing of processes, which included the use of eye-tracking,8 28 teach-back methods29 and spending time teaching older adults how to use technology in a classroom setting to better prepare them for engaging in the development process.14 While the latter two methods do not fit within the cataloguing of processes, the extent to which this involvement occurred provided a higher level of participation and associated learning outcome, suggesting that participants were empowered through these methods.

Learning outcomes were extracted from each study and classified using both Bateson’s levels of learning, and Argyris’ mutual learning model.25 26 Bateson’s learning framework consists of five learning levels, however only learning levels 0–2 are applicable to this review.25 30 31 Learning level 0 refers to when a participant is provoked to respond but where no change to their action takes place. Learning level 1 is an engaged response, in which a participant’s errors are reflected on and corrected. Learning level 2 uses context and tacit knowledge to draw insights for higher level learning to take place. Argyris’ model consists of single and double loop learning. Single loop learning (synonymous with learning level 1—reflective learning) is the ability to detect and correct an error, which is the result of an ineffective action, and double loop (synonymous with learning level 2—mutual learning) occurs when significant actions are evaluated by the extent to which they help participants generate valid and useful information.26 Learning level 2—mutual learning is where participants, designers and researchers learn from each other, or when participants learn from reflecting on their own efforts and errors during user testing,2 including when participants are taught or exposed to technology prior to user testing.14 29 As this review focuses on the development of EHTs, specifically the tools used to inform healthcare decisions, evaluating the role of learning through codesign approaches is necessary. If learning does not occur to the extent necessary to make qualified decisions, the information generated for consumption by the end user may not be validated.

Results

The initial database search returned 835 articles, including 495 from PubMed, 246 from Embase and 94 from Scopus, of which 210 were duplicates, leaving 625 articles for the title and abstract review. On completion of the title and abstract review, 156 articles remained for a full-text review. The further assessment resulted in 25 studies that met the inclusion criteria. The review process is depicted in the PRISMA flow diagram32 in figure 2.

Figure 2

PRISMA flow diagram. PRISMA, Preferred Reporting Items for Systematic Reviews.

Study quality assessment

All included studies were found to be of high quality, with no concerns regarding directness, consistency, precision and reporting bias. We did find some limitations, which we rated as ‘medium’ level, including two studies that did not indicate frameworks,8 33 and 6 of the 25 studies had limitations due to sample size.8 11 34–37 Overall, we did not find any studies that would rate as a ‘high’-level limitation.

Characteristics of included studies

Table 3 summarises the characteristics of all included studies. Twenty-five eligible studies were identified from our initial database search. While all studies included older adults over the age of 60, we were unable to define the overall mean age, as six studies simply provided the age range,7 29 37–40 and three simply stated participants were 65 or older.11 12 14 Overall, the studies included participants in the age range from 60 to 91, and of the studies which provided specific ages, we have a median age of 70.9. Of the 25 studies, 13 studies involved additional participants including caregivers, healthcare providers and others such as researchers,41 42 department heads and policymakers42 and control group participants.43 A total of nine studies targeted participants with specific diagnoses (table 3).

Table 3

Summary characteristics

First author’s last name	Year of publication	First author’s location	Digital health intervention	Co-design approach	Co-design approach defined	Number of phases	Learning level	Learning beneficiary	Number of stakeholders	Number of OA	OA/P	Ca	HP	Ot	Age of older adults (mean)	Diagnosis
Abujarad et al52	2021	USA	PHTCBRODISC	User centred design	Yes	–	Level 0	Research team	38	24	X	X	X		69
Ahmed et al34	2019	USA	TCCPHT	Participatory design	Yes	–	Level 1Sgl LoopLevel 2Dbl Loop	Research teamparticipant	9	7	X	X			67	Heart failure
Bogza et al29	2020	Canada	PHTODISC	User centred design	No	6	Level 1Sgl LoopLevel 2Dbl Loop	Research teamparticipant	19	12	X	X	X	X	(2)60 – 64 years (6) 65–74 years, (3) 75–84 years, (1)>85 years	Mild Cognitive Impairment (MCI)
Cornet et al12	2020	USA	PHTODISC	User centred design	Yes	3	Level 0	Research team	63	50	X	X	X	X	>65	Heart failure
Gonzalez et al33	2014	USA	PHT	Human factors	No	–	Level 0	Research team	12	12	X				77.7
Grimaldi et al11	2020	Italy	ODISC	Human centred design	Yes	4	Level 0	Research team	10	10	X				>60
Gustafson et al14	2016	USA	CCC	User centred design	Yes	5	Level 1Sgl LoopLevel 2Dbl Loop	Research teamparticipant	135	135	X	X	X	X	>65
Harrington et al9	2018	USA	PHT	Co-designhuman centred computinginteraction design processparticipatory design	No	2	Level 1Sgl LoopLevel 2Dbl Loop	Research teamparticipant	39	39	X				72.1
Harte et al7	2017	Ireland	PHT	Human centred design	Yes	3	Level 1Sgl LoopLevel 2Dbl Loop	Research teamparticipant	12	12	X				61–85
Hoffman et al39	2020	USA		Co-designuser centred design	No	3	Level 0	Research team	16	8	X	X			Phase 2: 65–84Phase 3: 61–89
Holden et al53	2020	USA	ODISC	User dentred design	Yes	3	Level 0	Research team	23	23	X				67.6
Kim et al38	2018	USA	PHTODISC	Participatory design	No	2	Level 0	Research team	17	17	X				65–80
Mansson et al35	2020	Sweden	PHT	Co-creation	Yes	2	Level 1Sgl Loop	Research teamparticipant	10		X				76
Martin-Hammond et al36	2015	USA	ODISC	User centred design	Yes	3	Level 0	Research team	7	7	X				70
Nguyen et al42	2019	Netherlands		Co-design	No	3	Level 0	Research team	56	56	X			X	68.9
Nielsen et al44	2018	Denmark	TCCCCCPHTODISC	Participatory designco-design	Yes	3	Level 0	Research team	54	36	X	X	X		70.3	Hearing impairment
Or et al43	2012	Hong Kong	TCCPHT	Usability evaluation approachusability test design	No	–	Level 0	Research team	57	50	X			X	71.6	One of the following: hypertension, diabetes, heart disease, asthma, prostatitis, hypotension
Petersen et al45	2019	Netherlands	ODISC	Does not state	No	3	Level 0	Research team	71	32	X		X		70	Larynx cancer
Poirier et al28	2019	USA	ODISC	Does not state	No	–	Level 0	Research team	45	45	X				72.2
Portz et al40	2020	USA	PHTODISC	Human centred design	Yes	3	Level 0	Research team	81	31	X	X	X		66–91	Heart failure (palliative care patients)
Scandurra et al37	2013	Sweden	TCCCCCODISC	Participatory designuser centreddesign	No	5	Level 1Sgl LoopLevel 2Dbl Loop	Research teamparticipant	8	8	X				65–80
Ummels et al41	2020	Netherlands	PHT	Co-creationuser centred design	Yes	4	Level 1Sgl LoopLevel 2Dbl Loop	Research teamparticipant	63	63	X			X	70.3
Van De Dijk et al54	2013	Netherlands	PHT	User centred design	Yes	–	Level 0	Research team	86	86	X				72.5	COPD
Van Velsen et al55	2019	Netherlands	CBRODISC	Participatory design	No	2	Level 0	Research team	82	82	X	X			71.5
Wikberg-Nilsson et al8	2018	Sweden	ODISC	Co-designparticipatory user experience design	No	5	Level 1Sgl Loop	Research teamparticipant	7	7	X				75	Reduced hearing, eyesight, mobility, sensibility, and loss of memory function

Ca, caregivers; CBR, citizen-based reporting; CCC, client to client communication; HP, healthcare providers; OA/P, older adults/patients; ODIFC, on-demand information services to clients; Ot, others; PHT, personal health tracking; TCC, targeted client communication.

Of the 25 studies, the number of stakeholders across all studies ranged from 7 to 135 participants, with a mean of 35.5 for participants classified as older adults, and a mean of 40.8 when including all stakeholders.

The settings and locations varied by study. Of the 25 studies, only 11 reported a specific setting in which the study was held, including laboratories, clinics, homes, community and senior centres as well as remote sessions via Zoom. 50% of the studies took place in North America, 46% in Europe and only one study (4%) was from Asia.

Codesign approaches

Table 3 includes codesign approaches, number of phases conducted in each study and whether they defined their approach. Overall, nine different codesign approaches were represented among the 25 studies. Of the 25 studies, seven studies indicated the use of more than one codesign approach,8 9 37 39 41 43 44 and two studies stated that stakeholders were involved in development but did not indicate a specific approach.28 45 Overall, 19 of the 25 studies conducted their study in multiple phases, ranging between 2 and 6 phases, and an overall mean of 3.4 phases per study. Of the six studies which did not employ iteration in development, two labelled user involvement as either a human factors or usability evaluation approach and did not provide definitions to the approach used.33 43 Reviewing the definitions of codesign approaches, eight studies refer to involvement with end user/stakeholder, 5 studies refer to system usability, 34 studies refer to user needs, 3 studies refer to using prototypes and 3 studies refer to iterative development. Studies did not determine which involvement processes were used when defining the approach.

Theoretical framework/codesign principles

Studies did indicate that specific frameworks or principles were utilised to guide them in selecting which involvement processes to follow. Online supplemental appendix C summarises the key theoretical frameworks or codesign principles employed for each study. Studies that reference the same terminology for both the approach and framework such as human-centred design (HCD) did not use iteration in their definition, but referenced standards such as HCD methodology—ISO 9241–210, which specifically states this is an iterative process.46

Of the 25 studies, 2 did not indicate that their study was guided by a particular framework or principle,8 33 although one study was defined as using codesign with a participatory experience, and the other as using human factors, neither provided clarification on their approaches. As 18 different frameworks or design principles were represented across the 25 studies (online supplemental appendix C), it is difficult to conclude which involvement processes should be included within each framework. The most commonly reported frameworks were Human-centred design methodology (ISO Standards) (n=5), International Patient Decision Aid Standards (IPDAS) (n=4) and user centre design (n=3). There are inconsistencies among the studies, which indicated using similar frameworks or principles, such as HCD, User Centered Design (UCD), and IPDAS, which warrants further exploration into what it means to state a study is employing a particular framework or principle. Interestingly, 6 of the 25 studies developed recommendations for frameworks that could be employed when collaboratively designing EHTs with older adults.

Electronic healthcare tools

Types of EHTs

Table 4 summarises the EHTs based on the WHO Classification of Digital Health Interventions. In 12 of the 25 studies, the EHT that was developed or redesigned fits within multiple categories, therefore, was catalogued into multiple categories. The overall objective was to ensure that the tools included in the selected studies fit within a categorisation that could be defined as those which support healthcare decisions, which was achieved through this classification process.

Table 4

WHO classification of digital health interventions

WHO classification of digital health interventions	Total studies (n=25), n (%)	References
1.1 Targeted client communication
1.1.2 Transmit targeted health information to client(s)	1 (4%)	44
1.1.3 Transmit targeted alerts and reminders	2 (8%)	37 43
1.1.4 Transmit diagnostics result, or availability	1 (4%)	34
1.3 Client to client communication
1.3.1 Peer group for clients	3 (12%)	14 37 44
1.4 Personal health tracking
1.4.1 Access by client to own medical records	3 (12%)	34 35 38
1.4.2 Self-monitoring of health or diagnostic data by client	7 (28%)	7 33 34 41 44 52 54
1.4.3 Active data capture/documentation by client	8 (32%)	9 12 29 34 35 39 40 43
1.5 Citizen based reporting
1.5.1 Reporting of health system feedback by clients	2 (8%)	52 55
1.6 On-demand information services to clients
1.6.1 Client look-up of health information	16 (64%)	8 11 12 28 29 36–40 42 44 45 52 53 55

Subjective and objective measures

We extracted 18 different measures, including 13 subjective, 3 objective, and 2 physiological measures, accounting for 19 of the 25 included studies, as indicated in table 5. These studies used between 1 to 6 measures, with a mean of 3.2 measures per study, with usability as the most common endpoint, being measured in 13 of the included studies. Of the 19 studies that used these measures, only nine studies used objective measures but were focused on three specific endpoints (errors, performance and need for assistance). Within the 18 studies that used subjective measures, we extracted 23 different endpoints.

Table 5

User testing—subjective and objective measures

Measures	Endpoint	Total studies (n=25) n (%)	References
Validated
Subjective measures
ACE (altarum consumer engagement)	Patient engagement	1 (4%)	34
ASQ	Efficiency	1 (4%)	7
CES	Confidence and computer efficacy	1 (4%)	52
Decisional conflict scale (values clarity)	Values clarity	2 (8%)	28 29
I-PANAS-SF	Emotional reactions	1 (4%)	52
Likert scale (specified validity)	comprehension/recall, graphicacy, numeracy	1 (4%)	28
NASA-TLX	Cognitive workload	2 (8%)	7 12
NVS	Health literacy	2 (8%)	12 34
Ottawa acceptability scale	Quality	2 (8%)	29 39
SURE scale	Values clarity	1 (4%)	39
SUS	Usability	5 (20%)	7 12 29 52 53
Us.E 2.0	Quality	1 (4%)	11
Non-validated
Subjective measures
Likert scale (non-specified validity)*		13 (52%)	7 9 11 12 14 29 33 37 41–45
Objective measures
Mistakes counted	Errors	7 (28%)	7 11 12 39 42 43 53
Time to complete	Performance	4 (12%)	7 28 43 53
Times request assistance	Need for assistance	2 (8%)	43 53
Physiological measures (eye-tracking)
Cognitive processes	Usability	1 (4%)	8 28
Gaze fixation	Usability	1 (4%)	8 28

*Satisfaction, user acceptance, appearance, comfort, functionality, reliability, usability, usefulness, ease of use, comprehensibility, effectiveness, value of information.

ASQ, Ages & Stages Questionnaire; CES, Computer Efficacy Scale; I-PANAS-SF, International Positive and Negative Affect Schedule Short-Form; NASA-TLX, National Aeronautical and Space Administration’s Task Load Index; NVS, Newest Vital Sign; SURE, Sure of myself, Understand information, Risk-benefit ratio, Encouragement; SUS, System Usability Scale.

We further categorised these measures to evaluate whether studies were using validated measures and found that only five studies (20% of all included studies) used validated subjective measures to evaluate their EHT. Other than the System Usability Scale (SUS), which was used in 20% of the studies to measure usability, the other validated subjective measures identified were only used in 4%–8% of studies, while 12 distinct non-validated Likert-scale assessments spanned across 52% of all studies. Usability was also the most specified measure in non-validated Likert-scale assessments, taking place in eight studies. Overall, 13 of the 25 studies reported measuring usability, but less than half used a validated measure.

Stakeholder involvement

Types of involvement processes

A variety of involvement processes were used in the 25 studies, as indicated in table 6, which reports 152 processes across all the included studies. Online supplemental appendix C includes the full list of involvement processes used within each study. The involvement processes that were most widely reported were interviews and prototypes (medium fidelity) (n=18), followed by prototype (functional) (n=15) and the least reported was remote evaluation (n=1). Each study used between 2 and 11 involvement processes, with a mean of 6 processes per study.

Table 6

Codesign involvement processes by learning levels

	Involvement processes	Level 0 (n=25), n (%)	Single loopLevel 1 (n=8), n (%)	Double loopLevel 2 (n=7), n (%)
Contextual inquiry (rapid ethnography)*	Shadowing in context including semi/unstructured interviews, observation, etc	6 (24%)	4 (50%)	4 (57%)
Participatory design	Survey	10 (40%)	3 (38%)	3 (43%)
	Interview	18 (72%)	7 (88%)	5 (71%)
	Cognitive walkthrough	8 (32%)	4 (50%)	3 (43%)
	Think aloud	11 (44%)	2 (25%)	1 (14%)
	Remote evaluation	1 (4%)	1 (13%)	1 (14%)
	Focus group	8 (32%)	4 (50%)	4 (57%)
	Task analysis	10 (40%)	3 (38%)	2 (29%)
	Scenarios	7 (28%)	2 (25%)	1 (14%)
	Prototype (low fidelity)	8 (32%)	4 (50%)	3 (43%)
	Working Group Workshop	9 (36%)	5 (63%)	3 (43%)
Product design	Use cases	4 (16%)	2 (25%)	1 (14%)
	User stories	9 (36%)	2 (25%)	2 (29%)
	Prototype (medium fidelity/throw-away)	18 (72%)	8 (100%)	5 (71%)
	Heuristic evaluation	5 (20%)	2 (25%)	1 (14%)
Software prototype as hypothesis	Prototype (functional/Beta version)	15 (60%)	8 (100%)	7 (100%)
Other	Other	5 (16%)	3 (38%)	2 (29%)
Total		152	64	48

Contexual inquiry Participatory design Product design Prototype Others (color-coded to reflect association in figure 3).

*The goal of contextual inquiry is to understand the context and the preliminary design challenges. It involves understanding the user, what the design challenges are and why they are being solved. Leinonen et al24 proposed ethnographic methods and benchmarking the environment to have a complete understanding of the major design challenges. Contextual inquiry is used as a category to remain consistent with Leinonen et al.’s terminology, but we recognise that this category can be broadened into ‘rapid ethnography’ which Leinonen recommended in his follow-up research contributions.56

Figure 3

Involvement processes (percent of total). *All studies (n=25) reached the ‘inform’ and ‘consult’ levels of participation.

Levels of participation

Defining each study based on the level of participation presented challenges. Each study was assessed to have reached a participation level based on Vaughn’s definitions3 and cross-classified by codesign approach (Online supplemental appendix D). While Vaughn’s definitions are related to research processes, they can be translated to the levels of participation in developing EHTs. The challenge in mapping each level against the codesign approach is that the studies did not define distinct points at which participants were involved. However, we can see that most studies employed the first three levels of participation, that of inform, consult and involve, as evidence suggests the information provided to the research team was valued and assessed for inclusion into various design elements of the tool. However, fewer studies indicated that participants had an equal level of decision-making when making iterative changes to the design, and even fewer empowered the participants to take a lead in engaging others on the usefulness of the tool.

Figure 3 depicts each level of participation and the percent breakdown of associated involvement processes. The most distinct comparison is between the studies classified as level 5-Empower (no) versus empower (yes). This comparison reflects a decrease in participatory design (from 69% to 55%) and an increase in functional prototypes (from 2% to 14%), respectively.

Learning outcomes

Table 3 summarises the learning levels of each study. All 25 studies were categorised as having met learning level 0, in which researchers benefitted from participants’ responses to specific user testing measures. While eight studies met learning level 1, in one of the eight studies, only a subset of the participants were involved in reflective learning36 as this occurred in later phases of the study, in which the number of participants had decreased by more than half. In addition to these eight studies, one study acknowledged to the participant they had committed an error but did not provide feedback43 and another study acknowledged it as a study limitation that they did not review results with participants.40 Seven studies reached learning level 2—mutual learning, using transference of knowledge that focused on prior experience with the technology. This benefitted both researchers and participants, indicating a bidirectional flow of knowledge occurred. It should be noted that these seven studies were also classified as having reached the empower participation level.

Table 6 summarises the difference in the types of involvement processes used when studies engaged researchers and participants in mutual learning. While additional studies engaged in observational activities that provided opportunities for additional context, these studies did not engage in mutual learning. For example, an additional two studies used contextual inquiry; however, in these studies, the learning was unidirectional and only benefitted the researchers.

Discussion

Key findings

This review focused on synthesising the current state of involving older adults in codesign approaches to develop EHTs and aimed to identify how the codesign approaches used mutual learning techniques to benefit older adult participants. We extracted evidence from each study to categorise the terms used to describe codesign approaches as well as to draw themes around how these terms were defined. We identified the EHTs and categorised them by health sector classifications as well as the subjective and objective measures employed. This review also explored the frameworks and principles that determined the stakeholder involvement, including types of involvement processes, levels of participation and learning outcomes.

As anticipated, heterogeneity exists within the definitions used to describe codesign approaches. While 13 definitions were represented across the studies, general themes presented suggest that the approaches could be used interchangeably. While this may further exacerbate a problem, and lead to inconsistent use of terminology, what is of notable importance within this review, is not so much on the name of the approaches used, but how they are interpreted and then employed to determine the methods for involving and codesigning with older adults. Not surprisingly, each study specified either definitively within their aim or subsequently, that end users should have some level of involvement or participation during the development life cycle process. However, studies did not specify the extent to which older adults should be involved, nor did they specify whether a learning model should be employed. Of interest, is that only 36% (n=9) of included studies had a mutual exchange of knowledge between the research team and study participants, even though all studies were classified as having used a codesign approach. For the remaining studies (n=16; 64%), participants engaged in processes in which they simply provided feedback on the EHT and did not engage in processes that facilitated a mutual exchange of knowledge. Future codesign studies involving older adults should use improved codesign frameworks and optimise mutual learning.

Frameworks and principles

We chose not to cross-classify this variable as to not misrepresent relationships; however, the diversity of frameworks extracted exemplifies a need for a framework that is specific to engaging older adults in the development of EHTs, which can be consistently used, and which also provides validation to the involvement processes that should be employed. This need is further exemplified as 25% (n=6) of the included studies suggested recommendations for a framework that could be used when engaging older adults in the development of EHTs. Further research is needed to determine whether the proposed, or a combination of existing and proposed frameworks can be validated to optimise older adults’ involvement, one that provides an environment for mutual learning and empowerment.

While the EHTs that were applied in the studies spanned multiple categories under the WHO classification, the majority focused on tools that provided the ability to search for information on a particular health topic or that allowed participants to track their health information through self-monitoring, including accessing personal health records, use of wearables or documenting health information. Each study involved participants in developing an EHT that would support older adults’ healthcare decisions. Previous studies have reported positive impacts to both older adults and improvements in technology when engaging them in the development lifecycle.1 2

Furthering the notion that a framework, specific to older adults in developing EHTs, needs to be evaluated, is concerning the inconsistency of user testing assessments. We extracted evidence that 19 different measures were used in 76% (n=19) of the included studies and found that only five studies (20% of all included studies) used a validated subjective measure to evaluate their EHT. Usability was the most specified measure, and while the SUS, a validated usability assessment was used, more than half of the studies used a non-validated Likert-scale assessment. This discrepancy in assessments for user testing makes it difficult to draw definitive conclusions as to whether studies are consistently measuring what they intended to measure. This is consistent with a recent review that found studies often use non-validated questionnaires rather than readily available validated assessments and that despite recommendations to measure learning, this is rarely employed.47

Stakeholder involvement, participation and mutual learning

While we found inconsistency in the involvement processes among the studies, over 70% included interviews and prototyping, others included a range of processes such as cognitive walkthroughs, think-aloud sessions, surveys, scenarios and focus groups, and only 20% of the studies used contextual inquiries and use cases. While fewer studies used contextual inquiries, the studies that employed this process were more likely to be categorised as having both empowered users and involved them in mutual learning. For studies in which participants and researchers benefitted from mutual learning, we observed a higher utilisation of specific involvement processes including contextual inquiry, cognitive walkthroughs, focus groups, working group workshops and functional prototyping and a decrease in think aloud sessions. This is consistent with previous research by both Bateson and Argyris as well as recent studies that evaluated learning outcomes30 48 and suggests the importance of understanding user context and involvement in group activities that facilitate reflective and mutual learning. While Argyris also suggests that when groups work on important topics, they are less receptive to corrective feedback, inhibiting learning and potentially producing information of lower value; we observed that these processes were effectively employed and solicited valid information on important topics, or that of improving the development of EHTs that are meant for healthcare decisions.

To the best of our knowledge, current frameworks or principles have not provided guidance on distinguishing the levels of participation, nor the validated involvement processes that should be used when engaging older adults in developing EHTs. To provide structure to this notion, this review drew on Leinonen’s framework for processes and Vaughn’s framework for levels of participation. After extracting evidence of both variables from the 25 studies, and overlaying these on top of the two frameworks, we began to recognise the areas in which we need to further explore. We need to explore what it means to involve older adults, specifically which involvement processes are more likely to lead to empowerment and mutual learning. This is important because the evidence extracted from the studies suggests that bidirectional knowledge, which benefits both older adults and researchers, may increase as participation levels increase.

It has also been suggested that exposure to the technology before usability testing has shown to be an effective means of eliciting users’ needs, such that the increased knowledge and awareness of the tool inspires ideation and innovation,9 37 and that tacit knowledge or real-life context increases the ability to portray specific user needs.42 However, there are concerns about bias if an individual is exposed to technology before soliciting their design ideas.49 Not to dismiss the concern for bias, but further evidence supports the notion that experienced users of technology can provide more meaningful insight during design sessions and could provide feedback more representative of the target population.50 We also see that allowing users to engage with the technology for an extended time or ensuring they understand how to use the technology, reduces frustration when contributing to design ideas.14 29 49 For example, when older adults were asked to participate in two similar design activities, they were more involved when they had familiarity with the topic, compared with being asked to innovate on a novel topic.34

Suggestions have been made to separate codesign processes from outcomes and focus on the role that participants have to be codesigners and coinitiators,51 which supports the values defined by the fourth and fifth levels of involvement, that of collaborate and empower, respectively. We should be looking at involvement as a process, one in which through various stages of the development life cycle, users have the potential to cross through the spectrum of involvement highlighted in Vaughn’s framework. By combining Leinonen and Vaughn’s framework, we can see how the involvement processes that span across the various codesign approaches can be used in iterative development, but through these various stages, users can participate in the processes at different levels, from being informed, consulted, involved, in collaboration or as a coleader, to empowering oneself and others. This is consistent with a review by Fischer et al that also suggest that involvement as a process should be a consideration for future research and policy, specifically the extent to which users are involved and receive benefit.18 Furthermore, we suggest that additional exploration into the extent to which a greater understanding of the tool can contribute to richer feedback is necessary.9 We also suggest the levels of participation must be bi-directional for mutual learning to occur, as depicted in figure 1, and when realised within the development lifecycle will result in greater contributions to the development of the tool.

Limitations

As our inclusion criteria spanned an array of terms that we categorised as codesign approaches, we had several approaches that were only represented by one study, which may not fully represent all studies that incorporate such approaches. One of the biggest limitations of this study was the difficulty in extracting and classifying the information; however, we used mitigation strategies (eg, established frameworks and classification methods) to support the review of variables including types of EHTs, stakeholder involvement and participation and levels of learning. Studies that employed codesign approaches were included in our review; however, we limited our review to the research groups’ initial study and did not evaluate whether subsequent papers by these groups provided a broader analysis of user experience.

Conclusions

This review depicted not only the heterogeneity in the terminology used to define codesign approaches but also the gaps in frameworks and design principles for developing EHTs for older adults. Most studies in this review developed EHTs specific to patient engagement and designed their study to include older adults’ feedback in the design, indicating a collaborative focus centred on meeting the needs of this population. However, we found that studies did not consistently use validated assessments for conducting usability testing. We also found gaps in the evidence to support which involvement processes should be used when codesigning with older adults. Additionally, gaps exist around which learning models should be employed when conducting user testing with older adults. Future research needs to evaluate how empowerment is associated with mutual learning, whether this bidirectional knowledge can occur at all participation levels, and whether the benefit to both older adults and researchers increases as the levels of participation increases. To ensure studies are optimising the involvement processes, participation and learning of older adults when developing EHTs, there is a need to place more emphasis on the approaches promoting mutual learning.