Developing an interactive mobile phone self-report system for self-management of hypertension. Part 2: content validity and usability.

Self-management support tools using technology may improve adherence to hypertension treatment. There is a need for user-friendly tools facilitating patients' understanding of the interconnections between blood pressure, wellbeing and lifestyle. This study aimed to examine comprehension, comprehensiveness and relevance of items, and further to evaluate the usability and reliability of an interactive hypertension-specific mobile phone self-report system. Areas important in supporting self-management and candidate items were derived from five focus group interviews with patients and healthcare professionals (n = 27), supplemented by a literature review. Items and response formats were drafted to meet specifications for mobile phone administration and were integrated into a mobile phone data-capture system. Content validity and usability were assessed iteratively in four rounds of cognitive interviews with patients (n = 21) and healthcare professionals (n = 4). Reliability was examined using a test-retest. Focus group analyses yielded six areas covered by 16 items. The cognitive interviews showed satisfactory item comprehension, relevance and coverage; however, one item was added. The mobile phone self-report system was reliable and perceived easy to use. The mobile phone self-report system appears efficiently to capture information relevant in patients' self-management of hypertension. Future studies need to evaluate the effectiveness of this tool in improving self-management of hypertension in clinical practice.

Introduction

Hypertension remains an important risk factor for cardiovascular disease worldwide, and more than 7.5 million premature deaths can be attributed to high blood pressure. Furthermore, it has been shown that the impact of hypertension on daily life is quite severe (1,2). Despite a profound evidence base for a combined medication and lifestyle intervention and despite guidelines and education programmes, only a quarter to a third of people receiving treatment for hypertension achieves well-controlled blood pressure (3,4). One factor explaining this might be that no more than 50% of people receiving treatment for hypertension adhere to their treatment (3), and another explanatory factor might be that lifestyle adjustments in general are hard to maintain (5). It is known that patients’ views of hypertension impact on their decision whether or not to stay on treatment and/or maintain lifestyle changes (6,7).

Increased understanding among patients as well as healthcare professionals of the interrelationships between blood pressure, treatment and well-being may support patients’ self-management of hypertension, including medication adherence and treatment effects. One approach to achieve such understanding may be to support self-management through a self-report system.

Mobile phones have previously been successfully used in several areas in healthcare. Even so, there are surprisingly few research studies that focus on mobile phone technology for disease management and health monitoring (8,9). Although literature on the use of mobile phones in the delivery of healthcare is emerging, the published evidence is limited. However, Glynn et al. (10) conclude in a review that self- monitoring is useful in hypertension care but that reminder systems require further evaluation.

To our knowledge, there are no existing fit-for-purpose (11–13) mobile phone self-report tools supporting self-management for persons with hypertension. Our earlier focus group study (14) suggests that a self-report system would be of value to use, for example during periods when blood pressure is difficult to control or at the start or after a change of medication. Moreover, the self-report system should preferably be available through the patient’s own mobile phone to be able to routinely capture relevant self-reports as well as be used to record blood pressure measured separately with an automatic blood pressure monitor at home. In addition, the self-report system should be interactive in the sense that it allows the patient him/herself to follow, and possibly develop an understanding of the possible interconnections between the blood pressure and the self-reports. The aim of the study was to examine comprehension, comprehensiveness and relevance of items, and further to evaluate the usability and reliability of an interactive hypertension- specific mobile phone self-report system.

Methods

A communication system for mobile phones, Circadian Questions (CQ), developed by 21st Century Mobile AB (http://www.cqmobil.se), was adapted for use in this study. In this system self-reports are registered by means of the patients’ own mobile phones and returned to and stored in a database. The system also includes a login-restricted web-based feedback module, which provides patients with the opportunity to examine for themselves how their self-reports of drug intake, lifestyle, blood pressure, symptoms and wellbeing may interact. Feedback is provided in the form of graphic displays showing plots of these variables over time.

Development process

The development process started with focus group interviews to lay a foundation from which to depart. It thereafter continued with item drafting based upon focus group results, complemented by a literature search and several design meetings. Furthermore, content validity and usability were ensured through cognitive interviews with an iterative analysis process using an Item Tracking Matrix (ITM). Finally, a set of items was presented and used for the interactive self-report system. The process was performed according to good research practice, as described in the FDA guidelines (13) and the ISPOR task force report on good research practice (11,12) with regard to developing Patient Reported Outcome Measures (PROM) (Figure 1).

Figure 1.

Flow chart of the development process.

Support needs

Five focus group interviews were performed, including patients with hypertension (n = 15) and healthcare professionals experienced in hypertension care (n = 12) to identify important aspects for inclusion in the self-report system. The interviews resulted in a number of aspects regarded as meaningful to include in a self-report system: blood pressure values, pulse, symptoms, medication intake, side-effects to medication, lifestyle and well-being (14).

Item drafting

The next step was to extract areas and concepts based on results from the previous focus group interviews (14) complemented by a review of the literature (15–17). Following this, a set of items was drafted. Items and response options were developed to be compatible with all types of mobile phones – classic mobile phones as well as smart phones – which placed a focus on the aspect of limited space in regard to item wording.

It was determined that a five-stage Likert scale would be used as it would be possible to present in a similar way in both classic and smart phones. The draft items were divided into items to be answered every day (items 1–12), including blood pressure values, and items covering side-effects to be answered once a week (items 13–16). The items assessing side-effects were decided to be an option for those to whom they are relevant. Furthermore, several (n = 10) interdisciplinary design meetings were held, focusing on the wording of items as well as response scale/response options.

Content validity and usability evaluation: cognitive interviews

Recruitment and participants.

Twenty-one patients who were currently undergoing medical treatment for hypertension were recruited by their responsible district nurse or physician for face-to-face cognitive interviews. Patients aged 30 years or under and those unable to understand and speak Swedish were excluded. To enhance the chances of effectively testing item understanding, attempts were made to achieve a sample representative for the topic in focus and with demographic diversity (18,19). Patients were recruited from two geographically separate locations: one primary healthcare centre in a multi-cultural city suburb and one internal medical outpatient clinic at a provincial hospital in a smaller town. The proportion of men was slightly higher than women as is the case in the middle-aged hypertensive population (20) and other demographics were also comparable with the general hypertensive population in Sweden (15) (Table I). The number of antihypertensive drugs prescribed ranged from one to a maximum of four (median = two). Nine of the 21 patients had co-morbidities, the most common being diabetes and high cholesterol.

Table I.

Participant characteristics.

Participants (n = 21)	Women (n = 9)	Men (n = 12)
Median age (range)	69 (54–73)	62.5 (49–82)
Median years with hypertension (range)	10 (< 1–40)	12.5 (< 1–30)
Marital status
Married	8	9
Unmarried	1	1
Widow/widower	0	2
Education
Compulsory school (≥ 9 years)	2	1
High school (9–12 years)	6	4
University	1	7
Employment status
Employed	2	5
Unemployed	0	1
Retired	7	6

Preparing the cognitive interviews.

A structured interview guide designed for using probes was developed with cognitive interviewing in mind, to determine the understanding and meaning of items (19). The interview questions are shown in Table II.

Table II.

Cognitive interview guide.

Introductory questions

Coverage and number of questions

What do you think of the number of questions?

Spontaneously, do you feel any questions are missing?

Comprehension and relevance: items and response options

Did you understand the question?

If no: what was it that was difficult to understand?

In your own words, what does this question mean?

Is the question relevant to you?

Would you pose this question differently? (probe to be used if needed)

If yes: how would you pose it?

Was it difficult or easy to find a response option when answering this question?

If difficult: how did you think when you chose a response option?

Usability

Measuring and registration of blood pressure

How was it to measure your own blood pressure?

How was it to enter the blood pressure value in the mobile phone?

Mobile phone application

How was it to use the mobile phone to answer the questions?

Easy?

Hard?

If hard: what was hard?

Do you have any suggestions of something that could make it easier for you to answer the questions in the mobile phone?

Closing question

Now, when we have gone through the questions again, do you feel anything is missing?

Conducting and analysing the cognitive interviews.

The 21 face-to-face interviews were conducted at the respondents’ (i.e. the interviewed patients) outpatient clinics in four rounds of three to five respondents each. They were audio-recorded and ranged in duration from 40–70 min. The first two interviews were designed as pilot interviews to test the process, but it was determined that they would be included in the total sum of 21 interviews, as the process did not change substantially. During the interviews, the interviewer made notes in an interview guide prepared for each respondent. The interviewer first instructed and showed the respondents how to handle the self-report system in the mobile phone, secondly, the respondents provided answers to two mock/training questions, and finally, they answered all 16 study items on their own. Retrospective probing was used (21), with respondents answering all items without being interrupted, in a mobile phone. This included automatic measuring of blood pressure and entering the blood pressure values in the mobile phone. Following this, respondents were interviewed about the items, one item at a time. Immediately thereafter, questions were asked about the usability of the mobile phone, the mobile phone application and the measuring of blood pressure, along with the other questions in the cognitive interview.

After each interview, the interviewer listened through the recording, taking thorough notes while listening. These notes, along with those taken during the interview, formed a detailed summary synthesis for each respondent. Based on this, an ITM (12) (a rigorous way to structure and systematize the analysis of cognitive interviews that tracks the development item-by-item and round-by-round) was developed. The ITM was continuously updated as the interview rounds were performed. Also, a Cognitive Interview Summary (CIS) (12) was constructed. The CIS and the ITM are available upon request. The interview process and analysis were iterative, as between rounds the findings were discussed and actions for the next round of interviews were decided on, during the interdisciplinary design meetings.

After the second round of interviews with patients, four interviews were held with healthcare professionals (two nurses and two physicians) with a consultative purpose, to gain their perspectives on the patients’ ability to understand each item as well as to get their perceptions of the self-report system.

By using the mobile phone as the mode of administering questions during the cognitive interviews, usability could be tested during the interview. By observing the participants as they entered answers into the data-capture system in the mobile phone and by asking specific questions about this afterwards, we obtained information about ease/difficulties and potential problems. The same procedure was used regarding measuring of the blood pressure. A home blood pressure monitor (Microlife BP A200 AFIB), validated according to the International Protocol of the European Society of Hypertension was used (22).

Reliability

The reliability of the mobile phone self-report system was examined by performing a test–retest. Twenty-one participants were asked to complete the ten mandatory items of the assessment twice, four hours apart. The clinical measurements, i.e. blood pressure and pulse, are not constant by nature, and the items describing side-effects were not answered on a daily basis or by all participants; hence, these items were not included in the test–retest. The test–retest reliability was analysed by calculating the intra-class correlation coefficient (r tt) (23).

Ethics

The study was approved by the Regional Ethics Board in Gothenburg, Sweden (study codes 551-09 and T-100-12) and was conducted in accordance with the Declaration of Helsinki (24). All participants were informed about the study both orally and in writing before giving their written informed consent. Transcripts were anonymized and the participants were ensured confidentiality.

Results

Item development

All items developed in this study are based on our previously performed focus group interviews (14). These resulted in six areas, 16 concepts and a draft set of 16 items. A map of the areas, concepts and items and how they connect to each other is shown in Figure 2.

Figure 2.

Map of areas, concepts and items. The map organizes the content of the interactive self-report system into six areas, 16 concepts and, subsequently, the final 17 items.

Content validity and usability evaluation

Summary of cognitive interviews

The respondents understood the majority of items and response options (11/16 = 69%), showing that they were familiar with the concepts related to hypertension presented in the set of items. Due to the space limitation in the mobile phone, the items were brief and direct. Problems were identified with five of the total of 16 items, three concerning the understanding of the question and two a response option. Furthermore, an action for change regarding an item response scale was initiated by the study team and was tested in the following interviews. All items were perceived as relevant by the majority of respondents, and the coverage was perceived to be good. One item (“Today’s pulse?”) was added, resulting in a total of 17 items. The mobile phone as mode of administration was perceived as easy to use, with little or no trouble connected to recording answers to items.

Actions taken or revisions of items

The cognitive interview process, including the analyses, was iterative and any changes or actions decided upon were evaluated in the next round of interviews. In general, all items were well understood, although some concepts were perceived as a bit vague regarding their meaning, or respondents explicitly asked about the meaning of the concept. For example, in relation to the item “How do you feel today?” respondents asked the interviewer: “Do you mean in general or in connection to my high blood pressure?”

An additional three items (item 1 on general well-being, item 6 on heart palpitations and item 14 on swollen ankles) showed a problem relating to understanding or interpretation. Heart palpitations were expressed by a participating patient: “I don’t quite know what real palpitation should feel like, but I’ve experienced that my heart has beaten harder than usual; I felt it in the rest of my body as well.”

The four interviews with the healthcare professionals confirmed this insecurity regarding understanding item 6 on heart palpitations, concurring that it might need better explanation. As a solution, we decided to create an information leaflet with clarifying explanations since no further explanations fitted on the screen. In the forthcoming rounds of cognitive interviews, the suggested explanations were presented along with the items they referred to. When the items along with the added clarifying information did not raise any new queries, no further actions were taken.

Actions taken or revisions of response alternatives.

In general, respondents found it “very easy” to find an appropriate response option. However, response options in relation to two items raised some queries. One concerned item 2 (“Taken your medicine today?”) and the relevance of one of the response options, i.e. “some of it”. Respondents wondered if it was relevant if they took just some of their medication. This resulted in a note in the information leaflet explaining why this response option was included. It was clarified that dose adjustments might be done, or that sometimes one of several medications might not be taken.

Another query related to item 9 concerning physical activity during the day and how to interpret what was meant by the different response options (none, light, moderate, heavy, very heavy). For example, one respondent wanted specification in min. This also resulted in an explanation in the information leaflet.

Two issues, concerning the response alternatives for items 2, 3–7, 10 and 14–16 on one hand and the response alternative for item 17 (final version) on the other, arose within the research team. The interviewer then explicitly asked for the respondents’ preferences in these cases and thereafter changed the response options after a discussion within the team.

Relevance and comprehensiveness of items.

The content of the items was experienced as relevant to the majority of respondents; thus all original item content was kept. The items concerning side-effects caused most hesitation regarding relevance, for example, “Swollen ankles today?” was experienced as irrelevant by nine respondents. Still, though, 12 respondents thought it was relevant and therefore it was kept. One additional item (pulse) was requested by four respondents and was thus added beginning with the 15th interview, after a discussion within the team.

Examples of the development from an initial to a final item, wording and response options, presented in a sample version of an ITM, are shown in Table III. The final set of items together with response options and explanations are shown in Table IV.

Table III.

Example of development of an item.

Item tracking matrix
Domain	Concept	Definition of concept	Item no	Original item	Original response options	Attribute to measure	Change or actions taken after first round of cognitive interviews	Rationale for change or action	Change or action taken after second round of cognitive interviews	Rationale for change or action	Patient quotations
Side-effects	Swelling of ankles	Swelling of ankles due to hypertension medication	14	Swollen ankles today?	Not at all Slightly Neither nor Severely Extremely	Degree	Explicitly ask what middle is preferred, “neither nor” or “moderately”, in the second group of interviews	The research team is not confident that the response option “neither nor” is to be preferred to “moderately”	Explanation added to the accompanying leaflet:“This item asks if your foot and/or ankle are swollen”Change response option “neither nor” to “moderately”	Respondents could relate to being swollen but how it manifests itself or how ankle was defined varied	“If my body is swollen”
										“Moderately” was preferred by patients	“Yes, they mean in the feet, in the ankles” “Swollen foot” “Swollen legs”
Change or action to take after third round of cognitive interviews			Final item			Final explanation of item			Final response option
None				Swollen ankles today?			“This item asks if your foot and/or ankle are swollen” added to the Information leaflet		Not at all Slightly Moderately Severely Extremely

Table IV.

Final items with correlating response optionsR1–6 and explanationsE1–5.

Item
1	2	3	4	5	6	7	8	9
How do you feel today?R1,E1	Taken your medicine today?R2,E2	Tired today?R3	Dizzy today?R3	Headache today?R4	Heart palpitations today?R4,E4	Restless today?R3	How was your sleep last night?R5	Physically active today?R6,E4
Item
10	11	12	13	14	15	16	17
Felt stressed today?R3	Systolic blood pressure today?	Diastolic blood pressure today?	Pulse today?	Swollen ankles today?R5,E5	Dry mouth today?R3	Dry cough today?R3	Passed water often today?R3
Response options version 1–6
1	2		3	4		5	6
Very well	No		Not at all	Not at all		Very good	None
Well	Some of it		Slightly	Slight		Quite good	Lightly
Moderately well	Yes		Moderately	Moderate		Moderate	Moderately
Not at all well			Severely	Severe		Quite bad	Heavily
Very bad			Extremely	Extreme		Very bad	Very heavily
Explanations to items version 1–5
1	2		3		4		5
State how you feel/have felt this very day	The alternative “Some of it” is used if you only took parts of your prescribed medication		Heart palpitation may feel as if the heart is beating harder or faster than usual		Answer with regard to what you think is physically active and do not compare with anyone else. The item is included in order for you to be aware of how much you activate yourself physically from one day to the next		This item asks if your foot and/or ankle are swollen

Usability

Mobile phones.

One respondent refused to answer the items in the mobile phone since he/she was interested in evaluating the content of the items but nothing else. Among the remaining 20 respondents who did complete the items in the mobile phone, 15 expressed it as easy or very easy, five as pretty easy and one as not so easy/difficult. Answering all items in the mobile phone took respondents between one and two min.

Blood pressure measurements

Measuring blood pressure was found to be easy or very easy by the majority of respondents (n = 17/20), testing it (n = 20/21) and recording the value in the mobile phone self-report system was found easy or very easy by all respondents testing it (n = 20/21).

Test–retest reliability yielded high correlations (r tt= 0.90) for questions 4, 7, 8 and 9 and satisfactory results (r tt= 0.70) for questions 3 and 6. Questions 2 and 10 were just below the threshold of 0.70 (r tt= 0.69).

Discussion

This study reported on the iterative development and evaluation of a hypertension-specific mobile phone self-report system with regard to item content and usability. Items and response formats were evaluated iteratively in a series of cognitive interviews in relation to their comprehension, comprehensiveness and relevance. Analyses showed that items were generally easily understood and represented a good coverage of core aspects relevant to patients’ needs for self-managing hypertension. Supplementary written information explaining a few potentially confusing items (due to their brevity) needed to be provided. Interviews and direct observations of patient–system interactions showed that patients could easily interact with the system.

Limitations of the study

The validity and reliability of our results were enhanced by conducting a larger number of cognitive interviews (25) among a demographically diverse and representative sample of the target population (19,21). For example, our sample was comparable with the middle-aged hypertensive population in the USA (20) and the general hypertensive population in Sweden (15), with respect to age and gender. The sample also included patients representing different education levels, employment and marital statuses. However, only one of the 21 interviewed patients did not have Swedish as their first language, hence further studies need to explore experiences and perceptions with regard to hypertension and treatment in the immigrant Swedish population

Methodological considerations

Cognitive interviews.

As shown in a previous study (26), misinterpretations may be attributional, i.e. the respondent understands the question but is unsure about whether or not it refers to the disease under study – and, consequently, whether or not to respond to the item in relation to their general status or their status as affected by their disease. For example, with regard to the concept of well-being, I might report “not at all well” if I feel bad because of something other than my disease or condition. This attributional problem was confirmed in our study, and shows the importance of thoroughly testing the understanding of items before moving on to psychometric testing (11,12).

Our approach to developing items suitable for both classic and smart phones further added to the space limitations cell phones have as a mode of administration. Hence, when we found an issue regarding the understanding of an item we chose not to change the item’s wording but instead to provide a brief explanation in the information material to be provided at the start of use of the self-report system. The reasons for this were, on the one hand, that we had to adapt our items to the space limitation in the mobile phone and, on the other, that respondents understood the actual words in the item but were unsure of the meaning in the particular context.

Misunderstandings may further be silent (27), i.e. they will not be discovered unless the interviewer probes in a pre-specified way, actively looking for misinterpretations. But this might also become a source of error, running the risk of spurious findings, at the same time as another source of error is missing valid findings (28). To try to avoid this we used pre-specified probing, but kept an open mind for emergent probes (21) when needed. We employed retrospective probing (21), in the sense that the respondents first answered all items in the mobile phone and were directly afterwards interviewed about the items. Hence, the problem that normally occurs in retrospective probing – that respondents need to provide information about responses given in the past (21,28) – was diminished.

Our study, like any other, could not disregard the sources of error when conducting cognitive interviews, but had to be of aware of them when building a structured, thorough cognitive interview design and when recruiting participants; this may have minimized the risk of error.

The usability of the mobile phone

When developing the items we had to adapt to the limits of the technology; in this case, the limitation of space. Although the technology would allow a great deal of space in a modern mobile phone, particularly a smart phone, this did not help us since we aimed to create a self-report system for use in all types of mobile phones. We further decided to use a Likert scale instead of VAS; thus items needed to be short enough for the whole Likert scale to be visible on the screen without the need to scroll, which had implications on our item development work:

Items had to be concise to fit on small displays. Short and direct items are generally easy to understand and leave little room for misinterpretation. However, if the respondent is unsure of the meaning of an item it may be hard to answer at all, since the text cannot be extended to give, for example, describing examples.

When an item was found to be poorly understood by respondents, we decided to keep the wording, and instead wrote explanations for the items. These were also tested together with the items in subsequent rounds of interviews. The explanations have been compiled in an information leaflet for future interventions. In a perfect scenario, such explanations would be included in the self-report system together with the actual item. However, we chose to be able to reach more people with our self-report system and thus had to accept this solution.

These types of issues might not be as problematic in the future, when more or most people will use smart phones, which will facilitate self-reporting performed via mobile phone. Furthermore, other eHealth applications will be developed, like health portals in web applications. In Sweden, the national eHealth strategy (29) is continuously updated and new approaches and innovations are being implemented. It is important to consider the development of our self-report in the light of the eHealth strategy and possible synergistic effects, and its potential usefulness in combination with, for example, a personal Health Record.

The developed interactive hypertension-specific self-report system will be used in future studies to evaluate the feedback loop of items via graphs to patients and healthcare professionals. We also aim to evaluate the effectiveness of the interactive mobile phone self-report system in clinical practice.

Conclusion

The included items were developed in a structured manner to ensure content validity, and the usability of the mobile phone as a mode of administration was established. The mobile phone self-report system is reliable, and appears efficiently and effectively to capture information relevant in patients’ self- management of hypertension.