Assessment of the quality of existing patient educational tools focused on sudden cardiac arrest: a systematic evaluation by the Sudden Cardiac Arrest Thought Leadership Alliance.

BACKGROUND: Conveying contemporary treatment options for those at risk of sudden cardiac arrest (SCA) is challenging. The purpose of the present research was to evaluate the quality and usability of available patient educational tools relevant to SCA and its treatment options, such as implantable cardioverter defibrillators (ICDs). We hypothesized that this review would identify gaps in areas of information for the enhancement of patient education and decision-making materials.
METHODS: We used a formal instrument to assess specific domains of content, development, and effectiveness of 18 available SCA and ICD educational tools. The multidisciplinary review panel included two electrophysiologists, two general cardiologists, a cardiac psychologist, a health services researcher, and a patient advocate.
RESULTS: Of the 18 education tools, four were rated as "good, may need revisions, but sufficient for use", 12 were rated as "marginal, needs revision prior to use", and two were rated as "poor, inadequate for use". None of the tools were rated as being of "very good" or "excellent" quality.
CONCLUSION: There appear to be opportunities to improve the quality and completeness of existing educational tools for patients with SCA and ICD. While many tools have been developed, they fall below current standards for supporting informed medical decision-making.

Introduction

Advancements in the treatment of sudden cardiac arrest (SCA) have been impressive.1 The ability of the implantable cardioverter defibrillator (ICD) to detect and treat potentially fatal arrhythmias demonstrates the power and potential of biomedical technology. However, ICD implantation requires regular device monitoring, avoidance of certain electronic or magnetic devices, and potential limitations on driving.2,3 A number of risks are also associated with ICD implantation, such as inappropriate shocks, a potentially negative impact on quality of life, and psychological distress, such as anxiety, depressed mood, hypervigilance, and avoidance.1

Ideally, the decision regarding whether an ICD implantation is “right” for an individual at risk for SCA should be one shared between the patient and their doctor after both parties have received adequate information on the risks and benefits of the available technology.4 To facilitate this process, many organizations and device companies have developed educational tools for those at high risk for SCA and potential ICD therapy (see Table 1).2,3,5–19

Table 1

Description of patient education tools related to sudden cardiac arrest

Developer and title of tool	Shortened title	Use	Web link	Sponsor
Cardiosmart: Arrhythmias5	CS-Arr	Arrhythmia and treatment education	http://www.cardiosmart.org/HeartDisease/CTT.aspx?id=232	American College of Cardiology
Cardiosmart: Sudden Cardiac Death6	CS-SCD	SCD Education and ICD advisement	http://www.cardiosmart.org/HeartDisease/CTT.aspx?id=724	American College of Cardiology
Boston Scientific: Your Heart Health7	Bos-Sci	Broad education on heart disease, risk, and treatment	http://www.bostonscientific.com/your-heart-health/index.html	Boston Scientific
IMPROVE HF Heart Failure Therapy: Devices for Heart Failure Rhythm Problems8	IHF-HFT	Education on devices	http://www.improvehf.com/	Medtronic
IMPROVE HF Living with an Implantable Device for Heart Rhythm Management8	IHF-ICD	Post-implant education	http://www.improvehf.com/	Medtronic
IMPROVE HF Sudden Cardiac Arrest and How to Protect Yourself8	IHF-SCA	ICD education for SCA	http://www.improvehf.com/	Medtronic
Heart Rhythm Society: Apples and Oranges9	HRS-A&O	SCA education	http://www.hrsonline.org/News/SCA-awareness/sca_research.cfm	Heart Rhythm Society
Heart Rhythm Society: Implantable Cardioverter Defibrillators (ICD)10	HRS-ICD	ICD education	http://www.hrsonline.org/PatientInfo/Treatments/ICD/index.cfm	Heart Rhythm Society
Heart Rhythm Society: ICD Frequently Asked Questions11	HRS-ICD-FAQ	ICD and SCA education	http://www.hrsonline.org/PatientInfo/Treatments/ICD/icd_faqs.cfm	Heart Rhythm Society
Heart Rhythm Society: Risk Assessment12	HRS-RA	Risk assessment and education for SCA and treatment	http://ceondemand.org/hrs/ram/splash.php	Heart Rhythm Society
Heart Rhythm Society: Treatment13	HRS-SCAT	SCA treatment options	http://www.hrsonline.org/PatientInfo/Treatments/index.cfm	Heart Rhythm Society
Medmovie.com: Electrophysiology Media Library14	HRS-Media	Broad cardiac educational library	http://www.medmovie.com/mmdatabase/MediaPlayer.aspx?ClientID=13&TopicID=886	Heart Rhythm Society
National Coalition for Women with Heart Disease: Support for Women15	Women-Heart	Includes women-specific education and support	http://www.womenheart.org/supportForWomen/livingHD/index.cfm	National Coalition for Women with Heart Disease
SCA Prevention Medical Advisory Team – SCA Prevention Pathways and Tools: What are Arrhythmias?16,18	SCA-Arr	Education on arrhythmias and treatments	http://links.lww.com/A1086	Medtronic
SCA Prevention Medical Advisory Team – SCA Prevention Pathways and Tools: Patient Discharge Contract16,19	SCA-PDC	Patient care instructions post HF diagnosis	http://links.lww.com/A1084	Medtronic
SCA Prevention Medical Advisory Team – SCA Prevention Pathways and Tools: Pre-ICD (Implantable Cardioverter Defibrillator) Placement2,16	SCA-Pre-ICD	SCD, AED, and ICD education	http://links.lww.com/A1086	Medtronic
SCA Prevention Medical Advisory Team – SCA Prevention Pathways and Tools: Post-ICD (Implantable Cardioverter Defibrillator) Placement3,16	SCA-Post-ICD	Post ICD implant education	http://links.lww.com/A1086	Medtronic
SCA Prevention Medical Advisory Team – SCA Prevention Pathways and Tools: Caring for a Person with Heart Failure16,17	SCA-Care	Heart failure education for patients and partners	http://links.lww.com/A1086	Medtronic

Abbreviations: AED, automated external defibrillator; HF, heart failure; ICD, implantable cardioverter defibrillator; SCA, sudden cardiac arrest; SCD, sudden cardiac death.

The primary aim of the Sudden Cardiac Arrest Thought Leadership Alliance (SCATLA), comprised of representatives of professional societies, academics, and patient advocacy groups, is to assist patients, families, and physicians, in making optimal treatment decisions for preventing SCA. The first step in realizing this goal is to understand the current status of patient education materials. Educational materials for patients undergoing cardiac surgery have been systematically evaluated.20 We are aware of only one study that evaluated ICD-related educational materials, and the primary objective of this study was to determine the readability of various print materials by a broad audience.21

Our specific aims were to evaluate the available SCA and ICD education material in terms of breadth and depth of content, empirical development of the tool, and the potential effectiveness of the tool in assisting patients in decision-making. To achieve this, a structured process was used to generate a template for evaluating the available tools. The goal of this evaluation was to identify gaps in the existing educational tools on SCA and ICD and propose means of enhancing educational tools related to SCA and ICD.

Materials and methods

Identification and evaluation of SCA and ICD education tools

The first phase of this project was identification of existing SCA patient education tools. We searched the Internet for patient education tools related to SCA and ICD therapy. Search terms included various combinations of the following: sudden cardiac arrest; implantable cardioverter defibrillator; treatment; education. We also reviewed the websites of major cardiology professional societies, patient advocacy groups, associations focused on SCA, and the major manufacturers of devices for SCA.

The second phase of the project involved implementation of an instrument to assess SCA-related education tools on the specific domains of content, development, and effectiveness.22,23 The instrument used in the current study included 16 questions derived from the 2005 International Patient Decision Aids Standards (IPDAS 2005) criteria for judging the quality of decision aids. The IPDAS criteria were developed from the work of the Cochrane Collaborative,22,23 where content validity was established by identification of content by a panel of decision aid experts. Using the IPDAS instrument, we were able to examine SCA-related materials using a validated instrument that reflects the current standard for patient education materials.

The items in the instrument reflect the importance of a comprehensive and patient-centered approach toward educating individuals at risk for SCA (see Table 2). We placed each item on a rating scale ranging from 1 (very poor) to 6 (excellent) with the addition of a choice for “not applicable”. These 16 items comprised subscales related to content (six questions), development (five questions), and effectiveness (five questions).

Table 2

Individual item evaluation

	Ratio of tools with relatively better ratings (≥3.5 rating)
Content items
1. Provide information about options, risks, benefits, financial impact, and alternatives in sufficient detail for decision-making?	13/18
2. Present probability of outcomes in an unbiased and understandable way?	7/18
3. Include information that reflects expectations for quality of daily life and long-term functional capacities?	15/18
4. Include methods for clarifying and expressing patient values?	13/18
5. Include attention to the effects of therapy on personal and family functioning?	6/18
6. Include structured guidance in deliberation and communication?	0/18
Development items
7. Present information in a balanced manner?	15/18
8. Has a systematic development process?	5/18
9. Use current scientific evidence that is cited in a reference section of the technical document?	7/18
10. Disclose conflicts of interest?	3/18*
11. Use plain, accurate, culturally appropriate, and standardized language?	3/18
Effectiveness items
12. Recognize a decision needs to be made and in what timeframe?	2/18
13. Know action steps, options, and their features?	8/18
14. Understand that values affect decision?	6/18
15. Discuss values with their practitioners?	6/18
16. Improve the match between the chosen option and the features that matter most to the informed patient?	2/18

Note:

All tools may not have had conflicts of interest to disclose.

In the third phase of the project, each educational tool was assessed by a panel of reviewers with a wide variety of expertise, including two electrophysiologists (SMA, KLT), two general cardiologists (GF, DH), a cardiac psychologist (SS), a health services researcher (GS), and a patient advocate with a master’s degree in public health (SC). These individuals were chosen to participate because of their experience in the treatment of patients with SCA, use of decision tools, and/or their nationally recognized expertise. Each of the reviewers responded to all the questions in the instrument for each of the educational tools. Reviewers could also make qualitative comments or suggestions regarding each tool. Tools were assessed on the basis of their individual qualities, even if they were part of a larger group of educational instruments.

Data analysis

Mean scores were calculated for each educational tool by total scores, subscale scores (ie, content, development, and effectiveness), and all individual item scores (total of 16). The total scores were divided by 16 (number of items in the full measure), and the subscales were divided by the number of items in each respective subscale. Items that were scored as “not applicable” by a particular reviewer were replaced using the average score of the other expert raters on the same item. Use of mean substitution with the other raters’ scores was considered the most conservative method because having a missing or zero score could negatively affect the rating of a particular tool. The maximum possible score for each question was 6. The following prespecified cut points were used for each question to interpret the results:

means from 1 to 1.4 indicate very poor quality and inadequate for use

means from 1.5 to 2.4 indicate poor quality and inadequate for use

means from 2.5 to 3.4 indicate marginal quality and need for revision prior to use

means from 3.5 to 4.4 indicate good quality and may need revisions but sufficient for use

means from 4.5 to 5.4 indicate very good quality

means above 5.5 indicate excellent quality.

In order to evaluate relative areas of strength and weakness among all 18 tools, we also examined how highly each of the 16 individual items was scored in each of the 18 tools. To do this, average ratings of ≥3.5 on each item were considered to have received relatively strong scores. Any items with a mean rating of <3.5 were considered to have received relatively poor ratings. A prespecified cutoff of 3.5 was chosen because this is the halfway point on the 1–6 rating scale. Next, the number of tools with relatively strong scores (≥3.5 average rating) on each item the number of tools with relatively weak scores (≤3.5 average rating) were tallied. The final statistic listed the number of tools out of the 18 that had relatively strong or relatively weak scores.

Results

Our search conducted from December 2009 through March 2011 yielded the 18 tools listed in Table 1. We included all the instruments found regardless of the source or who developed them. Of the tools included, two were developed by the American College of Cardiology,3,4 one by Boston Scientific,5 three by the IMPROVE Heart Failure Research Group,6 six by the Heart Rhythm Society,7–12 one by the National Coalition for Women with Heart Disease,13 and five by the SCA Prevention Medical Advisory Team.14–19

Inter-rater reliability (Kendall’s coefficient of concordance) was significant between the seven raters [W = 0.45, χ2 (286, n = 7) =898.92, P < 0.001]. In terms of total score on the 18 education tools, four were rated as “good, may need revisions, but sufficient for use” and 12 were rated as “marginal, needs revision prior to use and two were rated poor, inadequate for use”. None of the tools were rated as having “very good” or “excellent” quality (see Table 2).

On the content subscale, five of the educational tools were rated as “good, may need revisions, but sufficient for use”, 12 were rated as “marginal, needs revision prior to use”, and two were rated as “poor, inadequate for use”. On the development subscale, eight of the education tools were rated as “good, may need revisions, but sufficient for use”, seven were rated as “marginal, needs revision prior to use”, and three were rated as “poor, inadequate for use”. On effectiveness, five educational tools were rated as “good, may need revisions, but sufficient for use”, eight were rated as “marginal, needs revision prior to use”, and four were rated as “poor, inadequate for use”. None were rated as having “very good” or “excellent” quality. These results are presented in Table 3. In each of the areas of content, development, and effectiveness, none of the tools were rated as having “very good” or “excellent” quality.

Table 3

Eighteen patient education tools related to sudden cardiac arrest with subscale and total scores

Educational tool	Content	Development	Effectiveness	Total
HRS-A&O	3.55*	3.71*	2.34	3.22
SCA-Care	2.20	2.37	1.74	2.11
HRS-RA	2.13	2.19	2.76	2.35
HRS-Media	2.96	3.14	3.43	3.16
HRS-ICD	3.11	3.49	3.11	3.23
HRS-ICD-FAQ	2.90	3.26	3.71*	3.26
HRS-SCAT	3.71*	3.71*	3.29	3.58*
Women-Heart	3.39	3.17	3.17	3.25
IHF-HFT	3.37	3.56*	3.55*	3.48
IHF-ICD	3.37	4.04*	3.64*	3.67*
SCA-PDC	3.66*	3.71*	3.74*	3.70*
SCA-Arr	3.98*	3.87*	3.67*	3.85*
SCA-Post-ICD	3.96*	3.55*	2.77	3.46
SCA-Pre-ICD	2.84	2.83	2.40	2.70
Bos-Sci	2.66	2.49	2.35	2.51
CS-Arr	3.02	2.75	2.87	2.89
CS-SCD	3.44	3.02	2.31	2.96
IHF-SCA	3.13	3.73*	2.80	3.21

Notes: All total and subscale scores were converted to a 1–6 rating scale (by dividing the total by the number of items) to allow for interpretation on the quality rating index (ie, very poor to excellent).

Indicates scores in the “good” range and higher.

Abbreviations: A&O, Apples and Oranges; SCA-Care, SCA Prevention Medical Advisory Team- SCA Prevention Pathways and Tools: Caring for a Person with Heart Failure; RA, Risk Assessment; HRS-Media, Medmovie.com: Electrophysiology Media Library; ICD, Implantable Cardioverter Defibrillators; FAQ, Frequently Asked Questions; HRS, Hearth Rhythm Society; SCAT, Treatment; Women-Heart, National Coalition for Women with Heart Disease: Support for Women; IHF-HFT, IMPROVE HF Heart Failure Therapy; IHF-ICD, IMPROVE HF Living with an Implantable Device for Heart Rhythm Management; SCA-PDC, SCA Prevention Medical Advisory Team-SCA Prevention Pathways and Tools: Patient Discharge Contract; SCA-Arr, Prevention Medical Advisory Team- SCA Prevention Pathways and Tools: What are Arrhythmias?; SCA-Post-ICD, SCA Prevention Medical Advisory Team- SCA Prevention Pathways and Tools: Post-ICD Placement; SCA-Pre-ICD, SCA Prevention Medical Advisory Team- SCA Prevention Pathways and Tools: Pre-ICD Placement; Bas-Sci, Boston Scientific: Your Heart Health; CS-Arr, Cardiosmart: Arrhythmias; CS-SCD, Cardiosmart: Sudden Cardiac Death; IHF-SCA, IMPROVE HF Sudden Cardiac Arrest and How to Protect Yourself

Each particular item in the 18 tools was examined individually to determine how well it was addressed. Of the items on the content subscale, those that were addressed well included: provides information about options, risks, benefits, financial impact, and alternatives in sufficient detail for decision-making; includes information that reflects expectations for quality of daily life and long-term functional capacities; and includes methods for clarifying and expressing patients’ values. Items that were addressed poorly or not included at all: includes attention to the effects of therapy on personal and family functioning; and presents probability of outcomes in an unbiased and understandable way.

The development subscale item best addressed by the tools was: presents information in a balanced manner. The other items on the development subscale scored relatively poorly and included the following: has a systematic development process; uses current scientific evidence that is cited in the reference section of the technical document; discloses conflicts of interest; and uses plain, accurate, culturally appropriate, and standardized language.

With regard to individual item ratings on the effectiveness subscale, none of the items were well addressed by most of the tools. The tools generally scored poorly on their ratings to provide information effectively so that a consumer could do the following: recognize that a timely decision needs to be made; be familiar with the action steps, options, and their features; understand that values affect decision-making; discuss values with their practitioners; and improve the match between the chosen option and the features that matter most to the informed patient. The results of the individual item analysis are presented in Table 2.

The two educational tools that scored highest in our review overall were the SCA-PDC tool and the SCA-Arr tool. The SCA-PDC tool scored a mean of 3.66 on the content subscale, 3.71 on the development subscale, and 3.74 on the effectiveness subscale, and had a 3.70 total scale score. This measure was rated in the “very good” range for its ability to “present information in a balanced manner”. However, this measure also scored in the “marginal quality and in need of revision prior to use” range on the following items: provides information about options, risks, benefits, financial impact, and alternatives in sufficient detail for decision-making; includes structured guidance in deliberation and communication; discloses conflicts of interest; uses plain, accurate, culturally appropriate, and standardized language; and recognizes a decision needs to be made and with a certain timetable.

The SCA-Arr tool scored a mean of 3.98 on the content subscale, 3.87 on the development subscale, and 3.67 on the effectiveness subscale, and had a 3.85 total scale score. The tool scored in the “very good” range on the following items: presents information in a balanced manner; uses current scientific evidence with citations listed; includes methods for clarifying and expressing patient values; and includes attention to the effects of therapy on personal and family functioning. However, this measure also scored in the “marginal quality and in need of revision prior to use” range on the following items: includes structured guidance in deliberation and communication; uses plain, accurate, culturally appropriate, and standardized language; recognizes a decision needs to be made and with a certain timeframe; and improves the match between the chosen option and the features that matter most to the informed patient.

Discussion

SCA is the most common mode of death in the United States, claiming the lives of more than 300,000 people each year.24 To reduce the morbidity and mortality associated with SCA, it is critically important to educate patients about SCA and the therapies that have been proven to reduce risk. While many patient educational tools have been developed, they vary in form and quality. To our knowledge, this is the first study to evaluate areas of development, content, and effectiveness for existing patient education tools on SCA in a systematic way. Our review found that none of the existing tools met our predefined optimal criteria in terms of content, development, and effectiveness domains. However, four tools had total scores in the “good, may need revisions, but sufficient for use” range (ie, HRS-SCAT, IHF-ICD, SCA-PDC, and SCA-Arr). Only two of these tools were considered “good, may need revisions, but sufficient for use” (ie, SCA-PDC and SCA-Arr) in all three of the major domains of content, development, and effectiveness. None of the tools were considered to be very good or excellent in quality. Also, none of the education tools scored in the good quality range or higher on all of the 16 individual survey items, suggesting at least some need for revision in every tool assessed.

The purpose of this research was to identify gaps in the current education tools that could be addressed to enhance the value and potential effectiveness of such tools. We explored each of the individual items and identified some which were consistently rated highly among a majority of the tools; however, many of the items in the tools consistently received poor ratings. Lack of content information related to the probabilities of outcomes in unbiased and understandable ways, limited attention paid to the effects of therapy on personal and family functioning, and lack of structured guidance on deliberation and communication were the items most cited.

While the majority of tools appeared to present information in a balanced manner, most were lacking in systematic development, citation of current scientific evidence used by the tool, disclosure of relevant conflicts of interest, and use of suitable language. The effectiveness of the tools appeared to be weak for all items assessed. The tools generally scored poorly on helping patients to recognize that a timely decision needs to be made, knowing action steps and options, understanding that values affect decisions, understanding the importance of discussing values with their health practitioners, and choosing an option that matters most to the informed patient. Future tools should focus on developing materials that address all the identified deficits.

Limitations

There are a number of limitations to this review process to consider. While the overall quality rating of the instruments examined in this study may be somewhat discouraging, evaluation of these tools was completed using criteria that were not available to the creators. In addition, the tools were assessed on their individual strengths and weaknesses, although it may have been a more appropriate strategy to assess each set of tools by the individual device units created by each developer or organization. The feasibility of analyzing sets of tools was limited by the fact that the instruments often repeated information, were developed as separate educational units, and the length of the combined material would have made the analysis more complex, potentially limiting each reviewer’s ability to make judgments. In other words, the broad range of information would have created a large amount of information that would have been difficult to assess together. Although we used an existing method for assessing the quality of these educational tools, we also selected the evaluating criteria. We had limited reviewers who were not blinded to the makers of the tools, although there was a high degree of inter-rater reliability. The rating system was subjective, but was prospectively defined. Not including information available from some websites, such as those from various cardiology groups, was another limitation. There may be excellent tools that exist which we did not assess. Further, we did not assess the reading level of the educational material, in that our reviewers rated the tools based on the criteria: “does the tool use plain, accurate, culturally appropriate, and standardized language?” While this “language” criterion was drawn from the IPDAS 2005 criteria for judging the quality of decision aids,23,24 making the desired reading level known to the reviewers may have improved their assessment of these criteria. Despite these potential limitations, our goal of identifying improvements needed in future SCA and ICD educational tools was achieved.

Future directions

Future work will focus on development of tools that provide educational information to patients and their loved ones to promote learning and guide decision-making in a patient-centered framework. The focus will be on understanding heart disease, the risk of SCA, and treatment options, a second focus will be on understanding heart failure, and a third focus will be on caring for a person with heart disease. These educational tools to address the deficits identified this study. Our tools provide information on content and place, and emphasis on helping patients make informed decisions about their treatment. Realizing that major domains related to content were generally well represented among the tools, efforts were made to ensure that the most relevant information was provided in the most systematic manner. We also focused on patient decision-making by providing as much directive information as possible and described treatments concisely. These tools will be tested for their effect on patient outcomes, including quality of life, satisfaction with decisions made, and any ensuing psychological distress.

Conclusion

While the tools in this study could be considered well designed by previous standards, none of the existing SCA-related educational tools evaluated in our study met all the criteria for an empirically developed, content-appropriate, and effective educational tool. The primary deficit that reviewers noted was the potential effectiveness of the tools. Specific gaps were identified in: structured guidance in deliberation and communication; use of plain, accurate, culturally appropriate, and standardized language; helping patients recognize that a decision needs to be made in a certain timeframe; and helping patients by improving the match between the chosen option and the features that matter most to the informed patient. Using this information, the SCATLA has developed three education tools that will be pilot-tested and later disseminated to the public to enhance patient understanding of SCA and its therapies and to facilitate informed decision-making.