Readability assessment of commonly used urological questionnaires.

Purpose: This study was performed to assess readability of the most commonly used questionnaires in urology including a separate analysis of their single-items to identify questions that might be especially demanding for patients. Materials and
Methods: The guidelines of the European Association of Urology were screened for recommended questionnaires. Readability was analyzed for complete questionnaires as well as their single-items separately using well established readability assessment tools, including Flesch-Kincaid grade level (FKGL), Simple Measure of Gobbledygook grade level (SMOG), Coleman-Liau Index (CLI), Gunning-Fog Index, and the Flesch Reading Ease formula.
Results: A total of 13 questionnaires were included to the analysis. Calculation of grade levels (FKGL, SMOG, CLI, FGI) showed readability scores of 2.7th to 16.7th grade. Easiest readability as calculated by median grade levels was found for the short form of the International Consultation on Incontinence Questionnaires-Female Lower Urinary Tract Symptoms short form (FLUTS-SF) while the short form of the International Index of Erectile Function (IIEF-5) showed the hardest readability. Based on the FKGL between 0% (FLUTS-SF) and 80% (IIEF-5) of the single-items were written above the recommended grade levels. Conclusions: The questionnaires that are used most frequently in urology mainly show a satisfactory overall readability. Inadequate readability levels were not only found for individual questionnaires but also for single-items of the majority of assessed questionnaires. This requires consideration for the interpretation of results and when developing novel health-related surveys.

INTRODUCTION

Questionnaires represent an integral part in everyday medical practice and leading urological guidelines recommend their use in various sub-specialties of urology. They facilitate a time saving and structured assessment of current complaints and allow a systematic assessment of changes during a longer-term course. Many clinical trials are based on validated questionnaires as they represent the most reliable way to assess patient reported outcomes. Thus, a PubMed search reveals over 5,200 hits for the International Prostate Symptom Score (IPSS), over 3,300 for the International Index of Erectile Function (IIEF) and its short form IIEF-5 and more than 700 hits for the Chronic Prostatitis Symptom Index (CPSI).

Readability of health-related content represents a fundamental component of comprehensibility but is often neglected, which has especially been shown for patient education material [12]. Therefore, the USA National Institutes of Health (NIH) and the American Medical Association (AMA) recommend a 7th to 8th grade or 5th to 6th grade reading level for written health materials respectively [34].

Previously, Bergman et al. [5] analyzed the readability of 76 different health-related quality of life instruments and reported on a median 6.5th grade reading level. Like similar studies in different fields of medicine [67], they reported on generally satisfactory results. However, all of these studies evaluated the average readability of the complete questionnaires. This approach provides no information on potential variations in the readability of single-items. Thus, very difficult single-items can be covered up by easier ones and compromise a meaningful completion of a questionnaire and the informative value of its results [8].

The aim of our study was to assess readability of the most commonly used questionnaires in urology including a separate analysis of their single-items to identify questions that might be especially demanding for patients and, therefore, deserve particular attention when interpreting the results of such surveys.

MATERIALS AND METHODS

We reviewed the guidelines published by the European Association of Urology (EAU) for recommended urological questionnaires. In addition, the 36-Item Short Form Health Survey version 2 (SF-36v2) questionnaire [9] was included into the analysis as a benchmark. This survey represents the most widely used generic measure of health-related quality of life, has already undergone a revision to improve comprehensibility [10] and is also recommended by the EAU guidelines.

The English versions of the questionnaires were assessed by copying each single-item into a Microsoft Word document (Microsoft Corp., Redmond, WA, USA). Several items within the surveys (e.g., sentence fragments with subsequent question choices) are not recognized by the readability-test programs as a complete sentence or a question. Therefore, incomplete phrases were combined with potential question choices to form and test only complete sentences as recommended elsewhere [11]. Response options which were single words and therefore are likely to score as very easy to read, were not assessed. Copyright notices, disclaimers, acknowledgments, author information, citations, and references were excluded from the analysis. Each extracted text was analyzed for readability using the software package Readability Studio Professional Edition version 2015 for Mac (Oleander Software, Ltd., Vandalia, OH, USA). Readability assessment tools used for the analyses are described in Table 1 [212131415].

Table 1

Description of applied readability assessments (based on Kasabwala et al. [2])

Readability assessment tool	Characteristic	Analyzed parameter	Formula	Strength	Limitation
FKGL [12]	Used in the field of education. Results corresponding to USA grade level.	Average number of syllables per word and average number of words per sentence	0.39×average number of words per sentence+11.8×(average number of syllables per word)−15.59	Wide availability, extensively validated, quick and easy to administer, correlates highly with other readability formulas	Solely based on polysyllable words and long sentence, therefore may underestimate reading difficulty of medical jargon (short but unfamiliar words)
SMOG grade level [13]	Widely used for checking health messages. Results corresponding to USA grade level.	Number of words with 3 or more syllables	1.043×√ ([number of polysyllables]×[30/number of sentences])×3.1291	Considered as gold standard measure of readability for consumer-oriented healthcare material	Complex calculation; May underestimate reading difficulty of medical jargon (short but unfamiliar words); Only validated in English
Coleman-Liau Index [14]	Used for calibration of the readability of textbooks for the public school system (USA). Results corresponding to USA grade level.	Average number of letters and sentences per 100 words	0.0588×average number of letters per 100 words−0.296×average number of sentences per 100 words−15.8	Relies on characters instead of syllables per word, therefore more accurately counted by computer programs than are syllables	Designed samples of 100 words, not for entire documents; May underestimate reading difficulty of medical jargon (short but unfamiliar words)
Gunning-Fog Index [15]	Estimates the years of formal education a person needs to understand the text on the first reading. Commonly used to confirm that the intended audience can read the text easily.	Average number of words per sentence, average number of Gunning-Fog hard words (3 or more syllables)	0.4×([average number of words per sentence]+100×[average number of 3+syllable words])	Less time-consuming compared to FRE and FKGL if calculated manually	May underestimate reading difficulty of medical jargon (short but unfamiliar words)
	Results corresponding to USA grade level.
FRE [12]	Best suited for school textbooks and technical manuals. Possible value 0–100. Higher scores indicate material that is easier to read.	Average number of syllables per word and average number of words per sentence	206.835−(1.015×average number of words per sentence)−(84.6×average number of syllables per word)	Easily available through Microsoft Word Office package	Explanation points, colons and semicolons serve as sentence delimiters

FKGL, Flesch-Kincaid grade level; SMOG, Simple Measure of Gobbledygook grade level; FRE, Flesch Reading Ease.

Data from the article of Kasabwala et al. Otolaryngol Head Neck Surg 2012;147:466-71 [2].

Descriptive statistics using median and range were performed for all tests corresponding to a grade level (i.e., Flesch-Kincaid grade level [FKGL], Simple Measure of Gobbledygook grade level [SMOG], Coleman-Liau Index [CLI], Gunning-Fog Index [GFI]). The Flesch Reading Ease (FRE) formula is charted and results in a score between 0 and 100.

RESULTS

Totally 13 questionnaires were included to the analysis (Table 2) [916171819202122232425]. The median length of the questionnaires was 238 words with a range of 57 words (International Cystitis Symptom Index, ICSI) to 403 words (International Consultation on Incontinence Questionnaires-Male Lower Urinary Tract Symptoms long form, ICIQ-MLUTS-LF) for specific urological questionnaires. The SF-36v2 has a length of 546 words. Analysis of word complexity (3 or more syllables) of the questionnaires showed a range of 4% (ICIQ-Female Lower Urinary Tract Symptoms short form, ICIQ-FLUTS-SF) to 28% (IIEF-5). Analysis of word length showed 17.9% (IPSS) to 38% (Short form-Qualiveen, SF-Qualiveen) of words with six or more characters.

Table 2

Median grade-levels of the different surveys

Questionnaire	No. of item	FKGL	SMOG	CLI	GFI	Median (range)
ICIQ-FLUTS-SF [16]	24	2.7	6.4	3.9	4.9	4.4 (2.7–6.4)
ICIQ-FLUTS-LF [16]	33	2.9	6.9	4.1	5.4	4.75 (2.9–6.9)
ICIQ-MLUTS-LF [17]	43	2.9	7.1	4.5	5.6	5.05 (2.9–7.1)
ICIQ-MLUTS-SF [17]	26	3.3	7.0	4.8	5.7	5.25 (3.3–7.0)
I-QOL [18]	22	5.0	7.3	5.3	5.8	5.55 (5.0–7.3)
Qualiveen [19]	30	6.7	8.9	9.8	7.6	8.2 (6.7–9.8)
SF-Qualiveen [20]	8	7.3	8.8	10.8	8.1	8.45 (7.3–10.8)
SF-36v2 [9]	36	6.8	9.7	7.8	9.4	8.6 (6.8–9.7)
ICSI [21]	4	6.5	9.9	7.8	9.5	8.65 (6.5–9.9)
IPSS [22]	8	6.4	10.4	6.8	10.2	8.7 (6.4–10.4)
NIH-CPSI [23]	13	8.6	11.6	8.4	11.8	10.1 (8.4–11.8)
IIEF [24]	15	9.8	13.2	10.2	13.7	11.7 (9.8–13.7)
IIEF-5 [25]	5	11.2	14.8	13.1	16.7	13.95 (11.2–16.7)
Median	-	6.5	8.9	7.8	8.1	-
Range	-	2.7–11.2	6.4–14.8	3.9–13.1	4.9–16.7	-

FKGL, Flesch-Kincaid grade level; SMOG, Simple Measure of Gobbledygook grade level; CLI, Coleman-Liau Index; GFI, Gunning-Fog Index; ICIQ-FLUTS-SF, International Consultation on Incontinence Questionnaires (ICIQ)-Female Lower Urinary Tract Symptoms (FLUTS) short form; ICIQ-FLUTS-LF, ICIQ-FLUTS long form; ICIQ-MLUTS-LF, ICIQ-Male Lower Urinary Tract Symptoms (MLUTS) long form; ICIQ-MLUTS-SF, ICIQ-MLUTS short form; I-QOL, incontinence quality of life; SF-Qualiveen, Short form-Qualiveen; SF-36v2, SF-36 version 2; ICSI, International Cystitis Symptom Index; IPSS, International Prostate Symptom Score; NIH-CPSI, National Institutes of Health Chronic Prostatitis Symptom Index; IIEF, International Index of Erectile Function; IIEF-5, International Index of Erectile Function-short form.

Calculation of grade levels (i.e., FKGL, SMOG, CLI, GFI) showed readability scores of 2.7th to 16.7th grade (Table 2). Easiest readability as calculated by median (range) grade level of FKGL, SMOG, CLI, and GFI was found for the ICIQ-FLUTS-SF (4.4 [2.7–6.4]), while the IIEF-5 showed the hardest readability (13.95 [11.2–16.7]). These results were confirmed by the FRE, where scores of 90 points (corresponding to ‘very easy’ to ‘easy’ readability) were found for the short and long form of the ICIQ-FLUTS. With a score of 42 points (corresponding to ‘difficult’ readability) the IIEF-5 proved to be the most difficult questionnaire (Fig. 1) [12]. Using the SF-36v2 as a benchmark, seven of the urological questionnaires performed better, while five showed a harder readability.

Fig. 1

Comparison of Flesch Reading Ease-scores for the different questionnaires: International Consultation on Incontinence Questionnaires-Female Lower Urinary Tract Symptoms short and long form (ICIQ-FLUTS-SF and -LF), ICIQ-Male Lower Urinary Tract Symptoms long and short form (ICIQ-MLUTS-LF and -SF), 36-Item Short Form Health Survey version 2 (SF-36v2), International Prostate Symptom Score (IPSS), incontinence quality of life (I-QOL), International Cystitis Symptom Index (ICSI), Qualiveen, Chronic Prostatitis Symptom Index (CPSI), Short-Form Qualiveen (SF-Qualiveen), International Index of Erectile Function (IIEF), IIEF-short form (IIEF-5) (‘Plain English’ is recommended for health-related content [12]).

Analyses of single-items showed a wide range of readability scores for all of the different questionnaires. Fig. 2 shows the results of the three most commonly cited surveys according to a PubMed search (i.e., IPSS, IIEF, CPSI) using FKGL, SMOG, CLI, and GFI. Single-item assessments for the other questionnaires are presented in Fig. 3.

Fig. 2

Readability grade levels for single-items of the questionnaires. (A) International Prostate Symptom Score (IPSS), (B) International Index of Erectile Function (IIEF), and (C) Chronic Prostatitis Symptom Index (CPSI). FKGL, Flesch-Kincaid grade level; SMOG, Simple Measure of Gobbledygook grade level; CLI, Coleman-Liau Index; GFI, Gunning-Fog Index.

Fig. 3

Readability grade levels for single-items of the questionnaires. (A, B) International Consultation on Incontinence Questionnaires-Female Lower Urinary Tract Symptoms short and long form (ICIQ-FLUTS-SF and -LF), (C, D) ICIQ-Male Lower Urinary Tract Symptoms long and short form (ICIQ-MLUTS-LF and -SF), (E) incontinence quality of life (I-QOL), (F) Qualiveen, (G) SF-Qualiveen, (H) 36-Item Short Form Health Survey version 2 (SF-36v2), (I) International Cystitis Symptom Index (ICSI), (J) International Index of Erectile Function-short form (IIEF-5). FKGL, Flesch-Kincaid grade level; SMOG, Simple Measure of Gobbledygook grade level; CLI, Coleman-Liau Index; GFI, Gunning-Fog Index.

Assessing the single-items of all questionnaires by only using the FKGL, which has been used most frequently and is considered as the gold standard of calculation of reading levels [2627], readability scores still showed a wide range (Fig. 4). The widest range was found for the SF-36v2, incontinence quality of life (I-QOL), and Qualiveen-questionnaires. Based on the FKGL, a readability level above the recommended 8th grade level was found for the following numbers and proportions of single-items of the questionnaires: IIEF-5: 4 (80%), CPSI: 8 (61%), IIEF: 7 (47%), SF-36v2: 9 (25%), Qualiveen: 6 (20%), IPSS and SF-Qualiveen: 1 (13%), I-QOL: 2 (9%), MLUTS-SF: 2 (8%), FLUTS-LF: 2 (6%), MLUTS-LF: 1 (2%), ICSI and FLUTS-SF: 0%.

Fig. 4

Median grade reading levels and ranges of all single-items of frequently used urological questionnaires (based on the Flesch-Kincaid grade level). SF-36v2, 36-Item Short Form Health Survey version 2; CPSI, Chronic Prostatitis Symptom Index; IPSS, International Prostate Symptom Score; IIEF, International Index of Erectile Function; ICIQ-FLUTS-SF and -LF, International Consultation on Incontinence Questionnaires-Female Lower Urinary Tract Symptoms short and long form; ICIQ-MLUTS-LF and -SF, ICIQ-Male Lower Urinary Tract Symptoms long and short form; I-QOL, incontinence quality of life; ICSI, International Cystitis Symptom Index; SF-Qualiveen, Short-Form Qualiveen.

DISCUSSION

This study assessed the readability of questionnaires that are used in everyday urological practice and are recommended by leading guidelines.

A wide range of readability was found for the 13 questionnaires assessed, including results that clearly exceed the 7th to 8th grade or 5th to 6th grade reading level recommended by NIH or AMA respectively [34]. While questionnaires assessing male and FLUTS as well as incontinence symptoms (i.e., ICIQ-FLUTS-SF and -LF, ICIQ-MLUTS-SF and -LF, I-QOL, Qualiveen, SF-Qualiveen) are generally written very easy or easy, most of the other questionnaires (i.e., SF-36v2, IPSS, ICSI) did at least not clearly exceed recommended reading levels (Table 2). In contrast, the CPSI and the IIEF, including its short form, have a readability level that has to be considered as clearly too difficult according to all tests that were applied. These findings are largely consistent with assessments of questionnaires in other medical fields [56711].

As items exceeding patients' reading skills often induce them to give an invalid response or simply skip the item, readability analysis of questionnaires should not only report on mean results of the summarized text, but also assess each single-item separately [811]. Remarkably, even questionnaires with good average readability showed single-items clearly exceeding acceptable difficulty levels in this study (Figs. 3, 4).

Thus, based on the FKGL, the questionnaires IPSS, IIEF, and CPSI that are used most frequently in the literature have 13%, 47%, and 61% of question-items that are too difficult, respectively (Fig. 4). These numbers were even higher if other tests were applied (Fig. 2).

As reflected by the underlying formula of established assessment tools (Table 1), complexity of the sentence structure and the words used represent the main causes of poor readability. Koo and Yap [1] and Mossanen et al. [28] previously described that simplification of content is often accompanied by creation of longer sentences and words, and therefore, improved readability is not necessarily achieved. Dalziel et al. [29] could show, that the readability of patient education material could be improved by an average 3.1 grade level by simple substitutions of multisyllabic words. Further improvement has been shown to be achievable by adaption of sentence structure. Though such simplifications are not easily applicable to standardized and validated questionnaires, they might be considered if such questionnaires undergo revisions like it has been performed for the SF-36 (version 1) in the past [10] and more attention should be paid to readability if novel questionnaires are developed. Moreover, in the case of inconsistent or implausible results of questionnaires in clinical practice or trials, special attention should be paid to single-items that clearly exceed recommended readability levels (Figs. 2, 3).

The study has limitations that have to be addressed. As there is no consensus which readability formulas are most suitable for assessing questionnaires, we decided to use a combination of validated and well-established tests. Moreover, readability tests do not provide information about the complexity of the content and factors other than text quality (e.g., appealing layout, font types, images) also affect comprehensibility [30]. Only English versions of the questionnaires were assessed in the study and, therefore, the results are not transferable to the multitude of translations that are available.

CONCLUSIONS

The questionnaires that are used most frequently in urology mainly show a satisfactory overall readability. Inadequate readability levels were, however, not only found for individual questionnaires but also for single-items of all assessed questionnaires. This requires consideration for the interpretation of results and when developing novel health-related surveys.