Reliability of the Urinary Symptom Questionnaires for people with neurogenic bladder (USQNB) who void or use indwelling catheters.

STUDY
DESIGN: This is a descriptive psychometrics study.
OBJECTIVES: Neurogenic lower urinary tract dysfunction (NLUTD), also called Neurogenic Bladder (NB), is a common and disruptive condition in a variety of neurologic diagnoses. Our team developed patient-centered instruments, Urinary Symptom Questionnaires for people with neurogenic bladder (USQNB), specific to people with NLUTD who manage their bladders with intermittent catheterization (IC), indwelling catheters (IDC), or who void (V). This article reports evidence of reliability of the IDC and V instruments.
SETTING: Online surveys completed by individuals in the United States with NLUTD due to spinal cord injury (SCI), or multiple sclerosis (MS) who manage their bladder with IDC (SCI, n = 306), or by voiding (SCI, n = 103; MS, n = 383).
METHODS: Reliability estimates were based on endorsement of the items on the USQNB-IDC and USQNB-V. Reliability evidence was representativeness of these symptoms for a national sample (by determining if endorsement > 10%); internal consistency estimates (by Cronbach's alpha and item correlation coefficient, ICC); and interrelatedness of the items (by inferred Bayesian network, BN). We also tested whether a one-factor conceptualization of "urinary symptoms in NLUTD" was supportable for either instrument.
RESULTS: All items were endorsed by >20% of our samples. Urine quality symptoms tended to be the most commonly endorsed on both instruments. Cronbach's alpha and ICC estimates were high (>0.74), but not suggestive of redundancy. BNs showed interpretable associations among the items, and did not discover uninterpretable or unexpected associations. Neither instrument fit a one-factor model, as expected.
CONCLUSIONS: The USQNB-IDC and USQNB-V instruments show sufficient, multidimensional reliability for implementation and further study.

Introduction

Neurogenic lower urinary tract dysfunction (NLUTD, also referred to as neurogenic bladder, NB) is defined as “lower urinary tract dysfunction due to disturbance of the neurological control mechanism” and is a common and disruptive condition for individuals with spinal cord injury and disease (SCI/D) (including multiple sclerosis, MS, and spina bifida, SB).(1–3) While NLUTD or NB can result from any trauma or disease of the brain or spinal cord, people with SCI/D are highly affected: in the United States, 70–84% prevalence of NLUTD is estimated for those with SCI.(4) In Canada, up to 90% of SB patients are estimated to have “an active urological issue”, such as NLUTD.(5) NLUTD is associated with a disproportionately high risk of genitourinary complications, including urinary tract infections (UTIs(1–3,6–12)).

Although UTIs are the most common cause of emergency department visits and rehospitalization among people with NLUTD,(1,3,9) the most prevalent, bothersome, urinary symptoms as reported by patients with SCI/D and NLUTD in a recent US-based national sample of individuals who utilize intermittent catheterization for bladder management, cloudier- and more foul-smelling urine than normal(13), are excluded from discussions of clinical management for NLUTD(14), and are also typically excluded from UTI diagnostic criteria(15–18). Further, while clinical classifications effectively describe urinary symptoms that are consistent with the pathology and physiology of NLUTD (see e.g., Dorsher & McIntosh, 2012(3)), these do not incorporate or contemplate the patient’s experience of the disorder. Thus, urinary symptoms significantly affect consumer life and function, but instrument(s) specific to UTI (and related) symptoms in the context of NLUTD, that reflect the patient experience, have been lacking (see, e.g., Tsang et al. 2016; Patel et al. 2016; Clark & Welk 2016(19–21)).

To address this gap, and to promote and support research into new interventions, our research team has developed a set of Urinary Symptom Questionnaires for people with NLUTD (USQNBs) that are targeted for people with NLUTD specific to bladder management method (intermittent catheterization/no voiding (IC)(13); indwelling catheter (IDC); and voiding with/without some intermittent catheterization, i.e., primarily voiding to empty the bladder (not catheterizing more than once daily) (V)). Our focus has been on capturing and prioritizing the consumer’s perspective, to both represent, and facilitate self-management of, urinary signs and symptoms. The USQNB instruments are intended to strengthen UTI diagnosis for individuals with SCI/D, as well as supporting research and patient self-management with respect to urinary symptoms potentially attributable to UTI for NLUTD, depending on bladder management. By allowing clinicians and researchers to focus on symptoms specific to NLUTD that may be suggestive of UTI, we hope to enable better antimicrobial stewardship, as well as facilitating diagnosis, patient self-management, and research. We previously published our results on reliability and validity evidence for the USQNB for intermittent catheter users(13) and for general scoring considerations for the instruments.(22) In this paper we explore the psychometric reliability of consumers’ responses to the USQNBs for indwelling catheters (IDC) and voiders (V) from these specific populations, following the definitions and recommendations for reliability estimation and reporting of the international Consensus-based Standards for the selection of health Measurement Instruments (COSMIN) featuring reliability specifically.(23) The COSMIN criteria for reliability include estimates of internal consistency, intra-rater reliability, and measurement error. Since it is impossible to estimate measurement error for a patient’s experience, no such evidence was sought. We estimated internal consistency and intra-rater reliability as described in the Methods section (Psychometric Reliability).

Methods

All USQNBs were developed following our model for patient-centered patient reported outcomes(24), namely, we conducted focus groups and interviews with consumers who manage their bladders with the specific method (voiding; indwelling catheters, the latter inclusive of both urethral and suprapubic catheters). This paper describes the reliability evidence obtained from national samples responding to the resulting surveys, while a subsequent manuscript describes their respective validity evidence(25), following international consensus guidance on health related patient reported outcomes.(23) Approval for all parts of the studies was received from the MedStar National Rehabilitation Hospital Institutional Review Board (IDC: IRB# 2016-088; V: IRB# 2016-212).

Target Patient Population Participants and Recruitment

Participants in the groups these instruments were developed to assess (i.e., the target populations) were recruited in the United States by English-language advertising through Facebook, via email, and with the assistance of the national (U.S.) advocacy networks in SCI and in MS. All of these outreach and recruitment efforts were advertisements seeking respondents with NLUTD who use the specified management method to visit the URL we established for data collection. The target populations for these instruments are individuals with NLUTD and the specific bladder management; since SCI/D are major causes of NLUTD, our instrument development and these national samples focused on SCI/D -specifically traumatic SCI, and MS. No personal or identifying data were collected from any respondent, to protect anonymity, we required that participants complete the survey in one, single session. No identifiable information was collected. All responses remained anonymous, and data were transmitted from the SurveyMonkey form to the analyst anonymously (i.e., the data analyst did not have access to the original data file). We focused recruitment efforts on individuals with SCI/D who have NLUTD or neurogenic bladder, but apart from including questions within the survey about time since diagnosis (of SCI or MS) and whether they received a diagnosis of neurogenic bladder or NLUTD, or their experience with UTI and we did not confirm any of this self-reported information.

Participants followed the link to a SurveyMonkey page and chose their bladder management type. Respondents were specifically directed to consider their “normal” experience with each and endorse an item if it represented a change from “normal”. Each item (urinary symptom) was presented as a query about whether the respondent had experienced it during the past year (yes/no). Thus, each endorsement was intended to reflect a difference from usual, and the instructions reflect this difference. If an item was endorsed, there were three additional responses required: average frequency (0–365); average severity (usually not at all severe; usually somewhat severe; usually severe; always very severe); and average impact on, or importance in, daily life (rarely affects my actions or decisions to go about my daily life; sometimes affects my actions or decisions to go about my daily life; usually affects my actions or decisions to go about my daily life; always affects my actions or decisions to go about my daily life). Analyses reported here were focused solely on the endorsement (yes/no) of the items “for the past year”, although we intend for the instruments to be used in a much shorter time frame. This yearlong time frame was featured to ensure that the preliminary evidence we collected from our national sample on each instrument was as inclusive as possible. Basic psychometric reliability was focused on endorsement; frequency and the other aspects of the symptoms could not be meaningfully analyzed without this preliminary evidence of reliability.

Psychometric reliability

These analyses were planned to generate estimates of instrument reliability: specifically, minimization of measurement error; internal consistency or interrelatedness of the items; and maximization of variability that is due to “true” difference between levels of the symptoms across patients. Psychometric reliability is essentially the description of the consistency of an instrument (or rather, the scores it produces) across independent administrations of the instrument: either to the same person over time, or to different people at the same time.(26) However, there can be no diagnostic “gold standard” for what the urinary signs and symptoms represent, so “measurement error” cannot be effectively estimated. Instead, to approximate inter-rater reliability, we sought to understand how ‘representative’ the item pool, that was assembled following our model, is for the lived experience of NLUTD through our national sample. In a previous study, we defined a minimum endorsement rate of 10%(13), such that any item on our instrument that was endorsed by less than 10% of our target consumer group would be reconsidered for inclusion in the final instrument (suggesting it contributed low inter-rater reliability). With a national endorsement level below 10% within the previous year, an item might be only minimally representative of our targeted lived experience and as such, might lower the representativeness of that item for the target consumer group. Such items would be reviewed for potential duplication with other items, and also by clinicians and the investigative team to ensure that rare (infrequently endorsed) items are, in fact, consistent with the “construct on which the proposed …score interpretations are based”(26), namely, experience of urinary symptoms associated with NLUTD. Thus, observing a higher-than 10% endorsement rate for all items on the new instruments was included as a component of our reliability evidence. Our analyses focused on endorsement of the items on these two new instruments, because the impact, frequency, and severity ratings increase the difficulty of generalizing results, and are difficult to reconcile with the representativeness of these items for the patients’ lived experience.

Additional reliability evidence was sought by estimating internal consistency for the items on each of the two new instruments using Principal axis factoring (PAF) on tetrachoric correlations (between endorsements of each item, 0=no and 1=yes); Cronbach’s alpha; and inferred Bayesian Networks (BN) to uncover patterns of associations between signs and symptoms. Because the time period we asked about in our national samples was an entire year, internal consistency and interrelatedness are not likely to yield interpretable estimates. Instead we utilized these modeling methods to test whether a) a one-factor model would fit the items (i.e., “NLUTD causes these symptoms”); and b) expected patterns were observed in terms of symptoms that should logically associate, and also in those that logically should not associate.(27,28) In this manner, the BN particularly can contribute evidence on intra-rater reliability, because symptoms that are generally related (e.g., about urine quality) should be associated within-rater, while those that are not related (e.g., cloudy urine and leg pain) should not be associated within-rater. We did not expect a one-factor model to fit the data because items were derived from the patients’ perspectives, rather than a measurement model(29); and also because items were endorsed “in the previous year”, potentially compromising interpretability of latent factors influencing endorsement.

For the present report, our national samples can provide preliminary estimates of internal consistency of the items on each instrument (Cronbach’s alpha) but again, because of the time frame of our survey, we expected this to be an underestimate (lower bound) for consistency, and we plan to revisit this estimate in future with data from a shorter time interval in an independent sample.

Data analysis

Statistical analyses were carried out using SPSS v. 24(30) for descriptive statistics, ICC and Cronbach’s alpha computation, and to estimate the variance explained by the exploratory factor analysis; these analyses were based on the endorsement (binary, yes=1, no=0) of all items for the samples, by instrument. For each of the new instruments (USQNB-IDC; USQNB-V), exploratory factor analysis with tetrachoric correlations was carried out in MPlus v. 8 (Statsoft, Los Angeles, CA). Since our prior work demonstrated that an 8-factor model was the best fit to the national sample data on the Urinary Symptom Questionnaire for Neurogenic Bladder- Intermittent Catheter (USQNB-IC)(13), we fit models with 1–8 factors to the voiders and IDC data. Exploratory factor analysis was used to determine the quality of this range of factor models of the data (from 1–8 factors). The extraction method we used was principal axis factoring (PAF), to ensure an accurate estimation of the total variance that could be extracted from any solution, together with oblimin rotation, which permits correlated factors. This approach was used to extract from one to eight factors from the tetrachoric correlation matrix of endorsement (0= “no”/1=”yes”) of items, separately for each instrument. The fit of model(s) to the data was formally qualified using five different aspects of fit, assessed for each model, reflecting general data-model fit (Satorra-Bentler model chi square, x2 - lower is better), assessment of the fit of the model to data in hypothetical replications (Akaike’s Information Criterion, AIC –lower is better), incremental model fit relative to an independence model (comparative fit index, CFI – between .95 and 1.0 is desirable), error in approximation of the data by the model (root mean squared error of approximation, RMSEA- <0.08 is ideal), and the mean absolute value of the covariance residuals (standardized root mean square residual, SRMR - smaller and <0.09 is best). We chose the model with the best (lowest) AIC that also had acceptable other-fit indicators as “the best model” in each case. If any model had the best AIC but unacceptable other-fit indicators, it “did not fit” the data (i.e., was not an interpretable solution). Our objective with these exploratory factor analyses was to explore: a) whether the same solution held for all bladder management groups; and b) whether a one-factor solution was ever the “best” model. Variance explained by the best fitting model was also computed for each group.

BayesiaLab v. 6 (Bayesia S.A.S., Laval, France) was used for the Bayesian network modeling: unsupervised structural learning (searching for the overall data structure) using a Maximum Weight Spanning Tree (MWST) learning algorithm to constrain the structure so that only one “parent” per variable is identified. Minimum Description Length (MDL –information, not probability) was used for scoring because linearity in relationships cannot be safely assumed for these items. The resulting tree was visualized in automatic layout for interpretability (features discussed in Ch. 7 of Conrady & Jouffe, 2015).(31) The Bayesian network was inferred using unsupervised learning and the maximum spanning tree option, an approach that is similar to the employment of rotation in factor analysis solutions for “simple structure”, but has no model fit indices to characterize the results. The results are, therefore, associative and not specifically causal. We planned the Bayesian network analyses to identify whether each instrument yielded evidence of interpretable associations among the items, such as like items associating (vomiting-nausea), and did not generate uninterpretable or unexpected associations, such as unlike items associating (vomiting-leg pain).

These were done on all patient-respondents on each instrument separately; voiders were modeled separately by etiology (MS, SCI) because the sample sizes were large, whereas for those using indwelling catheters, there were so few with MS, all were analyzed together.

Results

Demographics

Table 1 describes the respondents in the national samples for the specific target groups the instruments were intended for.

Table 1.

Descriptive Statistics

	SCI with IDC	MS with IDC	SCI Voiders	MS Voiders
# of Respondents	306	8	103	405
Avg Age ± SD (range)	54.2 ± 13.2 (8–80)	49.4 ± 14.2 (21–65)	55.6 ± 13.8 (20–82)	52.8 ± 14.6 (10–80)
Avg time living with injury± SD (range)	19.9 ± 14.5 (1–64)	13.6 18.9 (1–50)	16.3 ± 16.9 (1–65)	14.9 ± 10.3 (0–53)
% of women	38.5	37.5	50.5	88.0

NOTES: SCI=spinal cord injury; MS=multiple sclerosis; IDC= using indwelling catheter;

AVG = average; SD=standard deviation.

We recruited for respondents for six months or until our minimum level of response (300 per group (IDC, V) was achieved), and found it difficult to find individuals with SCI who were also voiders; we had to close the survey at the 6-month time point and had only obtained 103 responses from SCI voiders. Similarly, we were only able to find 8 individuals with MS who used IDC for bladder management. Since the estimation procedures we planned to use all require substantial sample sizes, we grouped these 8 MS IDC respondents in with the SCI respondents.

Reliability evidence, endorsement rates.

Endorsement rates across items are given in Tables 2 (IDC) and 3 (V), and plotted in Figures 1A (IDC) and 1B (V).

Table 2.

Endorsement rates for IDC

USQNB-IDC Symptom	Endorsement Rate (%)
SCI+MS N=314
Bad-smelling, stronger of fouler urine	83.5%
Dark urine	85.4%
Blood in urine	50.3%
Blood clots in urine	35.0%
Cloudy urine	87.3%
Sediments/debris in urine/clogged catheter	78.3%
Bladder spasms	62.6%
Sensation of urinary urgency	34.8%
Urine leakage that isn’t normal	49.0%
Discharge, redness or sensitivity around catheter site	51.1%
Increase or decrease from your usual blood pressure	47.2%
Increased spasticity (not bladder spasms)	46.4%
Sweating	46.4%
Flushing of the chest, neck or face	31.2%
Diarrhea	51.9%
Numbness, tingling in your extremities	37.5%
Sense of burning in the extremities or throughout the body	27.8%
Autonomic Dysreflexia (AD)	46.0%
Felt feverish	42.7%
Fatigue/Lethargy	60.3%
Irritability, mental slowing/confusion	31.5%
Bladder discomfort	37.5%
Pain in the suprapubic region	16.3%
Generalized abdominal discomfort or pain: bloating, pressure or cramping	48.2%
Back pain	40.5%
Pain more than usual below the level of injury	25.1%

NOTES: USQNB-IDC: Urinary Symptom Questionnaire for Neurogenic Bladder or Neurogenic Lower Urinary Tract Dysfunction – for indwelling catheter; SCI=spinal cord injury; MS=multiple sclerosis were analyzed together.

Table 3.

Endorsement rates for Voiders with MS or SCI

USQNB-V Symptom	Endorsement Rate (%)
	SCI	MS
n	103	405
Cloudiness	72.8%	58.8%
Dark urine	72.5%	52.8%
Bad, foul or stronger smelling urine	72.3%	58.8%
Sediment discharge in urine	41.6%	15.1%
Blood in urine	34.7%	19.5%
Increased frequency of urination	69.3%	89.9%
Urinary urgency	55.4%	72.3%
Urge incontinence	72.3%	80.2%
Sensation of urgency/empty bladder	35.6%	38.8%
Small release of urine	69.3%	82.7%
Inability to fully empty bladder	62.4%	62.4%
Weak urine stream	50.5%	58.2%
Bladder spasms	49.5%	38.1%
Malaise, feeling unwell	74.3%	68.1%
Fatigue	83.2%	95.0%
Muscle weakness	60.4%	81.4%
Difficulty ambulating (walking)	48.5%	72.1%
Worsening of motor function	41.0%	46.8%
Increase in spasticity (not bladder spasms)	46.0%	46.8%
Chills	60.0%	44.3%
Sweating	57.0%	56.4%
Fever	43.0%	37.1%
Nausea	48.5%	43.1%
Vomiting	27.3%	21.3%
Difficulty thinking, mental fogginess, forgetfulness	51.5%	72.3%
Autonomic Dysreflexia **SCI ONLY**	43.9%
Multiple Sclerosis Flare-up (relapse, attack) **MS ONLY**		65.8%

NOTES: USQNB-V: Urinary Symptom Questionnaire for Neurogenic Bladder or Neurogenic Lower Urinary Tract Dysfunction – for voiding; SCI=spinal cord injury; MS=multiple sclerosis were analyzed separately.

FIGURES 1A (IDC) and 1B (Voiders)

NOTES: USQNB-IDC: Urinary Symptom Questionnaire for Neurogenic Bladder or Neurogenic Lower Urinary Tract Dysfunction – for indwelling catheter; USQNB-V: Urinary Symptom Questionnaire for Neurogenic Bladder or Neurogenic Lower Urinary Tract Dysfunction – for voiding.

SCI=spinal cord injury; MS=multiple sclerosis UTI= urinary tract infection.

Figure 1A: IDC group includes 306 individuals with SCI and 8 with MS, all presented together.

Figure 1B: Groups: SCI or D Voiders & UTIs= individuals with SCI and a history of UTI (i.e., the target group); MS Voider& UTIs= individuals with MS and a history of UTI (i.e., the target group).

Tables 2 and 3 show the symptoms on the two instruments. Each bladder-management-specific instrument contained items derived from focus groups, subject matter expert consultations, and the removal of duplicates following our previously published model24. There is some overlap in symptoms on the instruments (e.g., cloudy urine appears on both), but many differences. One important difference is that one item (autonomic dysreflexia) was identified by SCI voiders, and not by MS voider participants during instrument development, while another item was only relevant for the MS voiders (MS flare up). The endorsement rates shown in Tables 2 and 3 range from 16.3%, pain in the suprapubic region to 87.3%, cloudy urine for IDC and from 27.3%, vomiting to 83.2%, fatigue for SCI voiders, and 15.1%, sediment discharge to 95.5%, fatigue for MS voiders. Fatigue was the highest endorsed symptom on the voiders instrument by both groups (82% by SCI and 95% by MS). The endorsement rates by group are visualized in Figures 1A and 1B.

Figures 1A and 1B show that all items were endorsed at greater than the 10% cutoff we proposed; most items were endorsed by more than 30% of each sample. The items with the lowest endorsement rates for voiders with MS were sediment discharge in the urine (15%) and blood in urine (20%) -also among the lowest endorsed items by voiders with SCI; and vomiting (20% for MS, 27% for SCI). For the IDC (SCI and MS analyzed together because of the small MS sample), blood clots (35%) and blood in the urine (50%) were more common than for the voiders. The two least-commonly endorsed items for the IDC sample were pain related: pain in the suprapubic region (16.3%) and pain more than usual below the level of injury (25.1%); however, back pain was more common than these two pain items (40.5%), and two other items (numbness/tingling, 37.5% and sense of burning, 27.8%) reflecting sensations were also more commonly endorsed.

Table 4 summarizes the results for COSMIN criteria for reliability.

Table 4.

IDC and Voiders COSMIN Internal Reliability Evidence

COSMIN construct:	Group:
Reliability-internal:	IDC Whole Group (N=314, including 8 with MS)	VOIDERS: SCI (N=103)	VOIDERS: MS (N=405)
Principal Axis Factoring/oblimin	8f* would not converge. 7f best fit but 4f better than 5f, 5–7 f not better than 4f (p>.18).PAF 4 correlated factors explain 50.9% of the variance.	6–8 f would not converge. 4f had the best fit.PAF 4 correlated factors explain 47.6% of the variance	Neither 7f nor 8f converged. 6f best fit and better than 5f (p<0.01). Good fit on 4 of 5 indices.PAF 6 correlated factors explain 46.1% of the variance
Cronbach’s alpha	.784 (26 items) (based on 296 complete cases)	.851 (26 items) ‡ (based on 67 complete cases)	.785 (26 items) ‡ (based on 372 complete cases)
ICC†	ICC = .784 (95% CI .746–.818)	ICC = .851 (95% CI .792–.899)	ICC = .785 (95% CI .752–.815)

Notes: COSMIN= Consensus-based Standards for the selection of health Measurement Instruments.[23] All available data was used, no imputations were done. PAF= principal axis factoring.

f= factors; 8f=the model with 8 factors extracted. Oblimin= rotation in factor analysis where simple structure is favored in the model but factors in the solution are allowed to be correlated. Models with 1–8 factors (1f-8f) were tested.

ICC= item correlation coefficient; estimated from a two-way mixed effects (random rater, fixed measurement) model.

Etiology-specific items (MS flare up; Autonomic Dysreflexia) were excluded from these analyses.

Exploratory factor modeling

Principal axis factoring (PAF) with oblimin rotation (accommodating correlated factors) was applied to the tetrachoric correlation matrix of endorsement for each instrument separately, and for MS voiders separate from SCI voiders. The variance explained by any one factor solution is more accurate when PAF is used to extract factors, so the values reported in Table 4 (ranging from 46.1% for MS voiders) to 50.9% (for the full IDC sample) suggest that only about half of the variability in the responses can be explained by any model. Moreover, the best solution for MS voiders had 6 factors, while the 6 factor solution for SCI voiders data did not converge (possibly due to too small of a sample). The IDC solutions had a variety of fit issues depending on number of factors, with the best fitting model (7 factors) exhibiting better characteristics than any others, but not actually fitting significantly better than more parsimonious models (with 4–6 factors). Thus, as can be seen in Table 4, evidence for reliability based on the internal structure of these instruments was not consistent. We did observe that the one factor model fit in each sample very poorly (data not shown), with a minimum of four factors required to achieve even a marginally good fit for each instrument.

Inferred causal network

The Bayesian networks inferred from the endorsed items by all respondents on the USQNB-IDC (Figure 2A, SCI and MS together)), and by SCI (Figure 2B) and MS (Figure 2C) respondents to the USQNB-V, are represented in Figure 2 below.

Figure 2.

Inferred Bayesian Networks, IDC (2A) and Voiders (2B, 2C)

NOTES: USQNB-IDC: Urinary Symptom Questionnaire for Neurogenic Bladder or Neurogenic Lower Urinary Tract Dysfunction – for indwelling catheter; USQNB-V: Urinary Symptom Questionnaire for Neurogenic Bladder or Neurogenic Lower Urinary Tract Dysfunction – for voiding.

SCI=spinal cord injury; MS=multiple sclerosis UTI= urinary tract infection.

Figure 2A: IDC group includes 306 individuals with SCI and 8 with MS, all presented together.

Figure 2B: SCI or D Voiders & UTIs= individuals with SCI and a history of UTI (i.e., the target group);

Figure 2C: MS Voider & UTIs= individuals with MS and a history of UTI (i.e., the target group).

Arrows in the figures indicate whether there is an association between two variables given the modeling constraints (e.g., similar to “simple structure” in exploratory factor analysis, the modeling seeks to limit associations to one variable.) Attention is paid to whether any two variables that are shown to be associated (linked with an arrow) are plausibly associated (e.g., felt feverish-chills), and that variables that are not plausibly associated (e.g., felt feverish-leg pain) do not appear to have an association. No estimation is included in the figure because the exploration is meant to be qualitative, not quantitative.

The networks shown in Figure 2 highlight the complexity of the lived experience of urinary signs and symptoms in NLUTD for those managing their bladders with indwelling catheters (Figure 2A, mostly SCI, 8 MS) and by voiding (Figures 2B & 2C). As can be seen in Figures 2B (SCI) and 2C (MS), the inferred network of symptomatology over one year differs for MS and SCI patients who void; additionally, the instruments and networks include two etiology-specific symptoms (autonomic dysreflexia, only for SCI; and MS flare ups, only for MS). None of these structures show a single causal or originating factor for all or even some of the symptoms; and although symptoms tend to group together in interpretable ways, (e.g., weak stream and small release group together for MS voiders; muscle weakness and cloudy urine do not group together in any model), the overall associations among these symptoms are not clearly interpretable.

For the SCI subset of voiders respondents, USQNB-V items reflecting bladder function tend to associate together, but it is the urine changes that tend to associate with feverish and chills (more commonly associated with infection). By contrast, for the MS group, motor symptoms tend to associate more with bladder symptoms. Arrows in the figures indicate whether there is an association between two variables given the modeling constraints (e.g., similar to “simple structure” in exploratory factor analysis, the modeling seeks to limit associations to one variable.) Attention is paid to whether any two variables that are shown to be associated (linked with an arrow) are plausibly associated (e.g., felt feverish-chills), and that variables that are not plausibly associated (e.g., felt feverish-leg pain) do not appear to have an association. No estimation is included in the figure because the exploration is meant to be qualitative, not quantitative. While associations differ depending on etiology, the items that are associated within each group are interpretable, and differences between the networks are, as well. This suggests that the voider instrument, in particular, has the potential to respond to subtle differences in respondent characteristics without impairing its potential reliability for either type of respondent (i.e., one with SCI or MS).

Discussion

The purpose of this study was to describe the evidence for psychometric reliability of two new patient-centered patient reported outcome instruments for urinary signs and symptoms in people with NLUTD and who use either indwelling catheterization to manage their bladder (USQNB-IDC), or who void (USQNB-V). These instruments were all developed using explicitly patient-centric methods, so as to be specific to the lived experience of the NLUTD population, and so it is not possible to estimate the error with which these symptoms are assessed, which is a hallmark of the estimation of psychometric reliability. Instead, we sought to define one type of “error” as the inclusion of items on our instruments that fewer than 10% of our national samples endorsed. We observed that more than 10% of our national sample endorsed all items on their respective instrument, suggesting that the items on each instrument do represent recognizable urinary symptoms according to our national samples. This captures at least one level of “consistency” in terms of describing the experience of independent individuals with NLUTD who manage their bladders according to either instrument’s target group: all groups on both instruments generated sufficiently high levels of endorsement and internal consistency to support conclusions of reliability. Moreover, with acceptable levels of internal consistency(27) (all Cronbach’s alpha >.78) that did not suggest redundancy (i.e., upper bounds on 95% CIs for our internal consistency coefficients <.90), we compiled additional evidence of the reliability of these instruments, even though “error” cannot be meaningfully defined in this context because the only “true” or gold standard response comes from the respondent’s endorsement of the item.

The PAF results clearly show that the instruments do not fit a single factor model, but the variance explained by any of the multi-factor models was low, ranging from 46.1% for MS voiders) to 50.9% (for the full IDC sample). Thus, only roughly half of the variability in the endorsement responses can be explained by any model for either instrument, but none of the factor models fit well. This was expected given the origins of these items (i.e., from focus groups, and not with a specific measurement model in mind); and also because the survey asked about experience in the past year – just because a patient experienced two symptoms does not mean they were associated in time. This could be part of the reason for the inconsistent factor analysis results. We did observe that the one factor model fit in each sample very poorly, as was expected.

Like the PAF, the Bayesian network was based on the endorsement of all items for each instrument. This method was used to uncover basic structure but limited the associations that appeared to a single-neighbor (i.e., a Markov Blanket approach, similar to ‘simple structure’ in factor analysis). This method is not parametric, does not use or require inferences, and uses information instead of probability for estimating associations. Our PAF directly, and the Bayesian Network indirectly, support a conclusion that there is no single factor or dimension underlying all the urinary symptoms in these respondents with NLUTD. This evidence is not unexpected given that the development of the instrument did not formally follow measurement model-driven methods.(29) We observed a similar multi-dimensional result, with poor fit of the one-factor model, in our previously-published national sample for the intermittent catheter-specific instrument(13). Although we have discussed the use of symptom-type classifications, rather than total score summarization, of individual levels of symptom burden in the intermittent-catheter bladder management context(14,22), the overall ICC values for the instruments specific to voider and to indwelling catheter management are high. The PAF did not support a one-factor model, as expected, and the Bayesian networks suggest expected (structural) associations are present in the data, and unexpected ones are not.

The results suggest these new instruments have extensive and diverse evidence of their reliability. However, limitations to these analyses must be noted. Firstly, since this was a preliminary evidence gathering study, and we asked participants to endorse items they had experienced at least once in the previous year, the timeframe of the questions may partly underlie the failures to find structural coherence in the models. Further, because uncovering a measurement model structure (e.g., path weights, factor makeup) for either instrument was not our purpose in the studies, we did not estimate fit, factors, or strengths of associations between any of the items on either instrument -not with the exploratory factor analyses (Table 2) nor with the inferred networks (Figures 2A, 2B, 2C). Instead, the Bayesian Network represents inferred, and not estimated, structural relationships among symptoms as they are experienced (simple endorsement (yes/no)) over one year. The observation of expected associations (e.g., cloudy with foul-smelling urine for all models) and not unexpected ones (e.g., cloudy urine with nausea) within all three of the networks shown in Figure 2 is evidence that the USQNB-IDC and USQNB-V have sufficient structural interpretability for continuing the evaluation of their measurement properties. These methods were also applied to the previously-published instrument, for intermittent catheter users (USQNB-IC)(13) and these results are similarly supportive of a conclusion of reliability. Importantly, all three instruments were developed for the same “urinary signs and symptoms” independently (following the same method) with focus groups representing each bladder management method, leading to overlap (both instruments include the same items) or similarities (both items include similar items with wording differences). We are currently completing an analysis of the similarities and differences between the three instruments -and identification of redundancies – to clarify our understanding of urinary symptoms associated with NLUTD that vary according to bladder management method.

There are some similarities among items on all of the instruments, but crucial differences as well. We are in the process of examining the similarities and differences in the symptoms included on these instruments, in terms of how they may improve our understanding of NLUTD and patients’ experience of it, including an evaluation of concordance between patients’ reports of their symptoms and the International SCI Basic Data Set for lower urinary tract function.(32) The USQNB-IC has already been utilized in a clinical trial for treating bothersome symptoms associated with NLUTD, and we are hopeful that the two new instruments described here will be similarly useful for those with NLUTD who use other bladder management methods. Crucially, while a basic data set may support the uniform documentation of patient status across contexts and countries, the USQNB instruments are specifically intended to support decision making, so all items reflect something that has changed – i.e., something that might be targeted for change (improvement or prevention) with an intervention or management decision.

Of interest, the levels of endorsement for one item that appears on both of the instruments by individuals with SCI were quite different: fatigue was among the most commonly endorsed (83%) items among voiders with SCI, but was endorsed by roughly 60% among those with SCI who use IDC to manage their bladders – still one of the most highly endorsed items, but not as often as by voiders with SCI. This item was the most prevalent symptom for all voiders, endorsed by 95% of MS and 83% of SCI voiders, while cloudy urine (87.3%), darker urine (85.4%) and stronger smelling urine (83.5%) were the most prevalent for those using IDC for bladder management. We have reported previously that, among those using intermittent catheterization, urine quality symptoms are the most prevalent in our national sample(13), and while these tend to be excluded from UTI diagnostic consideration (see Tractenberg, et al.(14), Table 1), these are common and bothersome symptoms that can be treated. The reliability evidence presented here, together with validity evidence published elsewhere, will support the use of these instruments in further studies of interventions that can address these symptoms. We have also completed an analysis of the validity evidence(25) supporting these instruments, further demonstrating consistency with Mokkink criteria(23) beyond the reliability evidence discussed here. The clinical relevance of this investigation lies in the identification of patient-centered, patient-reported urinary symptoms, their prevalences among individuals with NLUTD who have experience with UTIs. Our results – the identification of so many items by individuals with NLUTD, and their documentation as reliable (this paper) and valid(25) - go beyond the symptoms identified by authoritative guidelines as indicative of UTI.(15,16) We previously compared the patient-centered, patient-reported urinary symptoms associated with NLUTD and intermittent catheterization(24), with other NLUTD-related UTI guidelines or recommendations, discovering that there is a disparity between the clinician-determined urinary symptoms indicative of UTI and those urinary symptoms reported by people with NLUTD and considered by them to be related to UTI. By having an instrument that allows for measurement of both patient- and clinician-determined symptoms that are potentially related to UTI, we can begin work to actually differentiate those symptoms which are definitely, probably, likely or unlikely related to NLUTD-related UTI. This can improve treatment and research into UTI and bothersome urinary symptoms(14), as well as promote antibiotic stewardship by focusing treatments where antibiotics would be most likely to be effective.

Conclusions

The results presented here are based on collection of preliminary evidence to allow the estimation of measurement properties that are of greatest interest(23), specifically, reliability. The results of our national samples suggest that the instruments have sufficient reliability to continue our work; we continue to develop scoring criteria(22) to complete the documentation of the instruments that describe the patient’s lived experience with urinary symptoms specific to NLUTD.