Comparative validity and reliability of the WeChat-based electronic and paper-and-pencil versions of the PISQ-12 for collecting participant-reported data in Chinese.

OBJECTIVE: The objective of this study is to assess the consistency between the WeChat-based Pelvic Organ Prolapse/Urinary Incontinence Sexual Questionnaire short form (PISQ-12) in Chinese and the paper version and to determine the test-retest reliability of the WeChat questionnaire.
METHODS: A total of 120 women aged between 24 and 69 years were recruited from the outpatient clinic at Peking Union Medical College Hospital and randomly assigned to two groups. All participants completed the WeChat and paper questionnaires twice. Group A completed the paper questionnaire before the WeChat version; Group B completed the WeChat questionnaire before the paper version. Two weeks later, all participants completed the questionnaires in the opposite order. Then, the reliability and validity of the two versions were assessed using Pearson correlation coefficients, intraclass correlation coefficients, and Bland-Altman graphs.
RESULTS: No significant difference in completion time was found between the two versions of the Chinese PISQ-12 (P = 0.67). Half of the participants (60/120) preferred the WeChat questionnaire, 15% (18/120) preferred the paper form (P < 0.01), and 35% had no preference (42/120). The response time was positively correlated with age (P < 0.01) and negatively correlated with the degree of education (P < 0.01). A Pearson correlation coefficient of 0.92 and an intraclass correlation coefficient of 0.94 indicated strong consistency between the two versions. The WeChat form exhibited strong test-retest reliability (Pearson correlation coefficient, 0.86; intraclass correlation coefficient, 0.86). The Bland-Altman plots supported these results.
CONCLUSIONS: The WeChat questionnaire was preferred over the paper version in a Chinese sample and had excellent consistency with the paper version and high test-retest reliability for collecting data on private topics.

Patient-reported outcomes (PROs) are frequently used to evaluate the effectiveness of treatments and changes in disease trajectory.[1] PROs can complement independent objective information obtained through the physical examination and clinician-reported outcomes. In fact, PROs are generally considered to be more meaningful outcome measures than results reported by clinicians.[2,3] Approximately 40% of women experience various types of sexual function problems throughout all stages of their lives.[4] Both the physical and psychological aspects of pelvic floor dysfunction (PFD) have direct effects on women's sexual function.[5] A number of effective questionnaires are available in Western countries to assess the sexual function of PFD patients. The most targeted questionnaire is the Pelvic Organ Prolapse/Urinary Incontinence Questionnaire (PISQ), with 31 questions, and the short form that appeared in 2003, namely, the PISQ-12.[6] The latter contains 12 items covering three aspects of patient health: physical, emotional, and partner-related.

Traditionally, paper questionnaires have been the main way to collect clinical data. However, the collection and collation of paper questionnaires consume a considerable amount of human resources, introduce the possibility of data omission and errors in the process of data collection, and involve difficulties with respect to proper preservation, which all restrict the clinical utility of paper questionnaires. Automated data collection and processing methods are becoming more extensively used in healthcare research, and professionally designed electronic data collection has the potential to gradually replace paper data collection in the field of health care because of its advantages.[7] In China, the most popular smartphone application is WeChat, and its diverse and simple operations are gradually changing the way people receive information. In the field of clinical research, the rapid growth of telemedicine also provides opportunities to use WeChat to conduct questionnaires and collect clinical information. We[8] validated the usability of a WeChat-based electronic questionnaire for collecting participant-reported data on female pelvic floor disorders that used the PFDI-7 questionnaire and concluded that the WeChat version of the questionnaire was more easily accepted by women with pelvic floor disease, with good data quality and reliability. However, the problem of female sexual function in China is a relatively private topic. Although the clinical utility of the Chinese version of the PISQ-12 has been verified and the instrument is widely used in the clinical diagnosis and treatment process,[9] there is a lack of relevant research on the validity of the WeChat PISQ-12 questionnaire, and no comparison with the original paper version has been performed. This may result in the failure of the promotion of the WeChat PISQ-12 questionnaire in the clinic due to the lack of equivalence comparison between the two formats.

The objective of this study is to explore the following: first, the consistency of the WeChat PISQ-12 questionnaire with the paper version; second, the test–retest reliability of the WeChat questionnaire; third, whether the time spent on the WeChat questionnaire was shorter than that spent on the paper version; fourth, whether participants preferred one of the two versions of the questionnaire; and fifth, whether the consistency decreased in older participants. Therefore, the purposes of this study are to verify the validity and reliability of the WeChat PISQ-12 by assessing the consistency between the WeChat PISQ-12 and the paper version and determining the test–retest reliability and validity of the WeChat questionnaire to provide a useful reference for clinical application and scientific research progress on private issues such as sexual life in China.

METHODS

Study design

Participants were recruited at the outpatient clinic of Peking Union Medical College Hospital from November 2017 to June 2019. They suffered from pelvic organ prolapse or stress urinary incontinence and came to the pelvic floor outpatient clinic seeking medical treatment. The study was approved by the Ethics Committee. Participants were required to be older than 18 years, have good reading and comprehension skills, and have been sexually active within the last 6 months. Only patients who had lived or worked in Beijing for a long time and could complete the two outpatient follow-up visits were enrolled. Before the study, the volunteers were informed of the purpose of the study and the requirements to fill out the questionnaire four times in 2 weeks and to not receive any treatment during the 2-week period. If a patient had doubts about the study or was still reluctant after the explanation or the follow-up after 2 weeks could not be completed, then she was not included in the study group. Participants were enrolled after providing consent and signing the informed consent form.

According to the suggestions of statistical experts, each question of the questionnaire requires at least five samples; the PISQ-12 questionnaire contains 12 questions, necessitating at least 60 samples for each group. Therefore, 120 people were required for this study, and the participants were automatically divided into two groups according to a randomization list. To eliminate any influence on the completion time of the two questionnaires, a random crossover design was used. Volunteers in Group A completed the WeChat questionnaire and then completed the paper questionnaire during their first outpatient visit. Volunteers in Group B completed the paper PISQ-12 and then completed the WeChat questionnaire during their first outpatient visit. All the volunteers were asked to return to fill out the two versions of the PISQ-12 in the opposite order after 2 weeks. The complete process is shown in Figure 1.

FIG. 1

The flow diagram. ta1: The time taken by group A to complete the electronic questionnaire at the first outpatient visit. ta2: the time taken by group A to complete the paper questionnaire at the first outpatient visit. ta3: the time taken by group A to complete the paper questionnaire at the second outpatient visit. ta4: the time taken by group A to complete the electronic questionnaire at the second outpatient visit. tb1: the time taken by group B to complete the paper questionnaire at the first outpatient visit. tb2: the time taken by group B to complete the electronic questionnaire at the first outpatient visit. tb3: the time taken by group B to complete the electronic questionnaire at the second outpatient visit. tb4: the time taken by group B to complete the paper questionnaire at the second outpatient visit.

Survey

The questionnaire was completed in an independent and comfortable space provided by the hospital. The content and order of the two questionnaires were exactly the same. As the paper questionnaire, the A4-printed PISQ-12 questionnaire was adopted, and the WeChat version was prepared by using a specialized information collection system. The participants scanned the WeChat QR code into the system and input basic information such as age, pregnancy and birth, height, weight, education level, occupation, and other basic information in response to the questions. The program required all questions be completed before submission. During their second outpatient visits, the participants scanned the QR code to log in and quickly complete the questions.

A research assistant was needed to explain the content and options of the PISQ-12 and how to use the WeChat questionnaire to the participants and to provide guidance if needed. Timing began when the participant began to read the first question of the paper questionnaire and ended when she ticked off the 12th question. The assistant recorded the time at the top of the questionnaire. The timing and results of the WeChat questionnaire were built into the program automatically. At the end of the first outpatient visit, as suggested by Bischoff-Ferrari et al,[10] all volunteers were asked “Which version do you prefer?” to determine participants’ preferences. Three answers to this question were possible: (a) “WeChat questionnaire,” (b) “paper,” and (c) “no difference.”

Statistical analysis

The completion time and the results of the paper questionnaire were manually input into Microsoft Excel 2016 (Redmond, WA), while the data required for the WeChat questionnaire were derived directly from the system database. After all the data were summarized, IBM SPSS Statistics v 22 (Armonk, NY), was used for statistical analysis, with a significance difference level of P < 0.05. Student t test and the chi-square test were used to identify significant differences in continuous variables and categorical variables, respectively. A multivariate linear regression was used to test whether the time spent on the first completion of the questionnaire was related to age, education, occupation, or other factors. To determine the consistency between the paper and WeChat questionnaire scores and the test–retest reliability of the WeChat questionnaire, the intraclass correlation coefficient (ICC) and 95% CIs of the mean values were used. An ICC over 0.75 indicates good reliability.[11] The Bland-Altman map drawn by MedCalc version 19.1 and Pearson correlation coefficients (PCCs) were also used to verify the consistency. In the Bland-Altman map, at least 95% of the difference should be within the 95% CI. PCCs over 0.8 were considered to have strong consistency.

RESULTS

A total of 120 patients with PFD aged between 24 and 69 years were enrolled in the study. All participants, except for one in group A, completed the two versions of the questionnaire twice. Therefore, missing cases were excluded when the two WeChat questionnaire scores were compared. The basic information of the participants in the two groups is shown in Table 1. No significant statistical differences in demographic characteristics such as age, BMI, education, parity, or occupation were noted between the two groups.

TABLE 1

Demographic information of the study participants

	Group A (n = 60)	Group B (n = 60)	P
Age (mean ± SD)	42.55 ± 11.09	44.35 ± 10.66	0.56
BMI (mean ± SD)	23.36 ± 3.20	24.44 ± 2.55	0.07
Education, %
≤High school	14, 23.3%	16, 26.7%	0.91
Bachelor's degree	35, 58.3%	34, 56.7%
≥Master's degree	11, 18.4%	10, 16.6%
Parity, %
0	1, 1.6%	0	0.50
1	40, 66.7%	43, 71.7%
2	19, 31.7%	16,26.7%
≥3	0	1, 1.6%
Job			0.84
Office worker	41, 68.3%	42, 70%
Manual worker	19, 31.7%	18, 30%

Group A: Volunteers completed the WeChat questionnaire and then completed the paper questionnaire during their first outpatient visit. Group B: Volunteers completed the paper PISQ-12 and then completed the WeChat questionnaire during their first outpatient visit. All the volunteers were asked to complete and return the two versions of the PISQ-12 in the opposite order after 2 weeks. Student's t test and the chi-square test were used to identify significant differences in continuous variables and categorical variables, respectively. Data are expressed as the mean ± standard deviation or % (n).

PISQ, Prolapse/Urinary Incontinence Sexual Questionnaire short form.

Fifty percent of the participants (60/120) preferred the e-questionnaire, 15% (18/120) preferred the paper form, and 35% (42/120) had no preference (Table 2). A significant statistical difference was found between the preference rate for the e-questionnaire and the paper questionnaire (P < 0.05), but no significant difference in preference was observed among participants aged over 60 years (P = 0.37). There was no significant difference in preference by age, education level, or occupation type.

TABLE 2

Participant preference for the questionnaire format

	Electronic	Paper	No preference	P
N, %	60, 50%	18, 25%	42, 35%	<0.01
Age (mean ± SD)	42.82 ± 10.52	44.00 ± 10.67	44.12 ± 11.62	0.56
Age≥60	6, 5%	2, 1.7%	5, 3.2%	0.37
Education				0.22
≤High school	11, 9.2%	5, 4.2%	14, 11.7%
≥Bachelor's degree	49, 40.8%	13, 10.8%	28, 23.3%
Job				0.60
Office worker	44, 36.7%	12, 10%	27, 22.5%
Manual worker	16, 13.3%	6, 5%	15, 12.5%

Student's t test and the chi-square test were used to identify significant differences in continuous variables and categorical variables, respectively. Data are expressed as the mean ± standard deviation or % (n).

After a detailed explanation of the tablet's operation, none of the participants required additional guidance during questionnaire completion. The average time participants took to complete the questionnaire the first time was 98.13 ± 41.88s. Participants in group A took 92.37 ± 42.93s to complete the WeChat PISQ-12 the first time (ta1), and 103.88 ± 40.34s on average was needed to finish the paper PISQ-12 the first time in group B (tb1). There was no significant difference between the two groups (P = 0.67). Participants took 79.02 ± 34.76s on average to finish the WeChat questionnaire for the first time (tb2) and 61.40 ± 31.47s on average to finish the paper version for the first time (tb2), with no significant difference between the groups (P = 0.18). A significant statistical difference was found between the first and second questionnaire completion times, which probably represents a “learning effect” of the questionnaire. The differences were 30.97 ± 27.07s (P < 0.05) (ta1−ta2) in Group A and 24.87 ± 27.22s (P < 0.05) in Group B (tb1−tb2). The average time taken to complete the WeChat questionnaire for all participants at the first outpatient visit (ta1 and tb2) was 85.69 ± 39.47s, and the average time needed to complete the paper questionnaire (ta2 and tb1) was 82.64 ± 41.87s. No significant difference was observed between the two groups (P = 0.39) (Table 3).

TABLE 3

Differences in the completion times for the WeChat and paper questionnaires at the first outpatient visit

Time, s	N	Electronic	Paper	P
ta1/tb1	60	92.37 ± 42.93	103.88 ± 40.34	0.67
tb2/ta2	60	79.02 ± 34.76	61.40 ± 31.47	0.18
ta1, tb2 / tb1, ta2	120	85.69 ± 39.47	82.64 ± 41.87	0.39

Student's t test was used for statistical significance. ta1: The time taken by group A to complete the WeChat-based questionnaire at the first outpatient visit. ta2: The time taken by group A to complete the paper questionnaire at the first outpatient visit. tb1: The time taken by group B to complete the paper questionnaire at the first outpatient visit. tb2: the time taken by group B to complete the WeChat-based questionnaire at the first outpatient visit. ta1/tb1: Comparison of the two groups’ completion times for the WeChat and paper questionnaires at the first outpatient visit. ta2/tb2: Comparison of the patients’ completion times for the WeChat and paper questionnaires without the influence of the “learning effect.”

ta1, tb2/ta2, tb1: Comparison of the two groups’ completion times for the WeChat and paper questionnaires at the first outpatient visit.

A multiple linear regression analysis was used to determine which factors affected how long it took volunteers to complete the questionnaire the first time. Education level was negatively correlated with answer time, and the higher the education level was, the shorter the answer time (P < 0.05, 95% CI −40.19 to −21.18). Age was positively correlated with completion time. Older age corresponded to a longer response time (P < 0.05, 95% CI 0.78-1.92). However, the correlation between occupation type and answer time was not statistically significant (P = 0.66) (Table 4).

TABLE 4

Multiple linear regression analysis of factors influencing the time spent completing the questionnaire the first time

	Unstandardized coefficients			95% CI of β
Model	β	SE	Significant	Lower limit	Upper limit
Education	−30.681	4.800	<0.01	−40.186	−21.176
Age	1.352	0.287	<0.01	0.783	1.920

β, regression coefficient.

The average scores from the first round were 31.80 ± 5.86 points for the WeChat PISQ-12 and 31.72 ± 5.86 points for the paper version. The correlation between the two was excellent and consistent (PCC 0.92; ICC 0.94; 95% CI 0.89-0.96). In the Bland-Altman map, 4.2% (5/120) of the points are outside the 95% distribution range, suggesting good consistency (Fig. 2). There were 13 volunteers over 60 years old, and there was still a strong correlation between the two questionnaires in this group (PCC 0.93; P < 0.05).

FIG. 2

Bland-Altman plots for PISQ-12 scores. Solid reference lines and dashed lines represent the mean difference between methods and the limits of agreement (mean difference ± 1.96 standard deviation), respectively.

The average score of the second WeChat questionnaire was 31.73 ± 5.94 points, which was strongly correlated with the first score, with good intragroup consistency (PCC 0.86; ICC 0.86; 95% CI 0.76-0.92). In the Bland-Altman map, 3.4% (4/119) of the points are outside the 95% distribution range, suggesting good consistency (Fig. 2). In volunteers over 60 years old, there was a strong correlation between the two WeChat questionnaires (PCC 0.93; P < 0.05).

DISCUSSION

This study compared the consistency and validity of the WeChat and paper versions of the PISQ-12 questionnaire in Chinese. The current study's key findings are as follows. First, the WeChat PISQ-12 questionnaire was highly consistent with the paper questionnaire, and second, the test–retest reliability of the WeChat questionnaire was strong. Third, there was no significant difference in completion time between the WeChat and the paper PISQ-12 questionnaires. Fourth, participants’ preference for the e-questionnaire was stronger than that for the paper questionnaire, and the difference was significant; fifth, no significant difference was found for the older respondents in terms of consistency and preference. From a clinical perspective, the WeChat PISQ-12 can be considered a substitute for the paper questionnaire.

Electronic technology greatly facilitates health assessment through excellent management, better adaptability, easier operation, and more realistic responses to sensitive issues.[12] A meta-analysis conducted in 2010[13] comparing the correlation coefficients between electronic and paper methods found that the average correlation coefficient between the two was 0.90, and 94% of coefficients were above 0.75. And an increasing number of participants prefer WeChat surveys.[12,14]

Haverman's study[15] showed that for sensitive topics, such as sexual behavior, electronic questionnaires are considered more secure than paper questionnaires and to some extent increase the participation of volunteers. Sexual activity-related topics have always been private, especially for Chinese women under the influence of traditional culture, and the participants were somewhat concerned about the safety of the questionnaire and whether information such as their phone numbers, ID numbers, and sexual history would be leaked. However, the background operation of the e-questionnaires on the hospital internal network offered far more security than careful handling of the paper questionnaires. The process of data input, transmission, storage, and output is specially encrypted, and only authorized doctors can decrypt and obtain patient information under the hospital internal network. Even if a hacker invades the system, only codes without any real information will be found.

A systematic review by Gwaltney et al[13] in 2008 showed that the consistency between WeChat and pen-and-paper methods was negatively correlated with the age of the respondents; however, this paper compared volunteers older than 60 years of age and obtained similar results as Caetano et al,[16] namely, no significant difference in preferences for WeChat or paper questionnaires was found. Perhaps after nearly 10 years of development, with the wide application of electronic equipment and the expansion of users, even the ability of older individuals to use electronic equipment has improved.

In this study, there was no significant difference in completion time between the WeChat questionnaire and paper questionnaire. In the design of the experiment, we considered whether some factors such as previous contact with the same questionnaire would affect patients’ initial response time. Therefore, we arranged two consecutive answers for different versions at one outpatient clinic. The results showed that though a “learning effect” may occur for patients during either the first or second time that a patient answered the questions, there is still no significant difference of finishing time between the two versions. Although the use of intelligent devices cannot significantly reduce the response time, such devices can reduce the time needed for collection,[17] like entering the answers and a double-check of a paper questionnaire, and prevent errors during the process of data collection and input.[18]

Due to the availability of web-based technology and its increasing reliability, the Internet is seen as a more convenient means of collecting data than paper questionnaires in terms of lower cost and simpler logistics.[19,20] Although the start-up cost is higher, the overall cost of establishing and running a WeChat data collection system is lower than that of a paper data collection system.[21,22] The Kenyan Ministry of Health[21] reported that the average cost of establishing a WeChat questionnaire collection system was 9.4% greater than that of a paper questionnaire collection, but the operating cost of WeChat data collection systems was 7% lower after 2 years.

WeChat, as one of the most popular smartphone applications in China, has shown positive effects for health promotion and promoting healthy lifestyles.[23-26] With full use of the WeChat platform and under the guidance of a professional team of gynaecologists, effective and meaningful discussions regarding sexual health issues can be carried out between doctors and patients. In addition, the WeChat questionnaire is convenient for clinical medical follow-ups,[27] especially in special circumstances such as the COVID-19 pandemic. Movsas et al[28] verified that electronic questionnaires increased patient compliance, improved follow-up surveys, reduced the rate of missed visits, and improved the response rate of patients.[29] If information security can be guaranteed, using a WeChat questionnaire for regular patient follow-ups may reduce workloads and improve the work efficiency of doctors. This advantage of using WeChat questionnaires may be more pronounced in subsequent larger-sample studies.

To date, we found that no studies have compared the validity of WeChat and paper versions of the PISQ-12 for evaluating the sexual life of women with pelvic floor dysfunction. In this study, we carried out a randomized crossover trial as suggested by the International Society for Pharmacoeconomics and Outcomes Research to validate the consistency of the paper questionnaires and e-questionnaires.[30] All the participants were required to complete the two versions of the PISQ-12 at one outpatient visit and to return approximately 2 weeks after the first visit, which substantially mitigated the impact of short-term memory on responses. We made considerable efforts to minimize the missing data rate to only 0.8%, thus guaranteeing the accuracy of the data. As mentioned in the study method, strict inclusion criteria were applied to the participants, and a physician was responsible for contacting patients in advance to schedule an appropriate time to complete the second follow-up. This study also had limitations. Although we consider that the number of women included in the study is adequate, and that the test effectiveness is sufficient, the sample size of 120 cases might not be sufficiently large for analyses with age stratification to obtain further information. In addition, the participants in this study were PFD patients from the pelvic floor outpatient clinic. If conditions permit, in future research, we would like to choose a suitable sexual function questionnaire, and the participants selected will be randomly chosen healthy women. Then, we can verify the superiority of the WeChat questionnaire in terms of private topics.

Potential clinical value

The results indicate that the WeChat-based electronic version of the PISQ-12 in Chinese was preferred to the paper version and had excellent consistency with the paper version and high test–retest reliability. It is recommended that this version be more widely used in clinical data collection of private topics.

CONCLUSION

Using the PISQ-12 to investigate the private issue of sexual life in women with pelvic floor dysfunction, we found that the WeChat questionnaire showed high consistency with the paper questionnaire and that the WeChat questionnaire showed extremely strong test–retest reliability. There was no significant difference in the completion time of the two questionnaires, but significantly more volunteers preferred the WeChat questionnaire. There was no obvious difference in preference for the two forms of the PISQ-12 between respondents of different ages, education levels, and occupation types. In the future era of big data, data collection systems should be sustainable, easy to use, and easy to extract data from and should ensure data security. The use of WeChat for information collection fulfills these requirements and should be recommended for wide use.