Utility of WhatsApp in emergency urological practice: An interrater reliability study.

INTRODUCTION: The messaging application 'WhatsApp' is used in clinical practice, often for communication between a medical trainee and a consultant. We designed this study to find the interrater reliability of the data transmitted through this application and validating its use in urological practice.
MATERIALS AND METHODS: Clinical details and computerized tomographic (CT) images of 30 patients visiting the urology emergency were posted in a closed WhatsApp group involving three consultants (SKD, APS, and KC). The CT images were posted in the WhatsApp group as Whole Image (WI) and Image of Interest (IOI) format and rated on a scale of 1-5. The consultants formulated a provisional diagnosis and initial management strategy. The interrater reliability of these responses was analyzed in the study.
RESULTS: Mean WI rating ranged from 3.03 ± 0.61 to 3.73 ± 0.64 (Cronbach alfa [α]-0.494, P = 0.006). Mean IOI rating ranged from 3.4 ± 0.56 to 4.13 ± 0.73 (α-0.824, P < 0.0001). For diagnosis, the proportion of observed agreement (P0) was 83.3% for SKD and APS, 76.6% for SKD and KC, and 73.3% for APS and KC. For management, P0 was 86.6% for APS and KC, 86.6% for SKD and APS, and 80% for SKD and KC.
CONCLUSIONS: WhatsApp Messenger serves to transmit good quality pictures of CT scan images. A reasonable diagnosis and management strategy can be formulated using this app with fair inter-rater reliability. Copyright:

INTRODUCTION

WhatsApp is a smartphone-based application used frequently for telecommunication.[12] As of July 2019, WhatsApp has reached over 2000 million users worldwide and 400 million users in India, thus having a firm digital footprint.[34] Despite this heavy base of digital communication platforms in India, WhatsApp was not officially approved for providing telemedical consultation until recently. Still, this app has increasingly been used as an “off-label” means of communication for teleconsultation in clinical medical practice before COVID (BC) era.[56] Due to the COVID-19 pandemic, its use was approved in the guidelines provided by the Indian Medical Council and vetted by the Ministry of Health and Family Welfare, India, for its use in teleconsultation within 3 days of the first lockdown on March 26, 2020.[5] With this official approval, the app came to used as a means of teleconsultation.

Another prime use of WhatsApp in clinical practice is communication between a medical trainee/resident and a consultant, where the resident seeks opinion regarding a particular case for proper management. However, studies validating the use of WhatsApp for this purpose, especially in urological practice, are lacking.[7] We are unsure whether the use of WhatsApp as a mode of transmission of information can have the same reproducibility as that of physical consultation. Is there a potential loss of data or misinformation or misinterpretation of data sent through WhatsApp? It also needs to be determined whether this makes any difference in decision-making for patients and clinical care. Hence, we designed this study to find the inter-observer variation and inter-rater reliability of the data transmitted through this app in real-world setting and thus validating its use in urological practice.

MATERIALS AND METHODS

The prospective study was conducted from June 1, 2020, to July 15, 2020 at a tertiary care center after taking approval from the institute ethics committee (NK/6316/Study/190). Thirty patients presenting for emergency urological consultations under care of three urology consultants (APS, SKD, and KC) of a single unit were included.

All the emergency consultations to the urology department under the three consultants of single unit (APS, SKD, and KC) with relevant cross-sectional imaging in the form of computed tomography (CT) scan were initially evaluated by the resident posted for emergency urology care. The clinical details and radiological images were posted in a closed WhatsApp group involving the three consultants and a senior resident (SR) after taking informed consent from the patient. The relevant radiological images were posted in the WhatsApp group as a Whole Image (WI) and Image of Interest (IOI) format. WI included the photo of whole CT scan sheet with multiple cross-sectional images and IOI included the specific cross-sectional image with the possible abnormality or pathology [Figure 1]. It may be prudent to mention here that the facility of Picture Archiving and Communication System is still not well established at our center.

Figure 1

Screenshot of CT scan images sent by the resident to consultant (a) Whole Image (b) Image of Interest

All three consultants independently reviewed the clinical details and the images provided within 15–30 min of receiving the images and data. They rated the quality of image both WI and IOI separately on Likert scale of 1–5 (1– very poor, 2 – poor, 3 – good, 4 – very good, and 5 – excellent). The resident rated the image quality of original image and noted it separately. He also formulated his provisional diagnosis and line of management. Similarly, the consultants after looking at the images formulated a provisional diagnosis and initial management strategy. They sent these responses (Quality of WI, Quality of IOI, provisional diagnosis, and management) as a personalized WhatsApp message to the resident separately to maintain blinding from each other's responses. All the responses were recorded by the resident in the case report form and later entered in a Microsoft Excel Sheet.

In case of gross discrepancy in the responses, the initial response was recorded for analysis, and then management options were discussed among the three consultants before making the final decision of line of treatment for a particular patient. Respective phone models used were as follows: Consultant 1-SKD (iPhone 11, Apple, USA), Consultant 2-APS (One plus 5T, China), Consultant 3-KC (iPhone 8, Apple, USA), SR-(iPhone 6s, Apple, USA). All the phones of consultants and resident are meant for personal use and are password protected. WhatsApp Messenger safeguarded the information by providing end-to-end encryption. The images were not shared out of this closed WhatsApp group and were archived in a safe encrypted folder on a computer and deleted from all the smartphones after completion of the study.

Statistical analysis

Data were entered in Microsoft Excel 2010 and were analyzed using IBM SPSS Statistics for Windows, version 21 (IBM Corp., Armonk, New York, USA). The data were expressed as number and percentage. Categorical data between the groups were analyzed from a 2 × 2 contingency table. The concordance for the rating of image was represented using Cronbach alpha, and interrater reliability was represented by the interclass correlation coefficient (ICC) for rating of imaging. For diagnosis, coding was done in binary fashion, 0 for observed disagreement and 1 for observed agreement to the diagnosis of resident. For management, coding was done as 0 for observed disagreement and 1 for observed agreement to the final management of the patient. P0 was defined as a proportion of observed agreement, which is given as sum total of agreements divided by the total response (a + d/N) where a = total positive agreements, d = total negative agreements, and N = Total responses (30). The interobserver agreement between two raters was calculated using kappa (κ) statistics. P < 0.05 was considered statistically significant.

RESULTS

A total of 30 patients (20 males and 10 females) were included in the study. Mean age of the patients was 45.47 ± 13.68 years. The demographic details, provisional diagnosis, and imaging details are provided in Supplementary Table 1. The mean WI rating was as follows: resident – 3.27 ± 0.58, SKD – 3.70 ± 0.75, APS – 3.73 ± 0.64, and KC – 3.03 ± 0.61. The Cronbach alpha for WI rating was 0.494 and ICC was 0.196 (P = 0.006). The mean IOI rating by resident (the resident rated the actual imaging) was 4.07 ± 0.78. The IOI rating by the three consultants was as follows: SKD – 4.07 ± 0.87, APS – 4.13 ± 0.73, and KC – 3.4 ± 0.56. The Cronbach alpha for IOI was 0.824 and ICC was 0.540 (P < 0.0001). None of the consultants asked for any added IOI to be sent separately.

Supplementary Table 1

Individual case details

Age (years)	Sex	Clinical presentation and provisional diagnosis
40	Female	Follow up case of carcinoma cervix with right HDUN
54	Female	T2DM with CKD with right HDUN and pyelonephritis
29	Male	Follow up case of carcinoma recto-sigmoid with bilateral HDN with AKI with pleural effusion on right PCN
29	Female	Septic abortion with DIC with hematuria and bladder clots
55	Male	CKD with bilateral HDUN with hematuria and bladder clots
65	Male	Bilateral HDUN with AKI with retroperitoneal lymphadenopathy
46	Female	Follow up case of neurogenic bladder (Cauda-Equina Syndrome) with right solitary functioning kidney and CKD presenting with renal mass and hematuria
65	Female	Recurrent carcinoma urinary bladder status postchemoradiation with left solitary functioning kidney and HDUN with AKI secondary to hematuria and clot retention
22	Male	Follow up case of CKD presented with AKI and Uremic encephalopathy secondary to right HDUN and small capacity bladder
46	Female	Recurrent carcinoma urinary bladder with right HDUN presented with hematuria
36	Male	T2DM with right emphysematous pyelonephritis (Type 3A) and right pleural effusion
56	Male	Chronic liver disease with portal hypertension with liver and left renal space occupying lesion
63	Female	Gall bladder mass with right pyelonephritis and HDN
41	Male	HTN with transient ischemic attack with Right HDN secondary to UPJO
45	Male	Right perinephric abscess with HDUN secondary to distal ureteric calculus
39	Male	Left solitary functioning kidney with obstructive uropathy secondary to left RSD and ureteric calculus
70	Male	BTA with bilateral renal cystic disease with right cyst rupture and fracture of maxillary and ethmoid bone
30	Male	Left lower ureteric calculus with HDUN
32	Male	Right HDN secondary to UPJO with flank pain
59	Male	BTA with right renal Grade 2 injury
43	Male	Urinary bladder mass with hematuria
60	Male	Urinary bladder mass with hematuria
24	Female	Follow up case of bilateral RSD on left DJ stent presented with slipped out right PCN and fever
28	Female	Locally advanced carcinoma rectum status post-neoadjuvant chemoradiotherapy with bladder infiltration
47	Female	UPJO status post-endopyelotomy presented with gross hematuria secondary to renal artery pseudoaneurysm
60	Male	Renal parapelvic cyst with spontaneous urinary leak
55	Male	Urinary bladder mass with left HDUN presented with hematuria
34	Male	BTA with pelvic fracture urethral injury and bladder rupture
55	Male	Right renal mass with hematuria
36	Male	Bilateral RSD with obstructive uropathy

CT=Computerized tomogram, HDUN=Hydroureteronephrosis, HDN=Hydronephrosis, AKI=Acute kidney injury, CKD=Chronic kidney disease, PCN=Percutaneous nephrostomy, T2DM=Type 2 diabetes mellitus, RSD=Renal stone disease, UPJO=Ureteo pelvic junction obstruction, DJ stent=Double J Stent, BTA=Blunt trauma abdomen

For diagnosis, proportion of observed agreement P0 was 83.3% for SKD and APS, 76.6% for SKD and KC, and 73.3% for APS and KC. For initial management, P0 was 80% for APS and KC, 70% for SKD and APS, and 70% for SKD and KC. As compared to he resident, the P0 was 86.67% for resident and KC, 80% for resident and APS, and 76.67% for resident and SKD. The corresponding kappa values are provided in Table 1.

Table 1

Inter observer agreement for management decisions: Kappa values for various readers

Pair	Kappa	P
Resident and SKD	0.430	0.015
Resident and APS	0.441	0.016
Resident and KC	0.627	0.001
SKD and APS	0.270	0.127
SKD and KC	0.270	0.127
APS and KC	0.441	0.016

SKD=Sudheer Kumar Devana, APS=Aditya Prakash Sharma, KC=Kapil Chaudhary

There was difference in formulating the management strategy in terms of the modality of diversion used for obstructive uropathy such as placement of double J stent versus a percutaneous nephrostomy (4/30). There also was difference in terms of approach to a particular procedure such as laparoscopic versus open radical nephrectomy (1/30). We clubbed the modalities “DJ stenting” and “PCN placement” as “Urinary diversion.” The modalities “laparoscopic radical nephrectomy” and “open radical nephrectomy” were combined as “radical nephrectomy.” After this recoding, the P0 and interrater reliability increased further and is shown in Table 2.

Table 2

Interobserver agreement for management decision: Kappa values for various readers after adjustment

Pair	P0 (%)	Kappa	P
Resident and SKD	90	0.609	0.001
Resident and APS	90	0.672	0.000
Resident and KC	90	0.609	0.001
SKD and APS	80	0.280	0.125
SKD and KC	86.6	0.586	0.001
APS and KC	86.6	0.586	0.001

DISCUSSION

In the era of social distancing, the use of teleconsultation is likely to increase.[89] Hence, studies regarding the validation of tools for teleconsultation are needed to optimize their use. In this study, we intended to determine the inter observer variation and calculate inter-rater reliability of use of clinical data transmitted through WhatsApp in emergency urological setting. In our study, the mean scores of the image rating reflected “good (3)” to “very good (4)” rating for all transmitted radiological images. The mean scores for the WI were less than that given for the IOI by all the observers. This entails from the fact that there is always a need to zoom the WI to look for the abnormality/pathology, which leads to loss of pixels of the WI, resulting in blurring of image when seen on a mobile phone [Figure 2b]. However, the IOI is transmitted after clicking an individual image which does not require zooming and hence has no such problems of pixel loss [Figures 1 and 2a]. Thus, there is better perception and readability for IOI than WI.

Figure 2

Image showing a section sent as Image of Interest (a) and the same section of CT scan after zooming in the Whole Image (b). Note the blurring of image after zooming and thus causing loss of information

There was wide inter-rater variability in rating the quality of WI when compared to rating of quality of IOI as is seen from the values of Cronbach alpha (0.494 vs. 0.824) and ICC (0.196 vs. 0.540). The plausible explanation for the same could be difference in the rater's subjective interpretation of the quality of WI and the phone model used. Scanning the whole CT image on phone and interpreting it leads to the difference in ratings assigned to WI. Furthermore, as mentioned earlier, the need for zooming in and consequential loss of pixels leads to blurring of image [Figure 2]. The perception of this blurred and zoomed image leads to a significant difference in subjective rating by the observer, while for IOI alone such zooming is infrequently needed.

Regarding diagnoses, it was seen that the proportion of observed agreement P0 values among consultants were good and ranged from 73.3% to 83.3%. Thus, it may be inferred that despite the difference in rating the quality of images, the data may be interpreted to reach a reasonable urological diagnosis. On further analyzing the data on nonagreement cases, we noticed that the consultants found additional findings on imaging other than those pointed by the resident in 5/30 occasions (SKD and APS) and 4/30 occasions (KC), respectively. This amounts to a very high likelihood (>80%) of users finding the said diagnosis, based upon the clinical data and imaging using WhatsApp, irrespective of the image quality.

The final and the most important outcome of any teleconsultation is its utility in providing a management strategy after reviewing the clinical data. In our study, when we looked at the formulation of management strategy, we found high P0 ranging from 70% to 86.7%. Despite this high proportion of observed agreement, we had low kappa values (0.270–0.627). This paradoxical low κ values despite the high observed agreement is due to prevalence dependency of κ, which has been described in detail by Cicchetti and Feinstein and must be interpreted with caution.[1011] In cases where exact prevalence or the “gold standard” is unknown the calculation of prevalence is done using marginal totals from a 2 × 2 table for observed agreements and disagreements. In case the marginal totals (observed agreement or disagreement) become very low either vertically or horizontally as in our study, the κ lowers drastically for the same value of the proportion of observed agreement (Po).[11] Thus, it is the proportion of observed agreement which become more relevant than κ alone in such situations.[12]

As mentioned in the results, the observed agreement and κ increased further after recoding to “Urinary diversion” and “radical nephrectomy” as mentioned in the result section. This recoding was done since approaches to a particular procedure are likely to differ even in physical consultation, and hence the recoded values possibly reflect the true agreement among the observers. This difference of opinion can be sorted after mutual discussion, as was done in our series, and these differences are likely to arise even in cases where the clinical data and the imaging are presented physically to a group of physicians.

In a study by Sener et al. on evaluation of interrater reliability of WhatsApp for evaluation of hematuria, 212 patients were evaluated for hematuria by two groups of urologists.[13] One group having direct access to the patients, while the other group comprising urologist blinded to patients’ data and received image on WhatsApp. The grade of hematuria was evaluated by them as follows: Hematuria with the following rating: 0 – no hematuria, 1 – hematuria that does not require invasive treatment, and 2 – hematuria requiring bladder drainage or any form of active treatments. They found almost perfect agreement between two groups, (kappa-0.992). Another study for cystoscopic/ureteroscopic image was conducted by Arada et al., who found significant agreement between consultant and attending plans.[14] The reply was in the form of “agree” or “disagree” to the formulated management plan by the attending. However, this being a conference abstract, the detailed methodology and results could not be ascertained. Contrary to these studies, the kappa values in our study are low (0.280–0.672) as previously mentioned. This difference may also be accounted for by the complexity of the problem being evaluated. Reading and interpreting CT scan images on phone is a complex task as compared to the interpretation of hematuria using color of urine and cystoscopic images.

In this current era of COVID-19, the use of telemedicine has increased exponentially. WhatsApp has provided a universal tool for teleconsults and telemedicine in resource-limited countries like India. The major advantages of using this app for telemedical consultation was its widespread user base, no extra cost for sending messages, calls (both audio and video), ability to share photos, videos and message using a single platform and with end to end encryption facility which maintained confidentiality of patient's identity and data.[1516] Amid the call for social/physical distancing the use of this tool for emergency consults and communication between the residents and consultants in each field is also bound to increase. The same app can now be used for seeking such consults and has been more so useful in after COVID-19 (AC) era where physical distancing is a norm. Thus, this study was designed to validate the use and assess the inter-rater reproducibility of teleconsults. This present study is the first of its kind addressing the use of WhatsApp for entire decision-making of the patients visiting emergency. The images and the clinical scenarios are much more complex than those in the already published literature. The study is a small step toward the incorporation of social media applications (such as Telegram, Facebook, Twitter, etc.) being used off label for clinical use into mainstream clinical practice. Of course, excessive use of such apps comes at the expense of a precious commodity, i.e., time and they have associated side effects related to increase in screen time such as eye and neck strains.

An important limitation of the study was the small sample size. The study had descriptive data for comparison (diagnosis and management); this derives from the fact that we wished to replicate the day-to-day clinical practice in our system. In this study, all the phones used can click and reproduce high-quality images. The quality of mobile phones used can also affect the interpretation of the image. We did not address the picture-taking skill of a particular trainee through this study. A comparator arm of physical control was not kept in view of COVID-19 situation.

CONCLUSIONS

WhatsApp messenger serves to transmit good quality pictures of cross-sectional imaging modality such as CT scans. These images along with appropriate clinical history can be used to formulate a reasonable diagnosis and management strategy with fair to substantial inter-rater reliability. WhatsApp can be used in emergency urological setting with significant agreement among the resident and consultant.