Using a Facebook Virtual Examination Room to Enhance ROP Training in Armenia.

Purpose: Retinopathy of prematurity (ROP) is a leading cause of worldwide childhood blindness with increasing incidence in low and middle income countries (LMICs) due to advances in neonatal care. There are insufficient numbers of healthcare professionals specialized in ROP management and few local training opportunities in LMICs. Social media platforms provide a promising solution to enable interactive medical education across geographic and logistic barriers. As an adjunct to an ROP training program for ophthalmologists in Armenia, we implemented a Facebook Virtual Examination Room (VER) for case discussion with a global community of collaborators and preceptors. To evaluate training through VER, we operationalized engagement as a multilayer meta-construct that includes cognitive, behavioral, and social-emotional domains.
Methods: A concurrent mixed methods approach was taken to collect and analyze data from comments and activities within VER. Quantitative data was analyzed for descriptive statistics on group utilization, participant activity, and clinical metrics. Qualitative data was analyzed by conducting thematic analysis involving initial and pattern coding.
Results: Ten participants (7 trainees and 3 preceptors) interacted in the Facebook group across 153 unique cases, with 218 threads, 414 comments, and 216 likes. Of the 120 unique cases eligible for consensus evaluation, treatment was mentioned most frequently (87), followed by location (68), staging (65), and plus disease (31), with agreement ranging from 67% to 97%. Based on the qualitative analysis, the most common themes of discussion were clinical management, dilemma resolution, knowledge growth, and gratitude.
Conclusion: A closed Facebook group for case-based discussions can be a useful adjunct to an existing ROP training program by engaging learners across social-emotional, behavioral, and cognitive domains. For international training partnerships, the financial and logistical advantages can be significant, though focus should remain on the primary curriculum and training modality.

Introduction

Retinopathy of prematurity (ROP) is a vasoproliferative retinal disease which affects premature infants and is the leading cause of childhood blindness around the world.1 In low and middle income countries (LMICs), where increasing numbers of newborns are surviving infancy and prematurity due to advances in neonatal care, the incidence of ROP has been increasing.2 This problem is compounded by insufficient healthcare professionals specialized in ROP management and few local training opportunities.3 International training partnerships show promising results in addressing workforce shortages, but scalability has been limited by factors such as cost, time, and logistics.4 Technology has been successfully implemented to enhance training paradigms for screening, diagnosis, and treatment.3,5 In particular, knowledge sharing networks and social media have been leveraged innovatively to enhance educational discourse, consultation, and collaboration internationally.6–9

In 2010, Children’s Hospital Los Angeles (CHLA) partnered with the Armenian EyeCare Project (AECP), a non-governmental organization (NGO), to address the pediatric eye care needs of the country. With the rise in ROP, a multi-institutional collaborative effort was initiated to manage this condition, and training ophthalmologists in screening, diagnosing, and treating ROP became an important aim of the initiative. We used a number of technologies to help address some of the logistic and financial barriers to our international training partnership, such as implementing live streaming of the surgical microscope video feed to allow international ROP experts to provide real-time guidance and feedback during surgery.10,11 To provide additional asynchronous support to the trainees in Armenia, we created the Virtual Examination Room (VER), a closed Facebook group for trainees to post and discuss cases with a global community of collaborators and preceptors.

Many studies involving medical education through social media have considered engagement as an evaluation parameter.12–14 From a theoretical perspective, engagement is a multidimensional construct that involves active cognitive processes, such as decision-making, problem-solving, and evaluation, as well as successful collaboration with team members within a learning environment.15 Engagement frameworks offer a unique conceptual approach to better understand learning through emotional, social, cognitive, and behavioral lenses.16 In alignment with the existing views, we operationalized engagement as a multilayer meta-construct that includes cognitive, behavioral, and social-emotional domains. Cognitive domain refers to thinking about the applicable content and context of learning such as participants reflecting on their own learning and the learning content. Behavioral domain refers to observable features of participation such as the number and kinds of tasks and activities within a project. Social-emotional domain refers to expressions of feelings about specific activities or other people, as well as interactions with other participants.

This study explored the utilization and engagement of the VER by our global community of ophthalmologists. Here, we present our design, implementation, and findings in using the VER as an adjunct to the ROP training program for ophthalmologists in Armenia.

Methods

ROP Training Program

The ROP training program in Armenia began in 2010 as a partnership between the AECP and CHLA.10 The program includes in-person and remote didactic lectures, practical training in the operating room, visiting observerships for Armenian ophthalmologists in the US and UK, sharing of wide-angle images using a RetCam (Clarity Medical Systems, Pleasanton, CA) to support diagnosis and treatment decisions, and live surgical teleproctoring.10,11 The VER was added as an adjunct to the training program to create a forum in which Armenian ophthalmologists could discuss cases asynchronously with international colleagues. The patients of the program were newborns with ROP at one of seven NICUs in Armenia – six from Yerevan and one in Gyumri. ROP screening began as a pilot program based on national guidelines developed by a multi-stakeholder coalition led by the AECP, which were later validated and verified by the Armenian Ministry of Health, and is now mandatory national health policy.

Research Design

This retrospective study used a mixed methods approach to explore participant engagement within VER. Mixed methods studies allow researchers to combine both qualitative and quantitative data collection, analysis, and inference techniques for reaching deeper understanding and corroboration.17 Specifically, we applied a convergent parallel mixed methods design which allows for concurrent collection and analysis of quantitative and qualitative data.18,19 Convergent designs often help to obtain complementary data on the same topic. For our three domain engagement framework, the behavioral domain was explored through quantitative analysis of user participation, and the cognitive and social-emotional domains were explored through qualitative analysis of text.

Participants

The clinical staff involved 6 groups: professionals, non-professionals, trainees, mentors, researchers, and administrators. These groups can be organized into 3 main categories: preceptors, trainees, and observers (administrators, Armenian clinicians not in the training program, researchers, and staff members). The trainees were from Armenia, while the preceptors were based in the US or UK. The preceptors were ophthalmology specialists trained in ROP. The trainees were all ophthalmologists participating in the AECP ROP training program.

Virtual Examination Room - Facebook Case Reviews

The VER was created by AECP staff as a closed Facebook group requiring permission to join. The group consisted of ROP specialists (preceptors), ophthalmologists-in-training (trainees), researchers, and administrators.

For the purposes of this paper we use the following terms:

A Case is a unique patient being discussed in the VER.

A Post is the first Facebook posting that introduces a case.

A Comment is a reply to a post or another comment.

A Thread is a conversation between two or more users that begins with a post and is followed by one or more comments. A given case may be discussed across multiple threads.

A Like is the use of Facebook’s native features to indicate that a user agrees with or likes a post or comment.

All members had the ability to like posts and comments. Generally, wide-angle images were posted by trainees along with gestational age, current age, and weight. Both preceptors and trainees commented on the suspected ROP stage, zone, and the presence of plus disease. Additionally, treatment options such as medications and surgical procedures (such as laser and vitrectomy) were discussed by both groups.

Data Extraction and Analysis

Quantitative Data Analysis

Descriptive statistics were used for analyzing group utilization, participant activity, and clinical characteristics.

Programmatic and Manual Extraction of Facebook Data

To extract data from Facebook, a python script was developed to extract and parse the HTML data (see for code). The appropriate HTML fields were parsed to organize Facebook data into csvs (comma separated values). Figure 1 contains a diagram of the methodology. One csv file aggregated numeric metrics such as likes, number of comments, and group metrics. Another csv file contained purely textual data from posts and comments. These data files were subsequently used for quantitative and qualitative analysis. The Facebook HTML data was also used to manually extract clinical characteristics into a csv file which contained both numbers and text.

Figure 1

Overview of how Facebook HTML data is extracted and parsed into Excel as a CSV file. (A) First, Facebook data is downloaded as an HTML file. (B) Key variables of the post are selected for analysis. (C) Using a python script, the corresponding tags are used to parse the specified data. (D) The parsed data is then formatted as a CSV file compatible with Excel.

Virtual Examination Room Metrics

The Facebook data collected include group utilization, participant activity, and clinical characteristics. Group utilization metrics include the number of trainees, preceptors, and observers, in addition to the total, average, and range for counting metrics involving posts, threads, likes, comments, views, and commenting period. Participant activity metrics include counting metrics on specific individuals such as preceptors, trainees, and observers. Clinical data was extracted by reviewing the text of the posts, and includes ROP zone, stage, presence of plus disease, gestational age, current age, and treatment options such as medications, laser, or other related surgeries.

To evaluate consensus, each post and comment was read for agreement. Agreement was defined as either stating agreement in a comment (eg, “I agree”, or “I think so, too”), or by liking. If the original poster (OP) gave a range (eg Stage 2–3), then the commenter was considered to agree if they gave the same range, or picked a single value within that range (in the example above, choosing stage 3). Giving a different range (eg Stage 3–4), was considered a disagreement. The priorities for achieving consensus were:

Team agreement.

Agreement between preceptor and trainee.

Agreement between observer and trainee.

Agreement between trainee and trainee.

Qualitative Data Analysis

Qualitative data was analyzed independently by authors JD, TG, PM, and EP for themes and tone using Excel. Qualitative text data extracted from comments by VER trainees (7), preceptors (3), and observers (6) were analyzed using thematic analysis technique. The analysis steps aligned with the general guidelines for qualitative data analysis.20 We applied two levels of coding. Level One coding involved open coding where a code was assigned to a specific segment of the data.21 This allowed for the generation of the initial codebook. Level Two coding involved pattern coding where initial codes were clustered into themes and broader categories. To eliminate researcher bias in the interpretation process, coding was conducted independently by three researchers of this study and then calibrated during multiple discussion sessions. Any discrepancy was discussed thoroughly until consensus was reached.

Regulatory Statement

This research was approved by the CHLA Institutional Review Board (IRB) and data was managed according to regulation and guidelines implemented by the Armenian regulatory bodies.

Results

Quantitative Results

Group Utilization Metrics

There were 82 participants in the VER: 3 preceptors, 7 trainees, and 72 observers (Table 1). Over a period of 6 years, there were 153 unique cases and 218 posts with an average of 1 like, 2 comments, and 3 views per post. On average, 54 minutes elapsed for the first comment to appear for each post and 87 minutes for the last.

Table 1

Facebook Group Utilization Metrics

Participants	Users (n = 82)	Users Who Made Posts or Comments (n = 14)
Trainee participants	7	5
Preceptor participants	3	3
Observers	72	6
Total	82	14
Group Metrics	Total	Average	Range
Unique cases	153	–	–
Threads	218	–	–
Comments	414	1.9	0–9
All posts	632	–	–
Likes	216	1	0–5
Seen by	664	3	0–46
Time to first comment	–	54 minutes	0–6.6 days
Time to last comment	–	87 minutes	0–14 days

Participant Activity Metrics

Trainees initiated 215 threads for 153 unique cases. They liked 71 posts and made 140 comments. Preceptors did not create any cases or threads, but made 95 comments and liked 2 posts. Observers made 3 posts for 3 unique cases along with 179 comments and 143 likes. Table 2 shows individual participant activity metrics, and Table 3 shows how participants interacted with each other across threads. There were 120 out of 153 (78%) unique cases that involved 2 or more participants; the remaining 33 (23%) cases were solo posts initiated by trainees but did not receive any replies.

Table 2

Individual Participant Utilization

Participant	Cases Created (n=153)	Threads Started (n=218)	Comments Made (n=414)	Total Posts (n=632)	Total Likes (n=216)
All Preceptors	0	0	95	95	2
Preceptor 1	0	0	2	2	0
Preceptor 2	0	0	21	21	0
Preceptor 3	0	0	72	72	2
All Trainees	150	215	140	355	71
Trainee 1	1	1	0	1	0
Trainee 2	0	0	0	0	5
Trainee 3	44	91	49	140	16
Trainee 4	103	121	62	183	44
Trainee 5	0	0	7	7	2
Trainee 6	0	0	0	0	0
Trainee 7	2	2	22	24	4
Observers	3	3	179	182	143
Grand total	153	218	414	632	216

Table 3

Participant Activity on Threads and Cases

Thread Participants	Threads	% of Total Threads	Unique Cases	% of Total Cases
1 (solo)	41	19%	33	22%
2+ (group)	177	81%	120	78%
Trainees only	16	9%	9	8%
Trainee + observer + preceptor	70	40%	53	44%
Trainee + observer + no preceptor	88	50%	56	47%
Observer + preceptor	3	2%	2	2%
Grand Total	218	100%	153	100%

Clinical Characteristics

Across 153 unique cases, the average birthweight was 1246.59 ± 346 grams, and the average gestational age was 29.08 ± 2.35 weeks. Figure 2 summarizes the information presented in the case discussions regarding location, staging, plus disease, and treatment options.

Figure 2

Summary of clinical descriptors of the 153 cases discussed in the Virtual Examination Room, including (A) location, (B) staging, (C) plus disease, and (D) treatment options discussed.

Consensus

Of the 153 unique cases, 120 included 2 or more participants and thus were eligible for consensus evaluation, shown in Figure 3. For each domain (location, staging, plus disease, and treatment), the number of times that domain was mentioned in the case is shown, followed by the number of times it was discussed (meaning another participant engaged with the concept, eg “agree zone 2” or “I would consider Avastin first”), and finally among those cases in which it was discussed, how many of those achieve consensus, as defined in the methods. Relative percentages are shown next to the bar graphs. Treatment was mentioned most frequently (87 out of 120) and achieved consensus in almost all discussions (97%). More than one treatment option was discussed in 5 out of 35 cases. Plus disease was discussed least frequently, in only 6 out of 31 cases. Overall, consensus ranged from 67% to 97% of discussed cases for each domain.

Figure 3

Case discussion achieving consensus across 4 domains: (A) location, (B) staging, (C) plus disease, and (D) treatment options discussed. 33 cases did not have any comments, and are not included in this figure. Each bar represents the number of times a domain was mentioned in a case, followed by the number of times another participant engaged with the domain, and finally the cases which achieved consensus. Relative percentages are shown next to the bar graphs. *Cases: included 2 or more participants and thus were eligible for consensus evaluation.

Qualitative Findings

The analysis of the qualitative data collected from the VER trainee participant comments created multiple themes that were grouped into two broad categories related to the participants’ engagement within the VER training. Those categories were (1) Cognitive Engagement, that included themes related to the VER learning content; and (2) Social-Emotional Engagement, that included themes related to the expressions of feelings and peer-to-peer /group interactions within the VER. Table 4 below provides information on the two broad categories, the related themes, and exemplar quotes from the participant comments.

Table 4

Thematic Analysis of Qualitative Data on VER Trainee Engagement

Broader Categories	Description	Related Themes	Exemplar Quotes
Cognitive Engagement	Expressions related to VER learning content, including case, diagnosis, and treatment information.	Providing case information; providing case-specific advice; providing diagnosis; requesting additional diagnostic information; providing additional diagnostic method; suggesting treatment options; providing case prognosis; offering timeline for treatment; confirming treatment option; providing treatment details; providing treatment update; providing clinical course advice; highlighting unusual syndrome; rejecting diagnosis; requesting physical exam findings; confirming patient details.	“The left eye is high risk and will likely require treatment. Either laser or avastin would be appropriate. I would probably treat within the next several days.”“Avastin was performed for both patients 2 days ago, from 4 days will update new photos, thanks for contacting, Dr [name].”“This appears to be on border of Zone 1 posterior Zone 2 and will likely need treatment in the future. This is not severe ROP at this time. Once very early stage 3 or very early plus disease occurs you may want to consider laser.”“ … my suggestion to do Avastin first then to f/up for laser.”
Social-Emotional Engagement	Expressions related to feelings and peer-to-peer /group interactions within VER.	Peer encouragement; peer expert affirmation; expressing gratitude; imploring for urgency; requesting peer input; promising a follow-up update; expressing confusion; expressing uncertainty; apologizing for late response; seasonal greeting; sharing personal anecdote.	“Please, your opinions, dear colleges!”“Thanks for your support, Dr. [name]!”“What do you think, dear colleges?” “Not sure if avastin is contraindicated.”“Glad to hear they are doing well!”“Dear colleagues, friends, please have a look and comment on this case.”“I agree baby will need treatment and avastin is a good choice.”

Discussion

The purpose of this study was to understand how Armenian ophthalmologists participating in an ROP training program used and engaged with the adjunct VER program. We situated our findings within an engagement framework which included the behavioral, social-emotional, and cognitive dimensions.16 By using a mixed methods design, we gained a complementary understanding of participant engagement as our qualitative findings were supported by our quantitative results. To our knowledge, this is the first study to use a closed Facebook group to supplement an ROP training program internationally.

The VER case discussion usually began within the hour (54 minutes). Trainees initiated almost all the threads (215 out of 218) and made 34% of all comments. This result was expected since the training program was aimed specifically at helping address questions and concerns brought to attention by trainees. Most of the case discussions (108 of 153) involved conversation with at least 1 trainee and a mentor or experienced professional. These trends show evidence of behavioral engagement due to active participation by group members, especially trainees. In addition, our qualitative findings show cognitive and social-emotional engagement occurring within the VER. For example, after a trainee provided retinal images and preliminary information for a case, one of the preceptors commented, “Either laser or avastin would be appropriate … treat within the next several days”, to which the trainee responded, “I considered to do avastin because zone 1, will perform tomorrow [sic] and let you [sic] know. Thanks”. This was a common example of how preceptors provided diagnostic and treatment options to trainees which highlighted cognitive engagement. Trainees were not only actively thinking about ROP content but also evaluating their clinical conclusions in context with preceptor recommendations. Furthermore, participants also showed evidence of social-emotional engagement by expressing gratitude, providing encouragement, and interacting with peers; all of which are important components of clinical training.22 Although few cases met our criteria for consensus evaluation, among those that did, diagnostic and treatment consensus was frequently achieved (67–97%). In some instances, we could see how engagement between the trainees and preceptors worked to improve patient care and clinical results. For example, in one case where the trainee posted follow-up wide-angle images, the preceptor noticed improvement and responded, “This looks much better. There is no traction so I would watch carefully and not do surgery at this time. This child needs to be seen every week … ” (Figure 4).

Figure 4

A case discussion within the Facebook group involving a preceptor and trainee.

Although research is limited, social media platforms have been used as an informal supplement to graduate-level medical education.23,24 Several studies have reviewed how online social media platforms can increase engagement through a cognitive, social-emotional, and behavioral lens. Physicians have joined various online communities to read new articles, listen to experts, and consult colleagues on patient issues.25 A recent paper studied the benefits of a Facebook group comprised of women physicians trained in physical medicine and rehabilitation (PM&R).25 The majority of respondents (63%) found that learning from posted cases and discussion and advice on complex patients was the most beneficial to them.25 While Facebook has been primarily used for recruitment, content dissemination, and marketing in ophthalmology,24,26 other fields have leveraged the social media platform for training in both undergraduate and graduate medical education.7,27 For example, a study using a closed Facebook group to discuss surgical cases, showed how participants (surgeons, residents, fellows, and medical students) can use this platform to cognitively engage with clinical content across 40 countries.7 One concern with larger, more open groups is the provenance and reliability of information, ideas, or recommendations shared.7 It may not be feasible to vet each member’s expertise or verify comments for accuracy, so active moderation, trust building, transparency, and healthy skepticism are likely important components of this approach. Social-emotional engagement has also been a valuable part of these communities. Almost 15% of the individuals in a PM&R Facebook group found emotional support and mentorship to be a valuable facet of the group. Those at earlier stages in their careers mentioned an increase in empowerment, and 75% of the group believed that the Facebook community reduced their professional isolation.25 In our study, we found that trainees consistently expressed gratitude to each other and to the mentors who helped answer their clinical questions, highlighting the support that was being provided in this environment. Posting comments is not the only interaction modality on Facebook. In a study reviewing the International Hernia Collaboration (IHC), Ghanem et al found that engagement was significantly higher in Facebook Live Q&A sessions compared to posts, and that participants often continued to discuss lecture content until the conversation naturally concluded.28 Other studies have shown that participants preferred engaging with content that includes videos and pictures rather than text alone.7,29,30 Overall, our findings are consistent with existing evidence that social media is a positive contributor to medical education in terms of engagement. It allows physicians to actively engage with meaningful content, feel supported by a clinical community, and leverage a convenient platform which allows for organic conversations without geographic and economic barriers.

Our study benefited from a mixed-methods design, through which we were able to extrapolate a more nuanced understanding of the VER by placing our quantitative results in context through our qualitative analysis of the comments. Additionally, using the engagement framework helped orient the qualitative findings by focusing on socially constructed dimensions of training. Our qualitative analysis was conducted by 4 different individuals to improve reliability and reduce bias. The approach itself was very simple, requiring almost no resources or coordination compared to international travel. The study period spans over 6 years, demonstrating the durability of the solution. Finally, despite occurring before the COVID-19 pandemic, this study provides evidence for the utility of virtual education platforms which are currently of great interest. Academic programs may benefit from using virtual technology and social media to provide remote education and promote educational engagement.

There are a number of limitations to this study worth noting. As a retrospective study, there was no way for us to verify or clarify any of the information found in the VER. Trainees were not asked to follow a specific format when posting cases, and so many cases had missing data. The lack of a pre-specified case posting format and resulting missing data may explain the lack of discussion of some of the more subjective elements of ROP diagnosis and staging, such as plus disease. Preceptors were also not provided instructions on following a specific format in their responses, resulting in many cases not being able to be evaluated for consensus because preceptors did not comment on all aspects of a case. We did not compare VER case discussions to traditional in-person discussions. It is likely that there are aspects of ROP training that cannot be replicated virtually; as an adjunct to an existing training program, it is important that the VER complement and reinforce the in-person components. The qualitative findings could have been influenced by observational bias, cultural bias, groupthink, bandwagon effect, and impression management due to working with international colleagues.7 The majority of the data gathered and presented here corresponds to attitudes and perceptions, the first level of Kirkpatrick’s hierarchy for evaluating educational programs.31,32 The VER program did not formally evaluate knowledge change (level 2), and though a few posts included anecdotes of how the trainees changed their treatment plan (level 3), or even included updates on the patients’ outcomes (level 4), there was insufficient data to meaningfully evaluate these concepts.31,32 Finally, technology is constantly changing, and as the features of Facebook evolve and the popularity of other platforms waxes and wanes, our specific approach could become less reproducible in the future. For researchers interested in recreating or adapting the VER, changes could be made to improve the methodology. Creating posting guidelines for trainees and preceptors, collecting demographics and outcomes data, a larger sample size, and a comparison to other modalities would all improve upon the current study.

Conclusion

The increasing number of infants with ROP in LMICs has created an urgent need for specialized training. Technology can be used to support training and capacity building through synchronous and asynchronous educational programs. Social media platforms provide a cost effective, geographically agnostic, and convenient medium for medical education and clinical communication. A closed Facebook group for case-based discussions can be a useful adjunct to an existing ROP training program by engaging learners across social-emotional, behavioral, and cognitive domains. For international training partnerships, the financial and logistical advantages can be significant, though focus should remain on the primary curriculum and training modality.