Free open access medical education: a critical appraisal of techniques for quality assessment and content discovery.

CONTENT DISCOVERY

Finding free open access medical education (FOAM) can be daunting for those new to this resource. This difficulty is reasonable as FOAM is entirely de-centralized without an individual platform from which content is delineated. However, there is a large and increasing amount of available content. According to Life in the Fastlane [1], the number of blogs and podcasts related to emergency medicine and critical care increased to 356 in February 2016. How is a novice user supposed to navigate these chaotic waters? The simple answer is to start exploring beginning with large well-respected websites and following along through included links and webrolls. Along with the 316 websites listed by Life in the Fastlane, the Social Media Index presents their 100 most impactful FOAM sites (Table 1) [2,3].

Table 1.

Top 10 sites per the Social Media Index 50 [3]

Site	Location
Life in the Fastlane	http://lifeinthefastlane.com
EMCrit	http://emcrit.org
Academic Life in Emergency Medicine	http://www.aliem.com
EMS 12 Lead	http://www.ems12lead.com
Dr Smith’s ECG Blog	http://hqmeded-ecg.blogspot.com
Emergency Medicine Cases	http://emergencymedicinecases.com
PulmCCM	http://pulmccm.org/main
CanadiEM	http://canadiem.org
REBEL EM	http://rebelem.com
the Nurse Path	http://thenursepath.com

Another resource, FOAM EM RSS (http://www.foamem.com) automatically posts recently published FOAM onto its website in real time. While they do not directly list the sites they gather content from, as of March 2016, they list 4,360 posts (www.foamem.com) [4]. In addition FOAM Search (http://googlefoam.com/#gsc.tab=0) allows the user to directly search for free resources by utilizing specific key words (http://googlefoam.com) [5]. Though their algorithm is not described, interestingly, their search engine also includes freely available journal articles. The Life in the Fastlane team also regularly presents their favorite recently published resources and can serve as a good review of recent FOAM (http://lifeinthefastlane.com/litfl-review-225/) [1]. Finally, twitter (https://twitter.com/) can be invaluable in notifying learners when useful or important FOAM is created. By following educators active on twitter, you can receive instant notification when content is released and even view the discussion on that content (https://feedly.com).

Keep in mind, most sites have a specific flavor—some focus on evidence-based medicine, others on critical care, others on toxicology, and more. As the learner continues to explore FOAM, he or she will develop their preferences for certain FOAM sites. Once the learner has their preferred list, tools such as Feedly (https://feedly.com/i/welcome) can notify you when new content is released.

QUALITY ASSESSMENT

Both Academic Life in Emergency Medicine (ALiEM) and a research group from McMaster’s University have attempted to evaluate FOAM to determine the reliability and reproducibility of its content and pedagogical model. To start ALiEM’s Approved Instructional Resources (AIR) series (http://www.aliem.com/aliem-approved-instructional-resources-air-series/) specifically addresses the issue of measuring quality in FOAM [4]. To evaluate online content, the AIR series executive board created a scoring rubric that demonstrated good inter-rater reliability as well as some evidence for external validity [5]. The tool is subdivided into five components [4,5].

First, a resource must clearly display its author, providing both the necessary credentials and demonstrating a willingness to stand by the information provided. The post must have references, preferably in-line, in order for the reader to easily verify the statements provided by the author. Second, a premium is placed on content supported or derived from evidenced based resources rather than pure opinion or polemical writing.

The third and fourth measures are more relevant to FOAM curation and content editing but may be less applicable to a specific learner. The AIR scoring rubric rewards posts with more numerous high quality pearls, though, if a learner had a narrow learning need, a post with less, or more specific, pearls may be preferred. Next, the Best Evidence in Emergency Medicine score is a validated measure for journal article impact and is theoretically applicable to the assessment of FOAM [6]. This inclusion, however, does skew towards newer, more practice changing posts, resulting in deflated scores for less impactful or newer topics such as core content.

Fifth, and perhaps most important is the assessment of accuracy. This is much more difficult for a learner to evaluate on their own, and, perhaps the crux of the problem. While journal articles and textbooks also have the potential for inaccuracies, there exists trust in the peer-reviewed process with its multiple layers of assessment to assure the potential for high and reliable quality. There have been some attempts, notably at ALiEM, to perform and present the peer-review process of particular content [7]. Additionally, most FOAM sites have a comment section, that can serve as a real time crowd-sourced peer-review process. This process occurs much faster than for classic peer-reviewed literature and is typically open for anyone to comment.

While no standardized method besides the AIR Series rubric exists to measure quality, other publications have investigated quality indicators for FOAM. In one systematic review 157 articles were analyzed yielding a total of 151 distinct quality indicators [8]. These were divided into three themes—credibility, content, and design—and then subdivided into subthemes. No attempts have been published to package these 151 indicators into a more applicable or easy to use structure [9]. The follow-up study did winnow these 151 indicators down to the 13 highest regarded quality indicators, which were: list conflict of interest, identify author, differentiate fact and opinion, accurate, universally available, references, statements consistent with references, differentiate between advertisement and content, transparency in creatorship, good quality, professional, relevant and useful, and well qualified author.