eGEMs: An Opportunity for Better Science.

eGEMs faces two early challenges in its effort to effort to disseminate knowledge and new ideas about the use of electronic clinical data for research: attracting readers and producing better science. The health services grey literature and the microchip industry are two areas of knowledge generation that provide important insight for how eGEMs can achieve these goals. In order to achieve its goals, eGEMs should aim to promote rapid sharing of ideas, engage sponsors and potential users of research findings early in the process, develop metrics for success to guide research efforts, and recruit diverse contributors. Success would allow for not just the generation of new research, but also new science, which has the potential to significantly improve patient outcomes.

AcademyHealth’s Electronic Data Methods (EDM) Forum, a project funded by the Agency for Healthcare Research and Quality (AHRQ), has launched a new “e-publication,” eGEMs, (Generating Evidence and Methods to improve patient outcomes). The aim is to publish and rapidly disseminate scholarly efforts that use electronic clinical data to “advance the science of comparative effectiveness research (CER), patient-centered outcomes research (PCOR), and quality improvement (QI).”1 On one hand this effort addresses a relatively modest, though expanding slice of topical and academic turf. However, it also offers an alternative for scholars to the traditional pace and audiences of peer-reviewed and academically oriented publication—a non-trivial consideration involving potentially complex trade-offs for authors and editors. The proximate test for eGEMs is whether it can attract researchers and readers. However, the larger and more significant question for the sustainability of this endeavor is how can this approach best contribute to better science?

There are many ongoing discussions within the health research and health publication communities about opportunities for improved dissemination to inform and foster discovery that I won’t attempt to summarize or address directly here. Rather, my intent is to supplement those discussions by briefly considering two additional and different areas of knowledge generation and dissemination that offer guidance for the eGEMs effort: the health care and health services grey literature and the microchip industry. Both have important implications for eGEMS in its quest to promote efficient communication among contributors and stakeholders about innovative uses of electronic data.

The “grey literature” is a major body of health and health services knowledge which has been in place and evolving for many years.2 It is not merely a scientific “minor league” or a vehicle to surface work that is not yet ready or below the standards of commercial academic publishing. The grey literature and the researchers and processes that produce it exist to meet requirements for timeliness, customization and stakeholder focus that are often not well served by academic publishers.3

Typical sponsors and producers of grey literature include government, business and industry, academia, foundations, professional and membership associations, private research organizations and consultants. Featured materials and reports can include, but are not limited to, technical and statistical reports, preliminary progress reports, subject syntheses and white papers, plus market research. While alternatively positioned, the grey literature often shares many contributors and readers with commercial publications.

Traditionally grey literature was circulated in printed form. However, access to the produced documents and their content is increasingly fostered by use of online and mobile media, often without need for subscription or other restrictions. Also, the “work product” being shared is evolving from discrete manuscripts, white papers, and essays to also include more dynamic and distributed formats such as blogs, comments, and even tweets.

At its best, grey literature reflects an environment that has evolved for directly addressing organizationally generated questions and meeting the sponsor’s requirements for thoroughness and timeliness. The relevance and rigor of grey literature products is fostered and reinforced through ongoing dialog, cooperation, and accommodation between sponsor and researcher during the knowledge generation process. The grey literature also commonly seeks to identify and promote collaboration among contributors and to foster spread of effective findings. Thus, many of the intentions of eGEMs have a close analog in the grey literature.

Obviously, a major contribution of academic journals has been the development and application of editorial and review standards, especially the processes of peer review and feedback, for ensuring the validity and utility of the content they convey. However, beyond the time requirements for full editorial and peer-review, concern has been raised that these approaches may themselves create bias toward both preferential publishing of positive findings4 and perpetuation of the topical preferences and priorities of editors and reviewers, rather than encouraging the innovative and at times disruptive ideas of investigators.

The structure and processes employed in producing and validating grey literature differ from commercial publication with purposeful adaption to meet the knowledge needs of the sponsoring organization. Approaches such as use of steering committees, expert panels, technical experts, and public comment are embedded during the discovery process to foster validity and completeness, and to meet key stakeholder requirements for timeliness, and adaptation of research design to specific locations, logistics and populations. However, as these processes are designed largely to meet specific sponsor needs as opposed to creating generalizable knowledge, availability of grey literature products may be limited and also incompletely reflected by common search strategies developed for the academic literature. Also, the quality and scientific rigor of some grey literature has been criticized as inconsistent, incompletely protected from conflicts of interest and bias, and inferior to science vetted in classical peer-review. Thus, grey literature contributions may be substantially discounted in academic career evaluations such as promotion and tenure review.

As eGEMs seeks to evolve a process that is both rigorous and relevant across a range of interested parties, it can learn from the collective and at times conflicting experiences of the contributors to the academic press and to the body of grey literature, ideally adopting key practices of both domains.

Other insights about the approach and impact of research result dissemination and awareness can also be gleaned from looking at other industries, especially in technology centered fields.

In a 2005 commentary, Andrew Grove offered a comparison of the knowledge development and sharing approaches of the microchip industry, where he has been a key innovator and leader, and the “health science/health care industry.”5 He positioned the remarkable evolution and success of the former as a source of creative tension for health researchers to emulate. He noted that the microchip industry construct of “Moore’s Law” that the number of transistors on integrated circuits doubles approximately every two years is a shared performance goal that guides and drives how microchip research is organized and managed.6 Meeting this expectation demands learning that is both rigorous and rapid. The key features of this rigorous and rapid learning are continuous innovation and reliable production of needed knowledge, both over a short time span and iteratively over many years.

Grove attributes the overall success of his industry to the ability to systematically create, through research oversight and planning, many coordinated “knowledge turns,” the time required to move from hypothesis to results that then lead to the next hypothesis. Key to overall management is the parallel but coordinated pursuit of a common goal—Moore’s Law—by researchers testing often diverse approaches, commonly of their own origination. Methods and standardized approaches for early detection of hypothesis failure are an important aspect of overall shared research design. Development and routine use of common criteria and tools to reliably and quickly gauge success reflects a significant commitment of as much as 10 percent of overall research investment. This contrasts with usual health care research processes that feature highly variable and less precise endpoints that often take a long time to be reached and may be difficult to measure.

Progress by all contributors is closely and collectively scrutinized. When failure of a hypothesis is likely, the finding is memorialized, the experiment terminated and resources rapidly redirected to another more promising concurrent test. Experimentation also occurs in a physical setting that anticipates eventual large scale production, so when an idea does achieve validation it can rapidly be pursued at large scale and incorporated into marketed products. While a knowledge turn in the microchip industry may occur in months, a similar degree of testing and vetting in health care often takes several years. Grove does note that some areas of health related research, such as drug discovery, have historically leveraged systematic parallel processing, standardized metrics and short knowledge turns to achieve a relatively high efficiency of discovery similar to what he promotes.

Thus, the consistent achievement of Moore’s Law over 40 years reflects:

the number and pace of concurrent and coordinated knowledge turns,

a predictable rate of collective success toward a shared goal despite expected failure of most initially promising ideas,

efficient standardized oversight and coordinated redirection of resources to the most promising ideas at the moment, and

an environment and processes that recognize and reward the timely demonstration of both positive and negative findings.

So, how do these alternative approaches to knowledge generation and sharing relate to opportunities for eGEMs? To find its niche and develop a “sweet spot” that is differentiated for both authors and readers, plus contribute to better science, a few considerations for editorial emphasis and recruitment of scholarly work drawn from these examples include:

The promotion of accelerated sharing of good ideas, addressed with appropriate rigor that have been shown to not work. Elevating a problem formulation at the earliest opportunity from “no evidence available” to “no evidence of effect” would be a huge service to other investigators and to advancing the field.

Preference for approaches that explicitly anticipate the issues of dissemination and spread within the experimental design. Examples could feature early and ongoing engagement of sponsors and eventual users in research design and discovery.

Pursuit of common and scalable metrics to guide collaborative and competing efforts – while probably short of Moore’s law, are there common metrics that would capture progress while attracting collaborators and fostering the sharing of ideas? (e.g. how does one best gauge the utility of approaches to Natural Language Processing?)

Recruitment of diverse contributors—Does eGEMs have a role in recognizing and disseminating selected grey literature that has achieved credible sponsor validation and approval? Are there specific editorial review principles that should be developed for this complementary work?

None of these would be simple. All may be controversial. Any would offer the potential for meaningful differentiation. A final suggestion, reflecting the commitment of sponsors to improve both the processes and communication of research, would be to give the “s” in eGEMs more meaning and prominence by envisioning the goal of the endeavor as Generating Evidence and Methods Science (eGEMs).