Are we making a difference? Outreach evaluation in practice.

Donald A.B. Lindberg, M.D., brought with him when he joined NLM an inquisitive mind, tech savvy, and new ideas. He was an early advocate of both outreach and evaluation innovation at NLM. Dr. Lindberg initiated and supported multiple pilot test and implementation projects to strengthen NLM's health information outreach to healthcare providers, research scientists, health science and hospital librarians, and the general public, including minority and underserved populations. He helped steer NLM's transition to the Internet, and NLM's development of a robust framework for evaluating Internet and Web-based health information dissemination and outreach to its many audiences. Dr. Lindberg's leadership led to numerous landmark accomplishments, including the capacity-building "Measuring the Difference" outreach evaluation Guide, and a multi-dimensional approach to Internet and website evaluation that placed NLM at the forefront of federal agencies using these new and emerging technologies to support their missions.

Introduction

Are we making a difference? That is a question we commonly ask ourselves in middle-age or, retrospectively, in our later years when the answer has become relatively clear. But it applies as well to organizations — and Donald A.B. Lindberg, M.D., Director of the National Library of Medicine (NLM) challenged the institution with that fundamental question not long after his arrival at NLM in 1984. Following the successful completion of the Library’s first formal long range planning effort that benchmarked existing programs against new opportunities, Dr. Lindberg wanted to know if we are asking how effective are NLM’s products and services [1]? Are we reaching the people who should be our users? Are we meeting their needs? What are the benefits and outcomes? How might our products and services be changed and improved in response? Dr. Lindberg was thinking specifically of NLM’s premier offering, the online MEDLINE (later to be remade as PubMed) database of biomedical literature citations and abstracts that is used all over the world and, at that time mostly by NLM-trained medical librarians who served as intermediary searchers on behalf of their end-user patrons.

The critical incident technique study of MEDLINE users

Dr. Lindberg suggested using the Critical Incident Technique (CIT) evaluation methodology to identify answers to some of these questions [2]. First used in World War II to identify characteristics and behaviors of successful bomber pilots, the CIT became widely used after the War by psychologists in personnel selection and in a variety of other settings; several in the medical community but none in the library community. Dr. Lindberg tasked Elliot R. Siegel Ph.D., then Assistant Director for Planning and Evaluation, to adapt the methodology for NLM’s use.

Siegel’s CIT study was designed to gather detailed reports of MEDLINE searches that were especially helpful (or not helpful) in carrying out the full range of professional activities of more than 550 physicians, scientists, and other professionals working in a variety of clinical care and other settings. Of these, two-thirds were direct users of MEDLINE who were purposefully recruited for the study; the remaining were academic health sciences and hospital librarians who recounted searches performed by them for others. More than eleven hundred CIT reports were systematically content analyzed from three different perspectives: why the information was sought, the effect of having (or not having) the needed information on professional decisions and actions, and the outcome of the search.

In clinical settings, the study documented that MEDLINE searches were being carried out by and for physicians to meet a wide diversity of clinical information needs. Rapid access to the biomedical literature via MEDLINE, for example, was critical to sound patient care and favorably influenced patient outcomes [3]. All interviews were conducted by telephone and typically lasted 30 minutes. The protocol benefitted greatly from Dr. Lindberg’s insistence that the first 50 calls be pilot tested and conducted personally by Siegel, rather than by the interview-skilled but non-medically experienced contractor’s staff, to sharpen and refine the interview questions. A valuable hands-on lesson that benefitted Siegel’s evaluation work in the years ahead.

Evaluation at NLM pre-Lindberg–a promising beginning

In the early 1980s, the traditional “wood” book catalog was replaced, and a microform system was installed in its place in libraries without the benefit of evaluation. It proved to be unpopular with both patrons and staff. Then NLM Director, Martin M. Cummings M.D., tasked Siegel with performing a comparative evaluation of two home-grown prototype online catalog systems in the NLM Reading Room, much to the consternation of some staff who were uncomfortable with the perceived intrusion by the front office and the prospect of discomforting patrons. The patrons never complained and seemed intrigued by the new computer monitors and the opportunity to help test a leading-edge technology that could make searching the collection far easier and more effective. A clear favorite amongst the two electronic search systems was identified. Its demonstrated reliability and superior functionalities proved superior to the other system, and of course to the static card catalog it permanently replaced [4].

Dr. Cummings was the inspiration for another notable early evaluation when he returned from an annual Congressional Appropriations hearing greatly frustrated that once again, he was unable to answer a simple question with any precision — Who uses MEDLINE? Cummings approached Siegel who created what turned out to be NLM’s first MEDLINE user study, an early precursor of the CIT study.

It was implemented in a far less rigorous manner; self-selected anonymous MEDLINE users responded to a brief two question online survey that identified their professional (or public) role and the intended purpose of their search. Staff resistance to the possibility of burdening searchers was overcome with the realization they would now have greater insight into the actual identity and needs of their users, especially those for whom the librarian-conducted searches were being performed. The study findings provided Dr. Cummings with a credible answer for the next hearing, much to his delight.

Investigators at NLM and at other institutions later undertook descriptive MEDLINE user studies of their own [5]. Many more would follow the pathfinding CIT study, benefitting from new analytic tools, technologies, and policies that evolved, and are recounted later in this chapter. An early example was a nationwide field evaluation of MEDLINE products on CD-ROM that was organized by NLM under Dr. Lindberg’s careful watch. He viewed the CD-ROM as a transitional technology soon to be replaced by Internet access. Nevertheless, the commercial products existed, and the Library needed to assure itself and the community of users that their technical performance and user acceptance met acceptable standards [6].

Dr. Lindberg’s influence – A mandate and a way forward

It was Don Lindberg and the CIT study he initiated and personally guided that would lead to the establishment of a robust tradition of evaluation research at NLM. Dr. Lindberg made evaluation an accepted and a necessary requirement to study our users and ask the hard questions of them and of ourselves as program managers — Are we making a difference? He expanded Siegel’s role to include responsibility for coordinating and guiding NLM’s outreach activities and evaluation research throughout NLM and, indeed, introducing new evaluation tools and methodologies to the entire National Institutes of Health (NIH) community as detailed in this chapter. Siegel was joined in 1995 by Frederick B. Wood, MBA, DBA who collaborated closely as lead evaluator in the newly established Office of Health Information Programs Development (OHIPD), within the Office of the NLM Director. Together, Siegel and Wood developed and coordinated new outreach initiatives that sought to include a strong evaluation component.

Evaluating outreach

Dr. Lindberg brought with him many new ideas that motivated staff to pursue exciting opportunities for change. Among these was the need for a comprehensive long-range plan, the first of its kind for NLM, that was completed in 1987 [1]. A variety of important program possibilities were identified, many of which were ultimately implemented and are discussed elsewhere in this book. One novel possibility subsequently recognized by Congress encouraged NLM “…to develop an outreach program aimed at… [the] transfer of the latest scientific findings to all health professionals … [7]’. The mission of NLM was also explicitly amended to add the function to “Publicize the availability of [its] products and services… [8]”.

This outreach function was pursued in 1989 as a pivotal update to the plan, which was known as the DeBakey Report, in honor of famed heart surgeon Dr. Michael DeBakey who chaired the outreach planning panel [9]. Its focus was on connecting unaffiliated health professionals in rural and underserved communities to medical libraries. When the Internet subsequently made possible access to NLM’s information resources without a telecommunications cost to the Library in the mid-1990’s, NLM’s outreach commitment was significantly expanded to include patients, families, and the general public.

Dr. Lindberg also strongly encouraged a focus on enhancing the capacity of underserved and minority populations to access health information, consistent with the societal goal of reducing health disparities amongst Black, Hispanic, and Native American populations. Selected outreach initiatives developed by OHIPD are discussed in the following sections of this chapter, along with the evaluation methodologies used to assess their outcomes and the effectiveness of the strategies employed in their development and implementation. Additional outreach initiatives carried out by NLM’s health sciences and public library partners are discussed in the accompanying chapter, NLM’s Library Network: A Force for Outreach [10].

Outreach evaluation: The first five years

With the availability of newly appropriated outreach funding in the five years between 1989 and 1994, NLM undertook and supported close to 300 outreach projects involving more than 500 institutions across the country. The key to these efforts was the close collaboration of the National Network of Libraries of Medicine (NN/LM).

In 1995, the OHIPD coordinated a comprehensive evaluation of these outreach activities [11]. The review found very substantial progress in engaging and training individual health professionals to use MEDLINE via the Library’s end-user-friendly interface, GRATEFUL MED. The number of user codes issued increased from approximately 30,000 to almost 100,000 during that time period. There were four million searches of NLM databases in 1989 vs. seven million in 1994. GRATEFUL MED users in 1989 accounted for less than one-third of the searches, in 1994 they conducted more than two-thirds of the searches.

These metrics were certainly very useful indices of impact, but with few exceptions these outreach projects did not get at the fundamental underlying question posed by Lindberg — Are We Making a Difference? — nor did they address the need for intensified outreach to minority populations, and the institutions serving their health information needs at the community level.

A significant part of the identified outreach evaluation challenge was that the library staff undertaking the outreach had little or no formal training in evaluation. They had limited ability to plan an evaluation project, and to articulate answerable questions that surpassed “counting heads”. Nor was there available an appropriate training resource that could fit their current skill set and bring them to the next level.

Measuring the difference: Guide to planning and evaluation health information outreach

We published an Outreach Evaluation Guide in 2000 as a capacity-building effort. It was a partnership of the NLM and the Pacific Northwest Regional Medical Library (PNRML) at the University of Washington. The lead author was Catherine Burroughs, MLS, at the PNRML, and the project officer was Dr. Wood. The project included an advisory group of community outreach and evaluation experts, several of whom prepared background papers. For an extensive list of advisors and staff contributing to this landmark effort, see the guidebook acknowledgements [12].

The underlying themes of the Guide were: planning and evaluating an outreach initiative is one and the same process; and asking the right questions at the beginning was essential to obtain useful results. Moreover, the Guide was practical in purpose, theory-based, and offered a range of methodological possibilities and strategies that could be adapted to the simplest or most complex of outreach projects. The NN/LM also established an Outreach Evaluation Resource Center (OERC) at the PNRML that could serve in a consulting capacity to member libraries using the Guide and needing hands-on outreach evaluation and related assistance.

As a follow-on to the original manual, the OERC issued a set of three shorter guidebooks (in booklet form) for key stages of outreach evaluation, initially released in 2006 and updated in 2013 [13]: The principal authors of the series were Cynthia Olney, Ph.D. and Susan Barnes, MLS, MS, both at the OERC. The booklets were intended to help address the need of NNLM outreach staff for stronger evaluation skills, with specific emphasis on the Logic Model methods of evaluation that provides a straightforward relationship between project goals, process, and outcome measures. For further discussion of Logic Model applications, see the companion chapters on the NLM’s Library Network: A Face of Outreach [10] and HIV/AIDS Community Information Outreach Program [14].

For further background on evaluation of community-based outreach projects, see the proceedings of the 2004 symposium on this topic co-sponsored by the NLM and NNLM, and especially the overview article, and an article by Dr. Charles Freidman on right sizing outreach evaluation and the use of smaller and incremental evaluations where appropriate [15-17].

Transitioning to the internet

The overall framework for outreach evaluation from the mid-to late 1990s onward increasingly reflected the emerging Internet platform for biomedical and health information dissemination. Dr. Lindberg was an early leader in the arena of health on the Internet. He developed a long-term interest in medical informatics and served as the first director of the Federal Government’s Office of High-Performance Computing and Communications (HPCC). For several years, Dr. Lindberg wore two hats, NLM Director, and HPCC Director.

Dr. Lindberg was well positioned to discern and foresee the coming Internet revolution and its profound implications for NLM. The Internet would transform how NLM conducted its core information dissemination activities and its outreach around those activities. And likewise, NLM’s evaluation activities needed to evolve and adapt to the Internet.

Under Dr. Lindberg’s leadership, NLM needed to address such topics as:

Access of minority, underserved, and minority communities to biomedical and health information via the Internet;

Technical performance of the Internet to assure that users are receiving NLM’s data and information in a timely, efficient, and accessible fashion;

Customer satisfaction with the NLM’s electronic information platforms, and increasingly NLM’s websites;

A comprehensive approach to monitoring and assessing NLM’s website and other electronic information resources.

Thus, the need for new or revised evaluation approaches that would work in the increasingly electronic customer service and outreach environment.

Outreach implementations and their evaluation–examples from the field

This section highlights several key community-based outreach initiatives that NLM implemented to address Dr. Lindberg’s outreach vision, and the means used to assess the effectiveness of their processes and the success of their outcomes.

Tribal connections

The five-year review of NLM outreach concluded that NLM outreach to Native Americans was too limited. In response, NLM reached out to Sherrilynne Fuller, Ph.D., director of PNRML and a committed advocate for Native American outreach. Tribal Connections became a long-running partnership between NLM and the PNRML and was one of NLM’s first major outreach projects with a Native American focus.

Tribal Connections was intended to use a community-based outreach approach to improve the Information Technology (IT) infrastructure and Internet access at select Indian tribes and Native villages and provide training in online access to NLM health information. Tribal Connections was designed to run in parallel with the aforementioned project to develop an outreach evaluation Guide. The intent was that Tribal Connections could benefit from and contribute to the evaluation methods project.

Tribal Connections Phase 1 reached 16 Indian tribes and Native villages in the Pacific Northwest (Washington, Alaska, Montana, Idaho, and Oregon). Each site was intended to have community outreach with one or more site visits, collaborative needs assessment, customized technical support, purchase of necessary IT equipment and services, Internet connectivity upgrades or partnerships as needed, and associated technical and other training. At the time of this project, Internet connectivity at the participating sites ranged from 56 Kbps dial-up, 128 Kbps ISDN, to full T-1. Each site plan was optimized for the baseline Internet connectivity and realistic upgrade options. For detailed discussion of Phase 1, see [18,19].

Later phases of Tribal Connections included the addition of four sites in the Southwest, more intensive training and outreach at selected sites, and a collaboration with medical libraries serving American Indians in the Four Corners Area (parts of Arizona, New Mexico, Utah, and Colorado) [20,21]. The latter project was known as Tribal Connections Four Corners (TC4C).

From an evaluation perspective, the projects early on relied heavily on performing needs assessments with local tribal leaders and key staff, whose engagement and understanding of the technical infrastructure improvements proved to be the single most important factor in determining a successful outcome. Internet connectivity was successfully established at nearly all sites, and training in its use was assessed which identified areas needing improvement. Implementations in the later project phases added measures to better understand the impact on information seeking behavior and actual health-related decision-making.

Dr. Lindberg requested periodic Tribal Connections briefings, several of which occurred at meetings of the NLM Board of Regents. Although not foreseen at the time, Tribal Connections helped lay the groundwork for the subsequent Native Listening Circles and Tribal Consultations that led to NLM’s Native Voices exhibition which is discussed in a companion chapter [22].

Health information hispanic outreach

The 1995 NLM outreach review also concluded that NLM outreach to Hispanics was limited and needed a boost. A key element of the NLM initial response was the “Health Information Hispanic Outreach” project in the South Texas Lower Rio Grande Valley. The NLM “HIHO” project was built in part on earlier NLM circuit rider outreach pioneered by the iconic Mary Jo Dwyer who made the rounds from San Antonio to visit small, rural libraires and hospitals in South Texas. She helped facilitate access to NLM health information, at first in paper and CD formats, and later via the nascent Internet. Under the direction of Virginia Bowden Ph.D., at the University of Texas Health Sciences Center at San Antonio (UTHSCSA), HIHO included a needs assessment, focus groups, and surveys. It identified additional outreach needs and two specific initiatives for further study: a high school peer tutoring project, and a Colonia Promotoras project [23].

The Peer Tutoring project emphasized the training of high school students at health-focused high schools to serve as “peer tutors” to other students. The peer tutors helped students understand NLM’s online health information resources, MedlinePlus, and MedlinePlus en espanol, and how to use the Internet and World Wide Web to access such information. Peer tutors also were coached on how to provide peer leadership and outreach to local communities, at events such as health fairs, to further enhance health information access. The school librarians and UTHSCSA librarians teamed with high school teachers and staff to provide guidance and training to the peer tutors. Most of the students in these schools were of Hispanic origin, and many would be among the first in their families to graduate from high school and go on to some form of higher education. The Peer Tutor project had an evaluation specialist from the outset, Dr. Olney, who assured that the “Measuring the Difference” manual was properly applied. Each phase of the peer tutor project was evaluated on a variety of process and outcomes measures [23-25]. For many peer tutors, the experience was life changing and helped motivate them on to community or four-year colleges. They continued to use MedlinePlus beyond their high school graduation, and to advocate for family and friend’s use of MedlinePlus. Dr. Lindberg very much enjoyed meeting with the peer tutors and school librarians when they visited NLM and presented at the NLM Board of Regents. Reaching out to and enabling the next generation of student and library leaders was always a priority for Dr. Lindberg.

The Colonia Promotoras project deployed Hispanic community outreach workers and health advocates, many of whom were bilingual [26,27]. The Promotoras served as intermediaries between NLM health information accessible at local computer labs, and low-income community residents of the Colonias which are unincorporated towns near the Texas border with Mexico. The Promotoras submitted written reports of the health topics they helped residents explore on MedlinePlus and the ways in which the residents used the information. These reports along with verbal interviews constituted a database of “stories” that were like those collected in the original NLM Critical Incident Technique study. Dr. Olney was the project evaluator.

Internet/web evaluation — A multidimensional approach

Under Dr. Lindberg’s leadership of NLM, evaluation was a watchword for understanding and assessing information technologies and services. The advent of the Internet and World Wide Web (WWW) are cases in point.

During the decade from 1996 to 2006, with Dr. Lindberg as an active helmsman, NLM went from the very beginnings of a framework for evaluating the Internet and WWW for biomedical information dissemination, to a robust and comprehensive framework.

This section recounts the development of an integrated, multidimensional framework for NLM website evaluation: The major dimensions of this framework include:

Internet technical performance;

Website usage data;

Website customer satisfaction data;

External benchmarking survey data; and

External benchmarking usage data [28,29].

Internet technical performance

The first evaluation component that NLM focused on was the technical performance of the Internet. New NLM policy encouraged use of the Internet to access the Library’s information resources from international locations. Soon enough, Dr. Lindberg was receiving complaints from colleagues in other countries, specifically the United Kingdom and Western Europe, about the slowness of the Internet traffic coming from the United States. Dr. Lindberg asked Dr. Siegel to investigate these concerns. Dr. Lindberg assembled a team including himself, Dr. Wood, and Victor Cid from NLM’s Office of Computer and Communication Systems (OCCS).

Dr. Wood was an electrical engineer with experience in technology studies and extensive prior work at the Congressional Office of Technology Assessment in the information technology arena. Mr. Cid, a computer scientist, had just completed a seven-year project with the BITNIS, a pre-Internet system for responding to user requests for information from NLM’s MEDLINE. BITNIS users originally were clustered in South America and later extended to other geographic areas. BITNIS ran on BITNET, which was a technical network using telecommunications and packet switching among then-dominant mainframe computers [30]. Mr. Cid was familiar with the Internet network architecture and operations at this very early stage. In 1995, BITNET was being phased out in favor of the then nascent Internet.

The OHIPD-OCCS team developed one of the first frameworks for monitoring and measuring Internet quality of service. At the time, there was no widespread commercial Internet performance monitoring system, but there were a few start up companies and engineering experiments.

NLM met with Internet engineering researchers and experts, and developed a system to monitor the bandwidth, speed, and routing of packets of information moving over the Internet. NLM’s system tracked packets between NLM’s computer center and host computers at partnering institutions in the U.S. and abroad. These were from the NNLM, and overseas from the International MEDLARS Centers which were distribution points for access to MEDLINE data stored in the participating home country’s library or information center. The MEDLARS Centers proved to be important for several years prior to and during the transition to the Internet.

The technical testing metrics and tools used by NLM included:

Transmission capacity, the bandwidth or size of the Internet “pipe” (aka, Bulk Transfer Capacity, megabits per second, using for example the TReno test);

Latency or delay (aka, Round Trip Time for packets of data to go from the sending to receiving location and back using the Ping test);

Network routing (aka., the number and sequencing of links or hops that the data take from origin to destination (using for example the Traceroute test). Note that physically a hop is a segment of telecommunications network between pairs of switching centers or routers through which packets of information transit to “hop” along an interconnected Internet pathway from sending to receiving locations.

With a test network at home and abroad, and using the metrics and test tools mentioned, NLM was able to answer Dr. Lindberg’s questions about Internet connectivity and performance.

The delays in Internet traffic between the U.S. and the UK and Western Europe were occurring not in the U.S. when Americans were sleeping, but from the UK/European side due to inadequate local infrastructure and the increase in their own Internet traffic associated with the beginning of their business day. This pattern held for several other countries as well.

Further testing showed that the speed of the Internet traffic depended on the bandwidth of the pipe, the time of business day at the sending and receiving locations, and the types of content being transmitted. This Internet performance evaluation approach was of great interest in the U.S., to the Federal Government, private companies, and academic institutions alike, and to international groups such as the G-7 [31].

Eventually, NLM migrated to commercial Internet performance companies, such as Keynote, that assembled their own monitoring networks and provide Internet performance data on a wide-scale basis. The data could be used for benchmarking, Internet capacity planning, and trouble shooting.

In 1998, the NLM team published a landmark paper on this topic in the Journal of the American Medical Informatics Association [32]. Also, the NLM team taught a continuing education class on this topic at the annual AMIA meeting. And Internet performance was included in other NLM and NNLM projects, see e.g. [33].

Internet performance and quality of service continues to be an integral part of Internet network management and engineering in the U.S. and globally, by the IT departments and/or commercial providers, for most organizations of all types and sectors, including the NLM OCCS.

Website usage data

In the late 1990s, NLM transitioned to website-based platforms for information dissemination and database access, using the rapidly expanding Internet. Dr. Lindberg was vitally interested in understanding the usage of NLM websites, as they became the primary information platforms for NLM.

Dr. Lindberg’s driving interest helped prompt the use of successive generations of web log software by most NLM websites. In order to meet Dr. Lindberg’s desire for usage data across NLM websites, the OHIPD coordinated a trans-NLM project to collect and make available key usage data for the larger NLM websites. The primary metrics selected were:

Number of website visitors;

Number of pages downloaded; and

Number of database searches (where applicable).

There was considerable debate about how many and which metrics to include, and how often to collect data. The consensus was to include a small number of required core metrics but allow individual websites to use additional metrics if needed. Most web log software then and now provide additional options, such as first time versus repeat visitors, time per visit, visits per use, and pages most frequently visited. The consensus was to collect data monthly but report on a quarterly basis. The NLM operating divisions assigned staff to manage the data collection and reporting. The data were accessible to all interested NLM staff via the NLM Intranet. This usage data collection and reporting framework worked well for many years.

At the same time, web log software and metrics continue to evolve, and NLM has periodically upgraded and transitioned to newer software packages and systems. NLM is well on its way to implement the type of website data management that Dr. Lindberg dreamed of years earlier but was not yet fully possible in his working lifetime. He would have been very pleased about the progress being made.

Web user satisfaction data

In the pre-Internet era, Dr. Lindberg was always interested in user opinions of and satisfaction with NLM’s health information resources and databases. Pre-Internet, NLM was able to conduct or sponsor surveys of library patrons, since most access to NLM information physically occurred in a library setting. In a transitional electronic environment, such as Internet Grateful Med, users were required to sign up and pay a connection fee for service. Thus, their names and contact information were known, and they could be contacted with in person, mail, or email customer satisfaction surveys.

NLM’s transition to the free Internet, with no sign up requested or required, fundamentally changed the customer feedback process. NLM no longer had user contact information, and thus could not use standard survey methodology. New, online survey methods were needed and developed.

NLM quickly found that customer response rates to online queries were much lower than in person surveys or surveys mailed to known customers. It was common to obtain 50 percent or even 75 percent response rate from paper surveys in library settings. In contrast, a typical response rage in online surveys was and still is typically five-ten percent at best, and frequently three to five percent. This raises serious questions about non-response bias.

NLM studied the options for online surveys and ended up heavily using the American Customer Satisfaction Index (ACSI) online survey services, for many years from about 2000 through 2018 [34].

Using the American customer satisfaction index (ACSI) to conduct online website user surveys

The ASCI had several advantages:

it was based on a rigorous survey methodology developed by University of Michigan survey and communications specialists,

it used a standardized methodology with core questions designed for website users;

it was offered at an affordable price via a U.S. Government-wide contract with ForeSee Results Inc. and

it had pre-approval from the Office of Management and Budget, which needs to approve all major U.S. Government sponsored surveys.

NLM found the ASCI via ForeSee Results to be more cost-effective than contracting for individual surveys. Also, the ACSI was good public relations for NLM, since some NLM websites, and especially MedlinePlus English and MedlinePlus en Espanol, were always among the best performers.

These results were shared with NLM management and, occasionally, the NLM Board of Regents.

NLM and some other NIH institutes and offices partnered to sponsor at the time the largest agency wide ACSI survey program in the Federal Government. About 60 NIH websites participated in the two-year project, with most funding provided by the NIH Evaluation Set-Aside Program.

The NIH team, co-led by NLM, conducted an evaluation that confirmed the value of online customer surveys.

However, after the special project funding ran out, more than half of the NIH participating websites could not be renewed due to insufficient funds.

NLM continued to use the ACSI for about a half dozen websites for many years, but eventually decided to phase out the ACSI and try other options. At the time of the lead author’s retirement, NLM had shifted to using Qualtrics as the primary survey provider, which provided more flexibility and customization compared to the ACSI, but did not provide the benchmarking possible with the ACSI. NLM also used newer versions of the Survey Monkey online survey.

The bottom line seems to be that Dr. Lindberg’s desire to have website user or customer feedback is still being fulfilled, albeit with changing survey platforms over time. Customer feedback is an important part of the NLM’s comprehensive website evaluation framework developed during Dr. Lindberg’s tenure [28,29].

External benchmarking

Dr. Lindberg also was interested in how NLM website performance compared with other websites, in other Federal agencies and in the private sector. In response, OHIPD pursued two avenues to obtain external data on online health consumer and physician use of and satisfaction with online health websites; and to obtain comparative data on health website usage [35].

Benchmarking NLM website data with external comparative data

Comparative customer survey data: NLM purchased access to the results of syndicated nationwide surveys of online health information consumers and health providers.

These surveys, known as Cybercitizen Health (consumers) and Taking the Pulse (physicians), were conducted annually with large stratified and randomized samples.

The surveys provided insights into how consumers and physicians behave online, what types of information they seek or provided, and their uses of this information.

Typically included were questions on the most frequently used information sources accessed by consumers and physicians. These data provided some sense of the relative frequency of use of comparative and competitive health information websites in NLM’s market space.

The Cybercitizen Health and Taking the Pulse surveys were first conducted by Fulcrum Analytics, which was later bought by Manhattan Research, which was later bought by Decision Resources, Inc.

Comparative web usage data: NLM purchased access to website usage benchmarking services that had their own large panels of online users that agreed to monitoring of their web usage. This allowed the collection of third-party usage data for different types and categories of users and websites.

NLM was particularly interested in comparative usage data in the following categories:

Federal agency websites.

Federal health agency websites.

Federal science agency websites.

Commercial and non-profit health information websites.

Subsets of disease and condition websites.

The first usage benchmarking service NLM used was known as PC Data, which was bought later by comScore Networks.

These syndicated survey and benchmark services are relatively cost effective since costs are spread over multiple websites and clients. However, after many years, NLM eventually decided not to continue these services, in part because, while the top line results were informative, they were relatively unchanged year to year. Thus, the perceived value added declined. While the services were phased out in 2017, they did provide a useful measure of comparative market intelligence not otherwise available during the formative years and full implementation of NLM’s presence on the Internet and WWW.

Conclusions

Among Dr. Lindberg’s many legacies, he put both evaluation and outreach on the NLM map. He had the foresight and perspective to understand early in his service as NLM director that evaluation needed to be an integral part of NLM’s health information outreach and dissemination portfolios.

Evaluation is a key to understanding and assessing how programs and projects are working, whether they are meeting their goals and objectives, and how they may be improved or modified or phased out going forward. And yes, assessment helps answer Dr. Lindberg’s initial question, “Are we making a difference?”

Within the outreach domain, two defining evaluation contributions developed during Dr. Lindberg’s tenure were: (1) a health information outreach program evaluation Guide and updated field manuals, along with a dedicated NNLM evaluation office that could support local outreach project managers where they are; and (2) a multidimensional approach to Internet and website evaluation that placed NLM in a leadership position at the forefront of federal agencies learning to use and benefit from these technologies.

The outreach evaluation manuals (aka field guides or booklets) are still in circulation and available on the NNLM website, and NLM is still supporting the current manifestation of the original Outreach Evaluation Research Center. OERC became the NNLM Evaluation Office (NEO and is now known as the NNLM National Evaluation Center (NEC) and has relocated from the University of Washington Health Sciences Library to the Northwestern University Feinberg School of Medicine, Galter Health Sciences Library & Learning Center.

The foundational multidimensional web evaluation framework is still largely operational in some form at NLM. It has managed to transition through multiple advances in the underlying information technology and evolution of web evaluation methods, metrics, and software.

Other components and elements of success of Dr. Lindberg’s evaluation legacy include: senior level leadership; adequate funding; staff support and training; and documented evaluation results (including peer-reviewed published papers where appropriate). With these ingredients, outreach evaluation at NLM can continue Dr. Lindberg’s evaluation legacy.