Characterizing physicians' information needs related to a gap in knowledge unmet by current evidence.

OBJECTIVE: The study sought to explore information needs arising from a gap in clinicians' knowledge that is not met by current evidence and identify possible areas of use and target groups for a future clinical decision support system (CDSS), which will guide clinicians in cases where no evidence exists.
MATERIALS AND METHODS: We interviewed 30 physicians in a large academic medical center, analyzed transcripts using deductive thematic analysis, and developed a set of themes of information needs related to a gap in knowledge unmet by current evidence. We conducted additional statistical analyses to identify the correlation between clinical experience, clinical specialty, settings of clinical care, and the characteristics of the needs.
RESULTS: This study resulted in a set of themes and subthemes of information needs arising from a gap in current evidence. Experienced physicians and inpatient physicians had more questions and the number of questions did not decline with clinical experience. The main areas of information needs included patients with comorbidities, elderly and children, new drugs, and rare disorders. To address these questions, clinicians most often used a commercial tool, guidelines, and PubMed. While primary care physicians preferred the commercial tool, specialty physicians sought more in-depth knowledge. DISCUSSION: The current medical evidence appeared to be inadequate in covering specific populations such as patients with multiple comorbidities and elderly, and was sometimes irrelevant to complex clinical scenarios. Our findings may suggest that experienced and inpatient physicians would benefit from a CDSS that generates evidence in real time at the point of care.
CONCLUSIONS: We found that physicians had information needs, which arose from the gaps in current medical evidence. This study provides insights on how the CDSS that aims at addressing these needs should be designed.

LAY SUMMARY

In this study, we explored the patient-related questions that clinicians have in their routine practice and for which they cannot find answers in published literature, clinical trials, or guidelines. We interviewed 30 physicians in a large teaching hospital, classified and analyzed their answers. We found that most of the questions were related to treatment, while others were related to diagnosis or public health and quality of care. Physicians who had been in practice longer and those who primarily practiced in inpatient settings had more questions. Such questions were often related to patients with multiple pre-existing conditions, elderly and children, new drugs, and rare disorders. We posit that analyses of observational data can be used to help clinicians answer such questions.

BACKGROUND

Evidence-based medicine aims to reliably improve patient outcomes and provide a solid scientific background for clinical decision-making., As opposed to variable “intuitive” practices, evidence-based medicine uses existing evidence (gold-standard randomized controlled trials and meta-analyses, expert reviews, and case reports). Instead of relying on individual experience, physicians can use the accumulated evidence from scientific studies and apply it to a specific clinical scenario to drive better patient outcomes. Evidence-based medicine facilitates clinical care standardization, which has the potential to reduce medical care misuse and overuse and eventually leads to healthcare costs reduction and improvement in effectiveness and quality of care.

On the other hand, current evidence has been reported to be inadequate or missing for specific clinical cases., For example, guidelines include a large proportion of recommendations based on expert opinion or case studies indicating a lack of reliable data to act upon and cannot always address physicians’ information needs.

Apart from insufficient evidence, general information needs have been studied since the early stages of electronic health record (EHR) systems adoption. Smith in 1996 summarized the studies related to doctors’ information needs, concluding that the prevailing part of the unmet information needs consists of treatment questions that are often complex and highly patient-specific. This finding was supported by Ely et al,,, who found that most of the immediate questions generated during consultations remain unanswered, mainly due to the lack of time. These studies did not distinguish between the information needs that could potentially be answered using existing evidence and those, for which no randomized clinical trials or clinical guidelines existed. While evidence exists, clinicians tend to use it to guide their decision-making., Nevertheless, it is unclear which part of the clinical questions remained unanswered due to evidence lacking.

After more than 20 years, a growing pool of evidence requires an up-to-date assessment of the current evidence utilization and its ability to cover physicians’ needs. More recent studies have mainly focused on specific cohorts of doctors: primary care physicians, family physicians,, and residents, while the other specialties have not been studied thoroughly. Meanwhile, specialty physicians usually face complex clinical cases that may shape additional information needs and remained unaddressed by the existing evidence. These studies addressed the immediate information requests mainly in outpatient settings where physicians have limited time and resources to answer their questions.,,,, Surprisingly, there is little knowledge on evidence utilization and sufficiency among specialty physicians or senior physicians working in secondary and tertiary care services.

In this work, we address this knowledge gap, shedding light on current physicians’ information needs that are specifically related to insufficient evidence and therefore cannot be answered within the evidence-based medicine paradigm. These needs are not only of theoretical significance: with the growing pool of healthcare data, it has become possible to drive its secondary utilization to guide clinical decision-making. Now, electronic health records and administrative claims data can be used to generate large-scale observational studies, which potentially can be fed into clinical decision support systems (CDSS) to guide clinical care in real time.

While there are many other CDSS, few of them have been adopted in routine practice. Among the other reasons, these systems might have been designed without a precise study of the current information needs and the proper target group identification. Unmet information needs to specify user scenarios and use cases, which drives the methodology behind such CDSS, their design and implementation. We intend to conduct a qualitative study of information needs arising from a gap in clinicians’ knowledge that is not met by current evidence to inform future CDSS, which will generate medical evidence specific to a concrete patient of interest. Such CDSS will be able to use this evidence to answer clinical questions that cannot be answered using current medical knowledge base.

MATERIALS AND METHODS

Unmet information needs

We used convenience sampling method to select 30 physicians affiliated with NewYork-Presbyterian/Columbia University Medical Center, a large teaching hospital in New York. Clinicians were selected based on their affiliation with no additional restrictions (30 participated, 8 declined participation). The interviews were conducted over a 4-month period in the location of physicians’ choice; one interview was conducted over the telephone. The open-ended semi-structured questionnaire included questions about perceived information needs and knowledge resources that physicians use to fulfill these needs (textbooks, electronic resources such as PubMed and commercial tools, clinical consultants, and pharmacists). During 1-h in-depth face-to-face interviews, we asked interviewees to provide the number of questions for which they found no appropriate or insufficient medical evidence, the time expended searching for evidence, and examples of the questions that occurred. We provided examples of possible questions from our practice as well as other participants’ scenarios. To facilitate recall, we provided scenarios related to different aspects of care (diagnosis of rare events or disorders, treatment strategies, quality of care, patient compliance).

We collected demographic data, number of years in practice, clinical rank, departmental affiliation, and clinical specialty. Physicians were additionally classified as practicing in an inpatient or outpatient setting based on the primary type of care they provide.

Thematic analysis

The interviews were analyzed by two independent investigators using deductive thematic analysis according to six phases defined by Braun and Clarke. First, we entered the transcripts into the N-Vivo data management program with manual semantic code identification, which was moderated to ensure their validity and consistency across all transcripts. By creating codes applicable to all the data, we reduced the volume of text for analysis and identified new structures to incorporate disparate responses. We applied cutting and sorting method to identify the low-level codes and then generated a coding framework to connect related codes together to discover themes that are closely related to physicians’ information needs. Merged codes formed themes that served as a basis for our themes of information needs related to a gap in knowledge unmet by current evidence. Each theme was refined to ensure proper association of its definition and name with the coded extracts and coherency across all interview transcripts. The final themes were adjusted based on the discussions with both interviewees and investigators. We achieved at least 80% inter-rater agreement between the investigators. Throughout the analysis, three co-researchers used pair debriefing to discuss disagreements and reach an agreement on the themes and subthemes.

Statistical analysis

We conducted additional statistical analyses to identify the correlation between clinical experience, clinical specialty, setting of clinical care, and the characteristics of information needs. As we intended to study clinical information needs, we excluded two interviewees as they were not engaged in active clinical practice. To approximate perceived information needs, we used the number of questions or clinical scenarios that physicians recall during interviews and three main themes of clinical needs (diagnosis, treatment, and public health and quality of care) that we identified based on our hierarchical set of themes.

We then compared the distribution of the types of questions across specialties working settings and clinical experience using the Pearson’s Chi-squared test. To assess if there was a difference in the information needs among specialized and non-specialized physicians and among physicians working in different settings, Wilcoxon rank sum test with continuity correction was used; Kruskal–Wallis rank sum test was used to test if there is a difference in information needs based on clinical rank or departmental affiliation. Significant differences were defined as those associated with a P-value less than .05.

RESULTS

We interviewed 30 physicians from a broad range of specialties: pediatrics (23.3%), general internal medicine (20%), nephrology (16.7%), cardiology (13.3%), neurology (6.7%), gastroenterology (6.7%), infectious diseases (6.7%), emergency medicine (3.3%), and intensive care (3.3%). On average, physicians have spent 13.4 ± 8.3 years in clinical practice and most of them have an academic appointment (77%): assistant (52%), associate (39%), or full professor (9%). Twenty-three percent of clinicians indicated outpatient practice as their main working setting and 77 predominantly practiced in inpatient settings. Clinicians raised on average 4.3 ± 2 questions per interview.

Twenty-seven physicians in our study said that they use evidence knowledge sources routinely. Others listed socio-economic determinants, patient compliance, and evidence irrelevance to the real-world practice as the obstacles to evidence use. Low patient compliance was said to influence medication prescribing and duration of therapy. For example, shorter courses of treatment or modified drug regimen were sometimes more convenient for patients but deviated from the current standards of care. Low income and hospital remoteness were also said to influence patterns of prescribing, favoring aggressive and short therapy. Some clinicians identified clinical guidelines and clinical trials as “overcautious” and prescribed formally contraindicated drugs as they had not observed listed adverse events in their practice.

Ninety-one distinct clinical problems related to absent or insufficient medical evidence were raised during the interviews. Figure 1 shows the themes and subthemes of information needs related to a gap in knowledge unmet by current evidence, organized according to the clinical scenarios that gave rise to questions.

Figure 1.

Themes and subthemes of information needs arising from a gap in clinicians’ knowledge that is not met by current evidence. Green boxes represent themes and subthemes and a bracket with blue boxes represents broad topics applicable to included subthemes.

The final hierarchy comprises 15 end leaves and 3 main themes: treatment, diagnosis, and public health and quality of care (Figure 1). Majority of questions regarded treatment (81% of all questions), while others involved diagnosis and public health and quality of care (11.6% and 7.4%, respectively).

Almost all of the questions in the treatment group were related to drug treatment. We obtained only a few non-drug treatment questions, which did not allow us to specify this theme further. The information needs related to drug therapy further fell into four main subthemes based on the steps in drug prescribing: decisions on drug necessity, choice of a drug and its dose, and decisions on when to discontinue treatment. Drug choice included three subthemes and had the broadest range of possible clinical scenarios that were not covered by the existing evidence.

Clinicians reported that the current studies appeared to fail to provide evidence for newly marketed drugs (Should a diabetic patient on ACE inhibitors, diuretics, and SGL2 inhibitors be taken off diuretics as SGL2 inhibitors act as diuretics?). The available evidence also was said to inadequately cover certain populations such as pregnant women, children, elderly, and patients with multiple chronic conditions and rare disorders. Such questions were usually formulated within the Population, Intervention, Comparison, Outcome, Time (PICOT) framework, where population corresponded to a group of patients with conditions of interest, intervention, and comparison—drugs or procedures that were intended to be used for a patient of interest, outcome—to a disorder or event to be used to compare interventions and time—to time-at-risk. Using this framework, we can see that some questions shared common population or intervention. For example, chronic kidney disorder fell into multiple subthemes: “Comparative effectiveness” (Which one of the new SGLT2 inhibitors is best for patients with chronic kidney disorder?), “Indication and Contraindication” (Do we know if rivaroxaban or apixaban should not be prescribed for patients with kidney failure in real practice?), “Drug Safety and Adverse Events” (In patient with chronic kidney failure and hypertension taking five drugs, how do we know which one caused an adverse event?), and “Drug Dose” (Tacrolimus for kidney transplantation: what is the start dose, how often its level should be measured in blood?). There was a small number of questions that do not fall into the PICOT framework (eg, “What are the risk factors for vancomycin-induced kidney injury?”) that were mainly related to characterizing patients of interest.

The “Diagnosis” theme mainly included questions related to laboratory tests as well as vague symptoms and syndromes. Finally, we classified questions related to screening, prognosis, and ancillary services into one broad “Public Health & Quality of Care” theme. We provide examples of questions in Supplementary Table.

While most of the questions were unambiguous and straightforward, others required additional clarification. For example, one clinician asked a question that at first glance could have been interpreted a diagnostic question: “How should patients be screened for dementia?” Further discussion revealed the rationale behind the question, which allowed us to classify it into the “Screening” theme: “It [dementia] often is confused with depression especially if a person has a history of depression. We need this information to properly educate primary physicians on how to take care of such patients.”

Additional analysis

Figure 2 presents the knowledge source preferences among the interviewed physicians.

Figure 2.

Knowledge resource utilization. The inner circle shows the primary knowledge source that clinician used, and the second circle shows the secondary source.

More than half of the physicians (60%) indicated an unspecified commercial tool as their primary source of evidence and only 10% turned to guidelines first to answer their questions.

Table 1 shows the distribution of primary source of evidence, number of questions, and types of questions depending on primary working settings, specialization, and academic rank.

Table 1.

Distribution of primary source of evidence, number of questions, and types of questions depending on primary working settings, specialization, and academic rank

		Primary setting		Specialization		Academic rank
		Inpatient	Outpatient	Primary	Specialty	1a	2	3
Primary source of evidence	Commercial tool	8 (38%)	6 (86%)	7 (70%)	7 (39%)	5 (71%)	5 (41.7%)	4 (44.4%)
	Guidelines	2 (9.6%)	1 (14%)	2 (20%)	1 (5.5%)	1 (14.5%)	1 (8.3%)	1 (11.2%)
	PubMed	9 (42.9%)	0 (0%)	0 (0%)	9 (50%)	0 (0%)	5 (41.7%)	4 (44.4%)
	Other literature	1 (4.8%)	0 (0%)	0 (0%)	1 (5.5%)	1 (14.5%)	0 (0%)	0 (0%)
	None	1 (4.8%)	0 (0%)	1 (10%)	0 (0%)	0 (0%)	1 (8.3%)	0 (0%)
Average number of questions		4.9 ± 2.2	2.7 ± 1.1	3.4 ± 1.6	4.8 ± 2.3	2.9 ± 1.1	4.2 ± 1.8	5.7 ± 2.6
Average number of questions per type	Diagnosis	0.4 ± 0.6	0.9 ± 0.7	0.7 ± 0.7	0.4 ± 0.6	0.5 ± 0.5	0.3 ± 0.5	0.8 ± 0.8
	Treatment	4.2 ± 2.0	1.4 ± 0.9	2.3 ± 1.8	4.2 ± 2.0	2.3 ± 0.9	3.7 ± 1.8	4.2 ± 2.9
	Quality of care	0.3 ± 0.6	0.4 ± 0.8	0.4 ± 0.7	0.3 ± 0.5	0.1 ± 0.4	0.2 ± 0.4	0.7 ± 0.8

1—none, 2—Assistant Professor, 3—Associate Professor.

We found no correlation between the primary source of evidence and primary clinical settings (X-squared = 6.16, P = .29) or academic rank (X-squared = 18.67, P = .54). On the other hand, we found a difference in specialty and primary care practitioners: primary care physicians preferred the commercial tool (X-squared = 9.85, P = .08), while specialty physicians used PubMed, guidelines, and the commercial tool.

We found that the number of questions related to insufficient evidence in clinical practice did not decrease with clinical experience, and in fact appeared to increase with experience (R = 0.55, P = .0023, Figure 3).

Figure 3.

Correlation between the number of questions and clinical experience.

We discovered that physicians who mainly work in an inpatient setting identified more information needs (W = 120.5, P = .01). The number of questions was unrelated to their specialty and did not change irrespective of whether they were primary care or specialty physicians (W = 58, P = .1).

When we analyzed the distribution of type of questions based on our themes, we found that inpatient physicians had significantly more questions related to treatment (W = 134, P = .001). On the other hand, the type of questions was unrelated to clinical experience or academic rank.

DISCUSSION

In this study, we found a significant gap in clinical evidence, which resulted in a large variety of information needs across different specialties and care settings. The overwhelming majority (90%) of physicians in our study indicated that they practice evidence-based medicine and use evidence knowledge sources routinely. The others all identified similar barriers to evidence use in real-world practice. The main obstacles to applying evidence in practice were low patient compliance and socio-economic determinants along with mistrust of clinical guidelines and clinical trials. While other studies, listed lack of time, personal unawareness, disengagement, and passivity as the strongest barriers, our findings just partially support it. Indeed, outpatient physicians could rarely find time for literature review and usually thought of their clinical scenarios as routine and straightforward. On contrary, the length of inpatient stay might give physicians more time to search for evidence. They also tend to face more challenging cases, which oftentimes requires more thoughtful research, team collaboration, and experience sharing. This may imply that inpatient physicians are more likely to practice evidence-based medicine and use additional knowledge sources to find additional evidence for their clinical cases.

Surprisingly, we found that experienced physicians and tertiary care practitioners not only have questions that cannot be answered using existing evidence but also have more information needs arising from a gap in current evidence than those who have less experience. The focus of previous studies was shifted toward the most vulnerable groups: residents, family, and primary care physicians,,,,,, assuming that these categories require information the most. We have discovered that the need for clinical evidence does not diminish with physicians’ maturity: while residents and practitioners at the early stages of their career may experience lack of clinical expertise and knowledge, more senior physicians have already gained this expertise and oftentimes require more complicated and precise evidence. This observation relates to the Dreyfus model of skill acquisition, according to which one progresses through five levels of proficiency: novice, advanced beginner, competent, proficient, and expert. At the expert level, clinicians no longer have to rely on the set of rules or analytical principles that they had been taught but use their background experience to “intuitively” make decisions. On the other hand, experts can identify knowledge gaps and acknowledge their relevance to clinical scenarios. Constant search for evidence when no established practice exists seems to be a feature of experts, who tend to apply analytical problem solving to novel or complicated clinical cases.

We classified such clinical cases based on the main areas where physicians struggled to find answers (Figure 1). These broad areas included multiple comorbidities, rare disorders, polypharmacy, elderly, and young patients. Our findings align with previous studies, which reported that the patient comorbidities and contexts play an important role in inability to answer clinical questions at the point of care.,,

When analyzing narrow question themes, most of them were related to drug therapy, which may reflect current paradigm of extensive drug use in medical society. This trend in information needs has remained unchanged for over 30 years and is well supported by the recent studies.,, Moreover, the pull of unanswered questions grows with an increasing number of newly marketed drugs and poor generalizability of clinical trials. The distribution of the types of questions also reflects this paradigm: while 30 years ago the percent of medication-related questions was around 30%,, in our study we saw that it doubled.

We observed that physicians practicing in inpatient settings were more likely to ask questions related to treatment and diagnosis, while most of the questions related to quality of care or public health-related questions were asked by the outpatient physicians. Due to its nature, outpatient care tends to be more influenced by non-clinical factors, such as socio-economic determinants or patient compliance and adherence. When prescribing treatment or diagnostic procedures, healthcare provider should account for patients’ capacity to pay for their treatment and diagnosis, compliance with medications, and follow-up appointments and other factors., For example, one of the questions (Should we prescribe Vitamin D in children knowing that compliance may be low?) considered the relative effect of the Vitamin D tablets that are prescribed once a week. As such regimen was observed to be associated with forgotten and missed doses, the clinician required additional information on its efficacy in real-world practice. As outpatient care requires relatively fast and cheap diagnostic procedures, outpatient practitioners may be interested in evidence related to symptoms and syndromes, which would allow to interpret routine diagnostic procedures more effectively (Can obesity inflammation alone get erythrocyte sedimentation rate up to a very high level?).

When searching for evidence to answer these questions, the commercial tool was the main knowledge source used by physicians in this study, which aligns with the previous findings., It was said to be used for the new drugs, unfamiliar rare disorders, and disorders outside of physicians’ specialty. Other physicians, especially those who preferred PubMed, characterized the commercial tool used as a superficial knowledge base that did not provide case-specific information or in-depth knowledge (It usually tells me what I already know. I may use it to browse new drugs, but otherwise look for new studies). Cook et al also reported its inability to answer complex questions, which, together with our findings, may explain why specialty physicians preferred to use PubMed. The choice of primary knowledge base may also indicate the predominant characteristics of information needs arising from a gap in current evidence in specialty physicians. The latter may require evidence that addresses specific patient contexts rather than the general knowledge about a disorder and its treatment. Therefore, specialty physicians may benefit from an additional knowledge source that accounts for these needs and provides them with the evidence for a specific clinical case.

These observations support the previously observed inferential gap between the guidelines and trials (which show effectiveness for selected groups of patients) and a real patient with a complex of individual disorders, medical history, and non-clinical characteristics. The breadth of this gap depends not only on the relevance of available evidence and patient-specific factors but also on the knowledge availability. Lack of evidence and its dissemination among colleagues shapes the current information needs. Not surprisingly, physicians within a specialty share the same needs and attempt to answer them based on their observations and experience. For example, we saw that some of the same questions in nephrology occurred across multiple interviews with physicians trying to answer these questions on their own. This may lead to the disparate solutions to the same clinical problem even within a single institution and, eventually, to suboptimal patient care.

Lack of evidence and inability of existing evidence to answer complex real-world clinical questions points to a need for a CDSS that can address clinicians’ information needs arising from a gap in current evidence at the point of care. As noted in other studies,,, lack of time and difficulty remembering questions had been contributing to the pool of questions that remained unanswered by clinicians even despite the availability of CDSS and knowledge resources. Thus, a CDSS should allow obtaining relevant and concise answers in a timely manner.

Another important aspect of CDSS design is proper target group identification. Specialty physicians, compared to primary physicians, used more knowledge sources and may have more time to search for additional evidence. Specialty physicians also reported an increased need for a knowledge source that helps to answer complex context-specific clinical questions, which implies that they are more likely to benefit from such CDSS. As we observed an association between clinical experience and the number of questions, experienced clinicians may also use the system that provides answers to these questions.

With the widespread use of EHR systems and modern initiatives like Observational Health Data Sciences and Informatics, we can bring new services to provide real-time evidence from large-scale observational studies. Such service (Data Consult Service) can generate evidence at the bedside to facilitate clinical decision-making and guide clinicians. Using EHR and administrative claims data and domain expert knowledge, such CDSS select a group of patients similar to the patient of interest and use statistical approaches to compare it to another group that represents an alternative scenario. For example, the question “Which one of the new SGLT2 inhibitors is best for patients with chronic kidney disorder?” can be answered by conducting a comparative retrospective new-user cohort study using institutional EHR data. In such study, patients with chronic kidney disorder are classified into a target or comparator group based on their SGLT2 inhibitor exposure, balanced using propensity score adjustment and analyzed to examine relative risk of an outcome of interest. Data analysis can vary from simple descriptive statistics to comparative effectiveness studies, but generally allows one to learn from previous patient care. For example, the question “How do we interpret Synacthen test in patients on long-term steroid therapy?” (diagnosis theme) can be answered by characterizing patients on long-term steroid therapy and comparing their outcomes based on the length of therapy and their adrenocorticotropic hormone level (measured using the Synacthen test).

It may seem paradoxical that we exclude novices from the target group for such system. While novices may have more broad general information needs and may struggle with applying evidence-based practices, they would rather require guidance on applying existing knowledge to practice. On the other hand, Data Consult Service would be beneficial to clinicians who routinely apply evidence-based medicine practices, identify gaps in current evidence, and deal with complex clinical cases.

As these clinical cases are often complex and non-trivial, the description of information need may require further clarification. As we saw, the rationale behind the question may differ from the actual question formulation. When designing a CDSS that aims at answering such questions, one must account for their meaning and wording. For example, the question “How should patients be screened for dementia?” (Supplementary Table) may be interpreted as a question about appropriate tools for dementia screening. Nevertheless, the further discussion with the clinician revealed that the question considered common misdiagnosis of dementia patients with a prior history of depression. We, therefore, would rather transform it into the comparison of two groups: patients with a history of depression who were diagnosed with dementia and patients with a history of depression and no dementia. These two groups can then be compared to identify the distinctive features of patients who had had depression and developed dementia. The identified features can then be used by clinicians to promptly diagnose dementia. Such difference in interpretations has implications on study design, the volume of EHR data needed to conduct a study and eventually on the ability to address the question. All the relevant details should be taken into consideration and a system should allow for further question interpretation.

Limitations

Our study has several limitations. First, we assumed that the number of questions physicians raised during the interviews could serve as an approximation of their information needs arising from a gap in current evidence. We believe that the time-consuming and effortful nature of literature search has made mentioned clinical cases memorable, which provides sufficient accuracy for our approximation. We did not cover all clinical specialties, but the variety of interviewed specialties allowed us to get a broad picture of information needs.

We conducted our study in a single institution, which only represents the unmet information needs in a large tertiary care hospital and does not characterize clinicians’ information needs elsewhere.

CONCLUSION

In this study, we examined and classified the information needs arising from a gap in current evidence of physicians in a large teaching hospital. We found that the information needs do not diminish with clinical experience, and in fact, the number of questions that cannot be answered with the current evidence increases on average with experience. The information needs were mainly related to drug treatment, specifically for new drugs, elderly and children, and patients with multiple comorbidities. Current medical evidence appeared to be inadequate in covering these patient populations and sometimes irrelevant to complex clinical scenarios.

We identified possible target groups for a CDSS (Data Consult Service) that aims at generating additional evidence for such scenarios and providing clinicians with the answers to their questions at the point of care. Although it remains to be shown, we believe that such CDSS would be beneficial for experienced, specialty, and inpatient physicians. When designing such system, one should account for the rationale behind the question and enable producing answers in a timely manner.

FUNDING

This work was supported by the National Institutes of Health grants U01 HG008680, R01 LM009886, R01 LM006910, and T15 LM007079.

AUTHOR CONTRIBUTIONS

AO, RC, and GH contributed to study design, implementation, data analysis and interpretation, and paper writing. LZ contributed to study design and paper writing.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

CONFLICT OF INTEREST STATEMENT

None declared.

Click here for additional data file.