Which factors predict the time spent answering queries to a drug information centre?

OBJECTIVE: To develop a model based upon factors able to predict the time spent answering drug-related queries to Norwegian drug information centres (DICs). SETTING AND
METHOD: Drug-related queries received at 5 DICs in Norway from March to May 2007 were randomly assigned to 20 employees until each of them had answered a minimum of five queries. The employees reported the number of drugs involved, the type of literature search performed, and whether the queries were considered judgmental or not, using a specifically developed scoring system. MAIN OUTCOME MEASURES: The scores of these three factors were added together to define a workload score for each query. Workload and its individual factors were subsequently related to the measured time spent answering the queries by simple or multiple linear regression analyses.
RESULTS: Ninety-six query/answer pairs were analyzed. Workload significantly predicted the time spent answering the queries (adjusted R (2) = 0.22, P < 0.001). Literature search was the individual factor best predicting the time spent answering the queries (adjusted R(2) = 0.17, P < 0.001), and this variable also contributed the most in the multiple regression analyses.
CONCLUSION: The most important workload factor predicting the time spent handling the queries in this study was the type of literature search that had to be performed. The categorisation of queries as judgmental or not, also affected the time spent answering the queries. The number of drugs involved did not significantly influence the time spent answering drug information queries.

Impact of findings on practice

The workload model can be useful to predict the time needed to process answers in drug information centres (DICs), and also for evaluating and comparing changes in the complexity of queries over time.

Availability of high quality literature sources and skilled personnel are of importance to handle judgmental queries in DICs.

Introduction

Various methods have been proposed to measure the complexity, and thereby the time needed, to answer queries to drug information centres (DICs). Grace and Wertheimer proposed the term judgmental to categorise “queries that require the integration of data or knowledge and experience in the process of making a decision regarding a specific therapeutic problem” [1]. Furthermore, they assumed that answers to judgemental queries could not be given in any single reference source. Queries not fulfilling this criterion were described as non-judgmental. Several authors have later used this method to evaluate the complexity of queries received at DICs [2-5]. However, complexity of a query is not necessary related to its clinical impact. Cardoni et al. [2] found that 54 (11%) of the 491 queries received at a DIC were judgmental. However, they found that 329 out of 350 answers to patient-specific queries were considered to have provided useful information, and in 157 cases, the information provided by the DIC had positive effects on the patients’ outcomes. Based on these results, the authors conclude that it was invalid to measure the utility of DICs using a dichotomized judgmental/non-judgmental categorizing, only.

In another study based on queries to a DIC, judgmental queries required more time for review of the literature, integration of data and formation of an answer [4]. However, the authors did not take into consideration the assumption that judgmental queries could not be answered by the use of a single reference source to be valid. Romá et al. carried out a retrospective study in which three levels of request complexity were defined: Level I could be handled using readily available sources like tertiary resources (i.e. textbooks, monographs, etc.), and no data interpretation was necessary. Responses to level II requests required the use of less freely available sources, both primary (i.e. original studies etc.) and secondary (i.e. review articles etc.). This category also included evaluation and interpretation of data. Even more documentation was required to answer level III requests of complexity, and in addition, the data obtained should be adapted to the specific situation and used in proposing solutions in the response. Thus, Romá et al. [5] also considered the process necessary to answer the queries when categorising the complexity of them.

Davies et al. [6] used yet another method to classify replies to queries. Replies that required only simple statements of facts, for example the constituents of a tablet, were called factual. The replies that required clinical advice on a specific case and usually entailed discussion with the inquirer on the possible benefits and hazards of one or more courses of action were called consultative replies. The type of literature search needed was categorised as none, simple and complex, meaning that the DIC personnel could give the information and advice from memory, that reference to information sheets or basic texts was necessary or that a detailed bibliographical search had to be performed, respectively [6]. The extent of use of primary reference sources and more subjective methods based on a scale of difficulty rank have also been proposed as surrogate variables to assess the complexity of queries [7, 8]. Furthermore, a high number of drugs in the query could make it more difficult and time consuming to answer. The time used to process the answer has been suggested as a difficulty or a complexity factor when handling queries to DICs, and time might be considered as a standard when estimating the human resources needed to provide a reliable answer to a query [9].

Aim of the study

The aim of the present study was to develop a model able to predict the time spent answering queries to DICs. The model aimed to include at least some of the elements of complexity suggested by others, and also to represent a more comprehensive and practical tool to predict the time spent answering queries to a drug information centre.

Method

Setting and study population

At the time of the study, there were five regional DICs (RELIS) in Norway, serving health care professionals. The centres are organised within or in close collaboration with the departments of clinical pharmacology at five university hospitals. Pharmacists and clinical pharmacologists, as well as physicians of other specialities, answer problem-oriented drug-related queries and publish the queries along with the answers and reference sources in a web-based, full-text query-answer database (the RELIS database). This database is searchable and freely available to all health care professionals (www.relis.no/database) [10]. Schjøtt et al. [11] have previously described the organisation of RELIS and the methods used when answering queries. Each RELIS centre randomly assigned drug-related queries to employees that had accepted to participate in the study, until each of the participants had answered a minimum of five queries. The study was performed from March to May 2007, and a total of 20 employees participated. Queries solely concerning herbal medicines, and queries answered by sending copies of previous answers from the RELIS database only, were excluded.

Definition of workload and scoring system

The number of drugs involved in the query, the type of literature search performed, and whether a query was considered judgmental or not, were scored by the participants for each query in advance, and the scores were added together to define the workload score. The number of drugs involved in the queries was categorised as one (score 1), two (score 2) or three or more (score 3). Groups of drugs, e.g. antipsychotics, were categorised as three or more drugs. If the query could be answered without searching the literature and without consulting colleagues, literature search was categorised as not needed (score 1). If it was necessary to search the RELIS database, databases containing monographs like the Micromedex, the summary of product characteristics (SPC) for the drug, reference books and/or colleagues/other health professionals only, the search was categorised as simple (score 2). If searches in databases like Medline, Embase or Cochrane to obtain original articles were necessary, the search was categorised as advanced (score 3). In addition, queries were categorised as non-judgmental (score 1) or judgmental (score 3) according to the definition by Grace and Wertheimer [1], but modified according to Merrit el al. [4]. The use of judgmental as a weighted variable, giving a score of 1 or 3, but not 2, was based on our experience of increased workload associated with judgmental queries where different reference sources disagree in their conclusions or where the current evidence has to be adapted to a specific clinical situation. The scores of these three variables (number of drugs, literature search and judgmental or not) defined the workload score to a minimum value of three and a maximum value of nine. Finally, the participants registered the time they effectively spent answering the query. Any answer, including answers provided within the first 30 min, was rounded off to the nearest half an hour and included the time used by colleagues to discuss and evaluate the answer. Gender and working experience (less than 2 years or 2 years or more) within the DIC system were registered for each of the participants.

Statistical analysis

Simple and multiple linear regression analyses were used to assess whether the individual factors in workload or the total workload score could predict the time spent handling the queries. A Mann–Whitney test was used to assess if gender or working experience within the DIC system influenced the time spent handling the queries. SPSS version 15.0 (SPSS Inc., Chicago IL) was used for the statistical analyses. P values less than 0.05 were considered statistically significant.

Results

The participants returned 102 query/answer pairs. Six of these (5.9%) were excluded according to predefined criteria: Three queries were related to herbal medicines only, and three were answered by sending previous answers from the RELIS database only. Thus, in total, 96 query/answer pairs were included, and all the 5 individual regional DICs (RELIS) provided between 14 (minimum) to 31 (maximum) of the pairs. The mean time to process an answer was 4.0 h, whereas the median time was 3.0 h (range 0.5–24 h). The distributions of scores of the different variables included in workload are shown in Table 1. The median workload score was 7, whereas the mean was 6.6. One query (1.0%) scored 3 points, 36 (37.9%) queries scored 4–6 points and 58 (61.1%) queries scored 7–9 points (n = 95; one rating missing).

Table 1

The distribution of scores of the individual factors in workload of 96 queries to five Norwegian drug information centres

Variable	Score 1	Score 2	Score 3
	Number of cases (%)	Number of cases (%)	Number of cases (%)
Number of drugs	One	Two	Three or more
	37 (38.5)	29 (30.2)	30 (31.3)
Judgmental	No	–	Yes
	42 (43.8)		54 (56.3)
Literature searcha	Not needed	Simple	Advanced
	4 (4.2)	29 (30.2)	62 (64.6)

aOne case missing, n = 95

The main findings of the regression analyses are shown in Tables 2 and 3. Literature search was the individual variable best predicting the time spent answering the queries (adjusted R 2 = 0.17, P < 0.001) (Table 2). In the multiple regression analyses, the model that best predicted time was the one including the number of drugs, the type of literature search and whether or not the query was considered judgmental (adjusted R 2 = 0.22, P < 0.001), closely followed by the same model (workload) excluding the number of drugs (adjusted R 2 = 0.21, P = 0.001) (Table 3). There was no relationship between the number of drugs and the type of literature search (adjusted R 2 = 0.01, P = 0.81).

Table 2

Simple linear regression analyses of the relationship between the total workload score, or the different factors in workload, and the time spent answering queries to Norwegian drug information centres

Predictive variable	βa	95% CIb for β	P value	Betac	R 2d
Number of drugs (A)	0.65	−0.32, 1.62	0.186	0.14	0.01
Judgmental/non-judgmental (B)	1.39	0.62, 2.16	0.001	0.35	0.11
Literature search (C)	3.01	1.70, 4.33	<0.001	0.43	0.17
A + B	1.01	0.44, 1.58	0.001	0.34	0.12
B + C	1.37	0.81, 1.93	<0.001	0.45	0.19
A + C	1.44	0.66, 2.21	<0.001	0.26	0.13
Workload score (A + B + C)	1.12	0.66, 1.59	<0.001	0.44	0.19

a β unstandardised coefficients

b CI confidence interval

c Beta standardised coefficients

d R 2 adjusted squared coefficients (for the number of observations)

Table 3

Multiple linear regression analyses of the three factors in workload as predictors of time based on 96 queries to five Norwegian drug information centres

Predictive variables	βa	95% CIb for β	P value	Betac	R 2d
Number of drugs	0.70	−0.19, 1.59	0.123	0.15	0.19
Literature search	3.04	1.74, 4.34	<0.001	0.43
Number of drugs	0.53	−0.39, 1.45	0.255	0.11	0.11
Judgmental/non-judgmental	1.36	0.59, 2.13	0.001	0.34
Literature search	2.45	1.07, 3.83	0.001	0.35	0.21
Judgmental/non-judgmental	0.89	0.10, 1.68	0.027	0.22
Number of drugs	0.63	−0.25, 1.50	0.160	0.13	0.22
Literature search	2.50	1.13, 3.88	<0.001	0.36
Judgmental	0.85	0.06, 1.63	0.035	0.21

a β unstandardised coefficients

b CI confidence interval

c Beta standardised coefficients

d R 2 adjusted squared coefficients (for the number of predictive variables)

All the five regional DICs (RELIS) in Norway participated in the study. Of the participants, seven came from the western, four from the middle, three from the southern, three from the eastern, and three from the northern regional DIC. In total, 69 queries were answered by females (n = 14) and 27 by males (n = 6). Twenty queries were answered by participants with less than 2 years of experience (n = 5) and 76 by participants with 2 years or more of experience (n = 15) within the DIC system. There was a non-significant trend that females used longer time answering the queries than males (mean time per query ± standard deviation: 4.5 ± 4.4 h for females vs. 2.8 ± 2.3 h for males; P = 0.09). There were no significant difference in time spent answering queries between participants with less than and more than 2 years of experience (mean time per query ± standard deviation: 5.7 ± 6.1 h vs. 3.6 ± 3.2 h; P = 0.19).

Discussion

In the present study, we describe a model to predict the time spent handling queries to DICs. The use of a workload score introduced new elements in addition to the previously suggested classification of queries as judgmental or not. Furthermore, by relating workload, or the individual factors in workload, to the measurement of time we could test their respective importance in the handling process of queries. The results showed that in particular the type of literature search, but also the classification of the queries as judgmental or not, were predictors of the time spent handling the queries.

A high number of drugs was expected to increase the time spent handling the queries, but the regression analyses could not confirm this. The minor impact of the number of drugs could be related to our experience that a high number of drugs relatively often are related to queries about drug interactions. In such cases, easily accessible tertiary sources like handbooks or databases significantly reduce the complexity of a query [12, 13]. This finding is also supported by the fact that there was no relationship between the number of drugs and the type of literature search.

The variable judgmental also predicted the time spent answering the queries (Tables 2 and 3). Thus, we have confirmed the results from previous studies, indicating that judgmental queries in general are more time consuming than simple non-judgmental/factual queries [1, 4]. However, whether or not a query becomes judgmental, is probably also related the employees’ experience and available resources at the DIC. Furthermore, similar to queries about drug interactions, DICs can handle some judgmental queries, like drug use in pregnancy, within short time limits. In such cases, the lack of documentation in primary literature for many drugs is often compensated by several reliable tertiary sources that systematically categorise this information according to documentation of risk [14, 15].

Studies using the judgmental categorisation have reported between 4.6 and 53.8% of queries to DICs as being judgmental [1, 2, 4]. However, two studies using the consultative term defined by Davies et al. reported that 78.5 and 80.0% of the queries belonged to this category [6, 16]. Thus, the range for the percentage of consultative questions is smaller than the range for judgmental questions based on the results of these studies. The justification for using judgmental classification versus consultative and other classifications is not clear-cut, and this makes comparisons between different studies difficult. In our study, about 60% of the queries were found to be judgmental, and the distribution of workload scores was skewed towards the high scores. DICs in Norway are closely related to the Swedish, which have reported a particular high proportion of consultative queries [16]. Moreover, compared to the number of health care professionals in Norway, the amount of queries to DICs is low, suggesting that health professionals generally ask the DICs in the most complicated and time consuming cases. Thus, the present results reflect DICs with a high proportion of this type of queries.

The type of literature search was of particular importance for the time spent handling the queries (Tables 2 and 3). This finding could be related to the proportional association between literature search and subsequent need for analysis of the retrieved documentation. For example, an extensive search for original articles can be time consuming, and will also increase the time needed for reading and analysing the studies. Moreover, lack of available relevant documentation can also increase the time spent handling a query, as extensive literature searches may have to be done to make sure no information is overlooked. Finally, queries regarding adverse events, which are the most common category of queries to RELIS, might require searches for case reports or case series in databases like Medline or Embase, and such searches might be time consuming, depending on the availability of suitable search terms [17]. This finding implies the importance of availability of high quality literature sources for DICs to be effective.

The present study has some limitations. First, the choice of time as a standard could be criticised; as the relationship between complexity and this variable is not clear-cut (i.e. complex queries can be handled within a short time-period, as described above). Our categorisation of number of drugs, with three or more drugs in one category, could have masked the workload of queries with a particularly high number of drugs. The process of handling queries is also dependent on various human factors, such as the formal qualifications and the personal skills of the employees. The results showed that there were no differences in time spent answering queries between participants with less or more than 2 years of experience. One could have expected that participants with more experience would spend less time answering the queries. The cut-off for experience had to be set to a minimum of 2 years, as only one participant had less than 1 year of experience at the time of the study. Normally it takes about 3–6 months to learn how to handle a query to RELIS, and a cut-off at 3 or 6 months might have revealed differences between participants with more or less experience. Prior training related to drug information provision could be a useful variable to measure when our model is applied on other DICs with different organisation and training than RELIS. However, workload as a model cannot account for the importance of all individual factors that certainly influences both perception and handling of queries. However, in our opinion, time nevertheless remains the most useful quantifiable variable to relate the complexity of queries to.

Second, as the proportions of different types of queries differ between DICs the present results cannot automatically be generalised to all other DICs. We cannot exclude that the model e.g. would be less useful for DICs with a higher proportion of factual questions and fewer problem-oriented queries. In this case, use of tertiary sources like product monographs, textbooks or SPCs would provide quick answers to most of the questions. Whereas we report a median time of 3.0 h to process the answer to a query, Joy et al. reported that only 2% of the answers took longer than 90 min to process [7]. Merritt et al. reported that only 21% of the queries took more than 60 min to process [4]. These differences might partly be caused by more complex queries to RELIS, as more than 60% of the queries in the present study had workload scores of 7–9, and most of the queries were patient-specific. In addition, we document the answers in the RELIS database. Other DICs might not have implemented such a procedure and might thus not produce written answers, which are relatively more time consuming, to the same extent. Also, at other DICs, a substantial proportion of queries may come from consumers, and in a previous study two of three such queries were answered within 10 min [7]. In contrast, in the present study, all queries came from health professionals, mostly physicians and pharmacists. Furthermore, the five regional DICs (RELIS) in Norway collaborate closely, and use similar sources and methods for processing queries. Thus, the fact that individual DICs provided more query/answer pairs than others was not expected to influence the results.

Third, the highest adjusted R 2 values in this study suggest that the predictive variables explains only about 20% of the variance in time spent answering queries. However, this is not unexpected, as there are several individual factors (e.g. profession, professional experience, personal qualities, special areas of interest, availability of colleagues to discuss the problem with) of the employees affecting how drug-related queries are handled. There was a non-significant trend that females spent longer time answering a question than did males. This possible difference might have reached statistical significance if the material had been larger, although such a conclusion remains speculative until the finding is confirmed by others. It should also be noted that only one male had a shorter experience than 2 years within the DIC system, and it can thus not be excluded that the possible gender difference found might be affected by a difference in experience between males and females.

In the future it would be of interest to study whether workload as a model is sensitive to describe changes in complexity of queries over longer periods of time. If so the model could be used to assess the need for allocation of financial resources to DICs (more complex queries, need of more trained staff). In addition, with the potential limitations described above, the model could be suitable for comparing various DICs in both national and international settings.

Conclusion

The workload factors best predicting time consumption related to answering drug-related queries from Norwegian DICs included the type of literature search and the judgmental categorisation, whereas the number of drugs seemed to be of less importance. The type of literature search was the individual factor best predicting the time consumption. We suggest that the workload model can be useful to predict the time needed to process answers in DICs, and also for evaluating and comparing changes in the complexity of queries over time. Moreover, we conclude that availability of high quality literature sources and skilled personnel are of importance to be able to handle judgmental queries in DICs.