Assessment of Search Strategies in Literature-Review-Based Candidate Theses Within a Nursing Program.

BACKGROUND: The majority of candidate theses in baccalaureate nursing programs in Sweden are written as literature studies. Being able to carry out a systematic and structured literature search is an essential part of thesis-related work. AIM: The aim of the current study was to investigate changes in nursing students' search strategies in candidate theses.
METHODS: A retrospective, quantitative study design was obtained. Librarians (n = 2) and teachers (n = 4) randomly examined selected candidate theses (every third thesis, n = 89) from the years 2012, 2014, and 2016.
RESULTS: The result showed a significant improvement over the years (from 2012 and 2014 to 2016) regarding the use of a sufficient number of synonyms, matching search terms to the respective database, use of the Boolean operator OR, and the use of subject headings and free text searches. Use of the title/abstract search largely disappeared. There was a significant change in the types of searches being done. The searches have become more structured in later years as the use of block searches increased significantly; in other words, more systematic and relevant searches have been done in recent years.
CONCLUSION: The result of this study shows that the quality of the students' search strategies improved significantly during the studied years. It is recommended that search documents are used in both formative and summative assessments to evaluate students' search strategies. Educational development in the form of enhanced collaboration between librarians and teachers in nursing programs is recommended because it might help to develop student search strategies in literature-based candidate theses.

Introduction

In students’ future occupational profession as registered nurses, it is important to develop skills and ability to search and critically review scientific literature since nursing and healthcare should be based on evidence-based knowledge. Therefore, nursing educators and librarians put much effort into teaching information literacy. Most candidate theses in baccalaureate nursing programs in Sweden are written as literature studies. Being able to carry out a systematic and structured literature search is an essential part of thesis-related work and students’ skills in search strategies must be evaluated continuously. There is, however, a lack of studies that focus on students’ search strategies and the quality of these strategies. Therefore, there is a need for further research in the development of search strategies in literature-based work within nursing programs.

It is relevant to document the search strategies of candidate theses in nursing programs to increase evidence-based practice (EBP). There are many definitions of the different steps contained in EBP, and Ciliska,4 has formulated the following EBP-related steps: Asking a clinical question (1), searching the literature for relevant research (2), critically appraising what has been found (3), implementing the change in practice (4) and evaluating the change in practice (5). The entire EBP process includes information literacy, such as searching, gathering, evaluating, and using information. Step two includes the information-retrieval process itself. A systematic search aims at finding all relevant documents for a certain purpose. Block search technology is a prerequisite for a successful systematic search. This means that students identify meaning-bearing concepts from the research question, and then create so-called blocks for each meaning-bearing concept. Each block search must contain all of the keywords and terms that are required to capture all relevant studies. Techniques in a block search include the subject heading searches and free text searches, both narrow and wide searches, truncation and phrase searches, and the use of different Boolean operators (AND, OR, and NOT). Not only the searches but also the entire workflow needs to be systematic. It is expected that exclusion criteria and limitations be documented.5,10

Systematic searches are used in medicine and nursing sciences for systematic literature studies and health technology assessments (HTA), and there are a number of instruments designed to support researchers during this work process. Examples of such instruments include PRISMA,11 AMSTAR,21 ENTREQ,24 and the Cochrane Handbook.5 Research support for nursing research and EBP is strong in Swedish nursing education,23 which makes the literature search extremely important. According to the Swedish Society of Nursing’s Qualification Description for Registered Nurse,22 a nurse should be able to “keep up with the development of knowledge” and “be able to systematically seek, critically evaluate and compile scientific literature”. Swedish laws and regulations, such as The Higher Education Act and The Higher Education Ordinance, contain similar writings about the expected learning outcomes for nursing students. In particular, The Higher Education Ordinance defines the knowledge and skills in information literacy that future nurses need to contribute to the development of the profession and their work.18,19

In nursing education, a so-called search document for documenting the searches used in a literature study is often used. The search document shows (in detail) how the search has been performed and provides a unique opportunity to evaluate the search strategies the students have used for their thesis. The search document is used both for formative and summative assessments of student searches. Through the search document, the supervisor, examiner, fellow students, and librarian can follow the students’ search process during the course of the work. Students can thus get feedback on the searches they have performed, both in the writing of the project plan for the thesis course and during the middle and final seminars and at supervisory times.

Previous Research

A few studies discuss the examination of candidate theses as the end products of student work in nursing programs. The study subjects of candidate theses range from purpose of studies to discussions and conclusions. In the case of searches, Langius-Eklöf and Forsberg,9 have examined whether the search terms were relevant, if the description of the literature search was approved, and if the searches were sufficiently well described to be replicated. The result showed that two-thirds of the theses had relevant search terms and approved descriptions, while shortcomings were found in how the searches were described. Kapborg and Berterö,6 investigated whether the literature is scientific, current, and international. In these two studies, the students’ approach to the selection of literature and how the searches were performed were not reviewed, but they touched on the approach on which a systematic search should be based.

A Norwegian study13 examined an intervention in teaching and how it affected the number of databases and articles that were used in candidate theses in nursing education. The intervention consisted of introducing the first four steps of the EBP model into nursing education. Students’ arguments regarding the choice of sources were also collected from the theses. The focus of the study was on the selected databases and not on the search strategies.13 Schilling and Applegate,17 investigated the pros and cons of common assessment methods in library instruction. It is possible to assess students’ learning both during the process of information retrieval (in-process) and in the end-products. One strength of end-product assessment is that the product is the final goal, compared to a published scientific article that is the final product of a research project. According to Schilling and Applegate, the end product is a genuine, authentic, summative, and valuable product. One disadvantage is that the citation analysis does not provide information about the information-retrieval process itself. Another disadvantage is that the literature list does not show the quality of the information search regarding the use of Boolean operators, the use of filters and delimitations, and the articles’ relevance. Sampson and McGowan,16 studied the quality of search strategies used in systematic literature studies conducted by researchers, but according to our knowledge, search strategies in candidate theses have not been studied before. There are studies that have assessed nursing students’ search strategies in assignments other than candidate theses. The parts that were evaluated were search history and worksheets.3,12 Many studies deal with self-assessed student knowledge and skills in information searching [see, for example Kuhlthau,8] but there are few studies that investigate students’ changing search strategies, particularly in higher education. Mahmood25 demonstrates that students often overestimate their skills in information literacy, according to the Dunning–Kruger effect.7

Objective

The current study aimed to investigate changes in nursing students’ search strategies in candidate theses.

Methods

This study has a retrospective, quantitative study design, and a sociocultural perspective. This theoretical standpoint assumes that knowledge is created in a certain context. With relevance to this study, it is about developing an understanding and familiarity with the way information is sought and used in a certain social context. Using a sociocultural perspective, information literacy is not possible to measure per se, as it is regarded as such a complex competence. However, the students’ performance and the product of information retrieval (i.e., the thesis and the search document, in this study) can be assessed.15 Ethical approval is not applicable since the study does not include human subjects, and the analyzed data was public according to Swedish law.20 The students’ course grades would not be affected by the study as the authors used retrospective data, and the data was blinded in preparation for the analysis.

Context

In Sweden, undergraduate nursing students follow a 3-year university program. The students achieve two degrees, one professional degree as a registered nurse, as well as a bachelor’s degree. Many nurses continue to postgraduate studies (master and doctoral level).22 The context of the current investigation is the nursing program at Kristianstad University in southern Sweden.

Continuous Educational Development

Since 2013, approaches to teaching information literacy within the nursing program at Kristianstad University have been developed in collaboration between librarians and teachers. Examples of learning activities are that teachers discuss and explain how to choose relevant search terms based on the study aim, and librarians teach students how to use the databases and how to use MeSH terms and headings (see Box 1). These learning activities are both lectures and workshops on campus as well as instructional films. Constructive alignment has been the foundation for this work, defining and developing intended learning outcomes, teaching and learning activities, and assessment, and it aims at creating coherence for both students and teachers.1 There has also been an intensified effort on progression within the students’ learning processes along with increased demands on students’ success in information searching. The entire work of changing instruction sessions and supporting the students’ reflective approach can be viewed as using a sociocultural stance, in that learning is considered as emanating from interactive activities.2

Box 1

Examples of Learning Activities

Developing the students’ systematic search strategies and techniques Increasing knowledge about the structures of bibliographic databases Guiding the students in their working process, for example, to design a research question, identifying the meaning-bearing concepts, and finding relevant sets of search terms Developing a reflective approach to search processes and working processes Repeated supervision sessions with reflection of search processes Search workshops, learning resources, and instructional films Using search documents for formative and summative assessment

Procedure

A project group consisting of librarians (n = 2) and teachers (n = 4) examined – individually first, then in pairs consisting of one librarian and one teacher – randomly selected candidate theses (every third thesis, n = 89) from the years 2012, 2014, and 2016. The examiners were blinded as the examining librarians and teachers did not know from what year the theses were or the identity of the theses’ authors or supervisors. Within the project, a template for reviewing the methodology, method discussion, and search document was developed. The template was structured with five principal areas: search terms; search strategies; systematic work procedure; which databases were being used; and methodological considerations (see ). Information on what the students were asked to include in the search documents is presented in . All the librarians and teachers met and discussed the ratings after the first 10 rated candidate theses, and also when all theses were rated to increase inter-rater reliability.

Statistical Analysis

Descriptive statistics with frequencies and percentages were used to describe the search strategies presented in the theses from 2012, 2014, and 2016. Differences in search strategies between the earlier years (2012 and 2014; coded 1) and later (2016; coded 0) were analyzed using the chi-squared test.14 The level of significance was set at 5%. Statistical analysis was performed using IBM SPSS version 24.

Result

The frequencies and percentages of the different search strategies in 2012, 2014, and 2016 are presented in Table 1.

Table 1

Frequencies and Percentages of the Search Strategies in a Random Sample of Literature-Based Candidate Theses (n = 89) from the Years 2012, 2014, and 2016

	2012 n=32	2014 n=29	2016 n=28
Was the use of synonyms sufficient?
Yes, n (%)	4 (12.5)	2 (6.9)	14 (50.0)
No, n (%)	28 (87.5)	27 (93.1)	13 (46.4)
Not clear, n (%)	0	0	1 (3.6)
Are the search terms matched with the database?
Yes, n (%)	2 (6.3)	3 (10.4)	21 (75.0)
No, n (%)	30 (93.7)	26 (89.6)	5 (17.9)
Not clear, n (%)	0	0	2 (7.1)
Is the Boolean operator OR used?
Yes, n (%)	7 (21.9)	6 (20.7)	27 (96.4)
No, n (%)	25 (78.1)	23 (79.3)	1 (3.6)
Type of search strategy
Cinahl headings, n (%)	2 (6.3)	8 (27.6)	26 (92.9)
MeSH, n (%)	4 (12.5)	10 (34.4)	21 (75.0)
Free text/all field, n (%)	14 (43.7)	11 (37.9)	27 (96.4)
Title/abstract, n (%)	28 (87.5)	18 (62.1)	0
Use of block search, n (%)	1 (3.4)	2 (6.8)	22 (78.5)
Is the description of the methods in accordance with the search strategy?
Yes, n (%)	13 (40.7)	12 (41.4)	18 (64.3)
No, n (%)	18 (56.2)	17 (58.6)	9 (32.1)
Not clear, n (%)	1 (3.1)	0	1 (3.6)
Awareness of strengths and limitations
Yes, n (%)	12 (37.5)	11 (37.9)	17 (60.7)
No, n (%)	18 (56.3)	16 (55.2)	8 (28.6)
Not clear, n (%)	2 (6.2)	2(6.9)	3 (10.7)

There was a significant improvement over the years (from 2012 and 2014 to 2016) regarding the use of sufficient number of synonyms (p < 0.0001), matching search terms to the respective database (p <0.0001), use of the Boolean operator OR (p < 0.0001), and the use of subject headings (p < 0.0001) and free text searches (p < 0.0001) (Table 2). Use of the title/abstract search largely disappeared between 2014 and 2016 (p = 0.001). There was a significant change in the types of searches being done. The searches have become more structured in later years as the use of block searches increased significantly during 2016 (p < 0.0001).

Table 2

Result from the Analysis of the Search Strategies in a Random Sample of Literature-Based Candidate Theses (n=89) from the Years 2012, 2014, and 2016

	2012 and 2014 n=61	2016 n=28	p-valuea
Was the use of synonyms sufficient?			<.0001
Yes, n (%)	6 (9.8)	14 (50.0)
No, n (%)	55 (90.2)	13 (46.4)
Not clear, n (%)	0	1 (3.6)
Are the search terms matched with the database?			<.0001
Yes, n (%)	5 (8.2)	21 (75.0)
No, n (%)	56 (91.8)	5 (17.9)
Not clear, n (%)	0	2 (7.1)
Is the Boolean operator OR used?			<.0001
Yes, n (%)	13 (21.3)	27 (96.4)
No, n (%)	48 (78.7)	1 (3.6)
Type of search strategy
Cinahl headings, n (%)	10 (16.7)	26 (92.9)	<.0001
MeSH, n (%)	14 (23.0)	21 (75.0)	<.0001
Free text/all field, n (%)	25 (41.0)	27 (96.4)	<.0001
Title/abstract, n (%)	46 (75.4)	0	0.001
Use of block search, n (%)	3 (4.9)	22 (78.5)	<.0001
Is the description of the methods in accordance with the search strategy?			0.229
Yes, n (%)	25 (41.0)	18 (64.3)
No, n (%)	35 (57.4)	9 (32.1)
Not clear, n (%)	1 (1.6)	1 (3.6)
Awareness of strengths and limitations			0.223
Yes, n (%)	23 (37.7)	17 (60.7)
No, n (%)	34 (55.7)	8 (28.6)
Not clear, n (%)	4 (6.5)	3 (10.7)

Note: aP-value = Chi Square.

There was a trend toward a better correlation between the text in the methods section and the search document in recent years, but this difference was not significant (p = 0.229). There was also a trend toward increasing awareness of strengths and weaknesses concerning searches in the method’s discussion, but the difference was not significant (p = 0.223) (Table 2).

Discussion

The result of this study demonstrated that the quality of the information-searching section in the students’ candidate theses during the studied period increased significantly. In recent years, the students developed more search strategies and performed more systematic and structured searches. However, the result also demonstrated that the systematic approach requires further development, for example, better knowledge of bibliographic databases and certain search techniques. Students also need to elaborate their method chapters and method discussions in relation to the search document (where the searches are described) so that the text and search document are correlated. The result of the study is incorporated into further educational development at the nursing program.

Even though search documents are used in different forms in nursing education at Swedish institutions of higher education, we have not found any studies that have used search documents to evaluate students’ search strategies or systematic approaches. Kapborg and Berterö,6 and Langius-Eklöf and Forsberg,9 are the studies that lie closest to our issue in that they evaluated parts of the systematic approach in candidate theses in a nursing program.

In the current study, the aim was to investigate nursing students’ search strategies. A quantitative approach, as performed in this study, can provide an objective overview of the knowledge of a student group and indicate changes over time. Although this study design cannot prove a relationship between educational development and the students’ skills in information literacy, there are clear improvements in students’ searching skills. Self-assessment methods might, on the other hand, not accurately reflect students’ real knowledge of information literacy, as most overestimate their abilities.25

When information literacy teaching is developed and students improve their competence in search strategies, the teachers supervising the candidate theses might lag behind, which might lead to difficulties in supervision. Therefore, it is important to offer teachers support in supervising candidate theses regarding their skills in information literacy. To facilitate this, we have developed an educational guide to overcome possible difficulties. This guide can also be used by students during the preparation of their theses.

Suggestions for further research include supplementing the data in this study with a survey where the information searching process of a small number of students can be followed via search logs and interviews. In this way, it would be possible to capture how students use reason in their search process, and it would enable us to test the effects of teaching. However, the method needs to be supplemented with formative evaluation tools for the purpose of learning support.

Conclusion

The result of this study shows that the quality of the students’ search strategies improved significantly during the studied years. It is recommended that search documents are used in both formative and summative assessments to evaluate students’ search strategies. Educational development in the form of enhanced collaboration between librarians and teachers in nursing programs is recommended because it might help to develop student search strategies in literature-based candidate theses. Future nurses could then systematically apply their knowledge, thereby increasing the quality of evidence-based and sustainable healthcare.