Criminological scoping reviews as part of a student assignment: methodological considerations.

In this article we reflect on the use of a scoping review method in the course of a student assignment. In total, 24 scoping reviews were conducted on police research topics, following the same procedure with similar yet topic-specific review protocols. We discuss each of the eight phases of the scoping review method and provide examples and good practices on how to conduct these steps. We also focus on difficulties that were experienced and how to overcome them in future scoping reviews. The aim of this article is twofold: providing criminological examples of scoping reviews that can help define new scoping review protocols and avoid difficulties experienced in these reviews on the one hand, and reflecting on students' participation in the assignment on the other hand to stimulate involving students in scoping review projects more often.

Introduction

The increasing amount of literature in several research domains, highlights the need for knowledge synthesis methods that aim to synthesise the available literature and help develop more evidence-based policies. Literature reviews are also increasingly being used in criminology to map the available research (evidence). Examples regarding, for instance, police topics are numerous: decision-making in regard to sexual assault or rape (Lapsey et al. 2021; Sleath and Bull 2017), profiling and racial prejudice (Carvalho et al. 2021; Dowden et al. 2007), use-of-force (Harris 2009; Patterson and Swan 2016), tactical aspects of police work (Engel et al. 2020; Higginson et al. 2017), police training (Belur et al. 2020; Javdani 2019; Khorasheh et al. 2019), policing strategies (Boels and Verhage 2016; Gill et al. 2014), (crime reduction Braga 2005; Braga et al. 2015; Carriaga & Worral 2015) and various other topics such as procedural justice and police legitimacy (Mazerolle et al. 2013a, b), police leadership (Pearson-Goff and Herrington 2014), stress management (Patterson et al. 2012), and the effects of body-worn cameras (Lum et al. 2020).

Grant and Booth (2009) distinguish fourteen review types (e.g. critical reviews, mapping reviews, rapid reviews, systematic reviews, scoping reviews), that all differ in purpose, rigour, time necessary to conduct the review and specific procedures to be followed. For instance, scoping reviews1 differ from systematic reviews by studying a broader topic or theme (Arksey and O’Malley 2005; Levac et al. 2010; Pham et al. 2014) and by usually not including a critical appraisal (Arksey and O’Malley 2005; Levac et al. 2010; Peters et al. 2015; Pham et al. 2014; Whittemore et al. 2014), thereby including all the studies found that meet the inclusion criteria despite their methodological quality. As with systematic reviews, scoping reviews are conducted according to a predefined protocol, by means of an extensive search procedure, and with the aim of being transparent and reproducible. These characteristics should enhance the reliability of the results (Munn et al. 2018). However, scoping reviews do not aim to answer questions that could introduce direct practical changes (e.g. the effectiveness of a certain intervention). They aim to give an overview of the available literature and the kind of studies that have been executed (Arksey and O’Malley 2005; Munn et al. 2018). Building on the research of Arksey and O’Malley (2005) and Levac et al. (2010), Munn and colleagues (2018) identify several reasons for performing scoping reviews: identifying the kind of evidence that is available, clarifying concepts and definitions, looking at the type of research that is conducted in regard to a topic, identifying factors that are related to the topic under review, identifying and analysing gaps in the literature and deciding whether it would be useful to conduct a systematic review. Scoping reviews are believed to be particularly useful when a certain topic has not yet been reviewed or is rather complex and broad, making a systematic review less appropriate (Peters et al. 2015).

Scoping reviews are also increasingly being used by Ghent University students (e.g. Decombel 2021, Decombel et al. 2022), but their educational program currently lacks a course that focusses on this method. As a result, there is a need for further education about conducting and reporting such reviews. For that reason, we organised an assignment as part of an optional course on policing to teach students how to undertake scoping reviews. Although these reviews solely focus on police research topics, it is useful to provide examples and good practices that can also be useful for other research domains. Therefore, the aim of this article is twofold. First and foremost, we wish to describe the scoping review method that was used in this group assignment, identify difficulties and good practices, and deduce recommendations for future scoping reviews. Secondly, we wish to discuss student research in the course of academic assignments – and more specifically scoping reviews – to reflect on its advantages and identify tips and tricks. Attaining these objectives allows us to provide valuable advice on using the scoping review method in general and on police research in particular (e.g. keywords, journals).

Group assignment

As part of the optional course ‘Public Police Studies’ at Ghent University, 24 scoping reviews were conducted by groups of students in criminology and law. While “research is a process by which researchers extend their knowledge and possibly that of the whole community (…), student research involves the development of competence in undertaking research and adding to knowledge” (Sharp et al. 2002, p. 25). That was also the case for this assignment. Students were involved in each part of the scoping review, rendering them able to get acquainted with the scoping review method and to learn how to apply the method to police research. Students could then use the method for their own dissertations. The scoping reviews also provide knowledge that we believe is worth sharing with the broader (criminological) research community. In this article, we solely focus on methodological aspects of the assignment. Part of the results of the scoping reviews in terms of content are included in another article (see Feys et al. 2022).

In order to support students in undertaking the scoping reviews, we provided guidance on multiple levels. First of all, students were given a lesson on the method of scoping reviews. They were introduced to the advantages (and disadvantages) of the method, the different steps to be undertaken, challenges that may arise, the way in which feedback would be given and how to report the results. Students were also given a point of contact that would provide guidance during the assignment. For the police decision-making topics (see later), this article’s author2 was the point of contact. Students were also given a step-by-step-guide explaining the scoping review method that needed to be followed for the assignment (written by the author of this article) and an explanatory note on how to search databases, journals and websites.

The lecturers3 provided several topics on police decision-making and police careers (see Table 1 later in this article for an overview of the included topics) on which the scoping reviews should focus. These topics were based on the PhD projects of the two supervisors because of their expertise on these topics, allowing them to provide sufficient feedback (both in terms of content and in terms of methodology). Students were allowed to form groups4 and choose a topic of their interest. Having the liberty to choose a topic is important as it can contribute to sustaining the motivation, which can unmistakably have an impact on the successful outcome of student research projects (Sharp et al. 2002).5 In order to support students in choosing their preferred topic, the supervisors gave a presentation about their research. The research goal, research questions, methods and intermediate results were addressed, but the main goal was to get the students acquainted with important concepts regarding each research domain (i.e. police decision-making and police careers). This proved important as one group originally wanted to focus on the topic of crime fighting, but they misunderstood the topic and eventually shifted to the topic of COVID-19.

Table 1

Overview of the scoping review topics

Police decision-making	Police careers
General decision-making processes 1. Integrity violations 2. Discretion 3. Moral dilemmas 4. Ethnic profiling 5. Police violence	Recruitment 15. Recruitment of ethnic minorities 16. Recruitment of ethnic minoritiesa 17. Candidate profiles
Trust and legitimacy 6. Complaints 7. Trust and legitimacy 8. Bodycams	Selection 18. Cognitive test 19. Personality test
Police strategies 9. Community-oriented policing 10. Hot spot policing 11. Predictive policing	Socialisation 20. Police education 21. Reality shock
Threats to police work 12. Violence against the police 13. (Social) media 14. COVID-19	Police culture 22. We vs. them 23. Cynicism 24. Code of silence

a Both topic 15 and 16 focus on the recruitment of ethnic minorities. The first scoping review focused on which minority groups are recruited, how attention is paid to this recruitment and which evolutions can be noticed. The second scoping review concerned recruitment in general with a focus on ethnic minorities to more specifically identify problems, successful strategies and good practices.

After having chosen a topic of interest, each group had to work out a review protocol by means of a template provided by the lecturers before a certain deadline. After the piloting phase (see later), the review protocol was sent to the responsible supervisor by e-mail. The supervisor provided feedback on all aspects and made sure that consistency throughout the scoping reviews was ensured (e.g. similar inclusion and exclusion criteria), which would be important when comparing the findings of the reviews in a later phase. The groups precisely applied the feedback.

Subsequently, the groups of students were asked to fully execute the search process and register their steps and results in an Excel-file. An Excel-template was provided by the lecturers to enhance the consistency between the different groups. In March 2021, when all groups of students were conducting their searches, a live online Q&A session was organised by the lecturers. Students were able to send in questions in advance on the online teaching platform from the university. That made it possible for the lecturers to formulate thorough and constructive answers. However, there was also room for live questions.

Students executed the analyses in group, both analysing methodological aspects of the literature and its content. They were asked to report these results in a paper and a presentation. The paper needed to elaborate on the method of the scoping review, the results, and recommendations developed by the students. The groups were specifically asked to reflect on what went well during the review, what they liked or disliked about the assignment and what could have gone better. In the presentation, the results of several groups were combined. The presentations followed the overarching topics as discussed in the next section of this paper. In those presentations, students needed to combine their results, look for similarities and differences, and present the main findings to the lecturers and fellow students. After each presentation, questions were asked by both lecturers and fellow students, and a discussion on the topics was organised. At the end of the session, an overall discussion was organised with regard to the assignment itself. During the discussion, students could reflect on their experiences.

During the assignment, students were free to choose how they divided the work, but were stimulated to be involved in each phase of the scoping review procedure to get fully acquainted with the method. In order to optimise the supervision, which also has an impact on the successful completion of a student research project (Sharp et al. 2002), both formal and informal opportunities to receive feedback were present. The formal opportunities were the feedback on the review protocol and the live Q&A session. Besides these formal, planned feedback sessions, students were also continuously able to contact the responsible supervisor for feedback and guidance (e.g. by means of different fora on the Ghent University course site6, by e-mail7 or by means of a Microsoft Teams-meeting8). This not only allowed students to quickly get feedback and resume the assignment, it also allowed the lecturers to optimise the consistency of the scoping reviews, which enables us to compare the findings in a transparent way.

Scoping review method

For the purpose of this article – and in line with the group assignment – we follow Grant and Booth’s (2009, p.101) definition of scoping reviews: “preliminary assessment of the potential size and scope of available research literature. It aims to identify the nature and extent of research evidence (usually including ongoing research)”. The scoping review method used for this assignment was developed by the author of this article and was based on her experiences with the Arksey and O’Malley (2005) framework (further elaborated on by Levac et al. 2010) and the article of Verhage and Boels (2015), who describe a method of systematic reviews within criminology and provide an overview of challenges in order to apply systematic reviews to qualitative research.9 The author used these frameworks to conduct a scoping review on police accountability (Feys et al. 2018). After completing the scoping review, she wrote down the different taken steps in a step-by-step guide and included some tips and tricks. That same step-by-step guide, which entails many similarities with the original frameworks that were the basis for the initial scoping review, was used for the assignment. It is not our aim to further discuss how the method was developed. We rather aim to identify good practices that can be included in future scoping reviews.

As such, we present the different steps followed by the groups in the course of their assignment, and we reflect on the difficulties and good practices. We provide examples for clarification and to help us explain the method more clearly. That information can also be usefully taken into account by other researchers. We also use quotes from a couple of groups to demonstrate what they liked about the method and which particular tips and tricks they found useful when carrying out the assignment.

Determining research frame

For the assignment, students first needed to determine the research frame of their scoping review. This related to the research topic, the context of the scoping review and the development of research questions. Students were made aware that the combination of these elements would determine the scope of their review process (e.g. inclusion and exclusion criteria) and were asked to actively consider what they exactly wanted to study and to explicitly demarcate what they did not want to be part of the review.

As mentioned earlier, the research topics were – in general terms – defined by the supervisors of the assignment (see Table 1). Students were able (and encouraged) to further define their topics based on their interests. We do not intend to discuss the content of the scoping reviews and will thus not include a discussion on the topics in this article. A more extensive discussion on these topics and some results in terms of literature characteristics can be found in Feys et al. (2022).

When refining the research topics, students needed to take into account the context of the review. The two most important aspects in this regard were the geographical setting of the scoping review and the target population. Concerning the former, the supervisors predefined the desirable geographical limitations of each topic to help students define a feasible review protocol. Regarding the police decision-making topics, most groups were asked to limit the review to Belgian research as that scope was believed to be broad enough for the assignment.10 An exception was the scoping review on COVID-19, which was not limited in scope because of the recent nature of this research topic. The reviews concerning police career topics had no geographical limitations. The supervisors had also predefined that public policing actors would be the focus of the reviews (excluding private police or intelligence services), but students were able to narrow this down if relevant for their topic. For instance, the group working on the topic of violence against the police decided to focus solely on operational police officers (thus excluding administrative personnel).

After the students had explored what they wanted to study, they had to define their research question(s). This proved to be a bit difficult for a couple of groups as we had not provided any examples. An additional shortcoming was that we failed to provide clear guidance on this: when giving feedback on the research questions (see step four), we took into account the PCC-framework11 recommended in the JBI manual (Aromataris & Munn, 2020), but we did not communicate this to the students in advance. The expertise of the supervisors was thus essential in revising the research questions. Some students formulated questions that were too narrow or questions that they would not be able to answer on the basis of the available literature (e.g. the cost-effectiveness of hot spot policing in Belgium). Although it was an exception, some groups had formulated multiple primary questions and were recommended to focus on one (in line with the JBI instruction; Aromataris & Munn 2020). This was the case for the scoping review on police use-of-force. The questions that were originally included in the protocol would have made the scoping review infeasible as the questions were too broad and focused on two different things. The first is the definition of police force, with sub-questions regarding the situations in which police force is admissible, how the police are armed and whether or not this leads to violence, how a supervisory body supervises the police’s use-of-force, and use-of-force training. The second focus was the use-of-force towards people with another ethnic background, with sub-questions about the definition of ethnic profiling, how people with foreign backgrounds experience ethnic profiling, in which situations it is most common, and the prevalence of police use-of-force towards ethnic minorities. These (sub-)questions focused on many different things, some of which the students would not be able to answer based on the available Belgian literature. As such, the questions were redirected into one overall question: ‘What does the available literature teach us concerning the use-of-force by Belgian police officers?’. It was then sub-divided into four questions: how are police officers trained concerning the use-of-force, in which situations do police officers use force, is force used more often towards certain groups (which was included to reflect the students’ interests), and how and by whom is the police’s use-of-force supervised? Sub-questions were included in each scoping review protocol to define what students wished to learn from the reviews in terms of content and, as such, help them structure their analyses (see later).

Determining search strategy

In the second phase of the student assignment, students developed their search strategy12, which relates to the keywords and databases they would use for searching relevant literature.

Keywords

The students first defined the keywords they would use to look for relevant literature. All the groups used approximately ten keywords that were combined into one Dutch and one English search string by means of Boolean operators. The keywords included synonyms (e.g. ‘bodycam’ and ‘body worn camera’), related aspects (e.g. ‘predictive policing’ and ‘hot spot’) and/or different characteristics of the topic (e.g. ‘aggression’, ‘violence’, ‘insubmission’, ‘indignity’ in the review concerning violence against the police). To the extent of which was possible, the English keywords were a literal translation of the Dutch keywords. The keywords were, at first, defined by the students based on a preliminary assessment of the literature. After a preliminary search (see step 4), these keywords were revised by the supervisor of the assignment. During this revision, the supervisors checked whether most related aspects were included in the keywords, which synonyms could be added and whether the English translation reflected the Dutch keywords well. In some cases, the supervisors also recommended to delete certain terms. For instance, in the scoping reviews concerning police violence and complaints, two specific Belgian oversight bodies were included as keywords (‘Comité P’ and ‘Algemene Inspectie Politie’13), although the scope of the review was broader than that. The groups were recommended to delete these keywords to avoid steering the results in a specific direction. As the students reported in their paper, deleting these keywords resulted in research concerning other oversight bodies as well. The supervisor also paid attention to the exclusion of keywords that were too far off topic (e.g. the Dutch equivalent of ‘detection and enforcement’ for the scoping review on ethnic profiling; ‘scientific research’, a keyword that will not generally be mentioned in the abstract of a publication).

In their papers, students were asked to reflect on the keywords used for the search. Several groups reported that the keyword ‘polic*’ generated too many hits related to ‘policy/policies’ instead of the intended ‘police’ and ‘policing’. As such, it is recommended to include ‘police’ and ‘policing’ as separate keywords instead. Some students also identified additional keywords such as BME (black minority ethnic) and BAME (black Asian minority ethnic) for the topic concerning recruitment of ethnic minorities in English literature. In Appendix 1, we have included an overview of the search strings used for the scoping reviews concerning police decision-making. In that same appendix, we give a per topic overview of the keywords that were used at least twice by the authors of the publications on the shortlist (as keyword or index-term). This, in order to explore potential additional keywords. The analysis shows that there are multiple keywords that are worth being added in future scoping reviews (e.g. ‘integrity’ and ‘ethics/ethical’ for the topic of moral dilemmas; ‘procedural justice’ for the topic of trust and legitimacy; ‘local’ for the topic of community-oriented policing; ‘predictive policing’, ‘forecasting’ and ‘modelling’ for the topic of hot spot policing; ‘forecasting’ for the topic of predictive policing). In this overview, we do not include keywords that refer to the methods used or the country in which the study was conducted – we exclusively focus on the keywords regarding the topic itself. The analyses in the appendix can prove useful for other researchers to define their keywords. They should, however, be careful when deciding which keywords to use. For example, different publications on the shortlist concerning predictive policing referred to keywords involving ‘crime’. Adding such a general keyword could help prevent relevant studies from being excluded, but can also make the procedure needlessly heavy. This implies that a balance needs to be found between inclusiveness and feasibility. The piloting of the keywords (see later) is essential in this regard.

On the course’s forum, one of the groups asked a question concerning the application of the search strings and whether or not it is important to include all keywords in one search. The importance of using the entire search string (consistently) was demonstrated by an example: the search string “Belgi* AND (polic* OR “law enforcement”) AND aggressi*” generates nearly no hits in Web of Science. However, when the keyword ‘violence’ is added (“Belgi* AND (polic* OR “law enforcement”) AND (aggressi* OR violence)”), it does generate 36 results. All students were thus asked to consistently use the search string in all the databases and the journal.

Databases

The databases and journal to be used for this assignment were predefined by the lecturers to enhance the consistency among the scoping reviews and to make sure that all students had access to it through their Ghent University-accounts. In total, each group searched three databases (Web of Science, Scopus, ProQuest), one scientific journal (Police Practice and Research: An International Journal)14 and two websites (specific to each topic). Again, the websites were chosen by students based on a preliminary literature search and tested during the piloting phase (see step 4), for example to see if there were other, more suitable websites. Afterwards, these websites were revised by the supervisor. For the databases and journal, students had to look at the abstract of the publications to find the keywords.15 For the websites, students either used separate keywords to look for relevant literature or scoured the entire website. All websites concerning police decision-making topics are, as an example, included in Appendix 1.

The scope of the assignment in terms of databases was rather limited. A wider range of (national) databases and police journals (e.g. lib.ugent.be, VABB/ANET, Cahiers Politiestudies, Handboek Politiediensten, Panopticon, Tijdschrift voor Criminologie, European Journal of Policing Studies) will definitely be useful in the future – specifically, the national databases and journals would be relevant for the scoping reviews limited to Belgian research. Some groups also came to the conclusion that databases or journals not centred around criminology would be relevant to include as well, for instance concerning the topic ‘we vs. them’ as lots of the studies found during the review procedure focused on sociology and not merely criminology. Some groups specifically found a couple of journals that provided lots of search results when the keywords were entered, for instance the Journal of European Policy and the Journal of Ethnic and Migration Studies for the topic of integrity violations, Ethnicity and Health and the Journal of Ethnic and Racial Studies on the recruitment of minority groups and The International History Review, Studies in Conflict & Terrorism, and Terrorism & Political Violence on police violence.

The supervisors provided an explanatory note on requirements for searching each of the databases and the journal. This was important to enhance consistency and facilitate the search process – students particularly favoured this kind of support. Such guidance is relevant as most databases have advanced search options that allow for a specific search strategy. For instance, with ProQuest, researchers can determine where the keywords need to be found (e.g. anywhere, abstract, publication title), if the search should be limited to peer-reviewed articles, if the search should be limited to a specific time range, if certain types of publications (e.g. conference proceedings, blogs, newspapers, magazines, reports) need to excluded from the search, and which languages to include. Furthermore, researchers can decide to exclude duplicates in advance and include spelling variants for search terms. These possibilities are appealing, for this assignment as well. As the focus of this assignment was on empirical literature (see 3.3), one group of students reported in their paper that they had excluded newspaper articles from their search as they generated lots of search results but fell outside the scope of their scoping review. According to the students, this was much more efficient as these news articles did not prove relevant at all. As such, knowledge on the databases and journals before the start of the search process is a definite plus as this optimises the efficiency of the scoping review.

We explicitly chose to include websites in the scope of the assignment so students would learn how to search such websites as the use of search strings is usually not supported. In that case, the researcher can decide to manually look through the publications or to enter the keywords individually and assess the publications found. Both methods were used by the students, sometimes in combination to make sure they would find all relevant publications. The use of websites was useful (or even necessary) for some topics to find relevant results, as was the case for the topic ‘police violence’ since 46 of the 52 publications on the shortlist were found through the use of websites.

Researchers might also want to look for other ways to access grey literature or research reports that are not published in scientific journals or books considering a majority of the publications resulting from the Belgian scoping reviews on police decision-making were written as reports (n = 50 of the 79 unique publications). Some topics mainly appeared in reports (e.g. police violence, complaints), whereas other topics were exclusively found in journal articles or book chapters (e.g. integrity violations, moral dilemmas, predictive policing; see also Feys et al. 2022). Merely focusing on scientific articles would lead to biased results. The inclusion of websites in the review protocol and analysing reference lists of publications on the shortlist might be helpful in this regard (see later). We also believe it would be useful to include the websites of research institutions in the review protocol to explore ongoing research not yet published as some of the topics that yielded few to none results on the police decision-making topics, have recently been incorporated in research projects (e.g. IRCP 2022a, 2022b, 2022c; KU Leuven 2021; Stacius and Verstichel 2021).

Determining inclusion and exclusion criteria

A third step in the assignment was the translation of the research frame to inclusion and exclusion criteria. Inclusion criteria encompass publications’ requirements in order to be included in the review, whereas exclusion criteria determine which publications should be left out. As such, publications should meet all inclusion criteria and if even one exclusion criterion is present, the study is immediately ruled out. Defining these criteria provides guidance for assessing the studies on the longlist(bis), as one of the groups explained: “The other inclusion and exclusion criteria made it quite easy to identify which sources were eligible and which were not. This is an important aspect of a high-quality scoping review and should therefore be worked out well in advance.”

For the assignment, five inclusion criteria were defined for each scoping review. The two first inclusion criteria relate to the geographical setting and target group (i.e. police) of the review, which have already been explained. Concerning the geographical aspect, it is important to note that this concerned the location in which the research took place. One group was unsure about this, they found a publication written in the US but the data were collected in Slovenia, causing uncertainty as to how to categorise this study geographically. Another group wondered how strictly one needed to enforce the geographical inclusion criterion as they also found relevant literature that went beyond the predefined scope. All groups were reminded about the importance of following the predefined scoping review protocol and that they needed to interpret the criteria consistently for each publication.

A third criterion concerns the timeframe, which the supervisors set between 2000 and the beginning of 2021 (when the scoping reviews were conducted)16. This timeframe was chosen for two specific reasons. On the one hand, this was due to feasibility as the students only had a couple of months to conduct the assignment. On the other hand, the Belgian police underwent a reform after 2000. Regarding this criterion, one group of students asked if it relates to the date of publication or the period of data collection as the publication they found fell within the timeframe, but the survey on which it was based did not. For this assignment, the time period related to the date of publication as this is easier to determine. Since authors do not always mention when the fieldwork was conducted, this would have been more difficult to assess, potentially obstructing the assessment of this inclusion criterion and hampering the consistency between the scoping reviews.

A fourth inclusion criterion was predefined as ‘empirical research’17, again an obligated inclusion criterion for all the assignments. Evaluating the empirical nature of studies proved to be difficult for several students and resulted in multiple questions on the forum and during the Q&A session. For instance, students were unsure whether empirical research also includes secondary data and whether or not literature studies could be classified as empirical. We answered each question individually, clearly explaining that a literature study is not empirical, but that the data collection method can be either primary or secondary as both were considered empirical. We provided examples of the most common methods used in criminology to give students an idea of what was considered empirical (e.g. secondary data analysis such as complaints or official statistics, surveys, interviews, focus groups and observations). The author of this article checked all publications on the shortlists concerning the police decision-making topics to assess different aspects, such as the empirical nature of the articles. All publications on the shortlists were indeed empirical, implying that the students correctly interpreted what was considered empirical research.

The final inclusion criterion regards content and was specific to each topic. The scoping reviews also had an exclusion criterion regarding content. Combined, both criteria determined the scope of the review in terms of content. As the focus of this article is on the methodological aspects of the assignments, we will not be discussing these criteria in full. The supervisors assessed the applicability of all inclusion and exclusion criteria formulated by the groups of students (see step 4), but these criteria were (generally) well-developed as they were mostly predefined by the lecturers for the purposes of feasibility and consistency. The assessment of the content criteria (which were not predetermined), however, proved to be necessary for some groups. We cite one of the groups: “Initially, the intention was to use ‘police use of social media channels’ as an exclusion criterion. For example, only the use of social media by citizens and what threats this can cause to the police were to be investigated. The feedback questioned this choice, leading the group to drop the exclusion criterion. In retrospect, this was also one of the better decisions throughout the group’s work. After all, almost all sources included in the shortlist concern the use of social media by the police and its influence on the relationship with citizens. If the initial exclusion criterion had been maintained, these sources would not have been part of the scoping review.”

Three other exclusion criteria were added on top of the content-related exclusion criterion. A first one concerns the languages of the publications to be included in the scoping review. Languages were limited to Dutch and English for practical reasons18, although we acknowledge that the inclusion of other languages would have been a great addition (e.g. French to give a more exhaustive overview of the Belgian research, which is in part published in French). We highly recommend for future scoping reviews to include other languages if possible as this should be beneficial for the findings. This is partly demonstrated by the results of the scoping reviews regarding police decision-making as 56 of the 79 unique Belgian publications19 on the shortlists were written in Dutch.

Another exclusion criterion related to the type of publication as students were asked to exclude bachelor and master dissertations. This decision was made since their quality is difficult to assess, as opposed to PhD dissertations for which experts in the field provide guidance and feedback. Therefore PhD dissertations could be included in the reviews. A final exclusion criterion was pragmatic in nature: the full text being unavailable.

Piloting

In the fourth phase, all groups first piloted their review protocol in Google Scholar. When testing the search strategy, students were asked to explore:

Which amount of literature could be found (i.e. is the search strategy too broad or too narrow?);

Whether the keywords were frequently used in the title or abstract, or mainly in the full text (i.e. where should the researcher look for the keywords?);

Whether certain keywords were frequently used as index terms at the beginning of an article (i.e. should certain keywords be added to the protocol?);

Whether certain journals and/or databases regularly included topics on the study (i.e. should certain databases or journals be added to the protocol?);

Whether there was a breakpoint in the studies found in terms of timing (i.e. can a reason be identified to limit the review in time?).

A second part of this piloting consisted of feedback provided by the responsible supervisor.20 This was done by means of a template including: (a) the research question(s) students wished to answer, (b) the keywords students planned to use to perform the search, (c) the websites to be included in the review, and (d) the inclusion and exclusion criteria. During the initial presentation on the scoping review method, students received some information in order to write out this protocol. For instance, they were asked to focus on public police actors (target group) and empirical research, and search for studies starting from the year 2000 (timeframe). Students were also encouraged to include ‘polic*’ and ‘law enforcement’ in their protocol as an equivalent to the Dutch keyword ‘politi*’. Besides these recommendations and predefined inclusion and exclusion criteria (see 3.3), to stimulate their learning process, we found it important for this assignment to first let students explore the different aspects of their review protocol so they become acquainted with the different steps preceding the search process. Feedback was only provided afterwards on the delineation of their topic, research questions, search strategy and inclusion and exclusion criteria. All these elements have been discussed in the previous parts of this article.

The initial test and feedback exercise allowed students to optimise their protocols and anticipate problems they may encounter. For instance, the piloting phase showed that four topics on police decision-making that were initially focused on Belgian research, had to be broadened geographically because too few results were found. Testing the keywords in Google Scholar was in part the cause for this. The Google Scholar test sometimes yielded very diverse and often irrelevant search results and was thus not suitable to correctly assess the efficiency of the search strategy. It especially resulted in a biased impression on the amount of literature that could be found. As such, based on the assignments – and in line with the guidelines of Peters et al. (2017) – we recommend to test the search strategy in (some of) the databases that would be used for the scoping review as it will provide more accurate results.

Overall, the feedback given by the supervisors was considered to be useful, and sometimes even essential to successfully conduct the scoping review. Nevertheless, we would also recommend to look for experts in the topic who can direct the researcher(s) towards ground-breaking research in specific fields and assess the review protocol on its feasibility.21 In the course of the assignment, this was only done by means of feedback by the supervisor, but it would be beneficial to ask different experts for feedback to optimise the protocol as much as possible. Such experts will be able to give valuable feedback and can identify gaps in the strategy (e.g. forgotten keywords, relevant databases, definitions of concepts). They may even be able to provide practical tips on how to conduct the review.

Searching and assessing studies

In the fifth phase, students were asked to search publications and assess them in terms of relevance for the scoping review. We cannot stress enough the importance of consistently and thoroughly registering the search process. This systematic registration facilitates the transparency of reporting and allows for replication of the scoping review procedure. In this regard, the Excel-template provided by the lecturers proved useful for the student assignments. In the Excel-spreadsheet, we predefined the different steps to be undertaken and the kind of information students needed to register.

In the first tab (search process), the students recorded when the search strings were inserted in the databases and journal, how many search results they generated and how many of these publications were retained after title-based selection (i.e. the title suggests that a publication may be relevant for the review). Although this step yielded few to none questions, one group indicated in their paper that assessing the publications based on title was challenging because several titles are too vague or do not clearly state the topic of the publication. If there was any doubt, the publication needed to be included in the longlist.

In the second tab (longlist), the students registered which publications were found (including duplicates) per search string and per database that were possibly relevant for the review. Students registered the title of the publication, the authors, the year in which it was published and the type of publication (e.g. journal article, book chapter).

In the third tab (longlist bis) all publications were included just once. This implies that the students deleted all duplicates to have a more realistic view on the number of unique (possibly relevant) publications that were found during the search process. All the publications on the longlist bis needed to be assessed in light of the inclusion and exclusion criteria. The students assessed the abstract and, if necessary, the full text to determine whether or not the inclusion criteria were met (and no exclusion criterion was present). Students registered different components for each publication on the longlist bis: the author(s), title, year and type of publication; the language in which the publication is written; the keywords used in the publication (as this may help to refine the keywords for future reviews); the geographical region in which the research was conducted; the nature of the publication (empirical research, legal, theoretical, etc.); the type of research (quantitative, qualitative, mixed-methods, non-empirical) and the specific methods used (e.g. interviews, surveys, observations); whether it concerns primary data or secondary data; the (sub-)topic on which the publication focusses; and whether or not the publication is included in the shortlist and if not, the reason why it was excluded. Students were also asked to register which inclusion and exclusion criteria were identified.

A couple of questions emerged concerning this step of the search process. A few of them related to how the Excel-spreadsheet needed to be filled in, particularly concerning unknown aspects (e.g. no keywords are mentioned, the country is not defined). Students were told to register this in the Excel-sheet, clarifying that the most important part of this step is consistent registration (e.g. putting all these fields on ‘unknown’). Other questions, as mentioned before, concerned the assessment of the empirical nature of the publications. Since there also seemed to be some confusion on when to assess the nature of the publication, we clarified that this was of no interest when compiling the longlist as the publications on the longlist are assessed based on the title alone – non-empirical research was however excluded from the shortlist when assessing the inclusion and exclusion criteria. Upon verifying the Excel-spreadsheets of the groups, this article’s author noticed variable interpretations concerning the categorization of research as mixed-methods, which we operationalised as combining quantitative and qualitative research methods. For instance, one group classified the combination of complaints analyses and a literature study as mixed-methods, whereas another group did so for complaints analyses and surveys. In both cases, the supervisor recategorized these publications as quantitative in the integrated Excel-file (i.e. the spreadsheet that includes all Excel-files from the fourteen police decision-making scoping reviews in which all aspects were verified by the supervisor – this Excel-file provides the basis for the analyses regarding literature characteristics and content).

We explicitly recommended for all inclusion and exclusion criteria to be registered for each publication. One group asked a question about this, as they thought that when one exclusion criterion is found, the other criteria do not need to be assessed anymore. However, it is still interesting to register the other criteria as well to explore the main reasons why publications that were included in the longlist were excluded from the shortlist. To explore this issue, the author of this article made an analysis of the reasons for exclusion in the police decision-making reviews. A large majority of the publications was excluded because of their focus on another topic and to a lesser extent because the publications were not empirical or did not meet the geographical inclusion criterion. The unavailability of the full text was the sole reason for exclusion in less than twenty cases. Other reasons for exclusion were rarely identified (e.g. another language than English or Dutch, outside the timeframe, dissertations). These are important reflections in regard to the scoping review methodology as it says a lot about the effectiveness and suitability of the inclusion and exclusion criteria, and can lead to important sidenotes to interpret the results (e.g. if lots of publications are excluded because their full text is unavailable to researchers, it may lead to biased findings).

Although it was exceptional, some scoping reviews resulted in edited books in the longlist and challenged students on how to register these publications. Two questions regarding the matter were asked on the forum, relating to the question whether or not the book should be included in the longlist as one single publication, or whether the individual chapters should be included as different publications. The same question was posed for regular, non-edited books. Concerning the latter, students were told to include the entire book as one publication on the condition that each chapter was written by the same author(s). The method was slightly different for edited books. An edited book potentially relevant based on the title, was included as one search result in the tab on the search process. This book was then looked up to assess the titles of the different chapters. Similar to the general search process, students needed to include all chapters that were potentially relevant in the longlist (to make sure that they could assess all the characteristics of the literature and the inclusion and exclusion criteria). They were explicitly asked to register these chapters as ‘book chapters’.

In the final tab, the publications that met the inclusion criteria (but not a single exclusion criterion) and were thus included in the analyses, were collected (shortlist). The shortlist included the same elements as the longlist, adding the full reference to the publication. When it came to the shortlist, several students were confused or concerned about not being able to include studies that did not make the shortlist in the analyses. The lecturers explained that such publications could still be used for a broader literature study, to introduce the topic or to reflect on the results in the discussion. However, they clearly outlined that the analyses should only be based on the publications on the shortlist as the scoping review protocol needs to be followed precisely and ad hoc decision-making should be avoided.

A shortcoming that came along with this phase was that we did not ask students to screen the reference lists of studies on the shortlist on other relevant references (i.e. snowball sampling) (Peters et al. 2015). Adding this strategy to the assignment would have allowed students to get acquainted with this part of scoping reviews as well and might have also helped to identify publications on relevant research that was not identified by means of the students’ scoping reviews (e.g. Feys and Verhage 2019; Gilleir 2013; Rummens et al. 2017). However, this was not feasible for the assignment due to time constraints.

Although there were some limitations and challenges, we identified some advantages and good practices as well throughout this phase of the scoping reviews. A first advantage relates to the students’ collaboration on the scoping reviews as it is usually recommended to let two independent researchers run through the procedure. Students were able to perform the scoping review – including searching and assessing studies – in group and were thus able to verify each other’s decisions. For instance, the group executing a scoping review on the code of silence explained that they verified each other’s performances throughout the procedure and looked out for typos and misinterpretations. It is unclear how the students settled potential disagreements, but we assume that they (re)discussed potential discrepancies and tried to reach a consensus. If that were impossible, students could always ask the supervisors for a third opinion – this however only occurred once on the forum concerning the assessment of the empirical nature of the respective publication.

Another advantage relates to comparing the literature characteristics of the publications on the longlist with the literature characteristics of the publications on the shortlist (which was included in a few papers) as this can help to contextualise the results. This was an interesting addition and resulted from systematically registering the search process from beginning to end. A third good practice can be found in the scoping review of the working student. He had contacted an author of a publication to gain access to a potentially relevant study. However, he did not receive the publication in time before the assignment’s deadline. Nevertheless, this is a good example of how to get around the exclusion criterion regarding the availability of the full text.

Although we would suggest reviewers to explore other software to support the search process, such as Covidence (2022) and JBI SUMARI (JBI, 2022), this was not part of the assignment. Excel was chosen as an instrument because of the author’s positive experiences with it during an earlier scoping review and because of its easy access to the students. Despite Excel certainly having its limitations compared to software specifically designed to support review processes, the use of Excel (and particularly the template provided) was evaluated positively. “The most accessible way for students to do this in a simple and clear manner is by using the Microsoft Excel software program. This allows us to record the number of sources selected and then narrow it down to the most appropriate ones within the boundaries of the research framework. By using multiple tabs in the spreadsheet, the different steps can be tracked clearly.”

Limited methodological assessment

Although scoping reviews usually do not include a quality appraisal of the studies (Arksey and O’Malley 2005; Levac et al. 2010; Peters et al. 2015; Whittemore et al. 2014), the students were asked to perform a limited methodological assessment of all studies on the shortlist based on reporting criteria. In spite of this assessment’s outcome, no publications were to be excluded from the analyses.22 We do not intend to propose this methodological assessment as a vital step in the scoping review procedure, but we do believe that this might be an interesting addition in the course of student assignments (see below).

Table 2 summarises the aspects included in the assessment of methodological reporting23 for all 79 unique Belgian publications found on the basis of the fourteen scoping reviews regarding police decision-making. It demonstrates that most authors report on the specific methods they use (n = 75), the number of participants or documents/files included in the study (n = 73), how they selected participants/documents (n = 71), and their analyses (n = 68). Two aspects were missing in more than half of the publications: the limitations of the study and triangulation of the findings. Triangulation was defined as (explicitly) comparing findings of different research methods, or comparing the empirical findings with the literature. We classified papers on their methodological reporting24: a publication that includes five or six of the aspects discussed in the table is considered ‘good’ in terms of methodological reporting, whereas a paper that includes three to four aspects is ‘sufficient’, and a paper that includes less than half of the aspects is ‘insufficient’. Most papers were classified as sufficient (n = 47) and four as insufficient. This may be the result of the types of publications included in the shortlists as more than half of the papers are reports, which are normally not peer-reviewed. This is also partly reflected in the results, as all insufficiently reported publications were reports and twenty of the well-reported papers were scientific articles. However, eight reports were considered to be well-reported as well, perhaps because they are less limited in wordcount to reflect on methodological aspects.

Table 2

Overview of the methodological reporting of the Belgian publications on the police decision-making shortlists

	Specific method?	Selection?	How many/ numbers?	Triangulation findings?	Analyses?	Limitations?	General assessment
Yes	75	71	73	28	68	30	Insufficient: 4 Sufficient: 47 Good: 28
No	4	8	6	51	11	49

Assessing these criteria might have been more important for the students’ knowledge of methodological aspects than for the scoping reviews themselves. For instance, some students experienced difficulties in assessing whether or not authors triangulated their findings25 – although we defined this as comparing results from different research methods or comparing empirical results with a literature analysis in the discussion – and it helped them understand the importance of triangulation. Furthermore, as was explicitly mentioned in one of the papers, one group was not really aware of different analysis methods so we can assume that assessing these aspects helped gain knowledge on this aspect. It may have also taught students to be critical about their findings and how to (sufficiently) report on their methodology in the course of their dissertations. For instance, the group working on the topic of hot spot policing concluded that some authors needed to report better on the reliability of certain parameters as many readers would have insufficient background knowledge to interpret them. They also reflected on the reporting criterion regarding selection: “By stating the selection criteria of the city or cities on which the analyses were based, there would be more clarity about the characteristics that are important for a hot spot analysis (e.g. urban context, location, size, political role, etc.).”

To conclude, we do not advise including a limited methodological assessment in all scoping reviews. However, we do believe that this is a useful exercise in the course of student assignments to get students acquainted with different methodological aspects (e.g. different research and analysis methods, assessing the validity and reliability of the results) and to sensitise them about the importance of triangulation. This may help students improve their own methodological reporting26 and may result in them being more critical of their own research findings. To improve this exercise in the future, it may be relevant to include more information on these aspects, such as the actual number of participants, the selection procedure, the specific analysis method, the manner in which triangulation was performed, etc. For this assignment, students were solely asked to report on the inclusion of these aspects in the publications (yes/no), but registering these aspects and taking them into account when assessing the studies on the shortlist, would better allow students to critically interpret the results.

Analyses

After the shortlists were compiled and the limited methodological assessment had taken place, students started analysing the publications in terms of literature characteristics (e.g. research methods used, types of publications, year of publication) and in terms of content. Concerning the former, this analysis was simplified by predefining these elements in the Excel template. After carefully registering these aspects, students could easily import these findings into their papers.

Although different methods exist to perform the analyses in terms of content (see for example Kastner et al. 2012), the students could limit themselves to a basic qualitative content analysis approach in which they descriptively reported the findings. Students were not asked to re-interpret the findings but to thematically summarize the main results. Several groups reported that they used colour coding to facilitate the analyses. Where most groups used Microsoft Word for this purpose, one group used Nvivo and another a shared Google Drive to enhance collaboration. One group explained that they had divided the publications on the shortlist amongst the students who, initially, performed the analyses individually, but afterwards collaboratively discussed the analyses. This is identified as good practice as it allows students to first explore how to analyse the papers by themselves, but they also optimise the consistency by discussing these interpretations and, as such, also receive feedback from their fellow students, which positively influences their learning process.

Only one type of issue was identified regarding this phase. A limited number of groups had coded the results according to the research questions and then presented their results per question; exceptionally, publications were discussed individually, summarising the components of these publications without any integration of the findings at all. This was not the purpose as students first needed to summarise and integrate the findings and, afterwards, combine these results to answer the research questions in the conclusion of the paper.

Reporting

The final phase of the scoping reviews concerned the reporting. In the course of the assignment, this was done in two ways. Firstly, the individual groups of students were asked to write down their findings in a paper. The structure of this paper was outlined in advance (front page, abstract, introduction, methodology, results, conclusion, bibliography, references, optional appendices). All groups followed this structure, but some students added additional information (e.g. comparison of the characteristics of the literature on the longlist and the shortlist, see earlier). Secondly, students were asked to combine the results of related topics (e.g. all the topics concerning policing strategies) in a presentation and reflect on these findings more broadly. Although such presentations are not part of the general scoping review method, we briefly discuss this as it provides some useful examples for future student assignments.

Papers

The papers contained a couple of obligated sections that allowed the supervisors to read through the decisions made throughout the scoping review and the findings it generated.27 The methodological section of the paper consisted of a discussion on the scoping review procedure, also indicating potential difficulties the students encountered. The results were divided into different components. The results of the search strategy were presented in the form of a search decision flowchart that summarises the selection process. Students were given a couple of examples of criminologically-oriented scoping reviews at hand (Boels and Verhage 2016; Feys et al. 2017a, 2018; Snaphaan and Hardyns 2020) and a standardised example that could be used for the assignment (see Fig. 1 for the standardised search decision flowchart that was provided as an example to students28, filled in with the results of the scoping review concerning police violence). However, students were free to choose how they presented these findings and some groups developed more complex and detailed flowcharts. One practical question in regard to the search decision flowchart was asked, namely how to report the reasons for exclusion for those publications that could not be assessed (e.g. because the full-text was unavailable) or were not mentioned in the publication (e.g. country). These groups were advised to provide a footnote with additional information, explaining that the numbers presented include only those publications that had been fully assessed and adding how many publications had not been included in this assessment.

Fig. 1

Search decision flowchart for the scoping review on police violence

Two other components of the analyses were the general characteristics of the literature (based on the characteristics registered in the Excel-spreadsheet, see above) and the results of the limited methodological assessment (see Table 2 for an integrated version of this for the police decision-making topics focusing on Belgian research). A final part of the results were the analyses of the findings in terms of content presented in the form of a narrative synthesis. This led to a broad view of the topic and the different sub-questions outlined in the review protocol. For instance, the group working on integrity violations found that the focus of studies on this topic is on the perceptions of such violations and factors that influence whether or not police officers report unethical situations.

Another important element of the paper was the critical reflection on the review protocol (i.e. critically reviewing all aspects of the scoping review procedure and identifying shortcomings, advantages and challenges). Students were also encouraged to include recommendations for future (empirical) research on their topic (based on gaps identified in the literature) and if relevant practical recommendations. In line with Peters et al. (2015) the latter was not obligatory but frequently generated interesting ideas and reflections.

The main questions regarding the paper involved questions on what needed to be reported in which part of the paper (e.g. if the results of the search processes should be reported in the methodology or the results section) and the word count of each part of the paper. Regarding the latter, no word limits per section were defined as this was part of the learning process of the students and this also depends on the results of the scoping reviews, the number of difficulties that arose, etc. A suggestion for future student assignments may be to include an obligatory part on the division of tasks or a logbook that outlines which students performed which tasks and to which extent the students verified each other’s decisions and results. As we had not included this as a specific objective of the paper, we were not always able to assess the students’ involvement in each phase of the procedure and how they cooperated for the assignment.

Presentations

As a final part of the assignment, all groups presented their results in two combined interactive presentation sessions (one for each research domain) that were organised within the framework of the course.29 They were asked to integrate their findings into the presentation based on the overarching subdivisions we elaborated on when discussing the review topics (see Table 1). As such, students reflected on their subtopics within the broader domain and compared the findings from the different subtopics. Besides three practical questions on the submission of the presentation, the presentation time given to each student and the (online) operating of the PowerPoint when collaborating with multiple groups, there were few questions on the content of the presentations and how to integrate the findings.

The focus of the presentations was on the content of the reviews, implying that students were asked to leave out the methodological discussion of the scoping review method. Most groups first presented the results of the individual scoping reviews and subsequently their overarching conclusions. For instance, regarding decision-making processes, students came to the conclusion that there was an overlap in topics and a great focus on police violence throughout the different reviews. They concluded that there is a focus on quantitative research and came to joint practical recommendations (e.g. concerning education, sensitization and complaints). Concerning police strategies, the students found that hot spot policing and predictive policing can be used as tools for community-oriented policing, especially regarding problem-solving, for instance, because predictive policing can initiate more goal-oriented patrols. Another example concerns the topics of police culture. These groups first explained how both subtopics related to the overall topic of police culture, used media examples to explain the topic a bit more, presented the results of the different reviews and ended with some critical reflections (both per topic and in general) and an overall conclusion. This exercise was very useful for students to get a grip on topics related to their scoping review topic and to integrate findings from different reviews. This resulted in lots of interesting discussions on several topics, but it is outside the scope of this article to fully present these reflections.

After the discussion of the scoping reviews in terms of results, an overall discussion was organised on the scoping review method, difficulties experienced, and how the support was evaluated. This was, after the presentation session, first done by means of a collective Microsoft Teams meeting, but students were also able to provide anonymous feedback in the general evaluation of the course. Many of the students gave positive feedback, stating that they enjoyed learning to work with the method, but also that it was a very time-intensive methodology. A number of these students have chosen to use the scoping review in their master dissertation in a later stage. This indicates that they perceive this method as a valuable way to carry out a literature review. Another positive result was that some students gave training sessions for other students who wanted to use the scoping review method but had not been enrolled in the course. Apart from a thorough knowledge of the methodology, students have also gained in-depth knowledge on a specific topic and have seen this in a bigger picture when they were asked to present their papers in the group session. The assignment was thus useful for students to get acquainted with the method and enlarge their knowledge on certain topics. More generally, we believe that learning students how to consistently and transparently conduct their literature search – as these were the main considerations throughout the assignment – will also help them to carry out other research projects that require transparency and consistency as well. A couple of improvements were identified (e.g. organizing multiple live Q&A sessions), but most of these have already been discussed in previous parts of this article.

Discussion and conclusions

This article is centred around a student assignment as part of their curriculum in an optional course that aimed to teach them how to conduct scoping reviews and get them acquainted with police research topics. Students followed an eight-step scoping review procedure to explore the extent and nature of the empirical research on police topics. These scoping reviews have a couple of strengths that may be underlined. One of the main strengths is the consistency of the reviews, yet still allowing for personalisation based on the topic. This allows for a consistent comparison of the topics on the one hand. On the other hand, we demonstrated that the systematic nature allows different students to work on the same scoping review at a time if properly supervised. Based on the assignment, both the lecturers and the students assessed the scoping review method, the (dis)advantages and the challenges that arose during the execution of the projects. In general, the method was perceived as valuable and lead to the desired results. We do not discuss the scoping review steps here as these have been critically reflected upon in previous parts of this paper. We will however discuss some good practices of this assignment that can help lecturers in setting up their own scoping review assignments for students. We divide these good practices into three phases: before, during and after the assignment.

A thorough preparation of the assignment was crucial for the successful execution of the scoping review projects. It starts by defining the goal of the assignment and the topics to be included in the reviews. Additionally, the lecturers need to anticipate what kind of support students will need to successfully execute the scoping reviews. We recommend lecturers to develop (or provide) several documents to support such assignment: a) an outline of the assignment and the lecturers’ expectations, b) an article explaining the scoping review method in detail, c) a step-by-step guide on how to search databases, journals and websites, and d) a template to register the search process. Throughout these documents, clear instructions need to be given as we noticed that some groups interpreted our guidelines slightly differently (e.g. what is mixed-methods research, some reporting criteria), implying that some things might need a more detailed explanation.

During the assignment, the main issue was to provide continuous guidance, support, and availability to allow a quick follow-up of problems and to ensure the consistency amongst the scoping reviews.30 This guidance relates to advice concerning methodology (which was given by means of a presentation session and supporting documents on the scoping review method and search of databases) but also to getting students acquainted with the research topics, which was done by means of a presentation on the general topics of the scoping reviews. The latter was also important for students to be able to choose their topic of interest and understand what that topic involved. Guidance also involved defining deadlines for students to give them an idea of the time management of each part of the scoping review and to stimulate continuous attention to the scoping review projects. Throughout the entire exercise, the quality of the supervisor is important for the outcome of the project (Sharp et al. 2002). In this regard, we believe supervisors need at least the three following qualities: having project management competences to oversee the general assignment, having expertise in the scoping review method, and having profound knowledge of the topics of the scoping reviews. The latter is one of the reasons why the supervisors chose the topics based on their PhD projects, allowing them to fully support the students in terms content.

After the assignments were completed, the author of this article invested a lot of time in assessing the scoping reviews on police decision-making. On the one hand, this is extremely important to provide valuable feedback to students and enable them to learn from their projects. On the other hand, this was necessary to enhance the accuracy of the findings. For this purpose, the author of this article made an overall Excel-spreadsheet including the results of the different scoping reviews. She verified all the numbers reported by students, the characteristics reported in the shortlists, etc. Furthermore, she removed the duplicates that appeared on multiple shortlists and included an overview of the limited methodological assessment of the studies on the different shortlists. This thorough assessment and verification of all phases of the assignments were needed to ensure the consistency of the findings. As touched upon, there were differences in interpretations concerning the classification of mixed-methods research and the methodological reporting criteria. The supervisor thus made sure that the overall Excel contains correct and consistent interpretations. Although this was a very time-consuming exercise, we strongly recommend lecturers to include a similar assessment in their student assignments to come to correct conclusions.

The assignments were also interesting for the lecturers as they were able to identify gaps in the current (empirical) literature and can direct future research efforts to these gaps. However, having students carry out assignments is not simply a quick solution to perform multiple scoping reviews. The assignment necessitated a high level of guidance and feedback, especially since the students had no foreknowledge of the method as it is not included in any of the other criminology or law courses. Lecturers need to be very accessible and able to answer questions in a flexible way. This need for constructive support was also identified by one of the groups, summarising the necessities for a qualitative scoping review project in their paper: “First, the procedure requires lots of time. To successfully complete this method, the study needs to start in time. Sufficient self-discipline is also required to go through the procedures with sufficient criticism. (…) For example, throughout the procedures, we checked each other’s performance by tracking potential typos or inaccurate interpretations. Although such verifications are time-consuming, the support of the lecturers saved us a lot of time. This scoping review support entailed feedback, a clear description of the assignment and the available Excel-template. Criticism and various checks, therefore, result in higher-quality work. In short: a good division of tasks, interim deadlines and a judicious research team are the basic ingredients for a high-quality scoping review.”

	Search strings	Keywords/index terms used at least twice in the publications on the shortlist	Websites
Integrity violations	Europ* EN politi* EN (integriteit* OF ethisch OF wangedrag)	• (Police/integrity) training/ police education • Police integrity • (Business) ethics	Algemene Inspectie van de federale politie en van de lokale politie 31
	Europ* AND (polic* OR “law enforcement”) AND (integ* OR ethical OR misbehavio*)		Comité P 32
Discretion	Belgi* EN politi* EN (“discretionaire ruimte” OF beoordelings* OF beslissings*)	• Decision-making • (Police) use of force	Comité P
	Belgi* AND (polic* OR “law enforcement”) AND (discretion* OR decision*)		Centre for Policing and Security33
Moral dilemmas	Belgi* EN politi* EN (moreel OF dilemma OF integriteit)	• (Police) integrity (training) • (Police/integrity) training/ police education • Ethics/ethical decision-making • Police use of force/ violence	Comité P
	Belgi* AND (polic* OR “law enforcement”) AND (moral* OR dilemma OR integrity)		Centre for Policing and Security
Ethnic profiling	Belgi* EN politi* EN (“etnisch profileren” OF racis* OF discrimin*)	None	Amnesty International Belgium
	Belgi* AND (polic* OR “law enforcement”) AND (“ethnic profiling” OR “racial profiling” OR discriminati*)		Centre for Policing and Security
Police violence	Belgi* EN politi* EN (geweld OF agressie OF dwang)	• (Police) use of force • Decision-making • (Police/organizational) culture	Comité P
	Belgi* AND (polic* OR “law enforcement”) AND (force OR aggression OR coercion)		Algemene Inspectie van de federale politie en van de lokale politie
Complaints	Belgi* EN politi* EN (klacht OF aangifte OF wangedrag)	None	Comité P
	Belgi* AND (polic* OR “law enforcement”) AND (complaint OR misbehavio*)		Algemene Inspectie van de federale politie en van de lokale politie
Trust and legitimacy	Belgi* EN politie* EN (vertrouwen OF legitimiteit OF gezag)	• Trust (in the police) • Procedural justice • Legitimacy • Social capital • Performance • Corruption	Statistiek Vlaanderen 34
	Belgi* AND (polic* OR “law enforcement”) AND (trust OR legitimacy OR authority)		European Social Survey
Body cams	Belgi* EN politi* EN (bodycam OF “body worn camera” OF “draagbare camera”)	None	Comité P
	Belgi* AND (polic* OR “law enforcement”) AND (bodycam OR “body worn camera”)		Sander Flight35
Community-oriented policing	Belgi* EN politi* EN (“gemeenschapsgerichte politiezorg” OF “community-oriented policing” OF citizen)	• Local (police effectiveness/plan)	Comité P
	Belgi* AND (polic* OR “law enforcement”) AND (“community-oriented policing” OR citizen)		Centre for Policing and Security
Hot spot policing	Belgi* EN politi* EN (“hot spot” OF preventie OF spatio-temporeel)	• Predictive policing • Hot spot (accident) analysis • (Crime/spatiotemporal) forecasting • (Risk terrain/predictive) modeling	Federal police statistics
	Belgi* AND (polic* OR “law enforcement”) AND (“hot spot” OR prevention OR spatio-temporal)		Database city of Antwerp
Predictive policing	Belgi* EN politi* EN (“predictive policing“ OF spatio-temporeel OF “hot spot”)	• Predictive policing • Crime (statistics/mapping/patterns) • Predictive (modeling/analysis) • (Crime/spatiotemporal) forecasting	Centre for Policing and Security
	Belgi* AND (polic* OR “law enforcement”) AND (“predictive policing” OR spatio-temporal OR “hot spot”)		Cahiers Politiestudies 36
(Social) media	Belgi* EN politi* EN (media OF online OF filmen)	None	Comité P
	Belgi* AND (polic* OR “law enforcement”) AND (media OR online OR record*)		Centre for Policing and Security
Violence against police	Belgi* EN politi* EN (agressie OF geweld OF weerspannigheid OF smaad)	None	Centre for Policing and Security
	Belgi* AND (polic* OR “law enforcement”) AND (aggression OR violence OR insubmission OR indignity)		Comité P
COVID-19	Politi* EN (COVID-19 OF corona OF SARS-CoV-2 OF pandemie)	None	Comité P
	(Polic* OR “law enforcement”) AND (COVID-19 OR corona OR SARS-CoV-2 OR pandemic)		Policing and Society (a second journal was chosen because of the recent nature of the topic and the current lack of research reports)