A report on the development and implementation of an assessment of competence scheme for the forensic sciences.

Background

There has been increased interest recently by forensic service providers in the general transferable and key practical skills attained by graduates in selecting them as potential new employees. As the forensic service providers represent key stakeholders of The Chartered Society of Forensic Sciences, the Society commissioned a detailed analysis to identify the key skills sought by employers with a view to designing and implementing a pre-employment assessment of competence (PEAC) which would be undertaken by a number of Universities holding the Society's Accredited course award, together with candidates from non course-Accredited universities. The approach adopted was to undertake (i) a review of the literature and current thinking on employers' need of a forensic science (or science) graduate, (ii) a review of the commitment of the Accredited universities to the development of key practical skills, and (iii) a review of the protocols available for the assessment of practical psychomotor skills. The employers canvassed were supportive of the venture and felt that the product could be of considerable help to them in their recruitment processes and training. The benefit of PEAC to employers would be to save time and money on the recruitment process and give reassurance to employers that any new employees can carry out the initial, necessary, basic tasks required at a scene or in a laboratory. The benefit of PEAC to education providers is to ensure that the curriculm content better matches employers needs in the fast-developing discipline area of forensic science.

Having established the key specific core skills and methodologies for their assessment (see later), the Society identified and approved key institutions capable of operating as national Competency-Based Assessment Centres together with a pool of Assessors (academic and forensic practitioners) in order to implement the PEAC scheme on behalf of The Chartered Society of Forensic Sciences. A number of supportive forensic practitioners and employers contributed significantly at the design stage of the study and maintained an active interest in its implementation and enhancement.

As part of the implementation, the PEAC Development Team designed a range of support documents including a Candidate's Handbook (which includes a detailed Candidate Code of Conduct), Assessor's Checklist, Delivery Manager's Checklist, Competency Assessment Centre Checklist, Briefing Notes for Delivery Manager, Assessor Marking Sheets, Assessor Feedback Sheets, Laboratory Protocols, Reasoning Questions (inductive and deductive), an MCQ Question Bank, General Examination Record Sheets, Assessor Final Report pro forma, and Delivery Manager Report pro forma. These helped to ensure a uniform/consistent approach to the assessments across a range of Competency-Based Assessment Centres and by a number of External Assessors and Delivery Managers. Some of the items used in the practical assessments were supplied by the Society, again this augmented consistency between Assessment Centres.

Objectives

The scope and level of skills attainment from undergraduate and postgraduate degree programmes is of significant interest to potential graduate employers (from large public or private organisations to sole traders and SME's). As indicated earlier, the Society developed a new formal self-funding scheme for the pre-employment assessment of competence (PEAC) in forensic disciplines. This takes the form of a formal certification process. The benefit of the PEAC project going forward for employers is to save time and money on the recruitment process and give reassurance to employers that any new employees who have the Society's PEAC certificate can carry out the initial, necessary, basic tasks required in a laboratory. The benefits of the additional PEAC award to students are evidence of knowledge, skills, reasoning and problem-solving assessments approved by industry; a real opportunity to enhance their CV and personal statement and thereby enhancing their employability and ‘attractiveness’ to employers; and gives the prospective employer confidence by having a certificate of competence endorsed by the professional body to supplement their academic qualifications. The benefits to the University are increased confidence in educational standards and skills for their students gaining the award as it is independently assessed; esteem (shows active involvement with professional body and industry); provides evidence of the full range of facilities sufficient to fulfill PEAC requirements; offers additional marketing capability; and provides competent Assessors who can provide feedback to the professional body and university on course content/facilities and resources.

It was concluded that the most obvious way to implement the PEAC scheme within the UK was to establish a series of Competency-Based Assessment Centres distributed nationally – each being based within a University with a formally Accredited course. In selecting an Accredited course within the university for Centre status the Society sought in the first instance assurances with respect to staff relevant expertise, internal quality processes, the teaching staffs’ declared commitment to the PEAC scheme and adequate and appropriate resources. In the steady state issues such as effectiveness, transparency and “customer” focus were also considered. The final goal (for both parties) was the adoption of a culture of excellence.

All Assessors were formally appointed by The Chartered Society of Forensic Sciences, following an initial Assessor Training Day. This was a parallel model to that for used for Educational Accreditation, with applicants submitting an online application form that was assessed by an appropriate panel.

A number of models were suggested for the operation of the PEAC scheme in which:

Candidates took their assessment at another university from the one they studied and using Local Assessors

Candidates took their assessment at the university they studied at with External Assessors present

Candidates could choose where to take the PEAC assessment with the Society ensuring an Assessor was not known to the candidate

The third route was chosen as being the most practical and transparent. It was agreed in the first year of implementation that there would be two External Assessors supported by an internal Delivery Manager (to help with local organisation and planning) and an Observer from the PEAC Development Team. In subsequent years the Observer would not always be required.

The length of validity of the PEAC Certificate was considered and was agreed that there should be no time bar, but the Certificate would be dated. Candidates would, if they so wish, be eligible to retake the assessment at a later date to refresh/update their Certificate. It was agreed for the first implementation that there was a need for three Pilot Events (one for each topic area – see next Section) in order to ensure an effective and functional scheme.

Scope

From meetings with forensic employers, forensic practitioners and academics it was concluded that there were two key types of core skills important in providing reassurance to employers that any new employees can carry out the initial, necessary, basic tasks required in a laboratory or scene, these being general transferable skills and key practical skills. Arising from the meetings, the following general transferable skills were identified:

Integrity

Translating skills

Technical expertise

Colour vision

Measurement – length, volume etc.

Use of calculation of units (both Imperial and SI)

Dilutions/concentrations – pipette usage

Manipulation of trace evidence

Compliance to method/process, ability to follow procedure

Note taking

Ability to challenge and question appropriately

Health & safety

Contamination

PPE/COSHH

Manual handling/lifting

Critical (constructive) analysis

Knowledge and understanding of ISO standards and quality management systems

Experimental design

Lateral thinking

Calibration

Motor skills

The importance of these basic skills were reflected as elements in the assessment protocols. A suggested model for testing these skills during a PEAC assessment would be to give a candidate a series of protocols which they would have to follow, each covering a number of items on the above list of fundamentals. The advantage not just for employers but also for graduates would be to help to make them confident of their competence within a laboratory.

The key generic practical forensic skills that were identified from the meetings were considered under two encompassing headings, these being Laboratory Analysis (subdivided into Forensic Biology and Forensic Chemistry) and Crime Scene Investigation. As such, these mapped very well to two of the three Society's Component Standards for the Accreditation of forensic science programmes of study [1]. The key skills identified under each heading were photography, fingerprinting, trace recovery and communication for CSI; DNA profiling, blood pattern analysis, hair and fibre analysis, body fluids and damage for Forensic Biology; and drugs and toxicology, analytical skills and contact trace for Forensic Chemistry.

Constraints

It was noted that it is relatively easy to assess an individual's competence in carrying out a laboratory technique by direct observation. However, it was noted from discussions with Assessors and forensic practitioners that many of the practical techniques would take considerable time to execute and hence it would be impossible to assess a range of skills within a restricted time period of e.g. one day. For this reason, innovative ways were required to overcome this limitation. Thus it was possible to observe a student setting up an experiment then letting it run and be completed and assessed by the assessment team. In a similar way, experimental results can be supplied to the student for their analysis and subsequent assessment.

Assessment protocols to consider how to assess the various key practical skills that had been identified for use in the PEAC assessments included:

direct observation of practical skills (undertaken by Assessors/Observers)

simulations

case studies

mini tasks/experiments

problem solving approaches

technique assessment

data processing

analysis and interpretation

oral testing

report writing

Assumptions

The Chartered Society of Forensic Sciences (CSFS) developed and introduced an accreditation scheme for Higher Education Institutions (HEIs) delivering courses that contain forensic components. The courses intended for Accreditation largely comprised of Bachelors’ degrees or a post graduate qualification such as a taught Masters degree. The scheme was developed to help establish and maintain standards of education in forensic science and involved major employers, forensic practitioners and relevant Government bodies. The scheme aims to assist HEIs in achieving these standards by providing access to advice and guidance. Accreditation is based on a series of Component Standards [2]. The Component Standards address specific areas of forensic practice. They are intended to augment, not replace, the underlying scientific knowledge of the forensic components. Accreditation through consideration of the Component Standards focuses primarily on the course content whereas the PEAC Competency Scheme focuses primarily on the individual student, a distinction that is central to the assessment.

As developments in technology and science are occurring rapidly within the forensic science sector, it is essential that the assessed skills will be regularly reviewed to ensure they remain up to date and fit for purpose for professionals in forensic science roles and hence enhance graduate employability.

Risks

A number of issues associated with the self-funding fee structure were considered for the first year of implementation. Part of the costs associated with the PEAC assessment were funded in kind by the Assessment Centre. As the candidates were already paying university fees, the remaining funds were subsidised in part by the Society in order to secure an acceptable level of fee for the candidates. This arrangement was reviewed and minor modifications were made for subsequent years. The opportunity for a retake on failure of the PEAC was discussed. It was agreed that one retake would be made available, within a set time, and the retake costs incorporated into the business model. This provided the opportunity for lower costs for retakes.

The CSFS would not publish a list of passes/fails but candidates would need to provide such information if challenged. A clear, transparent syllabus would be made available for candidates and Assessors alike to view. Application was open to all interested candidates who are members of the Society at any level.

Whilst this is ostensibly a national test, there is no requirement for this to be run on the same day at each Centre. If application numbers were high there would be a requirement to run the PEAC on more than one day. It was felt that it would not be necessary to have significant content change between the exam days. By having a pool of questions/practicals there would be enough variety to cater for such additional repeat PEAC events.

Clearly, there are a number of key considerations and decisions required in order to ensure the success of the PEAC scheme. These include:

extending opportunities for the Society's members

ensuring the PEAC scheme is attractive to potential applicants

being viable (in fiscal terms as well as sustained market demand)

enhancing the reputation of the Society

further strengthening the linkage between employers and forensic providers and the Society

the provision of an award that is recognised by employers

promoting harmonisation between Accreditation/Recognition and CPD/CPC

Deliverables

Two specific topic areas within Forensic Science for PEAC consideration were identified as being Laboratory Analysis (comprising Forensic Biology and Forensic Chemistry) and Crime Scene Investigation and it was propitious that these mapped exactly to the two Component Standards in the Accreditation scheme (the third Component Standard (IEPE) stands for Interpretation, Evaluation and Presentation of Evidence. As PEAC units are developed in future, it is likely that these could serve as stand-alone CPD thus providing a unified framework for all aspects of knowledge and skills operating within The Chartered Society of Forensic Science. It was believed that this scheme would represent a win:win:win situation for the Society, the course-Accredited universities electing to operate a PEAC Centre and the employer. For the Society, the benefits would be the provision of a continuum for Accreditation and CPD and as an aid to further developing links with employers and forensic providers. For the universities the benefits would be enhanced graduate employment, encourage student engagement, increased programme marketability, kudos for the university, increased employer contacts, closer links with the Society, opportunity to develop additional CPD and potential for external income generation. For the employer the PEAC certificate provides additional assurance of the candidate having subjected themselves to an independent assessment of competence in terms of transferrable skills and role-specific skills such as crime scene or laboratory - biology or chemistry. This assurance can give the employer the confidence to offer the candidate an interview and impact on the detail of employer training.

Issues in implementation

The selection of competency-based assessment centres (CAC) was important in terms of (i) the geographic location, (ii) the availability of specialist facilities within the University such as a crime scene or a range of analytical instruments and (iii) consideration of the candidates ability to travel.

Agreeing the Assessor to candidate ratio was a difficult dilemma with the need to ensure there was sufficient contact between the candidate and the Assessor while at the same time meeting the self-funding model to be cost effective. The ratio of one Assessor to five or six (maximum) candidates was considered to be viable and effective.

Assessors had ongoing discussions with the candidates throughout the assessment to ensure their knowledge and understanding as well as the actual skill to perform the task. To support the Assessor templates were provided to enable sufficient notes on each candidate was not overly time consuming. On occasions candidates would be unfamiliar with the particular technique being requested to carry out a task for example presumptive testing for blood or recovery techniques for footwear impressions such as the use of an electrostatic dust lifting apparatus. Under these circumstances the Assessor would have to explain and demonstrate before the candidate would carry out the task. This had the benefits that the candidate would actually learn something new as part of the assessment, would not necessary be penalized and the Assessor would feedback the information to the Society to enable future discussions with the candidate's university.

Although there was a ratio of Assessor to candidate there would always be two Assessors at any one CAC enabling both Assessors to communicate with the candidates – this meant the Assessors could independently complete an assessment as well as coming together at the end of the day to agree overall competency levels to make a pass/fail recommendation for consideration by the external examiners panel.

In the first year oral assessments were carried out with the candidate. Although valuable this was very time consuming especially for the candidates as they had to wait in turn after an intense demanding day. It was therefore agreed that this vital communication skill would be incorporated into the whole day as the Assessor and candidate interacted.

The PEAC indirectly created a dilemma for the Society in terms of its educational course accreditation scheme. Like any professional body which provides course accreditation there can be a grey area between the professional body role and that of an external examiner. The professional body does not necessarily go into the same depth or areas of the course as the external examiner in terms of exam papers, marking schemes and moderation.

The other dilemma for the Society was that of the employer view. The employer will have a view about a course being accredited by a professional body. In the case of the Society there are Component Standards which were drafted and agreed between practitioners (employers) and lecturers/researchers (university). But what the PEAC has shown is that there were some common inadequacies such as it was not uncommon for candidate to be ‘rusty’ with certain techniques because there was only sufficient time to be able to either demonstrate or use a technique only once in their course. The Society and employers may see this as a core practitioner skill such as searching and note writing, recovery, at a search bench, of contact trace material and subsequent evaluation and analysis. The forensic course being based on good science with the legal context, provided means that it is not always possible for the students to become proficient and therefore competent in that specific skill. The question for the Society Quality Standards Committee was how deep should the accreditation scheme be expected to probe?.

Fortunately the universities which have provided a cohort of candidates for the PEAC have been able to receive confidential reports on the overall candidate performance thus enabling the core skills to be continually tested thus ensuring that the employment-ready graduate becomes a reality. This model, involving specific university cohorts could prove an effective model going forward.

The PEAC Assessment comprises of a knowledge test (multiple choice questions, short answer questions and experimental reasoning questions) administered prior to the day of the test. Candidates are required to pass this at a level of 60% or more to be eligible to attend the practical PEAC Assessment Day. On the day of the test, the candidates undertake a diverse range of activities from measuring, reasoning, crime scene scenarios and performing a number of analytical skills in the morning and subject specific activities in the afternoon. Each element is observed by one or more External Assessors who also interact with the candidates to assess their understanding and rationale for the approaches they have adopted. For those candidates who possessed a recognised individual learning plan (ILP), a small additional time allocation was provided. The nature and format of the assessment that was developed independently for PEAC reflects well the approach derived from a comprehensive study undertaken by the University of Virginia's Office of Institutional Assessment and Studies [3].

Results of the first three years of implementation

The educational system has the underlying principle of knowledge and understanding. This was demonstrated in the on-line knowledge tests with 85% candidates passing first time. The aims of the practical assessment was to give candidates the opportunity to demonstrate competence in specific work-related skills. A summary of the assessment data is shown in the figure below. Whilst it was apparent that the candidates were able to demonstrate the underlying knowledge principles, they were not always able to consistently demonstrate the practical aspects and thus resulting in lower pass rates for the practical aspects.

Also interestingly was that the academically able candidates were not always the best candidates for practical focused roles.

Discussion

In analysing the approach to the teaching and learning of forensic science in universities there is a very evident parallel with the approach adopted for nursing education. Thus, we have theoretical education delivered through lectures and tutorials supported by asynchronous vocational practical learning (praxis) delivered through crime scene scenarios, case studies, placements etc. In their article on nursing education, Chapman & Clegg [4], considered the notion of a theory-practice gap. Thus, nursing students often find that what is taught in theory is “not the same as that experienced in the clinical environment”. Students of forensic science frequently share this experience. This highlights an important distinction between the academic and practical content of a professional programme. The emphasis and higher status on academic content is evident in most undergraduate programmes, as reflected in the relative contributions that factual knowledge and understanding make to the overall performance/degree classification [5]. By focusing on professional competence, the PEAC scheme helps to redress this mismatch and provides a more comprehensive measure of skills attainment from the degree programme. Recently, a similar disconnect between the theory and practice of digital forensics has been observed [6] and these findings will be considered when a PEAC scheme for digital forensics is designed and implemented.

A second parallel with medical/nursing education is the concept of the “golden hour” – a medical term which refers to a critical time period (not necessarily exactly 1 h duration) following an injury, during which there is the highest likelihood that prompt medical treatment will prevent death. It has been suggested that such a concept applies to forensic incident response [7]. The approach proposed is an OODA loop (this refers to the decision cycle of observe, orient, decide, and act) at the scene of an incident – in the case of a forensic crime scene this might comprise early detection, early reporting, rapid response, good on scene practices, care in transit and solid forensic analysis. These represent the key elements that the various activities undertaken on the PEAC Assessment Day measure. For example, the Crime Scene PEAC requests candidates to evaluate a scene, decide what to recover and in what order; to understand the intelligence value and overall value to the investigation and court in identifying the perpetrator.

Similarly with the laboratory PEAC, candidates receive a packaged item to log, search and recover debris; to identify any contact trace material, and like the scene PEAC give an intelligence value and overall value to the investigation and court in the identification of the perpetrator.

In the 2015 Annual Report of the Government Chief Scientific Adviser [8], Sir Mark Walport stated “to maximise benefits for the UK from the emerging analytical techniques that power forensic analysis, this will depend not only on our mastery of the tools themselves but also on a supply of people with the right spectrum of skills”. What the Society sought was a formal certificate of attestation of competence that would be of value to both the candidate and potential employers. From the experience and outcomes to date the PEAC scheme is considered to be a successful, valid and additional measure of forensic professional practice. Development of the PEAC model continues and will incorporate digital forensics going forward.

Declaration of interest

None.

Conflicts of interest

None.