Professionalism among paramedic students: achieving the measure or missing the mark?

BACKGROUND: Professionalism is a pillar of paramedicine. Internationally paramedic curricula emphasize valid assessment of three domains: cognitive, psychomotor, and affective domains (professionalism). Little is reported on competency measures for professionalism specific to paramedicine. Literature suggests that paramedic students, paramedic practitioners, medical directors, and patients believe that professional attributes should have an increased focus.
OBJECTIVE: The objective of this scoping review is to outline valid and reliable assessments that evaluate professional behaviors.
METHOD: This review used Arksey and O'Malley's six-stage scoping methodology. In September 2016, five databases were searched for articles of relevance; these were MEDLINE, Scopus, Google Scholar, PsycINFO/APA, and EMBASE.
RESULTS: A total of 1587 articles were identified after removal of 468 duplicates. Five articles met the inclusion criteria, two of the articles were from the US and three from UK. The studies range from 2004 to 2014. Three different scales were identified but only two were recommended for use. A US-based scale is composed of 11 items and one generic form of professionalism. The UK scale has 77 items and identified 11 factors within 68 items.
CONCLUSIONS: This scoping review serves to describe valid and reliable measures for professionalism among paramedicine by outlining the quantity of instruments evident in the literature. The scoping review aimed to report the scales supporting evidence of validity and reliability. Three scales were identified in a total of five different studies that specifically measured professional attributes in paramedicine. Currently, two scales are available: an evaluation with 11 items and a self-reported questionnaire with 77 items.

Introduction

Professionalism and professional behaviors are an intrinsic aspect of paramedicine. Thus, these core competencies must be assessed with a valid and reliable evaluation. Paramedic curricula emphasize assessment of cognitive, psychomotor, and affective domains with valid tools. A disparity exists between curricula, paramedic students’ beliefs,1 and agencies that employ paramedic practitioners,2 but reflects the most frequent complaint against paramedics.3 Professional behavior evaluations should be precise or the efforts of paramedic educators are diluted by a fundamental flaw of inaccurate measurements.4

The transition of an occupation being recognized is professionalization5 and these changes are initiated in the academic setting.1 As paramedic education transitions into higher education settings the landscape is primed for an overhaul.6 A professional status is earned;7 it is the responsibility of the occupation to act professionally. A professional status is earned, it is the responsibility of the occupation to act professional. The transition process was summarized by Williams et al and identify a publicly recognized certification agency at the next required step in professionalization.8 Achieving a professional status is arguably the most pressing issue in paramedicine.9

Historically literature does not have a universal definition of professionalism. In 1915, Abraham Flexner theorized six generic traits of an occupation to be seen as a profession: involve essentially intellectual operations with large individual responsibility; derive their raw material from science and learning; they work up on this material to a practical and definite end; they possess an educationally communicable technique; they tend to self-organize; they are becoming increasingly altruistic in motivation.10 In 1957, Greenwood theorized a generic list of five attributes that accounted for overlaps in occupations.5

In 1945, nursing identified six aspects of nursing professionalism. The body of knowledge must be taught in a higher education setting, must have practical knowledge transfer, and must constantly expand. Additionally, it should emphasize an altruistic approach to patient care while functioning autonomously.11 In recent years professional behaviors for physicians have been described by Cruess and Cruess12 Swick,13 Steinert et al.14 Table 1 describes professional characteristics dating back over 100 years. It is worth noting how certain traits such as altruism, autonomy, body of knowledge, scientific approaches, and code of ethics transcend time and occupations.

Table 1

Professional characteristics over 100 years

Flexner*10	Bixler and Bixler15	Greenwood*5	Cruess and Cruess12	Swick13	Steinert et al14
Altruism	Altruism	Authority	Altruism	Accountability	Altruism
Educationally communicable technique	Autonomy	Code of ethics	Authority	Advance field	Autonomy
Intellectual operations	Increasing body of knowledge	Sanction of the community	Autonomous	Altruism	Commitment
Practical applications of knowledge	Positioned in higher education	Self-organization	Code of ethics	Autonomous	Competence
Science and learning	Practical applications of knowledge	Systematic theory	Discrete body of knowledge	Code of ethics	Integrity and honesty
Self-organization	Well-defined knowledge	–	Formalized association	Commitment to excellence	Morality and ethics
–	–	–	Lawful	Patient oriented	Responsibility to society
–	–	–	Organized and independent	Regards humanistic values	Responsibility to the profession
–	–	–	Self-regulating (academic)	Self-reflection	Self-regulation
–	–	–	Serve community	Serve community	Teamwork

Note:

Generic occupational trait.

Paramedic programs are increasingly encouraged to standardize summative assessments that take a holistic approach to patient encounters, and these competencies are not limited to technical skills.16 This standardized competency approach “delineates key technical, cognitive, and emotional aspects of practice, including those that may not be measurable”17 albeit physically measurable is more accurate. Quantifying professionalism and the trajectory of learning a competency require a different approach that is a series of objective observations at individual, interpersonal, and societal/institutional levels.18 A single assessment in an isolated setting does not accurately reflect abilities and more observations result in a more precise assessment.19

Nurses and physicians have used students to validate professional behavior scales. Physicians identified professionalism as a core clinical requirement, and in 1998 the Physicianship Evaluation Form was validated20 and in 2005 the Amsterdam Attitude and Communication Scale was validated.21 Nurses also view professionalism as an essential part of patient care, and in 2000 the Nursing Professional Values Scale22 was developed, then in 2003 the Professionalism Inventory Scale.23 Pharmacy validated the Professional Assessment Tool in 2011 with seven institutions participating in the study.24

Professionalism is a requirement across numerous international paramedic curricula.25–29 In the US professional behaviors were originally included in the 1998 emergency medical technician (EMT)-Paramedic National Standard Curriculum30 and repeated in the 2002 EMT-Paramedic National Standard Curriculum.26 To ensure that these competencies are met, the Committee on Accreditation of Educational Programs for the Emergency Medical Services Professions requires documentation of a summative affective evaluation for each graduating student.31,32 Acting unprofessionally has significant consequence and may lead to dismissal.33 Internationally paramedicine moves toward being viewed as a profession but each country has a unique set of challenges ahead.

The Canadian paramedic curriculum requires paramedic students to display 10 professional behaviors throughout practice.27 In the UK, the College of Paramedics Curriculum Guidance is used in educating paramedics and identifies three domains of learning; cognitive, psychomotor, and affective. It specifically identifies while on clinical placement student paramedics are expected to: “Constantly demonstrate a high level or professionalism.”34 As a paramedic practitioner, professional conduct requirements are required for certain certifications.28

In Australia paramedics have made significant progress to become a well-respected profession. The public views paramedicine as a trustworthy profession35 and the government is poised to recognize it as a profession.36

How are key terminology defined?

It is important to calibrate nomenclature when referencing international topics. The affective domain attributes are feelings, appearance, emotional response, attitudes, and behaviors.25,27,31,32,37,38 The affective domain, professionalism traits/behaviors, and professionalism are used interchangeably in this paper depending on the context. Validity is how well evidence and theory support test scores. Common expectations require test developers to display the values of the measure for content validity.19,39 Reliability is the consistency of score across replications of the test.40 Questions are items and refer to how a scale is composed.

Methods

The methodology of a scoping review was considered appropriate for this study to examine the depth, breadth, and research activity. This study utilized a scoping review method to investigate if paramedics or paramedic students are evaluated with valid and reliable evaluations. This scoping review was guided by Arksey and O’Malley’s41 six-stage methodology for scoping reviews. These stages included:

Identifying the research question

Identifying the relevant studies

Selecting the study

Charting the data

Collating, summarizing, and reporting results

Consulting an expert (optional and included).

The methodology identifies five mandatory stages with an optional sixth stage of consultation. The authors selected to include the additional step to help validate results.

Identify the research question

Given the significance of professional behaviors and the scant amount of literature available on whether a professionalism scale is psychometrically appropriate for emergency medical services, the following research question was developed to inform the scoping review: “Are paramedic students evaluated with a valid and reliable measurement tool for professionalism?”

Identify relevant studies

Five databases were searched for articles of relevance; these were MEDLINE, Scopus, Google Scholar, PsycINFO/APA, and EMBASE. The grey literature sites http://www.greylit.org/ and http://www.tripdatabase.com/ were also searched for non-peer-reviewed literature, which yielded no additional results. Along with this dissertation, forward and backward reference searching and hand searching were also conducted. Publications were not limited to a date range because of the previous knowledge of the Brown et al article,43 and no relevant studies were identified before this year. The search terms used in the strategy included “paramedic,” “paramedic student,” “professionalism,” “affective domain,” “professional role,” “measurement,” “reproducibility of results,” “questionnaire,” and “soft skill.” The search was executed by one author (LMB) and refined by LS and BW. An expert librarian validated the search strategy. The inclusion and exclusion of articles was decided upon between LMB and BW. The Supplementary material outlines the complete list of keywords.

Study selection

The search of the five databases returned a total of 2055 articles and 468 duplicates. The 1587 articles remaining were screened by title and abstract. A total of seven articles met the inclusion criteria. After a full paper screening, five articles fulfilled the inclusion criteria:

Articles focused on paramedicine OR paramedics OR paramedic students

Measure professionalism OR professional behaviors OR affective domain

Articles that reported more than descriptive statistics (reliability, internal consistency, p values, and so on)

Self-reported or evaluated professional behaviors.

Articles not in English or with subjective measured results were excluded from the study. This screening process captured four articles included in the review. Figure 1 displays search results and process of study selection.

Figure 1

Search results and study selection.

Arksey and O’Malley41 describe the charting of the data stage as a narrative and analytical approach to identifying data from articles that meet the research aim. Articles were grouped by publication year, author, study location, study type, cohort size, and main finding. Table 1 outlines the five articles selected for this scoping review.

Collating, summarizing, and reporting the results

The five articles included for this review had three unique questionnaires with ordinal scales. Two studies originated from the US; these data were collected on working paramedics by peer reviews. The other three studies originated from the UK. Phase I investigated and developed the questionnaire and Phase II was data collection of self-reported responses from paramedics and paramedic students.

Consultation

In an effort to help authenticate that these four articles were a comprehensive list, seven subject matter experts (SMEs) were contacted independently to help identify additional peer-reviewed articles, essays, or grey literature. At this point, only four articles were known, and the fifth was identified later in the process while finding citations. Six of the seven SMEs responded to the expert consultation. The SMEs were provided the list of four articles and asked if the list was comprehensive for valid and reliable measures to evaluate professionalism within paramedicine. Each SME reviewed the list and provided no additional articles.

Discussion

Of the five articles that met the inclusion criteria, two of the articles were from the US and three from UK. Four of the studies were cross-sectional and examined paramedics and paramedic students. One study was a qualitative workgroup for scale development. The studies range from 2004 to 2014. The results are displayed in Table 2. The aim of this scoping literature review was to map available scales that measure professional behaviors among paramedics or paramedic students. These articles will be discussed now.

Table 2

Overview of included articles

Year	Author	Location	Study type	Cohort size	Main findings
2004	James and Lindstrom42	US	Prospective	1510	The variance in results suggests further refinement with the scale
2004	Brown et al43	US	Retrospective	222	The scale was “good” and paramedics were the most critical of their peers
2011	Burford et al60	UK	Focus group	–	This was a work group of thought leaders to develop a conscientiousness and professionalism scale
2013	Burford et al61	UK	Cross-sectional	323	Data collection and validation of an 11-factor 68-item scale
2014	Burford et al44	UK	Focus group	112	Examine the definition of professionalism across three health care occupations

Evaluation with a five-point scale

The James and Lindstrom42 study involved five paramedic educators and assessed 222 fire department-based paramedics with the Professional Behavior Evaluation (PBE), an instrument published in the 1998 EMT-Paramedic National Standard Curriculum.30 The educators followed paramedics on 8-hour shifts and rated professionalism with a 1–5-point rubric. The PBE has 11 items: integrity, empathy, self-motivation, appearance, self-confidence, communications, time management, teamwork diplomacy, respect, patient advocacy, and careful delivery of service. After principal component analysis, only one generic type of professionalism was identified. The Cronbach alpha coefficient values ranged from 0.46 to 0.79 suggesting that the variance within total score results by each educator was unreliable for recommended use. Researchers suggested that additional studies should focus on assessing additional behaviors.

Evaluation with a binary scale

Brown et al’s study utilized a modified PBE scale from James and Lindstrom with binary responses.43 An 11-item questionnaire was disseminated by randomly selecting nationally registered paramedics in the US and asked participants to evaluate the partner whom they worked most closely with in the past year. The questionnaire response rate was 62% (n=2443) with an overall mean score of 0.68. Integrity had the highest score (0.77) and self-motivation (0.61) the lowest. Integrity falls under code of ethics,33 which is a trait included in professionalism since 1957. The PBE has generic professionalism traits identified by Flexner10 and Greenwood5 such as altruism and patient advocacy. It is worth noting that the PBE has items not identified elsewhere in literature as a professional trait like “appearance and personal hygiene.” These behaviors could be unique societal expectations specific to US paramedicine.13,44 This evaluation has practical implications with collecting objective assessments in a variety of unique settings where peers, preceptors, and educators are evaluators.18,45

Qualitative scale development

The study by Burford et al was a qualitative one with two phases to assess professional behaviors and conscientiousness index.46 The aim of this study identified SMEs from an ambulance agency and a university. Paramedic educators communicating with an ambulance agency is a key calibration to develop fair, achievable expectations.47,48 The think tank group identified objective behaviors that reflected heightened levels of conscientiousness by collecting routine data about attendance and adherence to deadlines. The SMEs also created a framework for desirable professional behaviors. High instances of conscientiousness are evident with goal-oriented behaviors and attention to detail49 and correlate with agreeableness, a fundamental skill for patient assessment.50 Conscientiousness also correlates with empathy,51 and empathetic behaviors lead to decreased hospital stays and increased patient satisfaction ratings.52–54

Self-reports of professionalism

Burford et al’s was the second report of the 5-year study.44 The questionnaire consisted of 79 items. Seventy-seven items used a ranked 1–5 response: professional identity; professional status; adherence to ethical practice principles; interactions with patients; interactions with staff; reliability; competency, knowledge, and improvement; pride in the profession; appearance; flexibility; behavior outside work; and organizational context. The remaining two items were global measures utilized for anchoring purposes and acquired from the American Board of Internal Medicine’s Project Professionalism.55,56

A total of 323 responses were captured from either paramedic or paramedic students. A total of three locations were sampled, two were ambulance agencies and one a university. A total of 113 qualified paramedics responded to the questionnaire, the average years worked as a qualified paramedic was 11 and ranged from 2–37 years.

Exploratory factory analysis identified 11 factors within 68 items and a fit 0.85 (p<0.001). The reliability of 10 of the 11 factors meets or exceeds reliability threshold of ≥0.70 with value ranges from 0.68 to 0.81. Organizational support had the highest threshold (0.81) and adherence to rules had the lowest (0.68). This scale identifies a transparent approach to measuring self-reported professional behaviors, and psychometric findings suggest that the scale is valid and reliable. It is worth stating the overlapping attributes with Brown et al but required >60% more items. The researchers acknowledged the relatively high number of items but this scale fits their aim.

Qualitative views of cross-discipline professional traits

Burford et al’s was a qualitative study to investigate the attributes of professionalism across three disciplines: para-medicine, occupational therapy, and podiatry.44 A series of 20 focus groups consisted of 112 participants from an ambulance agency, two universities, and one college. Exact definitions of professionalism vary but most attributes stem from three categories. This paper was discovered during the reference and citation search. It was then added to results after author consultation.

The design and development of performance assessments such as professional behaviors should reflect the fidelity of paramedic practice and elicit identical responses. Clear expectations for performance assessments and providing care to patients must be a transparent and a predetermined standard, which bolsters validity and reliability.57

Limitations

Scoping literature review is a relatively new process in health care. While systematic reviews address specific questions, a scoping review flourishes when it seeks to identify a broad question. The focus was on identifying what instruments are available and report their values. The lack of medical subject heading terms designated for paramedics introduces unnecessary error into concept searches. Time constraints and resources may have affected the amount of data and literature found for this review. To the authors’ knowledge, no other scoping reviews are available to compare results.

Future research

This scoping review provided an initial description of articles that aim to measure professional behaviors in paramedicine. In doing so, a mismatch of requirements in paramedic curricula versus ability to evaluate professional behaviors with valid and reliable methods has been established. The number of guidelines, standards, and curricula that require measurement of professional competencies far exceeds the feasibility to measure those.

Questionnaires are useful to understand a baseline measurement of self-reported levels but more research is needed to provide an assessment designed for peer evaluators with different approaches to objective observations of individual, interpersonal, and institutional levels. Future studies should utilize qualitative approach for establishing guidelines and standard setting while quantitative for data collection of scales.

Future data analyses of polytomous items scales should focus on item response theory (IRT). Analyses with IRT are sample-independent estimates of ability that achieve stronger statistical profiles.40,58 A drawback of IRT is that it requires large sample sizes but this requirement can be met with collaboration between educational institutes. Additionally, item fairness should be analyzed with differential item function (DIF).40 A DIF test compares discrepancies between two groups such as males, females, or different ethnic groups with similar overall scores.59 It would be unfair for a group with the same ability to pass or even fail an evaluation for no other reason that being who they are. DIF analysis evaluates for fairness at the item level.

Conclusions

This scoping review serves to outline scales for measuring professional behavior among paramedics by describing what is available in literature. The scoping review aimed to report the scales supporting evidence of validity and reliability. Three scales were identified in a total of five different studies that specifically measured professional attributes in paramedicine. Currently two scales are available, an 11-item rubric and a self-reported questionnaire with 77 items.

This scoping review provides guidance with future research of measuring professional attributes in paramedicine by highlighting the suggested scales that should be used to identify strengths or weaknesses. Professionalism has significant connections between paramedics working collaboratively in a large health care system to meet individual, peer, and societal expectations while adhering to standards of excellence.

Table S1

A keyword search

Concept 1	Concept 2	Concept 3
“Emergency medical services”	“Cronbach’s alpha”.tw	“Affective domain”.tw
“Emergency medical technician”	“Factor analysis”	“Professional role”.tw
“Emergency medical technician$”	“Factor analysis”.tw	“Soft skills”.tw
“Paramedic student”.tw	“Internal consistency”.tw	*Professionalism/ed, st (Education, standards)
Ambulance	“Reproducibility adj1 results”	Profession*
EMS	Dimensional.tw	Professionalism
EMT	educat* measurement	Professionalism.tw
Paramedic	educat* standards
Paramedic$.tw	educat* tool*
Paramedicine.tw	measure* tool*
	Measurement
	Model.tw
	Questionnaire.tw
	Reliability.tw
	Statistical
	Survey
	Survey.tw
	Validity.tw

Abbreviations: EMS, Emergency Medical Services; EMT, Emergency Medical Technician; tw, text word; $.tw, wild card text word.