Continuing Development and Initial Validation of a Questionnaire to Measure Sonographer Skill-Teaching Perceptions in Clinical Practice.

OBJECTIVE: Medical ultrasound examinations are performed by diverse professional cohorts sonographers are one group. Little evidence exists regarding the teaching practices used in medical ultrasonography and their effectiveness. We report the continued development and validation of an instrument to measure sonographer skill-teaching practice perceptions (SonoSTePs).
METHODS: An online survey was administered to a convenience sample of sonographers who were employed in Queensland, Australia. This paper reports on the continued psychometric testing of the measurement tool.
FINDINGS: The 25-item scale demonstrated good internal reliability. Exploratory factor analysis generated four factors with acceptable internal reliability: Factor 1 (Skill execution feedback, Cronbach's α = 0.89), Factor 2 (Cognitive overload, Cronbach's α = 0.68), Factor 3 (Teach new skill, Cronbach's α = 0.70), and Factor 4 (Assist learners scanning, Cronbach's α = 0.67). The combined instrument value was 0.83. The weighted kappa of the test-retest items identified that the majority of items achieved an interrater level of agreement of ≥0.5.
CONCLUSION: Results indicate that the SonoSTePs instrument items and factors are underpinned by theories and principles related to teaching a complex psychomotor skill. The initial data suggest that the tool is both reliable and valid.

Introduction

Medical ultrasound is now a ubiquitous imaging modality, and is used by a diverse professional cohort, for a wide range of clinical applications and contexts. It is a salient point, that for more than 40 years, medical ultrasound imaging has been largely performed by sonographers and doctors, in established disciplines such as radiology, cardiology, vascular surgery, and obstetrics and gynecology. In recent years, the clinical application of medical ultrasound in the health and education contexts has expanded. For example, rheumatologists, rather than solely relying on palpation of anatomy, use ultrasound imaging to guide targeted injections into tendons, bursa, and joints [1]. Similarly, midwives are using ultrasound imaging [2] to determine fetal number, presentation, and placental location. This ultrasound information assists the clinical management of patients. Furthermore, medical students attending universities in the United States use ultrasound to assist cognition of anatomy and pathophysiology during their undergraduate education [34]. Nevertheless, the single largest cohort to use ultrasound imaging in a diagnostic and clinical capacity remains sonographers. Despite this, there is no literature that we could identify which outlines the instructional approaches used by sonographers to teach the basic scanning skills required for competent clinical practice. Therefore, there is no knowledge of the teaching practices used by the educators in the profession, and consequently they cannot be objectively reviewed, examined, and assessed.

Anecdotally, the ultrasound profession uses a master apprentice or two-step skill-teaching model [5] to guide the acquisition of essential psychomotor skills. The model relies on the educator demonstrating and describing the task steps to the learner. To date, this instructional model has served the profession well. However, contemporary skill-teaching and motor-learning domain literature has identified that additional instructional steps are needed when teaching multipart and complex psychomotor skills [6], where a skill must first be acquired, then performed, and lastly learned.

One method to measure sonographer skill-teaching practice perceptions is to use a validated survey instrument. A review of the literature failed to identify a suitable measurement tool for this purpose. Thus, the sonographer skill-teaching practices survey, labeled SonoSTePs, was developed to identify and measure the major skill-teaching practices and perceptions used by sonographers, who perform formal or informal clinical teaching and supervision roles. To date, the content and face validity of the SonoSTePs instrument has been established [7]. However, as identified from the literature review, the analytics of the theoretical principles and instructional behaviors required to teach a complex and multipart psychomotor skill have not been determined. The five theoretical subscales related to the domain of teaching a psychomotor skill in the clinical health arena include: teach new skill, visual exemplar, cognitive overload, immediate error correction, and skill practice. The purposefully written items that explore the subscales related to teaching a psychomotor skill in the SonoSTePs instrument are yet to be validated, and therefore determine how accurately this newly developed scale will measure perceptions of skill-teaching practice.

The purpose of this paper is to report on the continued development and validation of the SonoSTePs instrument. In particular, this paper aims (1) to determine the instrument’s reliability (test–retest and Cronbach’s α coefficient) and (2) to report on the steps taken and outcomes of an exploratory factor analysis.

Method

Continued development of the SonoSTePs instrument

In 2012, we commenced development of the SonoSTePs instrument using published principles of survey design and construction [89] to measure the perceptions of sonographer skill-teaching practices. The discriminant ability of the instrument was improved by using a 7-point Likert-type rating scale [7].

The revised and reworded SonoSTePs P3 instrument has two primary components. The first consists of 23 questions seeking demographic information, clinical practice roles and qualification, skill-teaching behaviors, use of simulation to teach scanning skills, and four validation feedback questions. The second component contains 28 items exploring five theoretical domains related to teaching a psychomotor skill contained within a Likert-type rating scale.

Recruitment and sampling

The population targeted to receive the P3 survey included Queensland sonographers registered with the national and compulsory regulatory agency, the Australian Sonographer Accreditation Registry (ASAR). Schleyer and Forrest [10] explain that it is important when targeting an online population that the validation cohort is representative of the broader professional population and possesses the skills to undertake the instrument validation. The targeted cohort was purposefully and strategically chosen to pilot test the instrument as this professional group: (1) was composed of sonographers who worked in a range of geographically disparate locations (which included metropolitan, semirural, rural, and geographically remote areas); (2) performed a wide range of clinical examinations (general, breast, vascular, cardiac, pediatrics, obstetrics and gynecology, and musculoskeletal); and (3) executed a diverse portfolio of professional practice roles (clinical, academic, and education roles). Therefore, the cohort possessed a variety of skills and expertise, and significantly included academics and clinical tutors employed at two universities.

Questionnaire dispersal and administration

The SonoSTePs P3 pretest, test–retest, and P3 statewide survey were administered electronically using Survey Monkey software (www.surveymonkey.com), an internet-based survey tool for questionnaire administration and data collection.

A pretest was performed prior to dispersing the P3 survey. This was done to ensure the survey was operational with no access or progress issues [711]. The “dummy” responses were deleted from the data repository, to avoid contamination of the test–retest and P3 data.

A convenience sample of sonographers who resided in Queensland, Australia, were invited to voluntarily participate in the test–retest surveys. Participants were e-mailed twice (18 days apart via a 3rd party) with an invitation, introductory letter, and a link to follow to the online test and retest SonoSTePs P3 survey.

In November 2013, staff working at the ASAR sent an introductory e-mail and hyperlink to 835 Queensland sonographers, who were registered with ASAR and had “opted in” to receiving professional electronic communication. Sonographers were invited to voluntarily and anonymously participate in the validation of the online version of the P3 survey. Two reminders were sent: the first at 7 weeks after the initial invitation, and the second one 4 weeks later. Therefore, responses were collected over a 14-week period. No incentives were provided to participate in this research project.

Statistical analysis

Descriptive statistics

The P3 data was downloaded from http://www.surveymonkey.com/ website onto an Excel spreadsheet and then imported into SPSS (IBM SPSS Statistics for Windows, Version 23.0; IBM Corp., Armonk, NY, USA) to perform data analysis. Descriptive statistics were used to describe the demographic characteristics of the cohort. Item correlation, and parallel analysis was performed using StataCorp. 2015 (Stata Statistical Software: Release 14; StataCorp LP, College Station, TX, USA).

Temporal stability

The SonoSTePs P3 temporal stability was calculated using test–retest response data and applying a weighted kappa (kw2) with quadratic weights for ordinal items (survey questions) [121314]. This statistic measures the interrater agreement at two times points, usually 14 days apart [1516].

Establishing the SonoSTePs item correlation, factor loading, and internal consistency

Questionnaire data suitability was assessed using the Kaiser–Meyer–Olkin (KMO) measure of sampling adequacy and Bartlett’s test of sphericity, where KMO and Bartlett’s values above 0.5 were considered suitable for exploratory factor analysis (EFA). We used Cronbach’s α to explore the strength of the relationship of each item to the factor [17]. Oblique direct-oblim rotation was used to further simplify the factor structure [18].

This study was approved by Flinders University Behavioural Research Ethics Committee (SBREC 5584).

Results

Assessing the temporal stability of the SonoSTePs instrument

There were 11 respondents who completed both the test and retest surveys. The kappa values ranged from 0.1 to 0.8, where k = 1 and k = 0 corresponded to perfect agreement and no interrater agreement, respectively. The majority (52%) of the 25 SonoSTePs P3 items achieved an interrater level of agreement of ≥0.5. This suggests acceptable internal consistency [16] for the SonoSTePs instrument. However, the small sample number precludes unmerited reassurance.

SonoSTePs P3 survey

After the initial invitation, 35 sonographers responded to the survey. Raffi et al [19] assert that the use of follow-up e-mails is a potent tool to increase the survey response rate. Therefore, two further e-mail reminders were dispersed, and this resulted in an additional 74, followed by 33, responses, respectively. A total of 142 of 835 sonographers responded to the P3 validation survey, giving a 17% response rate. Nineteen respondents did not complete the rating scale questions, and these responses were removed from the P3 factor analysis data set.

Participants ranged in age from 25 years to 66 years (mean, 44.8 years), and 81% were female. The majority were employed in private practice (55%), followed by public hospitals (35%) and private hospitals (8%), and 2% were employed in a university capacity. Regarding the area of sonographic practice, 55% performed general sonography, followed by cardiac (22%), obstetrics and gynecology (9%), breast (7%), vascular (6%), and pediatric sonography (1%). The participants identified that their primary role was to scan patients (83%), function as a chief sonographer (11%), and performed an academic or clinical teaching/tutoring role (6%). Most participants had not completed additional health education training or qualification (78%).

Qualitative results

The validation process was primarily focused on question clarity and survey content. A descriptive content analysis of the qualitative feedback found that most respondents found the survey questions to be complete and not difficult to interpret. There was unanimous feedback from 57 respondents, about the ability to provide written feedback in open text questions.

Correlation analysis

Inspection of a correlation matrix depicts the presence or absence of interrelationships among a set of variables, or the set of items in a scale. It is a summary of the associations between items in a scale [16]. Scrutiny of the matrix in Figure 1 reveals the presence of many coefficients with a numerical index ranging from negative to positive values, which indicates the strength and direction of the relationship between two items [11]. As a guideline, the strength of the relationship between variables can be broadly classified into small (r = 0.1–0.29), medium (0.3–0.49), and strong (0.50–1.0) [20]. Review of the matrix identified coefficients with a numerical index of ≥0.3, as well as a clustering of items. We identified that there were four clusters of items demonstrating this relationship between the items in the correlation matrix [16]. Therefore, we could justify progressing to perform EFA. This conclusion can be further tested using Horn’s parallel analysis [21], a technique to identify the number of factors that cluster within an item pool and can be extracted [11].

Figure 1

Correlation structure of 27 items for factors of skill practice feedback, cognitive overload, teach new skills, and assist learner’s skill acquisition.

Parallel analysis

The scree plot in Figure 2 depicts four factors with an eigenvalue >1, and this suggests that four factors maximize the total variance explained by the combined factors [22]. Horn’s parallel analysis (Figure 2, top red line) was used to determine the number of factors to retain within an item pool, and this step preceded performing EFA. The resultant parallel analysis plot was transposed over the Cattell scree plot (see Figure 2), and the factors that lay above the juncture of the two graphics, suggests the number of factors to be extracted from the item pool [82123]. For the P3 item pool, there are four factors above the point of intersection, suggesting that this number be retained when performing EFA.

Figure 2

Parallel analysis and scree plots (Figure 1) confirmed a four-factor model existed in the P3 questionnaire.

Changes to the item pool

Prior to further data analysis, the P3 instrument number was reduced to 25 items. This reduction occurred because one redundant question was removed from the item pool, and two additional items exhibited a negative correlation value.

Exploratory factor analysis

We theorized that the items within the SonoSTePs P3 rating scale were associated with teaching a complex psychomotor skill. To test this hypothesis, we analyzed the data using principal component analysis with maximum likelihood extraction methods, and oblique (direct oblim) rotations. The KMO value was 0.74, which exceeded the recommended value of 0.6 [24], and Bartlett’s test of sphericity, p ≤ 0.05 [25], reached statistical significance. Principal component analysis revealed four factors explaining 24.1%, 9.3%, 8.3%, and 6.9% of the variance, respectively. Therefore, using the four-factor model, the total variance across the items was 48.6%.

The component correlation matrix for the rotated four factors revealed a small correlation between the factors, and this suggests that the four factors are independent and uncorrelated.

The process identified groupings of items and four factors. The item groups were recoded according to the predominant instructional step they mostly closely represented when teaching a psychomotor skill. The four factors which were extracted from the scale items can be seen itemized and described in Appendix 1.

Four factors were extracted from the scale items in the SonoSTePs instrument and they have been indexed and described.

Click here for additional data file.

Reliability–internal consistency

Table 1 provides a comparison of the reliability information for each of the four factors and the total items. The initial reliability of the four-factor scale ranged from 0.67 to 0.89 for each of the factors. The reliability assessment for the combined rating scale items (n = 25) was 0.83. For two of the factors [Assist learners scanning (ALS) and Cognitive overload (CO)], this did not meet the generally accepted 0.7 minimum threshold for scale reliability [811]. However, Moore and Benbasat [26] suggest that the internal consistency, or the extent to which items within each scale are correlated with one another, should be of a value of >0.6 at the initial validation stages. Furthermore, the mean intercorrelations for these factors (“ALS” and “CO”) were 0.20 and 0.27, respectively, suggesting a moderately good relationship between the items. This further supports that the initial alpha results for these factors were acceptable. Also, both factors contained <10 items per construct, and mathematically this makes achieving a Cronbach’s α of ≥0.7 difficult [8]. Acknowledging these limitations and development guidelines, the internal consistency and initial reliability of the four-factor instrument are acceptable.

Table 1

Results of factor extraction and analysis.

Factor	Factor items	Factor loadings	Variance explained	Cronbach’s α	Alpha for all items (n = 25)
Factor 1: Skill execution	SEF1	0.75	24.1%	0.89	0.83
feedback (SEF)–7 items	SEF2	0.75
	SEF3	0.82
	SEF4	0.71
	SEF5	0.75
	SEF6	0.59
	SEF7	0.77
Factor 2: Cognitive	CO1	0.85	9.3%	0.68
overload (CO)–2 items	CO2	0.69
Factor 3: Teach new	TNS1	0.32	8.3%	0.70
skill (TNS)–7 items	TNS2	0.66
	TNS3	0.65
	TNS4	0.81
	TNS5	0.67
	TNS6	0.45
	TNS7	0.22
Factor 4: Assist learners	ALS1	0.34	6.9%	0.67
scanning (ALS)–9 items	ALS2	0.50
	ALS3	0.50
	ALS4	0.31
	ALS5	0.42
	ALS6	0.35
	ALS7	0.70
	ALS8	0.32
	ALS9	0.52

The Cronbach’s α for Factor 1 SEF was improved by the removal of item 13 H. Theoretically, this item was more aligned with the items in Factor 3 TNS and therefore was placed in this item pool. Also, item 13B is related to the theoretical principles of teaching a new skill and was therefore moved from Factor 2 CO and placed in Factor 3 TNS. Additionally, this change improved Factor 2 CO Cronbach’s α from 0.51 to 0.68.

Discussion

This paper describes the continued development of a new self-reporting instrument that explores sonographer psychomotor skill-teaching perceptions in clinical practice, labeled SonoSTePs.

The construct to “teach a psychomotor skill” was theorized from the literature review to be multidimensional, so EFA was used to aggregate items or summarize the underlying patterns of correlations between variables into groups that represented the same construct. The decision to adopt a four-factor model was premised on an iterative process that involved subjective assessment and statistically derived data. It confirmed the hypothesis that the items within the SonoSTePs P3 instrument do indeed explore the instructional practices used by health professionals to teach a complex psychomotor or procedural skill. Cronbach’s a for the four factors ranged between 0.67 and 0.89, and these values are acceptable for the initial validation of an instrument. However, using the four-factor model, the total variance across the items was 48.6%, and this metric suggests that the items within each factor may not be sufficiently diverse to glean reliable professional practice behaviors [8]. Additionally, the low response rate (n = 142) and cohort characteristics may have also influenced and attributed to the P3, overall data outcome.

The response rate to the SonoSTePs P3 instrument is representative of current online response rates to Survey Monkey, which range from 8% to 36% [19]. However, the factors influencing this result may be related to the survey being dispersed over a 14-week period.

The response rate to the P3 validation survey also poses an analytical conundrum. The sample size used to explore factor analysis was not sufficient (n = 123) because usually a minimum of 150 respondents is required to mitigate against item assignment errors and response bias [816]. Nunnally [15] reports that 10 respondents are required for every item being analyzed, to ensure a stable factor pattern is calculated. However, Tabachnick and Fidell [17] and Stevens [27] assert that a ratio of five or more participants per rating scale item is sufficient to perform EFA. The current item/respondent ratio is approximately 1:4.8 and, based on this criterion, may be insufficient for initial validation. Therefore, a larger cohort study would be required to gain reassurance of the item communalities and a stable factor pattern [8].

The representativeness of the cohort undertaking the validation of the P3 instrument may not, as we purported, have the pedagogical knowledge related to teaching a complex psychomotor skill. Therefore, the current factor pattern, and communalities may not be representative of a larger sample number. For example, a large majority (78%) of the cohort reported that they had no credentialing in clinical health education. The remaining respondents (22%) identified that they had either completed a graduate diploma in health education, or completed a course such as “train the trainer” or Certificate IV in Workplace Training and Assessment. Therefore, the validation cohort may not be cognizant of the pedagogical processes required to teach complex psychomotor skills, such as those used in medical ultrasound. These cohort attributes may have unexpectedly introduced reporting bias and error, which we suggest has been further magnified by a 17% response rate.

As with all research, there are potential limitations. The primary shortfall of this study relates to the small sample number and, consequently, the sample/variable ratio. A reduced respondent/item ratio may cause interpretation effects in sampling error, and low correlating items being misplaced within the factors [16].

Conclusion

The newly developed SonoSTePs P3 instrument after initial validation is composed of a 7-point Likert-type rating scale, with good discriminant validity, that contains 25 items. Initial EFA has identified four factors linked to teaching a psychomotor skill, and these are corroborated by contemporary skill-teaching literature. The instrument internal consistency for the total pool of items is good. The SonoSTePs instrument may, after undertaking additional exploratory and confirmatory factor analysis on a larger cohort, establish a reliable and valid instrument that can be used to tease out the skill-teaching behaviors of sonographers and other users of diagnostic medical ultrasound.