Study protocol: content and perceived educational utility of different modalities of clinical teaching visit (CTV) workplace-based assessments within Australian general practice vocational training: a cross-sectional study.

INTRODUCTION: Clinical teaching visits (CTVs) are formative workplace-based assessments that involve a senior general practitioner (GP) observing a clinical practice session of a general practice registrar (specialist vocational GP trainee). These visits constitute a key part of Australian GP training. Despite being mandatory and resource-intensive, there is a paucity of evidence regarding the content and educational utility of CTVs. This study aims to establish the content and educational utility of CTVs across varying practice settings within Australia, as perceived by registrars and their assessors ('CT visitors'). In addition, this study aims to establish registrar, CT visitor and practice factors associated with CTV content and perceived CTV utility ratings. METHODS AND ANALYSIS: This study will collect data prospectively using online questionnaires completed soon after incident CTVs. Participants will be registrars and CT visitors of CTVs conducted from March 2020 to January 2021. The setting is three Regional Training Organisations across four Australian states and territories (encompassing 37% of Australian GP registrars).Outcome factors will be a number of specified CTV content elements occurring during the CTV as well as participants' perceptions of CTV utility, which will be analysed using univariate and multivariable regression. ETHICS AND DISSEMINATION: Ethics approval has been granted by the University of Newcastle Human Research Ethics Committee, approval number H-2020-0037. Study findings are planned to be disseminated via conference presentation, peer-reviewed journals, educational practice translational workshops and the GP Synergy research subwebsite. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This study will sample from a diverse range of general practice training settings, including all urban and rural classifications.

Although the COVID-19 pandemic has impacted the amount of face-to-face clinical teaching visits (CTVs) that will occur during the study period, this has provided an opportunity to gather evidence on different CTV modalities and remote CTV options.

A limitation of this study is that outcomes will be self-reported rather than observations of participant behaviour.

Introduction

General practice training

In Australia, and in many other countries, specialist vocational general practice (family medicine) training occurs via an apprenticeship-like model.1 In Australia, the training of vocational general practice/family medicine trainees (known as general practitioner ‘(GP) registrars’) is overseen by the Royal Australian College of General Practitioners (RACGP) and the Australian College of Rural and Remote Medicine (ACRRM). Regional Training Organisations (RTOs)—geographically defined not-for-profit educational organisations—are tasked with delivering regionalised training and education, tailored to meet regional needs. The two colleges conduct summative licensing examinations and RTOs conduct a range of formative assessments, including in-training assessments.

Summative licensing assessments ensure that a registrar’s competencies have developed to a satisfactory, criterion-based standard. While formative assessments also examine a registrar’s competencies, the main objective is to provide actionable feedback to further their professional development across the course of their training.2 While both methods of assessment are essential components of medical training, increasing consideration is being placed on programmatic assessment, in which formative assessments play an integral part. Due to the timing of their delivery, formative assessments have the potential to both identify the need and provide the opportunity for tailored development via the delivery of feedback.3

Workplace-based assessments

A key mode of formative assessment undertaken by medical trainees, including GP registrars, is workplace-based assessment (WBA). During a WBA, a trainee’s competencies are assessed at the apex of Miller’s pyramid (does),4 by an assessor observing the trainee in practice. Ultimately, the goal of a WBA is to determine potential areas of improvement for the trainee and provide feedback that could be translated into practice. The psychometric properties of WBAs used in medical training have been substantiated by considerable research involving psychometric evaluations.5–9

Though the reliability and validity of a number of WBAs have been established, a WBA can sometimes have limitations in practice. For example, a recent study from Ireland found 20% of GP trainees and trainers perceived WBAs to be a single use, ‘box-ticking’ exercise.10 This suggests that, for a proportion of those participating, WBAs were used suboptimally. Contrary to this perception as a ‘tick-box’ exercise, WBAs must be part of an educational process. Holmboe et al2 remarks that competency-based medical education needs to continually link assessment of competency and the provision of feedback. Reoccurring assessment provides multiple opportunities for feedback, and, therefore ongoing reflection and development.

Other characteristics are important to ensure that the feedback is effective. For example, when feedback is too vague, it can inhibit learning.11 Furthermore, the literature suggests that, in addition to the provision of specific feedback, it is important to provide the trainee with an opportunity to reflect on their performance.12 Observational evidence involving GP trainees has found that the rate of translating feedback into action is higher if the feedback is specific (compared with nonspecific feedback), and higher again if the specific feedback is coupled with trainee reflection.13

While WBAs are, by definition, based within actual practice, there is only limited evidence for the utility of WBAs in affecting clinicians’ training and, particularly, their subsequent clinical performance.14 15 Most evaluations of WBA utility are conducted at lower levels of Barr’s evaluation hierarchy (learner’s reactions; modification of attitudes and perceptions; acquisition of knowledge and skills) rather than at higher levels (change in behaviour; change in organisational practice; benefit to patients).16

Clinical teaching visits

Clinical teaching visits (CTVs) are a particular WBA used throughout Australian vocational GP training. CTVs involve a trained and experienced GP (external to the practice), assigned by their respective RTO, assessing a registrar’s competencies.17–20 During a visit, of approximately 3-hour duration, the CT visitor observes the registrar’s consultations and engages in feedback-oriented discussion.21 22 Such discussion is intended to address both strengths and areas for improvement for the registrar, aiming to facilitate their professional development. While a CTV in itself is considered a WBA, due to the length of the visit, other forms of WBA can take place within a CTV. For example, a mini clinical evaluation exercise (mini-CEX) or random case analysis (RCA) can be nested within a CTV. As such, the assessment of the registrar’s competence and the feedback provided can be guided by various tools during the visit.

CTVs are mandated nationally by the RACGP, with GP registrars required to complete a minimum of five CTVs during their training period.23 ACRRM requires nine mini-CEX to be completed during training. These mini-CEXs are conducted within a CTV-like format.24

Educational content and utility of CTVs

Despite the central, mandated place of CTVs within Australian GP vocational training and their resource-intensive nature, there is a paucity of empirical evidence supporting their use. In fact, most of the evidence that exists concerning CTVs relates to the various WBAs that can take place within them. For example, the mini-CEX, a tool with established content and construct validity,5 9 25 26 can be used during a CTV. However, to our knowledge, the mini-CEX has not been empirically studied when used within a CTV setting. Regarding discrete tools that have been studied within a CTV setting, there is evidence to support the addition of RCA during a CTV. This evidence comes from a study involving qualitative analysis of registrar and assessor feedback on an RCA tool.21 27 While it demonstrated support for the tool’s use during CTVs, the frequency of its use is currently unknown.

There is also some evidence for structured feedback forms such as Hays’ Consultation Assessment Scale (CAS). Hays’ CAS has been shown to demonstrate validity and interrater reliability via evaluations of videotaped sessions28 and can be used to guide observation and registrar feedback. A small study of 21 CTVs investigating the use of Hays’ CAS during the CTV29 reported that registrars found feedback administered via this form useful for guiding further discussions with their medical educator (that is, useful at the lowest level of Barr’s evaluation model).16

GP Synergy, the RTO based in New South Wales (NSW) and the Australian Capital Territory (ACT), currently uses the General Practice Registrar Competency Assessment Grid (GPR-CAG) for assessment of registrar’s performance across the entirety of a CTV. The GPR-CAG has demonstrated initial content and construct validity.30 However, the perceived educational utility of the GPR-CAG itself as a CTV assessment instrument has not been evaluated.

The above studies of CTVs have focused on a single instrument (the RCA)21 27 incorporated within a portion of a CTV, evaluated the utility of an instrument for providing feedback following a CTV (CAS)29 or have derived and evaluated the psychometric properties of an instrument for use in evaluating registrar performance during a CTV (GPR-CAG).30 Also, the first two of these three studies were small in terms of sample size as well as geographical coverage. Thus, there is limited empirical literature concerning CTVs and no studies have assessed the utility of the CTV as a whole, particularly at the higher levels of Barr’s evaluation model. Additionally, the variability of CTV content and utility across different practice or training contexts are also largely undocumented.

There is some evidence from small-scale local evaluations conducted within RTOs to indicate that registrars and supervisors perceive CTVs to be a useful and important part of registrar training (Personal communication with: Christopher Starling 2012; Karen Flegg 2008), but these do not cover the views of the CT visitors and, to our knowledge, there is no peer-reviewed published evidence on this topic. This is a key evidence gap for a tool that is so broadly used and deeply integrated into Australian GP education.

CTVs in different Australian training settings

Frameworks are in place within each RTO to guide the way CTVs are conducted,17–20 however, CT visitor training can be ‘siloed’ between RTOs, meaning there is little coordination between RTOs regarding the guidance provided to CT visitors.31 Consequently, the way CTVs are conducted across different regional and practice settings can vary. This may be appropriate as it is unlikely an entirely standardised CTV would be able to adequately account for practice, cultural and socioeconomic variability. Factors such as patient demographics, practice physical environment and organisation, local health resources and local health referral pathways have the potential to affect performance on specific assessments.32–34 It is important to critically assess the similarities and differences of CTV elements across different training settings, taking into account these contextual factors.

Establishing the content and utility of CTVs in GP vocational training is of vital importance for informing best practice regarding GP training. In this study, we aim to establish the content and perceived educational utility of CTVs in three geographically and demographically distinct general practice vocational training programmes.

The objectives of the study are to establish:

The clinical and educational content of CTVs in three Australia RTOs.

Perceived educational utility of CTVs from the perspective of registrars and of CT visitors in three RTOs.

Registrar, CT visitor and practice factors associated with CTV content.

Registrar, CT visitor and practice factors associated with perceived CTV utility ratings.

Agreement of registrars’ and CT visitors’ reports of CTV content and of registrars’ and CT visitors’ ratings of CTV utility.

Methods and analysis

Study design

This observational study will collect data prospectively using online questionnaires completed soon after incident CTVs. Though participants may contribute data on multiple occasions, analysis will be cross-sectional (adjusting for clustering of CTVs within registrars or CT visitors).

Study setting

The study will include registrars and external CT visitors from three RTOs—GP Synergy, General Practice Training Tasmania (GPTT) and Northern Territory General Practice Education (NTGPE). GP Synergy is the largest of these RTOs, with an intake of approximately 500 registrars per year. Across all Australian RTOs, GP Synergy trains 32% of Australian registrars in general practice terms, covering both NSW and ACT. GPTT and NTGPE provide training to all registrars in Tasmania and the Northern Territory, respectively, which constitutes 5.2% (2.6% GPTT and 2.6% NTGPE) of registrars in general practice terms. These three RTOs cover a wide range of practice and demographics including urban, rural, remote and diverse socioeconomic settings. This will allow us to capture the variability of CTVs conducted throughout different Australian GP training settings.

Eligibility criteria

All registrars from GP Synergy, NTGPE and GPTT who are in general practice terms during the study period (March 2020–January 2021) will be eligible to participate after each of their CTVs. The external CT visitors who conduct these CTVs will also be eligible.

Immediately prior to the commencement of data collection for the study, the onset of the COVID-19 pandemic led to substantive changes to CTV delivery modalities. The previously practised face-to-face CTVs (as described in the introduction) were no longer practicable, and alternative CTV modalities were adopted. The three RTOs used variations of CTV modalities (see table 1). Participants involved with these assessments will be invited to participate, regardless of the CTV modality.

Table 1

Alternative CTV modalities used during the peak COVID-19 pandemic period

RTO	CTV modality	Description
NTGPE	Recorded video CTVs	Involves a CT visitor observing registrar consultations via a pre-recorded video session. This modality was in use prior to the COVID-19 pandemic, though with less frequency (ie, typically about one-fifth of mandated CTVs)
GPTT	Live video/phone CTVs	A new modality in which CT visitors conduct visits via videoconferencing software or telephone
GP synergy	Live video/phone CTVs	A new modality in which CT visitors conduct visits via videoconferencing software or telephone. These can also involve newly implemented simulated cases, in which CT visitors role play, as patients, with the registrars and then discuss performance.
	CNA-CTV	A new modality in which CT visitors engage in CNA with the registrar via videoconferencing software or telephone. Involves the registrar using a standardised selection method to obtain medical records for eight recent consultations, prior to the CTV occurring, to discuss with the CT visitor. The CT visitor then selects a minimum of four of the eight consultations to discuss with the registrar, during the CTV. This discussion involves the registrar reflecting on their clinical decisions during the consultation and rationales for those decisions.
	CBD-CTVs	A new modality in which CT visitors engage in CBDs with the registrar via videoconferencing software or telephone. These involve the registrar selecting any eight cases from recent patient encounters, of which the CT visitor will choose a minimum of four for in depth discussion, during the CTV.

CBD, case-based discussion; CNA, clinical notes analysis; CT, clinical teaching; CTV, clinical teaching visit; GP, general practitioner; GPTT, General Practice Training Tasmania; NTGPE, Northern Territory General Practice Education; RTO, Regional Training Organisation.

Patient and public involvement

No patients have or will participate in the design, implementation or interpretation of the study.

Recruitment

All registrars and external CT visitors will be invited via email by their respective RTOs to participate in the online questionnaire about each of their CTVs during the study period, within 2 days of the index CTV taking place. The study period is scheduled to be March 2020–January 2021. Due to the unforeseen impact of the COVID-19 pandemic on the scheduling of routine CTVs, data collection may continue beyond this period as required, to ensure adequate power is obtained for analysis. Completion of the questionnaire within 2 days of the CTV occurring will limit recall bias. As each CTV is different, participants who have completed more than one CTV during the study period will be invited to participate after each CTV they complete. Contact information held by each RTO will be used to distribute email invitations to participate. Contact information will not be shared between RTOs, in order to maintain confidentiality. Completion of the questionnaire will be voluntary for participants and no reward (financial or otherwise) will be offered for completing the questionnaire.

Data collection and management

Separate online registrar and CT visitor questionnaires were piloted in GP Synergy’s Hunter Manning Central Coast region during 2019. Registrar and CT visitor questionnaires took an average of 3 and 5 min, respectively, to complete. The questionnaires used in the current study are iterations of the pilot questionnaires. Questionnaire development was influenced by the analysis of the pilot data and further consultation with the project team, which included experienced GPs, GP registrars and GP educators (including CT visitors).

Each RTO will be responsible for collecting and managing data from participants within their respective regional settings, to maintain participant anonymity. This will entail the creation of a master list that will contain registrar and CT visitor contact and CTV schedule information, registrar practice characteristics as well as unique registrar, CT visitor and CTV identifiers. This information will be updated weekly to maintain accurate CTV scheduling information and will be used to distribute invitations, within 2 days of each visit, via Survey Monkey using its collector methodology. Survey Monkey’s collectors can be used to import data, such as contact information, from excel files into email invitations. In addition, the use of collectors allows for the association of CTV and participant identifiers with response data. The use of this method enables the establishment of registrar–CT visitor dyads for analysis of registrar–CT visitor agreement and for our analyses to be adjusted for clustering of CTVs within registrars or CT visitors.

Outcomes

As this is an exploratory study of an under-researched area, there will be a large number of descriptive outcomes. There will be three primary outcomes elicited from both registrars and CT visitors and two primary outcomes elicited from registrars only. Multivariable analyses will be conducted for these eight outcomes (for a full overview of outcomes, see table 2).

Table 2

Outcomes elicited from registrars and/or CT visitors, whether prevalence will be calculated, and whether multivariable analysis will be performed with that outcome

Item	Elicited from registrars	Elicited from CT visitors	Prevalence outcome	Multivariable analysis to be performed for associations
CTV content elements as indicated by:
Number of patients seen during visit*	Yes	Yes	Yes	No
RCAs reported to occur during the visit†	Yes	Yes	Yes	Yes
Specific discussion topics identified by the questionnaire (see table 4)‡	Yes	Yes	Yes	No
Whether the registrar consulted with their supervisor during the visit§	Yes	Yes	Yes	No
Rating of CT visitor/supervisor feedback consistency¶	Yes	No	Yes	No
Overall quality of the feedback received during the visit**	Yes	No	Yes	No
An opportunity for registrar to reflect on performance after consultation(s) occurring during the visit††	Yes	Yes	Yes	No
Feedback on specific element of registrar’s performance after consultation(s) occurring during the visit††	Yes	Yes	Yes	No
The CT visitor observing the registrar perform a physical examination in consultation(s) occurring during the visit††	Yes	Yes	Yes	Yes
CTV utility ratings
Educational utility rating of the CTV‡‡	Yes	Yes	Yes	Yes
Registrar rating of the likelihood they will change their practice as a result of the visit§§	Yes	No	Yes	Yes
Registrar rating of likelihood they will change their approach to learning as a result of the visit§§	Yes	No	Yes	Yes

*Variable type/response format for questionnaire: continuous variable ranging (0–20).

†Dichotomous (yes/no); number of RCAs if answer is yes.

‡Dichotomous (yes/no) tick boxes (multiple selection possible).

§Yes/no.

¶Likert-type ‘1—not at all consistent—5 very consistent’ plus ‘I did not receive any feedback’.

**Likert-type ‘1—broad, non-specific; difficult to translate into action – 5- focused/specific; easy to translate into action.

††Registrar responds as either ‘not for any consultation’, ‘yes for one consultation’, ‘yes for two consultations’ or ‘yes for 3 or more consultations’.

‡‡Likert-type ‘1—not at all useful—5 very useful’.

§§Likert-type ‘1—not at all likely—5 very likely’ plus ‘I didn’t receive any feedback’.

CT, clinical teaching; CTV, clinical teaching visit; RCA, random case analysis.

Some outcomes may also be used as covariates in other multivariable analyses. Additional independent variables are seen in table 3. Outcomes will encompass the occurrence of a range of specific clinical and educational elements within the CTV, plus a rating of the utility of the CTV, and ratings of the registrar’s likelihood of changing their practice, and of changing their learning approach, as a result of the CTV.

Table 3

Independent variables

Variable	Definition
Registrar specific variables
Training pathway	General or rural.
Fellowship	Fellowship(s) registrar indicated they are working towards, which includes FRACGP, FARGP, ACRRM.
Training term	The registrar’s current training term at the time of survey completion, either first term (first 6 months FTE), second term (second 6 months FTE), third term (third 6 months FTE) or ‘other’; with specification.
Fulltime/part time	Regarding the registrar’s current enrolment in training
Practice size	The number of other GPs (full time equivalents) working with the registrar in the practice at the time the CTV was conducted.
Rurality of practice*	Location of registrar’s practice at the time the CTV occurred, based on practice postcode, MMM 2019.
Socioeconomic status of practice*	Based on registrar’s practice postcode. SEIFA-IRSD 2016.
CT visitor specific variables
CTV experience	Number of years the participant has been a CT visitor.
Medical education experience	The CT visitor indicated they have had experience in any of the following roles: medical educator (vocational training), accredited GP supervisor, examiner for ACRRM fellowship exams, examiner for RACGP fellowship exams and a CT visitor.
Rurality of practice	Rurality of the CT visitors’ practice, indicated by—Major city (RA1), inner regional area (RA2), outer regional area (RA3), remote (RA4) and very remote (RA5).
Completed fellowship(s)	Completed fellowship of the CT visitor, indicated as FRACGP, FARGP, ACRRM or ‘other’; specified.
Years since fellowship	Number of years since the CT visitor has gained fellowship.
Registrar and CT visitor common variables
Age	Age of the CT visitor or registrar, in years, at the time of questionnaire completion.
Weekly hours	Number of hours worked in practice as indicated by the CT visitor or registrar.
Gender	Self-reported as either male, female, prefer not to say or ‘prefer to self-describe as (optional)’.
Country of primary medical degree	Australian medical graduate or international medical graduate (with specification of country).
CTV Format	Which type of CTV took place (for descriptions, see table 1)
Method conducted	How the CTV was conducted (either face-to-face, video, or telephone)
Session time	Time of the day the CTV took place (either morning or afternoon)
Telehealth consultations	Number of telehealth consultations that occurred during the CTV
Simulated cases	Whether simulated cases were reported to occur during the CTV (responses as either yes or no, if participant responds ‘yes’ they are also asked to document how many simulated cases occurred during the visit).
RCAs	Whether RCAs were reported to occur during the CTV (responses as either yes or no, if participant responds ‘yes’ they are also asked to document how many RCAs occurred during the visit).

*Elicited from routinely collected data from each RTO.

ACRRM, Australian College of Rural and Remote Medicine; CT, clinical teaching; CTV, clinical teaching visit; FARGP, Fellowship in Advanced Rural General Practice; FRACGP, Fellowship of Royal Australian College of General Practitioners; FTE, full-time equivalent; GP, general practitioner; MMM, Modified Monash Model; RACGP, Royal Australian College of General Practitioners; RCA, random case analysis; RTO, Regional Training Organisations; SEIFA-IRSD 2016, Socio-economic Indexes for Areas—Index of Relative-Disadvantage 2016.

Sample size

The pilot study of 175 CTVs demonstrated response rates of 49% and 53% for registrars and CT visitors, respectively. Using this information, with an estimated 1169 visits occurring within the study period, we estimate that the current study will yield a sample size of 561 individual CTV registrar responses and 643 individual CTV visitor responses. We estimate that there will be approximately 350 CTV registrar–CTV visitor dyads available for analysis. This will allow estimation of the prevalence of individual CTV elements, with 95% CIs, to within an error margin (half the CI width) of 3%–4%. The anticipated response rates are considered acceptable for cross-sectional study with GP participants.35

Statistical methods

The prevalence of categorical outcomes will be assessed as proportions with a 95% CI. Continuous outcomes will be assessed using means and SD with a 95% CI.

For the eight outcomes nominated in table 2 for multivariable analysis, the frequencies of categorical independent variables will be compared between outcome categories using χ2 tests for all variables, except when Fisher’s exact test will be used (due to an expected count of less than 5 in 25% or more cells). For continuous independent variables, means will be compared using a t-test or nonparametric equivalent if appropriate.

Univariate and multivariable associations of the eight outcomes nominated in table 2 will be tested using logistic regression within the generalised estimating equations framework to account for repeated measures within registrars and visitors. An exchangeable working correlation structure will be assumed.

Registrar–CT visitor agreement will be established using Cohen’s κ.

Ethics and dissemination

Consent or assent

The completion of the questionnaire by the participant will be considered implied consent.

Confidentiality

Each RTO will be responsible for downloading data from SurveyMonkey and deidentifying it before it is sent to GP Synergy for analysis. Master lists were held by each RTO and will be stored in password-protected files. Separate password-protected files will be used to store data and all complete, deidentified questionnaire data will be held and only be accessible by the study research team at GP Synergy.

Access to data

Data will only be accessible by investigators and statisticians committed to the project.

Dissemination policy

Study findings will be disseminated via presentations at general practice and medical education conferences as well as publication in peer-reviewed journals. GP Synergy will also disseminate findings via workshops run for GP Synergy stakeholders such as medical educators, CT visitors and supervisors. Each RTO will distribute the findings of the study to the participants within their respective domains. Thorough documentation of results and study methodology will be made available to interested parties on GP Synergy Research subwebsite: https://research.gpsynergy.com.au/. Publications will be written by the authoring team without the use of professional writers.

Data availability

Nonidentifiable data may be shared with other parties on reasonable request to encourage scientific scrutiny and to contribute to further research and public knowledge.

Discussion

Anecdotally, registrars and CT visitors acknowledge CTVs as a highly valuable educational experience (Personal communication with: Christopher Starling 2012; Karen Flegg 2008). However, there is little published evidence regarding the content of CTVs. While a small number of studies have examined the use of particular tools in a CTV setting,21 27 the overall content of CTVs as a whole has not been systematically documented. Similarly, there is little published evidence examining the educational utility of CTVs.29 This is a key evidence gap for an educational element that is so resource demanding and mandatory for all Australian registrars. As different regional settings can call for tailored variation to elements of training,32–34 it is important to systematically document the content of CTVs and the relative value they hold for registrars and CT visitors, as this information will help inform a best practice approach to GP training in varying settings throughout Australia.

The aim of this study is to address the current evidence gap regarding CTVs by eliciting registrar and CT visitor perceptions of CTV content and educational utility in varying practice settings via an online questionnaire. As our sampling strategy is broad and aims to include registrars and CT visitors from all urban and rural classifications, we expect the results to be generalisable to all Australian RTOs. The results may also be generalisable to other countries using direct observations in an apprenticeship-type model for GP training.

The findings of this study will be of importance in the domain of medical education, particularly regarding competency-based medical education in Australia. Demonstrating the prevalence of content elements and utility ratings as well as the corresponding associations between these outcomes across varying training settings could inform and influence the guidance RTOs provide to CT visitors regarding conducting visits. As a result, RTOs and CT visitors may be able to tailor the educational experience the registrar receives based on the needs imposed by the regional setting.

Limitations

A major limitation of this study involves the suspension of face-to-face CTVs in response to the COVID-19 pandemic, resulting in data being collected for a smaller number of face-to-face CTVs so far. At the time of writing, it remains unclear how long face-to-face CTVs will remain suspended. Although each RTO has implemented alternative CTV modalities, these differ from face-to-face CTVs and have not yet become well established. This, however, has provided us with a unique opportunity to collect evidence on a range of CTV modalities, which may inform how CTVs can be conducted in the future. This is of particular relevance for the ongoing development of effective yet resource-efficient alternatives to traditional, face-to-face WBAs.

Another limitation of this study is the subjective nature of determining the educational utility of CTVs. While our study includes assessments at a higher level of Barr’s evaluation model16 (level 3: change in behaviour) rather than purely lower levels (learners’ reactions and changes in attitudes, knowledge and skills) as studied in much previous research, we will employ self-reported intent as a surrogate measure of change rather than direct measures of behaviours. A self-reported measure, however, was deemed most appropriate as deriving an objective measure for the educational utility of CTVs was beyond the scope and resource capacity of this study. Also, an observational rather than experimental design was chosen. Implementing randomised control trials with CTVs as an intervention is not feasible, as CTVs are a mandatory element of GP training in Australia.19 24

Conclusion

This study will contribute valuable evidence regarding the content and perceived educational utility of CTVs within Australian vocational general practice training. This will help to inform the ongoing optimisation of CTV delivery in Australia and may help inform how other WBAs are conducted in similar apprenticeship-like vocational medical training programmes internationally.

Table 4

Specific discussion topics listed in the questionnaire

Response options available in the questionnaire regarding specific discussion topics
Identifying the reason for the consultation	Time management
Exploration of patient’s problems	Management planning
Consideration of the patient’s agenda	Appropriate medications
Organisation and flow	Appropriate investigations
Non-verbal behaviour	Medication prescribing processes
Developing rapport	Documentation in patients’ medical records
Appropriateness of physical examination components performed/ proposed	Referrals
Physical examination technique	Patient follow-up
Diagnosis	Dealing with uncertainty
Explanation of diagnosis to patient	Safety netting
Specific patient and/or location contextual factors relevant to the consultation(s)

The following are definitions that accompany particular response options in the questionnaire: Safety netting; Contingency planning with the patient to provide a management strategy for a change in symptoms, including explicit instructions for action(s) given specific circumstances. Specific patient and/or location contextual factors relevant to the consultation(s); for example, rural/remote, socioeconomic, Aboriginal and/or Torres Strait Islander, non-English speaking background factors.