Outcomes of the Northern Ontario School of Medicine's distributed medical education programmes: protocol for a longitudinal comparative multicohort study.

INTRODUCTION: The Northern Ontario School of Medicine (NOSM) has a social accountability mandate to serve the healthcare needs of the people of Northern Ontario, Canada. A multiyear, multimethod tracking study of medical students and postgraduate residents is being conducted by the Centre for Rural and Northern Health Research (CRaNHR) in conjunction with NOSM starting in 2005 when NOSM first enrolled students. The objective is to understand how NOSM's selection criteria and medical education programmes set in rural and northern communities affect early career decision-making by physicians with respect to their choice of medical discipline, practice location, medical services and procedures, inclusion of medically underserved patient populations and practice structure. METHODS AND ANALYSIS: This prospective comparative longitudinal study follows multiple cohorts from entry into medical education programmes at the undergraduate (UG) level (56-64 students per year at NOSM) or postgraduate (PG) level (40-60 residents per year at NOSM, including UGs from other medical schools and 30-40 NOSM UGs who go to other schools for their residency training) and continues at least 5 years into independent practice. The study compares learners who experience NOSM UG and NOSM PG education with those who experience NOSM UG education alone or NOSM PG education alone. Within these groups, the study also compares learners in family medicine with those in other specialties. Data will be analysed using descriptive statistics, χ(2) tests, logistic regression, and hierarchical log-linear models. ETHICS AND DISSEMINATION: Ethical approval was granted by the Research Ethics Boards of Laurentian University (REB #2010-08-03 and #2012-01-09) and Lakehead University (REB #031 11-12 Romeo File #1462056). Results will be published in peer-reviewed scientific journals, presented at one or more scientific conferences, and shared with policymakers and decision-makers and the public through 4-page research summaries and social media such as Twitter (@CRaNHR, @NOSM) or Facebook. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

The Northern Ontario School of Medicine (NOSM) is one of a few medical schools in the world with an explicit social accountability mandate employing a distributed medical education model.

The study started with the opening of the medical school in 2005 and includes all cohorts as they participate in NOSM's undergraduate (UG) or postgraduate (PG) medical education programmes.

Longitudinal tracking allows learners’ educational experience to be matched with intended and actual behaviours (eg, intended vs actual medical discipline) for at least 5 years into independent practice.

Natural comparison groups are used to investigate the effect of NOSM admission criteria and educational experience.

Data from overlapping surveys and administrative data will cover most gaps arising from missed surveys or low survey response rates.

Introduction

The Northern Ontario School of Medicine (NOSM), which first enrolled medical students in 2005, is a key initiative in the physician human resources plan of the Province of Ontario, Canada,1 and is an important strategy2 to overcome the long running shortage of medical doctors (MDs) in Northern Ontario.3–5 NOSM's mission statement includes a mandate to be socially accountable to the needs and the diversity of the populations of Northern Ontario’, and to actively involve the ‘Aboriginal, Francophone, remote, rural and underserviced communities’ of Northern Ontario.6 NOSM seeks to increase ‘the number of physicians and health professionals with the leadership, knowledge and skills to practice in Northern Ontario’.

NOSM's approach is based on evidence that if medical schools select learners who have lived in underserved areas such as rural and Northern Ontario and train them in a positive manner in similar environments, then these learners are more likely to practice in these areas. This evidence comes from Canada,7 Ontario,8 Northern Ontario9 10 and is synthesised at the international level in several systematic reviews.11–14

Northern Ontario has over 800 000 km2, an area larger than France, and a population density that averages 1 person/km2 with approximately 56% of the population clustered in and around five of the larger urban areas (Timmins, North Bay, Sault Ste Marie, Thunder Bay and Greater Sudbury), which range in size from 43 000 to 161 000 people.15 Northern Ontario includes a larger proportion of two cultural-linguistic minority groups than the province as a whole. Francophones represent 18% of Northern Ontarians versus 5% in the province and Aboriginal people represent 14% vs 2%, respectively.16 17 Northern Ontarians have poorer access to and lower use of medical care services than the rest of Ontario.5 18 19 People in Northern Ontario also have poorer health status than the rest of Ontario, and the health status of Francophone and Aboriginal people is worse.20–23

The Centre for Rural and Northern Health Research (CRaNHR) in conjunction with NOSM and funded by the Ontario Ministry of Health and Long-Term Care (MOHLTC) has tracked learners since 2005, the year in which NOSM admitted its first cohort of undergraduate (UG) medical students.24 The study's objective is to understand how NOSM's socially accountable admission criteria and medical education programmes set in rural and Northern Ontario communities affect choice of medical discipline, practice location, medical services and procedures, inclusion of medically underserved patient populations and practice structure (eg, solo, interprofessional team).

This tracking study is unique as NOSM is one of a few medical schools in the world with an explicit social accountability mandate25 and with medical education provided in communities away from large cities and regional hospitals.24 There is emerging global interest in how well NOSM and similar schools can fulfil their mandates.26 27 For example, the Training for Health Equity Network is a worldwide movement of schools committed to improving health equity by transforming education of health professionals.26 Eleven schools in nine countries are committed to measuring how well they match educational outcomes to the needs of the areas they serve.28 A second unique aspect is that the study started with the opening of the medical school and includes all cohorts as they participate in NOSM's UG or postgraduate (PG) medical education programmes. Third, longitudinal tracking allows learners’ educational experience to be matched with intended and actual behaviours (eg, intended vs actual medical discipline) as learners are tracked from the time of their arrival at NOSM and continuing for at least 5 years into independent practice. This is important, as previous and ongoing research demonstrate the utility of longitudinal tracking studies linking admission criteria, medical education and other factors with outcomes.29 30 A strength of the study resides in the use of natural comparison groups to investigate the effect of NOSM admission criteria and educational experience. Six groups are defined on three dimensions: (1) learners’ medical school (ie, NOSM vs other medical schools); (2) medical education level (ie, UG vs PG) and (3) medical discipline (ie, family medicine vs other specialties). This paper describes methods developed since the study started in 2005.

Methods and analysis

Study design, participant recruitment and data collection

All learners are tracked through administrative databases (eg, medical school admissions and educational programmes databases, medical licencing agencies registration databases), which provide basic demographic data (eg, table 1), details of the learner’s educational experience at NOSM and selected information on outcomes (eg, provincial health insurance (billing) databases). Additional demographic data as well as the learner's perspective on factors that influence key outcomes plus detailed information on the outcomes are obtained by surveys or interviews.

Table 1

Demographic characteristics of Northern Ontario School of Medicine UG medical students and PG residents

Characteristic*	2005–2013 UG cohorts	2009–2013 PG cohorts†
Age at entry: mean (SD)	26.0 years (5.15) n=537	30.9 (6.04), n=433
Female	67.6%, 363/537	63.6%, 269/423
Aboriginal	7.3%, 39/537	8.2%, 29/355
Francophone	21.6%, 116/537	26.5%, 100/378
From Northern Ontario	90.5%, 486/537
From rural community in Northern Ontario	30.0%, 162/537
From rural community in other regions	8.6%, 46/537
Married partnered		53.2%, 223/419
Canadian citizen		96.5%, 418/433
NOSM UG		63.7%, 276/433
Family medicine		63.0%, 273/433

*Refer to table 2 for definitions. Cultural, linguistic and background data are based on NOSM's administrative records. All other data are from CRaNHR's surveys and/or NOSM administrative records.

†Includes learners who completed their UG at NOSM or at other medical schools.

CRaNHR, Centre for Rural and Northern Health Research; NOSM, Northern Ontario School of Medicine; PG, postgraduate; UG, undergraduate.

Table 2

Primary research questions, study outcomes and explanatory variables

Research question	Outcome/variable*	Categories (if any) and definition	Data sources
Practice characteristics outcome group
Will NOSM medical learners practice in family medicine, generalist specialties such as paediatrics, general surgery and internal medicine or other medical/surgical specialties or subspecialties?	Medical discipline or specialty	CFPC or RCPSC certificationSpecialty within RCPSC (eg, paediatrics)Specialties as defined by CFPC or RCPSC	†‡§
What types of medical services and procedures will learners offer to their patients? (ie, what will be their scope of practice?)	Types of services or procedures	CFPC certified MDs65 Procedure skills37 3899 Priority topics and key features for assessment in family medicine37	†§¶
		RCPSC certified MDsSkills and procedures identified in ‘objectives of training’ documentation for each Royal College specialty39	†§¶
Will learners provide services to special populations such as Aboriginal and Francophone peoples or the elderly?	Practice languages	MD is able to practice medicine in specified languageLearners’ cultural/linguistic background as proxyAboriginal learners40Francophone learners41	†§
	Cultural group or ethnicity of patient population	Adapted from criteria for learners	†§¶
	Age profile of patients	Actual age of patients	†§¶
How will learners organise their practices?	Practice administrative type	Solo, group practice, etc	†§
	Practice operational type	Independent practice, interprofessional care teams, other	†§
	Hospital privileges, on-call duties, ED coverage, etc	Name and location of hospital at which the MD has privileges, provides on-call coverage, ED coverage, etc	†§
Practice location outcome group
Will learners practice in medically underserved regions such as those in rural and Northern Ontario?	Practice location—region	Geographic regionNorthern Ontario defined by the 2003 boundaries of the 3 District Health Councils of Northern Ontario)42This area is 0.5% larger and has 7.5% more people than the 2015 provincial definition of Northern Ontario. The older definition represents NOSM's service areaSouthern Ontario defined as other location in Ontarionorthern Canada defined by ministry of health of applicable province or territory	†§¶
Will learners practice in the smaller communities?	Practice location—rural–urban continuum	Measures of rurality or medical underservice:Rural–urban classes based on Government classifications of population size, distance/commuter flow to urban centres, etc43 44Rural Index of Ontario score45	†‡§¶
Explanatory variables
What is the effect of the selected demographic characteristics on outcomes listed above?	Socioeconomic and demographic characteristics	Rural or northern background, culture/ethnicity, other demographic characteristics	†‡§¶
What is the effect of the medical education experience on outcomes listed above?	Educational experience	UG and PG medical education at NOSM or other medical school	†‡§¶
What are some of the other factors that influence the decisions listed above?	Influential factors	Factors such as, opportunity, personal, familial, and societal imperatives	†

*The study measures intended and actual outcome/influential factor.

†Data source=CRaNHR survey/interviews with learners/physicians.

‡Data source=Medical schools or medical education agencies.

§Data source=Medical licensing or regulatory bodies.

¶Data source=Provincial Health Insurance Plans (billing data).

CFPC, College of Family Physicians of Canada; CRaNHR, Centre for Rural and Northern Health Research; ED, emergency department; MDs, medical doctors; NOSM, Northern Ontario School of Medicine; PG, postgraduate; RCPSC, Royal College of Physicians and Surgeons of Canada; UG, undergraduate.

This prospective comparative longitudinal study follows multiple cohorts from entry into NOSM's UG or PG programmes, and at least 5 years into independent (fully qualified) practice. A purposive sampling strategy invites all NOSM UG and PG medical learners to voluntarily participate in surveys or interviews. Every year, 56 UG students (64 since 2010) are tracked throughout their UG education and into PG residency, when they are joined by additional 40–60 PG residents who are new to NOSM. NOSM UG students are asked to participate in surveys and interviews at the first-year orientation, end of second year, and end of fourth year (figure 1). NOSM PG residents are invited to participate in surveys during orientation and just prior to completion of residency. NOSM UG students who go elsewhere for their PG residency training (30–40 residents per year go to other medical schools for PG training, included in the count of NOSM UGs) are invited to continue their participation in surveys at entry and completion of their residency. To summarise, the study tracks learners who finish (1) their UG education and PG training at NOSM, (2) only their UG at NOSM and go to other schools to complete their PG training or (3) only their PG at NOSM having completed their UG medical education at other schools. These three groups are followed for residents who become family physicians or other specialists (combined) to yield six groups for comparison.

Figure 1

Flow of medical learners through the NOSM as of January 2014 (CFPC, College of Family Physicians of Canada; NOSM, Northern Ontario School of Medicine; PG, postgraduate; PGE, postgraduate entry survey; PGX, postgraduate exit survey; PGY, postgraduate year; RCPSC, Royal College of Physicians and Surgeons of Canada; UG, undergraduate; UGE, undergraduate entry survey; UGM, undergraduate midway survey; UGX, undergraduate exit survey; UGY, undergraduate year).

CRaNHR researchers invite NOSM medical students, but not NOSM personnel, to a CRaNHR-sponsored meal, to explain the study and distribute the survey in paper or electronic format, or by a web page link. Wherever possible, a similar event is organised for PG residents at NOSM. NOSM UGs who go elsewhere for their residency training are invited by email or mail to start or continue their participation in the study. All subsequent contact with participants is by email or mail. A modified tailored design method31 is used for all surveys with at least two reminders, excepting those who have opted out. Each UG survey round lasts until all learners have responded or for 3 months, whichever comes first. PG survey rounds last up to 4 months and are initiated throughout the year because of staggered starts and exits due to, for example, parental leave or extra training requirements for internationally trained medical graduates. Since, contact information for residents at other medical schools can be difficult to obtain, we send an invitation whenever we have updated contact information. Participants can complete an on-line questionnaire, electronic MS Word document or paper form. For each survey round, a draw is held for a $C50 gift card from a national retail store as an incentive to participate.

During the UG entry survey, students in the first 5 years were also invited to participate in short-duration semistructured interviews in their first year and again in their fourth year. Interviews were conducted face-to-face, by telephone, or by Skype, depending primarily on learner preference. All interviews were digitally recorded with the interviewee's permission and conducted by Dr Hoi Cheu (CRaNHR Faculty Investigator) using a six-question interview guide, with prompting questions as needed. Questions were shared at least 1 day prior to the interview. Interviewees were given an honorarium of a $C25 gift card from a national retail store.

Exposure

NOSM's UG and PG admissions criteria and medical education programmes comprise the exposure. NOSM serves as the Faculty of Medicine of Laurentian University in Sudbury (2011 census metropolitan area population: 161 000) and of Lakehead University in Thunder Bay (2011 census metropolitan area population: 122 000)—located 1000 km apart by road. NOSM selects medical school (UG) applicants with a grade point average (GPA) of ≥3.0 of 4.0 in science and non-science university degrees, and does not require the Medical College Admission Test.32 Mean GPA was 3.8 for NOSM students starting in 2015, and this falls within the 3.7–3.9 range for all other Canadian medical schools.33 Preference is given to students from northern, rural, remote, Aboriginal or Francophone backgrounds so as to reflect Northern Ontario demographics. Learners must also have a strong interest in the understanding of, and aptitude for, practising medicine in Northern Ontario. NOSM provides medical learners with educational and clinical experiences in different health service settings in over 90 rural, remote and northern communities.34 35 For instance, all first and second year medical students undertake a 1-month Integrated Community Experience in Northern Ontario Aboriginal and rural or remote communities. In the third year, all medical students complete an 8-month Longitudinal Integrated Clerkship, grounded in family practice, and located in 1 of 15 large rural or small urban communities in Northern Ontario, away from Sudbury or Thunder Bay. Similarly, NOSM's PG residency programmes combine learning at the regional hospitals in Sudbury and Thunder Bay with clinical rotations throughout rural and Northern Ontario. NOSM offers PG residency training in family medicine and in eight additional specialist programmes.36 All of this is designed to select learners from rural areas or who are aware of the healthcare needs of the rural underserved, and enable learners to be trained and mentored by physicians who have chosen to live and practice in Northern Ontario so as to prepare learners for practices with fewer resources than in major population centres.

Research questions, study outcomes and explanatory variables

The main research questions and key variables were derived from the literature with selected input from the funder (MOHLTC). Questions and variables were outlined in a research framework adopted by the advisory committee, updated annually and critically reviewed in the 5th and 9th study year. All tools and methods are being reviewed in 2015—the 11th study year. The main research outcomes (table 2) include:

Clinical and organisational practice characteristics: medical discipline, medical services and procedures, patient population, practice organisation (eg, solo, interprofessional care team);

Practice location: categorised by geographic region, population size/density and rural–urban continuums.

Explanatory variables include:

Learner traits: selected socioeconomic and education demographic characteristics including rural or northern background and language/culture/ethnicity.

Medical education: level (ie, UG or PG) and medical school (ie, NOSM or other school).

Influential factors: opportunity, personal, familial and societal imperatives that affect decision-making around the main outcomes.

Many study outcomes are collected first as intention and then as actual outcome (eg, intended and actual medical discipline). Intended influential factors are those considered by the respondent as important prior to decision-making, and actual influential factors are those that respondents report in hindsight as having affected their decision.

Development and assessment of study tools

Tools to extract administrative data, questionnaires and interview guides were developed as the charter class progressed through their medical education: UG entry questionnaire and interview guide were developed in the academic year 2005/2006; UG midway questionnaire in 2006/2007; and the UG exit questionnaire and interview guide in 2008/2009. PG residents were tracked since 2009 by using administrative data. However, funding delays meant that the PG entry and exit surveys were not developed until 2011/2012. Measures to fill this data gap are described in the section on limits and strengths.

Operationalisation of outcomes, linkages among independent and dependent variables as well as question wording were based on the literature available when the study began,11 informed by a workshop to evaluate the impact of medical education initiatives in Canada,46 and updated with literature being published as the study progressed.12–14 To provide additional content validation and to facilitate comparisons with other medical educational programmes, most questions were based on similar CRaNHR studies.47–49 Other questions were based on the literature, including a block of questions to measure student attitudes on working and living in rural areas,50 modified to the Canadian context and used with permission (Adams ME, Dollard J, Hollins J and Petkov J, personal communications, 2005). Questions from earlier studies were revised to reflect choices available to NOSM learners. NOSM UG medical students, PG residents and practising MDs (two of each and all located in Sudbury, Ontario) reviewed surveys for content validity and readability. Interview questions inquire about selected key outcomes and related decision-making in greater detail.

Multiple data sources (eg, surveys, interviews or administrative data) for several outcomes improved content validity and allowed checking consistency of response. Test–retest reliability of the questionnaires was not assessed because the research team judged that the likelihood of respondents remembering their answers would be too high over the short term, and that answers to many questions would be expected to change in as little as a few months as respondents became immersed in NOSM's distributed medical education programmes.

Dealing with potential bias in surveys and interviews

To reduce social desirability bias, CRaNHR researchers ensure that NOSM faculty or staff are absent during surveys or interviews. Learners are told that their responses will not affect their academic standing and that only aggregate data would be published or shared with NOSM and other stakeholders. Researchers seek to reduce non-response bias by providing multiple mediums (ie, paper, electronic, or online surveys) for up to 3–6 months, to facilitate participation at the learners’ convenience. Recall bias may be an issue only for selected questions about the geographic location of where respondents or their spouses have lived previously. Researchers use administrative data to assess non-response and recall bias for selected information on demographics and outcomes.

Analytical approach

Data comprise multiple measurements on individual learners generated from an (in)complete census of each cohort, and therefore, descriptive statistics or randomisation tests will be used to determine associations or group differences.51 52 The χ2 tests, logistic regression and hierarchical log-linear models will be the most frequently used statistical methods. Cohorts are stratified by medical school (ie, NOSM vs other), education level (ie, UG vs PG) and medical specialty (ie, family medicine vs other specialties). The use of multiple imputation techniques to handle missing data will be considered in the context of the specific analytical method or research question. Every effort will be made to contact non-respondents, provided they have not explicitly declined to participate. Administrative data are used to track learners, obtain basic demographic information, details of the learners’ NOSM education experience as well as selected outcomes, while surveys and interviews allow for collection of more detailed data.

Interview transcripts and responses to open-ended questions are analysed using an iterative analytical and inductive approach to group findings within each question.53 Transcripts and electronic recordings are re-examined to ensure that context is preserved, and that confirmatory and contradictory findings are noted. Researchers’ interpretations are distinguished from key informants’ statements,54 while anonymised quotes illustrate the scope and depth of groupings plus exceptions, if any.

Dissemination

All data are stored on a secure server hosted by Laurentian University with access to individual-level data restricted to CRaNHR researchers directly involved in the study. CRaNHR shares only aggregated data (cell size >5) with NOSM personnel or other stakeholders and researchers, and follows other Statistics Canada guidelines to reduce identity, attribute or residual disclosure.55

Results will be published in peer-reviewed scientific journals and presented at one or more scientific conferences. Research highlights will also be shared with policymakers and decision-makers and the public through 4-page reader-friendly summaries of research results (Research In FOCUS On Research), and by social media such as Facebook (http://www.facebook.com/cranhr) and Twitter (@CRaNHR, @NOSM and researchers’ accounts).

Limits and strengths of the approach

One limitation arises in assessing the exposure because NOSM selects UG medical students (but not necessarily PG learners) with rural or northern Canada backgrounds. Given that rural background is strongly associated with practice in rural areas,7 8 11–14 there may not be much variation remaining among NOSM medical students to predict outcomes such as location of rural practice. However, the evidence for the influence of other factors, such as northern Canada background, language/culture, gender or marital/partnership status, on outcomes such as medical discipline and practice location varies among contemporary studies11–14 and may be evolving over time and so the study will assess these influences. In addition, the tracking study is able to isolate the influence of different medical schools (ie, NOSM vs other) at different levels (ie, UG vs PG), and for different medical disciplines (ie, family medicine vs other specialties).

Small population size may limit some analyses given that there are 56 new UG students each year (64 since 2010) and a lower number in some PG programmes, especially specialties other than family medicine. Groups may be combined to achieve adequate numbers for analysis, albeit at the loss of some detail.

Choice of outcome measures derived from medical care needs of Northern Ontario and situated in the political context may be interpreted as a limitation as well as a strength. Perhaps a more important limitation is that study outcomes (ie, practice location and scope of practice) are proxies of the ultimate outcome—the health of Northern Ontarians. However, choice of proxy outcomes is reasonable given that NOSM is an important step in ensuring that there are sufficient numbers of skilled and locally trained MDs in Northern Ontario.1 2 The expectation is that improved access to MDs will help improve the health of Northern Ontarians.

Other limitations include delays and gaps in execution of surveys. UG surveys and interviews have been on schedule since early 2006 (a 6-month delay), while PG surveys have been on schedule since 2012 (2 prior cohorts had incomplete coverage). Fortunately, administrative data is available from NOSM, and missed PG entry surveys had near-temporal equivalents in the UG exit survey, and so gaps in PG entry survey coverage exist only for PGs new to NOSM in 2009 and 2010. Changes in the wording of questions, or response options, create challenges for temporal continuity that are addressed by a detailed codebook that facilitates appropriate comparisons and provisos.

Study tools and methods are reviewed in-house, which increases internal utility, but may reduce external validity. Although there is no third-party review, many of the indicators and outcomes are copied or derived from the international literature. In addition, several advisory committee members are experts in rural or distributed medical education in Canada, the USA and Australia, and the study benefits accordingly.

Future study

The tracking study will be integrated within a broader research programme assessing the medical, social and economic impact of NOSM on Northern Ontario communities building on previous research.56 Detailed individual-level data allows for investigations into the relationship between specific aspects of NOSM's programmes and medical education outcomes or socioeconomic impacts. For instance, practice characteristics (eg, medical discipline, geographic location) of medical students with science backgrounds could be compared with students with non-science backgrounds.57 Other examples would be to compare performance and practice characteristics of students who had their third year clerkship in larger versus smaller communities (Ellaway RH and Graves L, personal communication, 2011), or to assess the effect of cultural safety training (Jacklin K and Maar M, personal communication, 2012). The integrated study will include investigations into NOSM's admission criteria and processes.

Conclusion

This paper describes a prospective, comparative, multicohort, longitudinal study of NOSM UG and PG medical learners that tracks learners as they progress through the medical education system beginning at admission into NOSM and at least 5 years into independent practice. The tracking study also serves as a platform upon which other research can improve understanding of the role of learner background and medical education experience on outcomes germane to the health and well-being of people living in sparsely populated and medically underserved areas such as Northern Ontario.