Validity evidence for the measurement of the strength of motivation for medical school.

The Strength of Motivation for Medical School (SMMS) questionnaire is designed to determine the strength of motivation of students particularly for medical study. This research was performed to establish the validity evidence for measuring strength of motivation for medical school. Internal structure and relations to other variables were used as the sources of validity evidence. The SMMS questionnaire was filled out by 1,494 medical students in different years of medical curriculum. The validity evidence for the internal structure was analyzed by principal components analysis with promax rotation. Validity evidence for relations to other variables was tested by comparing the SMMS scores with scores on the Academic Motivation Scale (AMS) and the exhaustion scale of Maslach Burnout Inventory-Student Survey (MBI-SS) for measuring study stress. Evidence for internal consistency was determined through the Cronbach's alpha for reliability. The analysis showed that the SMMS had a 3-factor structure. The validity in relations to other variables was established as both, the subscales and full scale scores significantly correlated positively with the intrinsic motivation scores and with the more autonomous forms of extrinsic motivation, the correlation decreasing and finally becoming negative towards the extrinsic motivation end of the spectrum. They also had significant negative correlations with amotivation scale of the AMS and exhaustion scale of MBI-SS. The Cronbach's alpha for reliability of the three subscales and full SMMS scores was 0.70, 0.67, 0.55 and 0.79. The strength of motivation for medical school has a three factor structure and acceptable validity evidence was found in our study.

Introduction

Motivation drives behaviour and effort towards success (Grolnick and Ryan 1987; Maslow 1970). Not much is known about motivation of medical students. There exists literature on the motivations or motives of students in entering medical study (McManus et al. 2006; Millan et al. 2005; Nieuwhof et al. 2005; Vaglum et al. 1999), but once the students start their medical study, very little is known about what happens to their motivation (Sobral 2004). We performed a review on research done on motivation in medical education (submitted) and found that motivation can be both, a dependent and an independent variable in medical education. As a dependent variable, motivation could be enhanced or manipulated by changes in the curriculum and learning environment (Von Below et al. 2008; Diemers et al. 2008; Cantillon and MacDermott 2008; McHarg et al. 2007; White 2007; Williams et al. 1997) and as an independent variable it does stimulate learning (Sobral 2004, 2008; Wilkinson et al. 2007; Dolmans et al. 1998) and academic success (Sobral 2004; Moulaert et al. 2004; Hoschl and Kozeny 1997; Price et al. 1994). The number of studies investigating motivation as a dependent variable was few. Also, although motivation for medical study has been found to correlate significantly with better performance in the study, more research is required to establish the causal relationship between the two variables. Knowledge of factors influencing motivation could provide medical educators with concrete means to enhance motivation of their students.

To carry out studies for exploring such relationships, we needed an instrument which would result in a valid measurement of the strength of motivation for medical school. Such an instrument was developed in our institute (Nieuwhof et al. 2004). The initial analysis suggested that the strength of motivation for medical school might have a uni-dimensional structure and the evidence for this was reported locally by our institute (Nieuwhof et al. 2002, unpublished). The analyses were limited and involved only first year medical students. Therefore we decided to delve deeper and look more rigorously into the evidence for the internal structure of the strength of motivation for medical school and also its relations to other variables.

Keeping in mind that, “validity refers to the degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests” and not “to the test itself” (Downing 2003; American Education Research Association, American Psychological Association 1999), we would first like to clarify the proposed uses of the SMMS questionnaire. The SMMS questionnaire is a useful tool to carry out studies to uncover relationships between motivation, teaching–learning processes, academic successes, but is not intended or recommended to be used for high stakes examinations, for e.g. for selection of students for medical study, because of the possibility of socially desirable answers in high stakes situations (Nieuwhof et al. 2004). We carried out the present study to evaluate the evidence for validity of measurement of strength of motivation for medical school using the SMMS questionnaire for these proposed uses.

Methods

Design

SMMS, AMS and MBI-SS (exhaustion scale) questionnaires were sent out electronically to 4,162 medical students from two universities, UMC Utrecht and VUmc Amsterdam, in The Netherlands. This was done using an internet based data collection programme called Survey Monkey (www.surveymonkey.com).

Ethical considerations

According to Dutch regulation, ethical permission is not mandatory for educational research. The work was carried out according to the Declaration of Helsinki i.e. the participation was voluntary, written informed consent was obtained from all the participants, confidentiality was guaranteed and the data was kept anonymous. The participants were assured that non-participation would not cause them any harm or disadvantage and that they could withdraw from the study any time without giving any reason for doing so.

Instruments

The Strength of Motivation for Medical School (SMMS) questionnaire (Nieuwhof et al. 2004) contains sixteen items that are scored on a Likert point scale of 5, ranging from “Strongly Disagree” to “Strongly Agree”. The higher the score, the greater is the strength of motivation. The measurement with this questionnaire has been previously reported to have favourable psychometric properties and good content validity (Nieuwhof et al. 2004).

Academic motivation scale (AMS) by Vallerand et al. (1989, 1992, 1993) measures the quality or type of motivation. It is based on the Self-determination theory (SDT) of motivation by Ryan and Deci (2000) and Deci et al. (1991). SDT basically distinguishes two types of motivation: intrinsic and extrinsic, and a state called “amotivation”, signifying a lack of motivation. Having intrinsic motivation for a study is to pursue the study solely out of genuine liking or interest. Having extrinsic motivation for a study is to pursue it for an external gain or benefit. Extrinsic motivation is further decomposed into several stages on a continuum from ‘very extrinsic’ to ‘approaching intrinsic motivation’ (Fig. 1). The motivation in these stages is regulated by four different mechanisms: external regulation, introjected regulation, identified regulation and integrated regulation. External regulation means, when we use education as our example, studying without any interest or perceived relevance, only because of pressure or expectation of others, e.g. parents. In this case, the locus of causation for studying is fully external. “Introjected regulation” means that a student realizes the importance of the study but the locus of causation is still external. “Identified regulation” means that a student has come to value the importance of his study, has identified with it and accepted the regulatory process. “Integrated regulation” means that the importance of the study has been fully integrated into the individual’s coherent sense of self; now the locus of causation is internal. Self-determination, the regulation type that fits with intrinsic motivation, means that one determines one’s own motivation; the motivation is self-generated and autonomous. External regulation is the least and integrated regulation is the most autonomous regulation of extrinsic motivation. Thus motivation has a spectrum which is most autonomous at one end and least autonomous at the other, the far end being amotivation (Fig. 1) (Ryan and Deci 2000).

Fig. 1

The self-determination continuum

AMS (Table 1) has 28 items grouped into 7 subscales, i.e. 4 items each, scored on a Likert scale of 1–7. An average of the total scores on each subscale is taken as the score. The subscales are “Intrinsic Motivation to know”, “Intrinsic Motivation towards accomplishment” and “Intrinsic Motivation to experience stimulation”, “Extrinsic Motivation- Identified Regulation”, “Extrinsic Motivation-Introjected Regulation”, “Extrinsic Motivation-External Regulation” and “Amotivation” (Vallerand et al. 1992). Intrinsic motivation (IM)-to-know is to study for the pleasure and satisfaction experienced while learning. IM-towards-accomplishment is to study for the pleasure and satisfaction experienced while accomplishing things (Vallerand et al. 1992). IM-to-experience stimulation is to study in order to experience stimulating sensations (Vallerand et al. 1992). There is no subscale available to measure EM-integrated regulation. There exists good evidence for the validity and reliability of the abovementioned construct of motivation as measured by the AMS (Vallerand et al. 1993).

Table 1

AMS and subscales (Vallerand et al. 1989)

Academic Motivation Scale (AMS)

Using the scale below, please indicate to what extent each of the following items presently corresponds to one of the reasons why you go to school (medical)

Intrinsic motivation—to know

2. Because I experience pleasure and satisfaction while learning new things

9. For the pleasure I experience when I discover new things never seen before

16. For the pleasure that I experience in broadening my knowledge about subjects (medical) which appeal to me

23. Because my studies (medical) allow me to continue to learn about many things that interest me

Intrinsic motivation—towards accomplishment

6. For the pleasure I experience while surpassing myself in my studies (medical)

13. For the pleasure that I experience while I am surpassing myself in one of my personal accomplishments

20. For the satisfaction I feel when I am in the process of accomplishing difficult academic activities

27. Because school (medical) allows me to experience a personal satisfaction in my quest for excellence in my studies (medical)

Intrinsic motivation—to experience stimulation

4. For the intense feelings I experience when I am communicating my own ideas to others

11. For the pleasure that I experience when I read interesting authors (medical)

18. For the pleasure that I experience when I feel completely absorbed by what certain authors have written

25. For the “high” feeling that I experience while reading about various interesting subjects (medical)

Extrinsic motivation—identified regulation

3. Because I think that a school (medical) education will help me better prepare for the career I have chosen

10. Because eventually it will enable me to enter the job market in a field (medical) that I like

17. Because this will help me make a better choice regarding my career (medical) orientation

24. Because I believe that a few additional years of Education (medical) will improve my competence as a worker

Extrinsic motivation—introjected regulation

7. To prove to myself that I am capable of completing my school (medical) degree

14. Because of the fact that when I succeed in school (medical) I feel important

21. To show myself that I am an intelligent person

28. Because I want to show myself that I can succeed in my studies (medical)

Extrinsic motivation—external regulation

1. Because with only a high-school degree I would not find a high-paying job later on

8. In order to obtain a more prestigious job later on

15. Because I want to have “the good life” later on

22. In order to have a better salary later on

Amotivation

5. Honestly, I don’t know; I really feel that I am wasting my time in school (medical)

12. I once had good reasons for going to school (medical); however, now I wonder whether I should continue

19. I can’t see why I go to school (medical) and frankly, I couldn’t care less

26. I don’t know; I can’t understand what I am doing in school (medical)

AMS available online from http://www.er.uqam.ca/nobel/r26710/LRCS/echelles_en.htm

The AMS-C 28 (i.e. college version) was used as it was most appropriate for the sample, but the words “college” were replaced by “school (medical)” as this is the term commonly used for medical study

We were looking for a measure of actual sacrifice for medical study and found the Maslach Burnout Inventory-Student Survey (MBI-SS), which contained an Exhaustion scale (Schaufeli et al. 2002). Burnout among students has been defined as feeling exhausted because of study demands, having a cynical and detached attitude toward one’s study, and feeling incompetent as a student. Only the exhaustion scale was used in our study expecting that if a student had a higher score on exhaustion from study his strength of motivation for medical school would be lowered. This scale had 5 items scored on a Likert scale of 0–6. Construct and concurrent validity evidence for this is good (Schaufeli et al. 2002).

Statistical analyses

The statistical analyses were carried out using the software programme SPSS version 15.0. An exploratory factor analysis for SMMS item scores was carried out. These items were analysed for grouping into subscales by using principal components analysis using promax rotation with Kaiser normalization, as the factors were not expected to be completely independent of each other (Costello and Osborne 2005; Field 2005). The ‘Relations to other variables’ were tested by Pearson’s correlation analysis of the SMMS subscales with the scores on the different scales of the AMS and exhaustion scale of the MBI-SS. Cronbach’s alpha for internal consistency was determined for establishing the reliability of the subscales.

Results

Out of the 4,162 students approached through their university e-mail accounts, there were 28 failed addresses, 2 students were out of the country, 1 had left to join another course and 9 explicitly refused to participate. Only 2,366 students viewed the questionnaire and only 1,494 completed it. Thus the sample size amounted to 1,494 (N) and the response rate to 36.2%.

The gender distribution among the responders was 71.9% females (1,075/1,494) and 28.1% males (419/1,494). This distribution was slightly, but significantly different from the actual population in both the schools. In percentage terms the difference was less than 5%, so we find it acceptable and representative. This difference was found to be approximately similar in both medical school populations. We also performed t-tests for independent groups on the SMMS total and subscale scores of males and females and did not find significant differences.

The factor analysis with promax rotation yielded three separate factors or subscales explaining 40.8% of the variance in the SMMS scores. The criteria for accepting the 3-factor structure were: eigenvalues above 1 (Field 2005), the scree plot (Fig. 2) and the amount of variance explained by the factors. This was also supported by theoretical explanation of three separate subscales as covered in the definition of strength of motivation for medical school. The three subscales for strength of motivation for medical study could be labelled as:

Fig. 2

Scree plot of SMMS questionnaire factor analysis

Willingness to sacrifice (Subscale 1)

Readiness to start (Subscale 2)

Persistence (Subscale 3)

Subscale 1: willingness to sacrifice

This subscale measures the willingness of a student to sacrifice for his/her medical study. It explained 26% of the variance in the SMMS scores. Five items i.e. 5, 7, 9, 10 and 12, based on their factor loadings (>0.40), fitted into this subscale (Table 2). Item 15 had low factor loadings i.e. 0.27, 0.21 and 0.06 on the subscales. We decided to check whether there is increase in internal consistency of Subscale 1 if this item is deleted and then make a decision on whether to retain this item in the subscale or not. The Cronbach’s alpha of this subscale was 0.69 and increased to 0.70 when item 15 was deleted, so we decided to drop this item from the subscale. All further analyses have been carried out after omitting item 15. Item total correlations of all items were >0.3, which is the recommended value (Field 2005).

Table 2

Factor loadings of the SMMS questionnaire items

S. no.	SMMS item	Factor 1	Factor 2	Factor 3
5.	Even if I could hardly maintain my social life, I would still continue medical training	0.75	0.02	−0.08
7.	I would still choose medicine even if that meant I would never be able to go on holidays with my friends anymore	0.70	0.05	−0.02
9.	If studying took me more than an average of 60 h a week, I would seriously consider quitting	0.45	−0.01	0.22
10.	I intend to become a doctor even though that would mean taking CME courses two evenings a week throughout my professional career	0.57	0.11	0.08
12.	I would like to become a doctor, even if that would mean giving precedence to my work over my family	0.76	−0.12	−0.04
15.	I would like to study medicine, even if I have to spend a lot of time on topics that later turn out to be a waste of time	0.27	0.21	0.06
1.	I would always regret my decision if I hadn’t availed myself of the opportunity to study medicine	−0.08	0.76	−0.04
3.	I would still choose medicine even if that would mean studying in a foreign country in a language that I have not yet mastered	−0.10	0.51	0.16
6.	I wouldn’t consider any other profession than becoming a doctor	0.11	0.66	−0.04
11.	It wouldn’t really bother me too much if I could no longer study medicine	0.04	0.66	0.09
16.	I would be prepared to retake my final high school exams to get higher marks if this would be necessary to study medicine	0.10	0.57	−0.16
2.	I would quit studying medicine if I were 95% certain that I could never become the specialist of my choice	0.06	−0.33	0.65
4.	As soon as I would discover that it would take me 10 years to qualify as a doctor, I would stop studying	0.02	0.17	0.57
8.	I would stop studying medicine if I started scoring low marks and failing tests often	−0.02	0.17	0.47
13.	I would quit studying as soon as it became apparent that there were no jobs or resident positions after graduation	0.01	−0.07	0.69
14.	I would not have chosen medicine if it would have caused me to accumulate substantial financial debts	−0.07	0.21	0.50

Factor/Subscale 1—willingness to sacrifice

Factor/Subscale 2—readiness to start

Factor/Subscale 3—persistence

Subscale 2: readiness to start

This subscale measures the readiness and will to enter medical study. It explained 9% of the variance in the SMMS scores. Five items i.e. 1, 3, 6, 11 and 16, with factor loadings >0.40, fitted into this subscale (Table 2). Internal consistency of this subscale was 0.67. Item total correlations of all items were >0.3.

Subscale 3: persistence

This subscale measures the persistence in medical study in spite of unfriendly circumstances during or after the study. This factor explained 8% of the variance in the SMMS scores. Five items i.e. 2, 4, 8, 13 and 14, with factor loadings >0.40, fitted into this scale (Table 2). Internal consistency of this subscale was 0.55, which is lower than recommended. Corrected item total correlation was 0.220, which is also lower than the recommended 0.3 (Field 2005).

There were no cross loadings of items in one subscale onto another subscale which were greater than 0.40 (Table 2). The three subscales were significantly correlated with each other with correlations of 0.485 (subscales 1 and 2), 0.369 (subscales 2 and 3) and 0.360 (subscales 1 and 3). This was expected, as they are not completely independent, which was also the reason why a promax rotation was used in the factor analysis.

The SMMS scores for Subscale 1, 2 and 3 and full SMMS (without item 15) correlated significantly positively with intrinsic motivation (0.248, 0.276, 0.175 and 0.308) and the correlation decreased and became negative as one moved from the intrinsic to the extrinsic motivation end of the spectrum (Table 3; Fig. 3). The correlations of Subscale 1 and 2 and full SMMS with the Extrinsic Motivation-External Regulation score (0.018, 0.037 and −0.023 respectively) and SMMS Subscale 3 with Extrinsic Motivation-Introjected Regulation (−0.036) were not significant. The correlations with amotivation (−0.367, −0.450, −0.293 and −0.472 respectively) and exhaustion from study (−0.184, −0.151, −0.175 and −0.206 respectively) scores were negative and significant.

Table 3

Pearson correlations between SMMS subscales and AMS and MBI-SS

Subscales of AMS and MBI-SS	Willingness to sacrifice	Readiness to start	Persistence	Full SMMS without item 15
Intrinsic motivation to know (AMS)	0.248*	0.296*	0.175*	0.308*
Intrinsic motivation towards accomplishment (AMS)	0.225*	0.294*	0.145*	0.286*
Intrinsic motivation to experience stimulation (AMS)	0.258*	0.341*	0.144*	0.327*
Extrinsic motivation—identified regulation (AMS)	0.257*	0.340*	0.114*	0.305*
Extrinsic motivation—introjected regulation (AMS)	0.104*	0.126*	−0.036 (p = 0.169)	0.087*
Extrinsic motivation—external regulation (AMS)	0.018 (p = 0.494)	0.037 (p = 0.148)	−0.112*	−0.023 (p = 0.375)
Amotivation (AMS)	−0.367*	−0.450*	−0.293*	−0.472*
Exhaustion from study (MBI-SS)	−0.184*	−0.151*	−0.175*	−0.206*

All positive and negative correlations >0.037 and <0.450 are significant at *p < 0.001

Fig. 3

Graphical representation of correlations between SMMS and AMS and MBI-SS exhaustion subscales. All positive and negative correlations >0.037 and <0.450 are significant at P < 0.001

Discussion

The SMMS questionnaire is the only existing questionnaire, to the best of our knowledge, for measuring the strength of motivation of students particularly for medical school. Validation of the measurement by this instrument is therefore important. If this measurement has good psychometric properties, the SMMS questionnaire could be used on a post-hoc basis for measuring strength of motivation of students admitted to medical study or for studying motivation of students through the duration of their medical study and how it can be manipulated, finding a link between motivation and academic success and so on.

The SMMS questionnaire has previously been proposed to have a uni-dimensional factor structure (Nieuwhof et al. 2004). Though in our analyses the first factor alone could explain 26% of the variance in the SMMS scores, we accepted a 3-factor structure for the questionnaire because the factor loadings of 11 items in the questionnaire onto the first factor were less than 0.40. The 3-factor structure was also theoretically sound.

The findings for reliability of the Willingness to sacrifice and Readiness to start are acceptable since they are 0.70 and 0.67 respectively. The reliability for Persistence subscale (0.55) was lower than expected, but we think that it is acceptable for comparing groups of students. These reliabilities are good enough for use of this questionnaire to assess and understand relationships between motivation and teaching–learning processes or academic successes. We would not advocate the use of this instrument for selecting medical students in which case it would qualify as a high-stakes examination and would need higher reliability values. We would recommend the use of the Persistence subscale with caution and encourage further studies for establishing further validity evidence. The use of the full scale (SMMS-R, Table 4) would also be acceptable. The reliability of the full scale is higher than that of the subscales (Cronbach’s alpha = 0.79). Similar reliabilities have been found in other studies (Kusurkar et al. 2010).

Table 4

SMMS-R questionnaire People have diverse reasons to study medicine. Please indicate how much the following statements reflect your personal situation by marking 1–5 (Strongly disagree to Strongly agree) with each statement below

S. no.	SMMS-R item	Subscale	To be reverse scored
1.	I would always regret my decision if I hadn’t availed myself of the opportunity to study medicine	2
2.	I would quit studying medicine if I were 95% certain that I could never become the specialist of my choice	3	√
3.	I would still choose medicine even if that would mean studying in a foreign country in a language that I have not yet mastered	2
4.	As soon as I would discover that it would take me 10 years to qualify as a doctor, I would stop studying	3	√
5.	Even if I could hardly maintain my social life, I would still continue medical training	1
6.	I wouldn’t consider any other profession than becoming a doctor	2
7.	I would still choose medicine even if that meant I would never be able to go on holidays with my friends anymore	1
8.	I would stop studying medicine if I started scoring low marks and failing tests often	3	√
9.	If studying took me more than an average of 60 h a week, I would seriously consider quitting	1	√
10.	I intend to become a doctor even though that would mean taking CME courses two evenings a week throughout my professional career	1
11.	It wouldn’t really bother me too much if I could no longer study medicine	2	√
12.	I would like to become a doctor, even if that would mean giving precedence to my work over my family	1
13.	I would quit studying as soon as it became apparent that there were no jobs or resident positions after graduation	3	√
14.	I would not have chosen medicine if it would have caused me to accumulate substantial financial debts	3	√
15a.	I would be prepared to retake my final high school exams to get higher marks if this would be necessary to study medicine	2

The SMMS-R has been translated from Dutch to English for publication purposes

aOld item 15 deleted and old item 16 renumbered as item 15

We found significant correlations with a more established motivation scale i.e. the AMS, and a pattern of decreasing correlations from the intrinsic to the extrinsic motivation could be clearly observed. This supports the construct validity of the SMMS scale. We expected all AMS subscales for intrinsic motivation and the extrinsic subscale for “identified regulation” to correlate highest with the SMMS scores and its subscales. This was supported by the results. The size of correlations was not very big. This may be explained by the fact that the SMMS measures strength of motivation and the AMS measures type of motivation. We also expected all three SMMS subscales to have a strong negative correlation with amotivation, which was also supported by the results. These findings are in concurrence with the earlier study, which had found that SMMS scores correlated significantly negatively with ambivalence towards studying (Nieuwhof et al. 2004). We also expected no significant correlation between the SMMS subscales and external regulation subscale of the AMS. This was because earlier studies indicate that a higher number of students have altruistic motives for entering medical study than monetary or status motives and the external regulation subscale measures the latter rather than the former (McManus et al. 2006; Millan et al. 2005; Vaglum et al. 1999). We expected the identified regulation subscale of AMS to be positively correlated with the SMMS subscales because the unique nature of medical education would make most students answer the items positively, as all the statements seem to be true in the absolute sense of the profession. Considering this as a limitation of these items, we would not recommend deep interpretation of this particular finding. The significant negative correlation with exhaustion from studies was expected as exhaustion or stress of study would tend to lower the motivation in the students. Low motivation has previously been proposed as both a cause and consequence of student distress (Dyrbye et al. 2005). It would perhaps have been better to use the entire MBI-SS rather than using the exhaustion scale in isolation, since the other scales measuring cynical and detached attitude towards study and feelings of incompetence would have added on to the findings and perhaps even increased the size of correlation. The reason not to do so in the current study was the anticipated length of all questions combined and its adverse effect on response. The use of the entire MBI-SS is however recommended in any further studies.

This study has a number of limitations. The biggest limitation is the response rate, which was quite low, therefore we cannot rule out a response bias. One of the reasons for this response rate could be that the students found the survey too long. The other reason for a response bias could be that the more motivated students in the sample were also the ones taking part in the study. The construct validity of this questionnaire needs to be evaluated and established in a wider variety of samples as all the studies which have used SMMS questionnaire as the instrument have been carried out in the Netherlands on Dutch students; even if they have been done in different institutes in the country. We recommend this because most of the students in The Netherlands enter medical study through a weighted lottery system and there have been studies which show that students entering by selection have higher motivation than those entering through lottery as measured with the SMMS (Hulsman et al. 2007). Therefore, additional validation needs to be carried out in other countries where selection is the method for entry into medical school. Further studies are also proposed to confirm the 3-factor structure through a confirmatory factor analysis.

Conclusion

We established support for the validity of measurement of the strength of motivation for medical school using the SMMS questionnaire, both for its entirety and for its subscales. The SMMS questionnaire has a three factor structure, signifying Willingness to sacrifice, Readiness to start and Persistence.