An international validation study of two achievement goal measures in a pharmacy education context.

BACKGROUND: Achievement goal theory helps us understand what motivates students to participate in educational activities. However, measuring achievement goals in a precise manner is problematic. Elliot and McGregor's Achievement Goal Questionnaire (AGQ) and Elliot and Murayama's revised Achievement Goal Questionnaire (AGQ-R) are widely used to assess students' achievement goals. Both instruments were developed and validated using undergraduate psychology students in the USA.
METHODS: In this study, our aims were to first of all, assess the construct validity of both questionnaires using a cohort of Australian pharmacy students and, subsequently, to test the generalizability and replicability of these tools more widely in schools of pharmacy in other English-speaking countries. The AGQ and the AGQ-R were administered during tutorial class time. Confirmatory factor analysis procedures, using AMOS 19 software, were performed to determine model fit.
RESULTS: In contrast to the scale developers' findings, confirmatory factor analysis supported a superior model fit for the AGQ compared with the AGQ-R, in all countries under study.
CONCLUSION: Validating measures of achievement goal motivation for use in pharmacy education is necessary and has implications for future research. Based on these results, the AGQ will be used to conduct future cross-sectional and longitudinal analyses of the achievement goals of undergraduate pharmacy students from these countries.

Introduction

For more than three decades, achievement goal theory has been one of the most important motivational theories in the field of education and has undergone significant conceptual development during this time.1–3 Achievement goals are precise types of goals that consider “competence” as the aim for any individual.4 Achievement goals are defined as a “future-focused cognitive representation that guides behavior to a competence-related end state that the individual is committed to either approach or avoid”.5 Current understanding centers around four types of goals that are seen to influence motivation for students’ achievement in learning environments. These are: 1) Mastery-Approach (M-AP), where the individual is motivated to learn or improve his/her skills; 2) Mastery-Avoidance (M-AV), where the individual is motivated to avoid failure to learn or declines in skill; 3) Performance-Approach (P-AP), where the individual is motivated to outperform others or appear talented; and 4) Performance-Avoidance (P-AV), where the individual is motivated to avoid doing worse than others or appearing less talented.6–10

A number of studies have linked the M-AP goal to positive outcomes, such as high interest,11 high persistence,12 using deep learning strategies,13 and seeking help when needed.14 However, despite these beneficial outcomes, no significant positive relationship between this type of achievement goal and academic achievement has been found.5,9 The P-AP goal, however, is associated with different effects. On the one hand, it is linked to memorization instead of deep learning15 and on the other, this type of achievement goal has a significant positive correlation with academic achievement.9,16–18 The avoidance types of achievement goals (ie, M-AV and P-AV) are associated with negative outcomes, such as low intrinsic motivation, anxiety, and low academic achievement.19–25

Despite the positive contributions achievement goal theory has made to the field of education, achieving precision in measuring these achievement goals has been difficult,5,26 and this is reflected in researchers’ continued endeavors to examine the theoretical underpinnings of achievement goal motivation. For example, one well-known instrument is the Achievement Goals Questionnaire (AGQ),6,27 reported by Elliot and McGregor.6 This instrument was developed and validated in higher education settings in the US, using a cohort of psychology students. More recently, the AGQ underwent further refinement in an attempt to develop a more precise instrument. According to Elliot and Murayama,26 some items on the AGQ assess either a value (eg, “It is important for me to do better than other students”) or a concern (eg, “I worry that I may not learn all that I possibly could in this class”), instead of a goal. In addition, the authors argued that one of the items intended to measure the P-AV construct was instead, measuring the goal with the reason behind this goal (eg, “My fear of performing poorly in this class is often what motivates me”). According to the authors Elliot and Murayama, the AGQ was designed to measure the goal regardless of the reasons behind it. Furthermore, Elliot and Murayama argued that the word “grades” that appears in one item intended to measure the P-AP construct could be applicable for both mastery and performance goals. Based on these concerns, a Revised Achievement Goal Questionnaire (AGQ-R)26 was developed by Elliot and Murayama, which was administered to undergraduate psychology students enrolled in US universities. In both questionnaires (Figure 1), each achievement goal is measured by three variables (ie, 12 variables for each questionnaire). Elliot and Murayama26 used confirmatory factor analysis to compare the construct validity of the AGQ with the AGQ-R, and the latter was found to provide a better fit to the data and to be superior to the AGQ in predicting and determining achievement goals.26

Figure 1

Schematic model of relationship between construct and questionnaire items.

Abbreviation: I, item.

More recently, Elliot et al2 developed a new questionnaire that builds on the achievement goal construct. This questionnaire measures six types of achievement goals: task-approach, task-avoidance, self-approach, self-avoidance, other-approach, and other-avoidance.2 These achievement goals have some similarities with the “classical achievement goals”, for example, “task” goals are mirrored in the performance goals and “self” goals are mirrored in the mastery goals. However, these new developments are beyond the scope of the current study.

Comparative face validity review of the content of AGQ and AGQ-R suggests that some items in the revised questionnaire are confusing and hard to understand. However, face validity review can be influenced by subjectivity and is less empirical28,29 than an examination of the construct validity of the two instruments. The construct validity approach provides a more rigorous and defensible method of assessing the relationships between the questionnaire items and the achievement goal constructs they are purported to measure.30–32 Furthermore, it is possible that in a different educational context and discipline area, such as a pharmacy education setting, these two measures may not be as precise in their measurement of university student achievement goal orientations. In addition, very little research has been conducted to investigate the utility of scales for measuring achievement goals in different educational settings, including pharmacy education. The research that has been conducted in this field has related more to learning styles,33,34 rather than achievement goals.

Validation of the AGQ and AGQ-R in an Australian pharmacy education setting is an important first step in determining the usability of these scales at a local level. However, since multinational data might influence the validity of these questionnaires,35 inclusion of participants from different countries will provide a more rigorous and generalizable investigation of the validity of the AGQ and AGQ-R measures. To our knowledge, there have been no cross-national validation studies of the motivational preferences of pharmacy students. Thus, results of this study will lay a foundation for future studies into undergraduate pharmacy students’ achievement goals and will facilitate comparative and longitudinal research between different countries. Knowing pharmacy students’ achievement goals will provide academics with invaluable understanding of how their students respond when they encounter academic activity.23 Yet the first step is to determine a precise instrument to use for measuring these goals.

Therefore, the aims of this project were to, first, assess the construct validity of the AGQ and AGQ-R, using a cohort of Australian undergraduate pharmacy students and, subsequently, to test the generalizability and replicability of these tools in schools of pharmacy in other English-speaking countries. Ultimately, the most psychometrically appropriate version of the model can be determined.

Methods

Ethical approval was granted by human ethics committees at the six participating universities.

Sample and procedure

Study 1

Australian participants for this study were undergraduate students enrolled in the 4-year Bachelor of Pharmacy degree at the Faculty of Pharmacy, University of Sydney. The study was initiated during the first semester of 2012.

The researchers invited students to participate in the study during normal lecture or tutorial periods. They were advised that participation was voluntary and that if they chose to participate, they could withdraw from the project at any time. In addition, students were advised that their decision to participate would not impact on their academic results or influence their student–teacher relationships. Researchers approached the students as a group and not individually. The questionnaires were administrated to students in paper form by the researchers. Completion of the questionnaires took approximately 15 minutes.

Study 2

International participants were those students enrolled in a professional pharmacy degree program at universities in the US (two universities), UK (two universities), and New Zealand (one university). The locations for data collection were selected by the first and last authors, who contacted researchers in different countries of interest at pharmaceutical conferences. The three locations were purposefully chosen as they are comparable in terms of language, education, and culture. The data collection method for Study 2 (international study) was the same as for Study 1 (Australian study).

English proficiency was an essential criterion for admission at all the participating universities. Such proficiency is measured either by International English Learning Testing System (IELTS) or Test of English as a Foreign Language (TOEFL) exams.36–41

Materials

The AGQ and AGQ-R6,26 were used. Both questionnaires contain 12 items that are intended to measure the constructs underpinning achievement goal motivation, known as latent factors. In the AGQ and AGQ-R models, these latent factors are the four goal orientations (P-AP, M-AV, M-AP, and P-AV). The AGQ uses a seven-point Likert scale, scored from 1= “not at all true of me” to 7= “very true of me”, and the AGQ-R uses a five-point Likert scale, scored from from 1= “strongly disagree” to 5= “strongly agree”. The questionnaires were combined into one survey, a total of 24 questions. Sociodemographic indicators included in the survey were sex and age.

Analysis

SPSS Statistics for Windows, Version 21.0 (IBM Corp, Armonk, NY, USA) was used for descriptive statistics regarding year group, sex, and age for all participants. Confirmatory factor analyses, using IBM SPSS AMOS 21.0 (IBM Corp.) software, were conducted on the data, for both the AGQ and AGQ-R, to determine whether the data replicated the expected factor/scale structure. The analyses were conducted on covariance matrices, and the solutions were generated on the basis of maximum likelihood estimation. No modifications were made to the model, which was a direct replication of the original model developed by Elliot et al (ie, Elliot and McGregor, and Elliot and Murayama).

The quality of any instrument is evaluated by its goodness of fit to the data.42 The most commonly used and reliable fit indices are the Comparative Fit Index (CFI), Tucker-Lewis Index (TLI), chi-square degrees of freedom ratio or normalized chi square (χ2/df), Akaike Information Criterion (AIC), and root-mean-square error of approximation (RMSEA).42–46

On this basis, therefore, several indexes were used in this study, to compare the fit of the models to the data: χ2/df, CFI, TLI, AIC, and RMSEA. The following criteria were used to assess the adequacy of model fit: χ2/df ≤2.0,47 CFI ≥0.90,48 TLI ≥0.90,48 AIC (the minimum value of the two models),49 and RMSEA ≤0.08.48

The Australian and UK sample sizes were sufficient to conduct separate confirmatory factor analyses; however, the New Zealand and US sample sizes were not (n<5 participants per observed variable).50 For this reason, we combined both countries into one group (NZ/US). The Australian data set was analyzed first, followed by the UK and NZ/US data set.

Results

Study 1: Australia

A total of 209 students (122 female and 78 male), with a mean age of 21.4 years, completed the questionnaires (Table 1).

Table 1

Participant demographics

Country	Age (mean/SD)	Sex: female/male (N (%)/N)	Total (N)
Australia	21.40/2.49	122 (58%)/78 Unspecified: 9	209
UK	20.80/1.81	311 (69.4%)/132 Unspecified: 5	448
New Zealand	21.30/2.65	75 (71.4%)/30	105
US	25.80/1.59	67 (58.3%)/47 Unspecified: 1	115

Note: N=877.

Abbreviation: SD, standard deviation.

Factor loadings and correlations

The results of factor loadings for AGQ and AGQ-R are shown in Table 2. For the AGQ, the model shows overall high to very high loadings between observed indicators (questionnaire items) and their related latent factors, ranging from λ=0.67 to λ=0.95. Similar results were obtained from the AGQ-R model. However, in this revised model, one observed indicator (Item 3) in particular showed a weak relationship (λ=0.49) with its latent factor (M-AV).

Table 2

Factor loadings: AGQ and AGQ-R

Goal orientation/Item	Australia	UK	NZ/US
Performance-Approach – AGQ
1. It is important for me to do better than other students	0.95	0.90	0.94
2. I t is important for me to do well compared to others in this class	0.91	0.89	0.91
3. My goal in this class is to get a better grade than most of the other students	0.90	0.83	0.87
Performance-Approach – AGQ-R
1. I am striving to do well compared to other students	0.92	0.84	0.89
2. My aim is to perform well relative to other students	0.91	0.79	0.84
3. My goal is to perform better than the other students	0.90	0.83	0.79
Performance-Avoidance – AGQ
1. I just want to avoid doing poorly in this class	0.74	0.81	0.79
2. My goal in this class is to avoid performing poorly	0.91	0.89	0.85
3. My fear of performing poorly in this class is often what motivates me	0.67	0.54	0.62
Performance-Avoidance – AGQ-R
1. My goal is to avoid performing poorly compared to others	0.82	0.75	0.67
2. I am striving to avoid performing worse than others	0.88	0.85	0.92
3. My aim is to avoid doing worse than other students	0.88	0.86	0.85
Mastery-Approach – AGQ
1. I want to learn as much as possible from this class	0.83	0.79	0.79
2. It is important for me to understand the content of this course as thoroughly as possible	0.89	0.86	0.90
3. I desire to completely master the material presented in this class	0.78	0.71	0.78
Mastery-Approach – AGQ-R
1. My aim is to completely master the material presented in this class	0.70	0.71	0.72
2. My goal is to learn as much as possible	0.85	0.79	0.81
3. I am striving to understand the content of this course as thoroughly as possible	0.85	0.67	0.69
Mastery-Avoidance – AGQ
1. I worry that I may not learn all that I possibly could in this class	0.81	0.79	0.78
2. Sometimes I’m afraid that I may not understand the content of this class as thoroughly as I’d like	0.83	0.83	0.84
3. I am often concerned that I may not learn all that there is to learn in this class	0.93	0.95	0.92
Mastery-Avoidance – AGQ-R
1. My aim is to avoid learning less than I possibly could	0.73	0.81	0.82
2. My goal is to avoid learning less than it is possible to learn	0.80	0.79	0.79
3. I am striving to avoid an incomplete understanding of the course material	0.49	0.52	0.59

Abbreviations: AGQ, Achievement Goal Questionnaire; AGQ-R, Achievement Goal Questionnaire – Revised.

As shown in Table 3, correlations between the latent factors in the AGQ were weak, suggesting the presence of distinct constructs. In contrast, the correlations between the latent factors in the AGQ-R were somewhat higher, especially between the M-AP and M-AV, and P-AP and P-AV constructs (Cronbach’s α =0.84 and =0.79, respectively).

Table 3

Factor correlations for AGQ/AGQ-R

	Mastery-Avoidance	Mastery-Approach	Performance-Avoidance
Australia
Performance-Approach	0.33/0.54	0.32/0.57	0.18/0.79
Mastery-Avoidance		0.40/0.84	0.26/0.64
Mastery-Approach			0.22/0.45
UK
Performance-Approach	0.13/0.23	0.08/0.21	0.11/0.69
Mastery-Avoidance		0.24/0.35	0.06/0.50
Mastery-Approach			0.08/0.08
New Zealand/US
Performance-Approach	0.03/0.25	0.21/0.34	-0.07/0.71
Mastery-Avoidance		0.24/0.41	0.02/0.44
Mastery-Approach			0.16/0.16

Abbreviations: AGQ, Achievement Goal Questionnaire; AGQ-R, Achievement Goal Questionnaire – Revised.

Fit indices

Table 4 shows the results of fit indices for both models. The AGQ model showed good fit for data (eg, χ2/df =1.80, RMSEA =0.06). However, the AGQ-R showed poor fit for the Australian data (eg, χ2/df =2.58, RMSEA =0.09).

Table 4

Goodness of fit summary for AGQ and AGQ-R*

	χ2/df	CFI	TLI	AIC	RMSEA
Australia
AGQ	1.80	0.98	0.97	138.81	0.06
AGQ-R	2.58	0.96	0.94	154.98	0.09
UK
AGQ	1.92	0.98	0.98	152.15	0.05
AGQ-R	5.01	0.92	0.89	300.38	0.09
New Zealand/US
AGQ	1.65	0.98	0.97	139.36	0.06
AGQ-R	3.82	0.90	0.86	243.39	0.11

Notes:

Recommended criteria: χ2/df ≤2.0, CFI ≥0.90, TLI ≥0.90, AIC – minimum value of the two models; RMSEA ≤0.08.

Abbreviations: AGQ, Achievement Goal Questionnaire; AGQ-R, Achievement Goal Questionnaire – Revised; AIC, Akaike Information Criterion; CFI, Comparative Fit Index; RMSEA, root-mean-square error of approximation; TLI, Tucker-Lewis Index; df, degrees of freedom.

Study 2: UK and NZ/US

A total of 667 out of 721 students (92.5%) (483 female, 232 male, and six with undisclosed sex), with a mean age of 21.7 years, completed both questionnaires in this study. We deleted cases containing incomplete data (54 participants).31 Descriptive statistics for the participants, by country, are reported in Table 1.

Table 2 presents the factor loadings for the AGQ and AGQ-R models. For the AGQ, in UK and NZ/US samples, the model showed overall medium to high loadings between the observed indicators and their related latent factors, ranging from λ=0.94 to λ=0.54. Similar factor loading results were obtained for the AGQ-R (Table 3), with factor loadings ranging from λ=0.92 to λ=0.52.

In both the UK and NZ/US samples, the AGQ produced a weak correlation between the model’s latent factors, thus suggesting the presence of distinct constructs (Table 3). However, the correlations between the latent factors (Table 3) were somewhat higher in the AGQ-R, especially between the P-AP and P-AV constructs (Cronbach’s α =0.69 and 0.71, for the UK and NZ/US, respectively).

The AGQ model showed good fit for the UK and NZ/US data (eg, χ2/df =1.92, RMSEA =0.05 for the UK; χ2/df =1.65, RMSEA =0.06 for NZ/US). However, the AGQ-R showed poor fit for the UK and NZ/US data (Table 4) (eg, χ2/df =5.01, RMSEA =0.09 for the UK; χ2/df =3.82, RMSEA =0.11 for NZ/US).

Discussion

Although the positive impact of achievement goal theory on education in general and higher education specifically is well known, measuring achievement goals in a precise manner is problematic.5 The AGQ and AGQ-R are validated instruments widely used to assess students’ achievement goals. In this study, our aims were to assess the construct validity of the AGQ and AGQ-R, using a cohort of Australian pharmacy students, in order to determine the most psycho-metrically appropriate version of the model and to assess the applicability and generalizability of both questionnaires across a range of pharmacy disciplines in English-speaking countries.

In contrast to Elliot and Murayama’s findings,26 our results show the AGQ to be a more robust measure of pharmacy students’ achievement goal orientations compared with the AGQ-R, in all six study sites. The factor loadings, correlations, and fit indices all indicate that the AGQ demonstrated better construct validity when using an international pharmacy student cohort. Results indicate that students from six schools of pharmacy in four different countries were better able to understand and interpret the questionnaire items for the AGQ than the AGQ-R, that the AGQ was a more appropriate measure of achievement goals in our pharmacy cohorts, and that the AGQ was a more psychometrically robust measure than the AGQ-R.

Item 3, “I am striving to avoid an incomplete understanding of the course material”, in particular, appears to be problematic. It showed low factor loadings across all samples in our study (λ ranging from 0.49 to 0.59). Such a low factor loading may be attributed to the double negative construction of this item, which is in general, hard to understand.51 Interestingly, this finding mirrors those reported by Hart et al,52 whose validation study utilizing a sample of African American high school students revealed that Item 3, with its latent factor M-AV, had a low factor loading (λ=0.42). Furthermore, Hart et al52 also found high correlations between achievement goal constructs in the AGQ-R, especially between P-AP and P-AV, suggesting that the model cannot measure separate latent factors effectively.

The findings in our study, contradictory to those of Elliot and Murayama, may be attributed to the differences between the cohorts used in the original validation study and the current study. To the best of our knowledge, there are no studies that compare pharmacy and psychology students’ learning styles and achievement goals, and therefore further work is warranted to better understand any differences between the two subject areas. Overall, these results emphasize the importance of confirming the validation of measures of achievement goal motivation in different educational settings.

Limitations

In interpreting the study’s findings, it is important to note to its limitations. The findings might not be generalizable to all pharmacy students as only four countries were included in this study. Additional construct validity studies for both questionnaires, using pharmacy students from other cultures, is required before we can generalize our findings globally. This study has laid a foundation for future studies into pharmacy students’ achievement goals and will facilitate comparative and longitudinal research between different countries to better understand students’ motivations.

Conclusion

The AGQ met the criteria for a good-fitting model in the context under investigation, while the AGQ-R did not, which is in contrast to the findings of Elliot and Murayama. Based on these results, the research will proceed to cross-sectional and longitudinal studies of the goal orientations and approaches to learning of pharmacy students, using the AGQ. Furthering our understanding of achievement goal constructs and their relevance to pharmacy education may facilitate future improvements to pharmacy education teaching and learning.