Performance assessment to investigate the domain specificity of instructional skills among pre-service and in-service teachers of mathematics and economics.

BACKGROUND: Key elements of instructional quality include the teacher's ability to immediately react in domain-specific classroom situations. Such skills - defined as action-related skills - can only be validly assessed using authentic representations of real-life teaching practice. However, research has not yet explained how teachers apply domain-specific knowledge for teaching and to what extent action-related skills are transferable from one domain to another. AIMS: Our study aims to examine (1) the relationship between action-related skills, content knowledge, and pedagogical content knowledge, and (2) the domain specificity of action-related skills of (prospective) teachers in the two domains of mathematics and economics. SAMPLE(S): We examined German pre-service and in-service teachers of mathematics (N = 239) and economics (N = 321), including n = 96 (prospective) teachers who teach both subjects.
METHODS: Action-related skills in mathematics and economics were measured using video-based performance assessments. Content knowledge and pedagogical content knowledge were assessed using established paper-pencil tests. Correlation analyses, linear regressions, and a path model were applied.
RESULTS: In mathematics and economics, we find a similar pattern of moderate correlations between action-related skills, content knowledge, and pedagogical content knowledge. Moreover, a significant correlation between action-related skills in mathematics and economics can be explained almost entirely by underlying relations between content knowledge and pedagogical content knowledge in both domains.
CONCLUSIONS: Our findings suggest that action-related skills empirically differ from domain-specific knowledge and should be considered as domain-specific constructs. This indicates that teacher education should not only focus on domain-specific teacher knowledge, but may also provide learning opportunities for action-related skills in each domain.

Background

Teacher training programmes at university particularly emphasize the development of practical skills that will later enable teachers to perform successfully and professionally in instructional praxis. Key elements of instructional quality include the ability to teach a lesson as intended, anticipate problems, and perform immediately and under time pressure in domain‐specific classroom situations (Hattie, 2009). Such skills – defined as action‐related skills (AS, Knievel, Lindmeier, & Heinze, 2015) – can only be measured validly if the assessment involves an authentic representation of real‐life teaching practice. Although previous attempts in assessing teacher skills using traditional test methods (e.g., multiple‐choice tests, text vignettes, or teacher reflections) were found to measure relevant aspects of teacher expertise, there is growing agreement that measuring teachers’ skills to react in a real‐time instructional situation requires consideration of the complexity and contextual nature of real classroom instruction (Darling‐Hammond & Baratz‐Snowden, 2005; Kersting, 2008). Our approach for measuring AS, therefore, uses video clips of realistic instruction scenarios that teachers have to react to in a naturalistic way (e.g., providing feedback) under time pressure, as is the case in the classroom.

AS are considered bridging constructs that create links between cognitive dispositions (such as knowledge) and real‐life performance (Blömeke, Gustafsson, & Shavelson, 2015). In previous studies, mathematics teachers’ instructional skills were found empirically separable from knowledge (Knievel et al., 2015) and to predict student learning (Kersting, Givvin, Thompson, Santagata, & Stigler, 2012). As these kinds of studies are typically conducted in one domain – domain referring to a school subject in this paper – it is currently unknown if those findings regarding the domain of mathematics also apply to instructional skills in other domains. Validated performance assessments to replicate those findings are currently missing for the majority of teaching domains. The cognitive relationship between teacher knowledge and AS is still insufficiently researched, especially with respect to the acquisition of AS (Blömeke, Busse, Kaiser, König, & Suhl, 2016). For instance, only little is known as to whether AS are to be acquired specifically for one domain (e.g., in the sense of an ‘enacted’ PCK) or should be considered as transferable skills that enable teachers to teach different domains given that they possess the relevant domain‐specific knowledge (e.g., in the sense of general teaching skills). Regarding those questions, the available research is limited as most studies either focus only one domain or contrast different groups of teachers teaching only one domain.

In this study, we addressed this gap. We examined the domain specificity of teachers’ AS using the fact that teachers in Germany are equally trained to teach two school subjects (i.e., teachers major in two domains). We used aligned performance assessments for AS in two domains, mathematics and economics, and applied our assessment to (prospective) secondary teachers who teach both subjects. We examined (1) how AS relate to domain‐specific teacher knowledge in mathematics and economics and (2) how AS in mathematics and economics relate intra‐individually if teachers are trained to teach both domains.

Theoretical background

Teachers require domain‐specific knowledge and must be able to apply it in order to understand a situation and react immediately according to the standards of the profession (Borko, 2004; Hill et al., 2008; Stürmer, Könings, & Seidel, 2013). Thus, the three main requirements for teaching a specific domain in the classroom are as follows:

domain‐specific knowledge of the content,

knowledge about effective ways of teaching and learning this content, and

application of this knowledge in a teaching situation under time pressure.

Research on teachers’ domain‐specific knowledge and instructional skills

According to Shulman (1986), teachers’ domain‐specific knowledge can be conceptualized as two main components: content knowledge (CK), which represents an understanding of the subject matter ‘per se’, and pedagogical content knowledge (PCK), which represents the pedagogical aspects required to teach the subject matter, such as knowledge about students’ cognition, and practical exercises (p. 9). It is assumed that CK is a crucial prerequisite for the acquisition of PCK and both knowledge facets are mainly acquired during the university phase of teacher training (Baumert et al., 2010; Depaepe, Verschaffel, & Kelchtermans, 2013). Based on the work of Shulman (1986), refined models of mathematics knowledge for teaching were developed, which furthered our understanding of the rich professional knowledge base of teaching (Ball, Thames, & Phelps, 2008). Those conceptualizations were investigated thoroughly and have proven to predict both instructional quality and student learning (Baumert et al., 2010; Hill et al., 2008).

Despite this predictive power of teacher knowledge, more recent research emphasizes the distinction between the knowledge a teacher may possess and the application of knowledge that becomes relevant in practice (Santagata & Sandholtz, 2018). This is especially true with respect to teachers’ instructional skills to apply knowledge for teaching a domain in the classroom, which we define as AS. Corresponding performance assessments were found to be even more predictive for instructional quality and student learning than measures of teacher knowledge (Kersting, Givvin, Sotelo, & Stigler, 2010; Kersting et al., 2012). AS are required for teachers to react in domain‐specific instructional situations, for example when students need assistance with an exercise, ask domain‐related questions, present incorrect solutions, or demonstrate a misconception. In contrast to demands mirrored through the assessments of teachers’ CK and PCK, those real‐life teaching demands are typically characterized by immediacy, complexity, spontaneity, context‐dependency, and interactivity as teachers have to directly respond to possibly unexpected issues in conversation with the students (Borko & Shavelson, 1990; Jackson, 1990).

While there is an agreement among researchers that teachers need skills to apply CK and PCK for mastering instructional situations, the nature of those skills is so far underresearched. Most conceptualizations share the notion of CK and PCK being crucial knowledge bases for teachers’ actions. Some researchers propose that teachers’ knowledge and performance are mediated by situation‐specific skills, that is perception, interpretation, and decision making, that might be understood as general skills (Blömeke, Gustafsson et al., 2015). Other researchers describe teachers’ skills to apply knowledge to be embodied in different levels of knowledge usability (Kersting, 2008; Santagata & Sandholtz, 2018) or in an enacted form of domain‐specific knowledge (e.g., enacted PCK; Carlson & Daehler, 2019). Overall, there is only little empirical evidence as to which conceptualization is more appropriate to describe AS.

Domain specificity of action‐related skills

Education and training programmes for pre‐service teachers strive to provide optimal learning opportunities for both teacher knowledge and AS. Such programmes are usually designed for a specific domain (e.g., for prospective mathematics teachers). Learning opportunities for domain‐specific knowledge and for general pedagogical knowledge are mostly offered in a side‐by‐side design.

Subsequently, a large body of research, especially with focus on the development of teacher knowledge in teacher training, has examined domain‐specific and generic components of a teachers’ knowledge base, such as the domain‐specific CK and general pedagogical knowledge which is considered not domain‐specific. Teachers’ PCK is described to encompass both domain‐specific components and general educational knowledge, for example, regarding the developmental readiness of their students and effective instructional strategies that may be generic and not connected to a specific domain (Carlson & Daehler, 2019). With regard to Shulman's definition, PCK itself is thus often illustrated as an ‘amalgam of content and pedagogy’ (Shulman, 1987, p. 8). Current discussions also point out that PCK in one domain may complement CK in another domain (e.g., mathematics‐specific PCK complementing economics CK), which suggests that PCK is not completely confined to one domain (Kirschner, Verschaffel, Star, & Van Dooren, 2017).

In contrast, research on the domain specificity of teachers’ skills to apply that knowledge is scarce. Early findings of expertise research indicated that the knowledge and skills of experts can barely be transferred from one domain to another, suggesting that expertise is likely a domain‐specific construct (Glaser & Chi, 1988; Van Overschelde, Rawson, Dunlosky, & Hunt, 2005). However, more recent discussions point out that teachers need to combine domain‐specific expertise on learning and instruction with a general pedagogical expertise, for example regarding classroom management, in order to respond to the demands they face in the classroom (Fauth, Decristan, Rieser, Klieme, & Büttner, 2014). In order to approach the domain specificity of teachers’ skills, Blömeke and colleagues attempted to empirically separate teacher skills of one domain (mathematics) from general pedagogical skills that are not related to a domain but are operationalized in a similar way (for the context of classroom management; Blömeke et al., 2016). These results indicated that, in a sample of practicing mathematics teachers, skills for applying mathematical knowledge are more closely related to skills for applying pedagogical knowledge than to mathematics CK and PCK, giving first evidence that teachers’ skills for applying CK and PCK may not be specific to the domain of mathematics. As this and most of the currently available studies focused on teacher skills of only one domain and did not compare teachers’ abilities to apply knowledge in two domains (e.g., mathematics and economics), they are limiting the explanatory power of hypotheses on the generic or domain‐specific nature of AS.

Research framework and research questions

In order to address the research gaps, we needed to ensure that potential differences in (prospective) teachers’ AS can be attributed to their education and training within the domains. For this, two domains that are different yet still comparable – in that their foundations lie in related disciplines – are focused in our framework. We chose the combination of mathematics and economics as they are different domains, but share an overlap due to the fact that mathematical means are used in fundamental models of economics (Anderson, Benjamin, & Fuss, 1994).

Accordingly, we formulated the following research questions (RQ):

How do AS in mathematics and AS in economics relate to domain‐specific knowledge (CK, PCK) in pre‐service and in‐service teachers of those domains?

Is there a relationship between AS in mathematics and AS in economics that goes beyond relationships of underlying CK and PCK in teachers who were trained in both, mathematics and economics?

Method

Study design and sample

To address our first research question, we administered test instruments for AS as well as for domain‐specific teacher knowledge (CK, PCK) in mathematics and economics to separate samples of (prospective) teachers. Pre‐service teachers were recruited at 20 German universities and had to be in the second half of their teacher education programme for upper secondary level in order to participate. In‐service teachers for upper secondary level were recruited at 38 public schools throughout Germany. The overall sample comprised N = 239 participants for mathematics (48% female) and N = 321 for economics (55% female, 2 missing). Participation was on a voluntary basis. The mathematics sample comprised n = 101 pre‐service (mean age M = 25.9 years, SD = 5.1, 1 missing) and n = 138 in‐service teachers (mean age M = 34.4 years, SD = 8.5, 3 missing). The economics sample comprised n = 162 pre‐service (mean age M = 26.8 years, SD = 4.1) and n = 159 (mean age M = 37.0 years, SD = 8.9) in‐service teachers.

To investigate the second research question, we benefited from the fact that upper secondary teachers in Germany major in two domains. Therefore, we focussed on the n = 96 pre‐ and in‐service teachers (55 % female) from the sample above who teach both subjects, mathematics and economics, leading to n = 54 pre‐service teachers (mean age 27.6 years, SD = 5.7) and n = 42 in‐service teachers (mean age 36.7 years, SD = 7.8).

Test instruments

To measure AS, we used performance assessments based on the typical demands related to teaching mathematics (M‐AS: 9 items; Lindmeier, 2011) and economics (E‐AS: 7 items; Kuhn, Zlatkin‐Troitschanskaia, Brückner, & Saas, 2018). Video vignettes of about one to two minutes in length were used to authentically portray the complexity of typical teaching situations. The assessments were delivered on a computer, and responses were recorded. Participants had to respond to the vignette immediately (speed condition) and directly in natural language (audio recording). The assessments focus on typical teacher actions during mathematics and economics instruction, for instance, providing an explanation to a student's question or adaptive feedback that helps students with a mathematics or economics problem. The responses were scored (0, 1, 2 points) by two trained raters using a scoring scheme developed and tested prior to this study. For this scoring, inter‐rater agreement of two raters was κ = .60–.89 (M = 0.76) for E‐AS and κ = .77–.90 (M = 0.84) for M‐AS.

To assess domain‐specific teacher knowledge, we used

short scales of established paper–pencil tests for mathematics CK (M‐CK: 12 items, Dreher, Lindmeier, Heinze, & Niemand, 2018) and for mathematics PCK (M‐PCK: 13 items, Loch, Lindmeier, & Heinze, 2015),

short scales of validated paper–pencil tests for economics CK (E‐CK: 14 items, Zlatkin‐Troitschanskaia, Förster, Schmidt, Brückner, & Beck, 2015) and for economics PCK (E‐PCK: 11 items, Kuhn, Alonzo, & Zlatkin‐Troitschanskaia, 2016).

The open‐ended PCK responses were scored (0, 1, 2 points) by two raters: κ = .70–1.00 (M = 0.89) for mathematics and κ = .60–.89 (M = 0.78) for economics.

Sample items for all our video‐based and paper–pencil assessments in mathematics and economics are provided in Appendix S1.

In our sample, the scales showed internal‐consistency reliabilities between .60 and .64 (Table 1), which is marginally sufficient considering the scale lengths and the conceptual heterogeneity of the constructs that have previously been reported to cause low internal consistencies (Blömeke, Hoth, et al., 2015; Hill, Schilling, & Ball, 2004).

Table 1

Pearson correlations and descriptive statistics for the scales CK, PCK, and AS in mathematics (N = 239) and economics (N = 321)

Variable	M‐CK	M‐PCK	M‐AS	E‐CK	E‐PCK	E‐AS
M‐CK	–
M‐PCK	.38***	–
M‐AS	.33***	.29***	–
E‐CK	.34***	.12**	.27**	–
E‐PCK	.22*	.29***	.19	.31***	–
E‐AS	.21	.15	.21*	.31***	.25***	–
Mean	8.90	10.07	7.80	7.29	6.85	5.87
SD	3.98	3.66	3.40	2.67	3.13	2.76
Theoretical scale maximum	22	25	18	14	17	14
Empirical scale maximum	17	21	16	14	17	13
Cronbach's Alpha	.60	.61	.61	.60	.63	.62

*p < .05, **p < .01, ***p < .001.

All our test instruments were validated according to the standards of AERA, APA, and NCME (2014) (E‐AS: Kuhn et al., 2018; E‐CK: Zlatkin‐Troitschanskaia et al., 2015; E‐PCK: Kuhn et al., 2016; M‐AS, Lindmeier,2011 ; M‐CK and M‐PCK: Heinze, Dreher, Lindmeier, & Niemand, 2016).

Data analysis

For all scales, the item data were summarized to sum scores. Missing values were treated as 0 points as long as at least one item was answered on that scale. This led to 2–14 missing scores which were imputed using a full information maximum likelihood (FIML) estimation. To answer the first research question, correlation analyses as well as multivariate linear regression models were applied. The second research question was examined through a path model. All statistics were computed using the ‘lavaan’ package (Rosseel, 2012) for R (version 3.5.1).

Results

RQ 1: How do AS in mathematics and AS in economics relate to domain‐specific knowledge (CK, PCK) in pre‐service and in‐service teachers of those domains?

To address our first research question, we first correlated the scores of all six measures (Table 1). AS in each domain correlated significantly with CK (mathematics: r = .33, p <  .001; economics: r = .31, p < .001) and PCK (mathematics: r = .29, p <  .001; economics: r = .25, p < .001) but accounted for less than 15 per cent shared variance. However, as Pearson correlations do not control for shared variance with other variables, those relations might, at least in part, be due to mediation effects.

To gain a deeper insight into the relational structure between AS, CK, and PCK for each domain, we analysed multivariate linear regression models to examine the relationships controlling common variance between CK and PCK.1

Overall, the results revealed no substantial differences in the relational patterns between AS, CK, and PCK in mathematics and economics (Table 2). In both domains, correlation analyses and linear models indicated only moderate relationships (β ≤ .26 and β ≤ .28, respectively) between domains of professional knowledge and AS. In the combined samples of pre‐ and in‐service teachers, we found the relationship between AS and CK (mathematics: β = .26, p < .001; economics: β = .28, p < .001) tending to be stronger than between AS and PCK (mathematics: β = .19, p = .005; economics: β = .16, p = .003), which is also consistent across both domains.

Table 2

Multivariate linear regression between CK, PCK, and AS in mathematics and economics for pre‐service teachers and in‐service teachers

Domain	N	AS by CK		AS by PCK
		β	p‐value	β	p‐value
Mathematics	239	.26	<.001	.19	.005
Economics	321	.28	<.001	.16	.003

RQ 2: Is there a relationship between AS in mathematics and AS in economics that goes beyond relationships of underlying CK and PCK in teachers who were trained in both mathematics and economics?

To address our second research question, we examined relationships between AS, CK, and PCK in mathematics and economics for n = 96 teachers that were trained in both domains. Without any control for domain‐specific knowledge (CK, PCK), Table 1 shows a weak but significant correlation between M‐AS and E‐AS (r = .21, p = .047). In order to additionally control for the common variance between M‐AS and E‐AS that is due to relations of the underlying CK and PCK, we computed a path model including AS, CK, and PCK of both domains. Given that the subgroup of teachers teaching in both domains is too low to compute path models of six variables (Kline, 2016), we used all available data to estimate the model and enhance statistical power. To make sure that the use of all available data can be justified, we first tested to what extent teachers with both domains (n = 96) exhibit the same relational patterns regarding AS, CK, and PCK to those who teach only mathematics (n = 143) or only economics (n = 225). A pairwise group comparison with constrained parameters revealed that the relations between AS and CK as well as those between AS and PCK were invariant across those groups (fit values for constrained models for both mathematics and economics: CFI = 1.00, RMSEA = 0, SRMR < .02). The model based on data from participants teaching both subjects resulted in similar regressions parameters as the model based on data from teachers with either only mathematics or economics (AS and CK: |Δβ| ≤ .026; AS and PCK: |Δβ| ≤ .023). Thus, it can be assumed that the approach of using all information for model estimation does not bias the results.

`The resulting path model (Figure 1) shows the relations between all measured constructs of this study. Apart from relations between AS, CK, and PCK within the domains, which are comparable to those in the previous section (Table 2), the model shows expected relations between domain‐specific knowledge in mathematics and economics. First, we find the expected relations between CK and PCK in both domains (mathematics: β = .38, p < .001; economics: β = .32, p < .001), which is in line with the assumption that CK is a prerequisite of PCK. Second, M‐CK shows moderate significant relations to E‐CK (β = .26, p < .001) and E‐PCK (β = .26, p = .0098). This might indicate that knowledge about the content in mathematics is a crucial part of economics knowledge (both CK and PCK). As can be expected from a theoretical perspective, the relation between E‐CK and M‐PCK is not significant (β = .05, p = .62). Finally, with respect to our second research question, we find that the relationship between M‐AS and E‐AS is no longer significant (β = .08, p = .41) either. Thus, the weak but significant Pearson correlation between M‐AS and E‐AS (Table 1) can be explained when shared variance between knowledge constructs in mathematics and economics is considered.

Figure 1

Path model with standardized weights relating AS in mathematics and economics (M‐AS, E‐AS) to CK and PCK in mathematics and economics (M‐CK/E‐CK, M‐PCK/E‐PCK). Significant paths are bold and marked with asterisks (* p < .05, ** p < .01, *** p < .001). Model fit: CFI  = 1, RMSEA  = 0, SRMR  = 0 (saturated model).

Discussion

Our study draws on recent performance assessment approaches in order to validly measure the AS of teachers in mathematics and economics, and investigate their relationship with teacher knowledge as well as their domain specificity. The Germany‐wide recruited sample provides first empirical evidence of

the relationship between AS and domain‐specific knowledge (CK, PCK) in mathematics and in economics and

the relationship between AS in mathematics and AS in economics for teachers who were trained or experienced in both domains.

For (1), our results indicate that AS show only moderate relationships for CK and PCK of both domains. This suggests that, in mathematics and economics, teachers’ skills to apply domain‐specific CK and PCK in teaching situations do indeed differ from teacher knowledge. This is in line with previous studies focusing primary school teachers of mathematics (Knievel et al., 2015). However, previous research did not examine whether this relational pattern can be generalized across domains. Our findings indicate that the relationships between AS and CK as well as AS and PCK are comparable across the domains of mathematics and economics. Particularly, there is tentative evidence that AS in both domains are more strongly related to CK than to PCK. This finding might conflict with the role previous studies ascribed to PCK for instructional processes in primary and lower secondary level instruction (Baumert et al., 2010; Hill et al., 2008). Our study, in contrast, focuses on teachers for upper secondary level, where teaching content is more complex. Thus, CK could play a more prominent role for the interpretation of and reaction to classroom situations.

For (2), our results indicate that the correlation between AS in mathematics and AS in economics can be explained to a large extent by the underlying relations between CK and PCK of both domains. In particular, we did not find a significant relation between M‐AS and E‐AS when relationships in the underlying domain‐specific knowledge are controlled. Thus, our results indicate that the teachers’ skills to apply domain‐specific knowledge for instructional purposes differ for mathematics and economic (i.e., for two related domains). The findings suggest that AS for one domain are not transferable to another domain for which the teachers possess CK and PCK.

Based on our results, several conclusions and implications can be suggested. Our study indicates that domain‐specific teacher knowledge and AS are only moderately related in the domains of mathematics and economics. This implies that teacher training should focus not only on fostering teachers’ professional knowledge base (Hill et al., 2008) but also on providing learning opportunities for AS. This further supports current findings on assessing teachers’ skills close to real‐life teaching performance (Santagata & Sandholtz, 2018). Our findings regarding the domain specificity of AS suggest that those learning opportunities might be most effective if designed with close relation to the domain. For instance, with respect to the professional development of out‐of‐field teachers, the domain specificity of instructional skills implies that training programmes focusing solely on the provisions of domain‐specific knowledge may fall short of their aim if the acquisition of AS is neglected. Future research should explore which skills and abilities beyond knowledge contribute to AS and find effective ways of fostering AS. For example, Santagata and colleagues designed a professional development programme for fostering mathematics teachers’ instructional practices using video‐taped lessons (Santagata, Kersting, Givvin, & Stigler, 2010). Although the programme's impact on teachers’ knowledge and practices was not detectable, an effect on student learning was found.

The domain specificity of AS implies that findings about instructional skills in one domain may not necessarily be transferable to other domains. Further studies should aim at replicating these results in other teaching domains as this study focused only on (prospective) teachers of mathematics and economics. Our findings do not conflict with current suggestions of AS being considered a form of teacher knowledge (enacted or usable knowledge; Carlson & Daehler, 2019; Kersting, 2008), although such hypotheses regarding the nature of instructional skills cannot be derived from our data. However, our results contradict the findings of Blömeke et al. (2016) as they found a close relation of teachers’ instructional skills in mathematics and pedagogy. This may suggest that teachers’ skills for teaching a domain are related more closely to knowledge and skills in pedagogy than skills for teaching a different school subject.

As every empirical study, our study also faces limitations. First, although AS were assessed using realistic video clips of classroom situations, the current technical resources do no permit representation of all demands related to real classroom situations. Second, to consider the demands of instruction, the assessment methods of AS are video‐based and under time constraints and, hence, are different to the assessment methods of subject‐specific knowledge (paper–pencil tests). This could have affected the found relations between domain‐specific knowledge and AS. Third, although the sample was collected throughout Germany and is well‐distributed, it cannot be considered to be representative. Fourth, there are generally only few teachers in Germany who have been trained in and teach both mathematics and economics. Thus, our analysis of relationships between mathematics‐related an economics‐related variables in particular relies on a small sample size. This might be an influencing factor as well, and corroboration of our results with larger sample sizes is needed.

Despite these limitations, the present study provided insightful initial evidence of how teachers’ AS differ in two major teaching domains, and furthers the discussion of the nature of instructional skills like AS. This is an important starting point to improve current teacher training programmes and to foster teachers’ instructional skills more effectively.

Appendix S1 Example Items for AS, CK and PCK in Mathematics and Economics.

Click here for additional data file.