A meta-analysis of single-subject reading intervention studies for struggling readers: using Improvement Rate Difference (IRD).

The purpose of this study is to examine and identify effective variables related to single-subject reading interventions for the struggling readers. This study conducted meta-analysis of 84 single-subject research articles of which 54 were published in Korean and 30 were published in English. Utilizing the Improvement Rate Difference (IRD), a total of 586 effect sizes were produced with the mean effect size of 0.77. Intervention variables were categorized into two types - participant-related and intervention-related variables. Collected studies were analyzed for each effect size calculation. Variables having large effects within grade levels, disability types, implementer and session duration are discerned along with further discussion and educational implications.

Introduction

Reading interventions on struggling readers are regarded to be linked to improving their basic skills. As students who do not successfully read in early age have a heightened chance of struggling with general reading skills throughout their lives, effective reading intervention plans in their early educational paths are critical (Hurry and Sylva, 2007; Wanzek et al., 2018; Zentall and Lee, 2012). On-time screening and early interventional approach for such at-risk students may reduce potentially more serious negative consequences and deficits that may follow, and adequate interventions and proper care prior to actual time of learning disabilities-related diagnosis may remedy reading problems and lead to a positive implication (Bender and Larkin, 2009; Lee and Son, 2010). With an emphasis on early reading intervention, especially for struggling readers, there is a high plea for steadier and more systematic reading support system (Kim et al., 2009, Kim et al., 2009; Vaughn et al., 2014).

According to earlier studies, reading abilities develop as sequential stages by acquiring mastery in reading constructs (Chall, 1983; Frith, 1985; Spear-Swerling and Sternberg, 1994). Chall (1983), in particular, argued that reading mastery follows five stages. Firstly, stage 0–1 involves pre-reading and text decoding phase in which children use visual, auditory and information processing mechanisms to acquire phonological awareness and start decoding words for recognizing printed materials. Once these stages are mastered, children move on to stage 2 where fluency of literal reading emerges, which increases their reading accuracy. Then, stage 3 involves reading in relation to finding meanings of contextual knowledge through acquiring new vocabulary and more advanced comprehension. Thereafter, children emerge into stage 4 during which children read for extending perspectives and begin to compare and contrast within texts. Finally, in stage 5, with added skills acquired thus far, reading occurs in a synthesized process for more complex text evaluation along with critical thinking skills. For this complete mastery of reading, essential constructs of reading abilities – phonological awareness, word recognition, fluency, vocabulary acquisition, and comprehension – cannot be underestimated in any parts of the reading process.

Struggling readers are not only found in the population of specific learning disabilities but also found in children with other disabilities. Many scholars firmly argue that people with disabilities also have the human right to be active citizens in open and accepting society (Flynn, 2013; Liasidou, 2016; Stein, 2007). Many sociologists, including Luhmann (1995), have advocated that reading and communicating about written context certainly is part of a “social system.” Unfortunately, children with disabilities often lack literacy competence in school and life beyond those school years, and, therefore providing adequate reading intervention and continuous support is vital to embrace them into the social system (Lundberg and Reichenberg, 2013). Historically, it is not uncommon to find that students with disabilities receiving more individual instruction when compared with students without any disabilities, and it is still the trend in recent years that small group or more individual intervention for differentiated instruction in classrooms again has increased (Donegan et al., 2020; Swanson et al., 2012; Vaughn and Wanzek, 2014). In an endeavor to ensure the quality of reading instruction, many research studies have supported explicit (direct) instructional strategies among other instructional methods.

As a type of evidence-based research method often used, including special education field (Bouck et al., 2018; Horner et al., 2005; Maggin et al., 2011), research studies done using single-subject research designs often attempt to demonstrate the effectiveness of interventions implemented for particular participants, in depth. As part of the experimental design of study, single-subject designs are often used to demonstrate the effectiveness of an independent variable through quantifying any changes in behavior within the same subject over a certain period of time. Many of such studies utilize multiple baselines for accurate pinpointing of the starting introduction of the subject's initial behavior. With efforts to control confounding contexts or time variables, each baseline is believed to respond to the actual intervention being introduced (Kazdin, 1982). However, factors involved in the design may contribute to different outcomes of reading intervention programs in single-subject designs. Some findings suggest that embedding behavioral strategies within reading intervention phases, for instance, could result in more advantageous outcome (McKenna et al., 2017). Yet, the key criteria of the What Works Clearing House (WWC) assert that a moderate to strong case of causal relationship can be drawn if a single-subject design study has at least three demonstrations with one non-effect within or with three intervention effect demonstrations, and it has been reported that at least 80% show positive effects across all single-subject design experiments (What Works Clearingouse, 2017). Furthermore, single-subject research designs possess significant methodological advantages over large-sample research designs in that it may help to yield faster clinical research results for demonstrating distinct causal relations between intervention and behavior change (Byiers et al., 2012; Nock et al., 2007).

When examining effectiveness of research studies that have already been published, meta-analysis is often used in order to provide and filter highly contributing variables among other variables employed in targeted studies; this type of analysis, by nature, is used for measuring effect sizes in order to yield objective and reliable outcomes (Kavale, 2001). Often, studies collected for meta-analysis are experimental designs utilizing experimental and control group setting (e.g. Hall and Burns, 2018; Kim et al., 2018), rather than single-subject designs. In order to scrutinize single-subject studies done on reading intervention programs, the present study attempts to collect and synthesize only single-subject research studies for a thorough examination of their reading intervention effectiveness. For this purpose, the present study attempts to discern variables highly contributable to successful reading intervention outcomes. Therefore, we pose the following research questions for this meta-analysis:

What is the overall mean effect size of reading interventions for students with reading difficulties in single case research designs?

To what extent do participant-related and intervention-related variables affect the reading improvement for struggling readers?

Method

Data collection procedure and inclusion criteria

Computerized literature searches were done using ERIC, PsycInfo, ProQuest, Research Information Sharing Service (RISS), Korean studies Information Service System (KISS), and Nurimedia DBpia online databases to identify articles from 1980 to 2015. The following keywords were used in various combinations to locate related articles: single-subject, reading intervention, struggling readers, learning disabilities, low achievers, reading difficulties. Then, the abstracts of yielded articles were scanned to determine studies that meet inclusion criteria described below. Reference lists were also inspected to detect additional articles. The following criteria were further applied to finalize the list of articles that were included for this meta-analysis: (1) single-subject design reading intervention studies addressing phonological awareness, word recognition, vocabulary, fluency, and comprehension for struggling readers with disabilities or at-risk, (2) studies providing sufficient information enabling to calculate the effect sizes, (3) only the peer-reviewed articles to heighten the quality of the study, (4) articles written either in English or Korean for more inclusive analysis. However, studies that focus on reading skills of English Language Learners (ELLs) as their second language were excluded. A final number of 84 research studies were selected as they met all of the eligible criteria. Of those, 30 studies were published in English, while 54 studies were published in Korean.

Coding procedure

This research based the coding categories on study features used by the What Works Clearinghouse (WWC) in study reviews and used in previous meta-analyses (e.g., An and Kim, 2018; Perry et al., 2012). The extensive coding categories included the participant (e.g., grade level, disability type) and intervention (e.g., type of intervention, implementer, and session duration) related variables, and other characteristics of studies such as study design, sample size, and reported findings.

The gold standard method (Gwet, 2001) was adopted to establish interrater reliability. To achieve coding accuracy and objectivity for the study, the first author, a researcher with previous experience in meta-analyses and syntheses, created a coding sheet and discussed each coding step through five training sessions for two hours each. The coders, four other authors, spent the training sessions practicing the coding process for sample intervention studies with different design types. Following the training session, the coders were given a task to code two additional intervention studies independently to establish interrater reliability. The first author and the four coders then discussed any discrepancy found until the interrater reliability marked score of 1. The interrater reliability was determined by the number of items in agreement divided by the total number of items. The process continued every other week to check on the progress of coding and resolved any coding issues, and final decisions were derived from discussion consensus.

Data analysis

The effect size index used in this meta-analysis was Improvement Rate Difference (IRD; Parker et al., 2009). IRD was first introduced to compensate empirical limitations of the percentage of non-overlapping data (PND), which has been the most widely used index for the single-subject research studies (Scruggs and Mastropieri, 2013). A study by Maggin et al. (2011) indicates that 47% of research synthesizing method of single-subject studies utilized the PND index from 1985 to 2009. Nevertheless, the PND has been criticized for some weaknesses including high sensitivity to outliers. Therefore, this study used IRD in which the effect sizes are calculated based on the difference of improvement rates in each baseline and treatment phase. Its name has also been known as “risk difference” or “risk reduction” in medical field, and is known for its reliability and easiness to calculate (Parker et al., 2009; Waddell et al., 2011) along with other advantages, such as better discrimination for effect size, production of forest plot, and CI in effect size calculation.

All data points from both baseline and intervention phase are examined for improvement. Improved or unimproved are determined by overlap data points between phases. In the baseline phase, data points that are equal to or exceed any intervention phase points are defined as improved. In the intervention phase, data points that exceed those of baseline phase are defined improved. Subsequently, the proportion is defined as the number of improved data points divided by the total data points in that phase. The difference between independent proportions from each phase yield IRD. An example of IRD calculation is illustrated in Figure 1. It is important to note that the fewest data points should be removed to eliminate overlap between phases (Parker et al., 2009). Statistical significance is determined by confidence intervals (CIs). Measurement precision is interpreted by the width of CIs; narrow width is interpreted as high level of precision. Also, when comparing with more than two groups, non-overlapping CIs show significant independence among various groups. For this study, NCSS 7 was used for analysis. Forest plots were also used for the best visual distribution display, suggested by Lewis and Clarke (2001).

Figure 1

An example of IRD calculation.

Results

Overall effects of reading interventions for students with reading difficulties

A total of 84 single-subject studies (30 in English, 54 in Korean) on reading interventions were selected with 589 effect sizes. Overall effect size of reading intervention studies was 0.77, as shown in Figure 2, representing very large effects, according to the standard suggested by Parker et al. (2009). Specifically, effect size of .50 and below are interpreted as very small or questionable effects. Ranging from .50 to .70 is considered moderate effects, .70 to .75 is large effects, and .75 and higher is interpreted as very large effects.

Figure 2

Result of overall effect sizes and confidence interval.

Results of variable analysis

Participant-related

The participant-related variables are to examine how the effects of the intervention vary according to the attributes of participants. Wexler et al. (2008) argued that the effects may be different on the same intervention given depending on characteristics of students. The results of participant-related variables are described in the section below.

Grade level

In order to examine the effect size according to grades, all of the participants included in this study were classified into five subgroups: (a) preschooler, (b) grade 1–3, (c) grade 4–6, (d) grade 7–9, (e) grade 10–12, and (f) mixed. Mixed is defined as the subgroups that included more than one grade level in participants. The analysis revealed that grade 1–3 had the largest number of effect sizes (n = 201, 34.13%) which was followed by grade 4–6 (n = 121, 20.54%), mixed (n = 106, 18.00%), preschoolers (n = 103, 17.49%), grade 7–9 (n = 30, 5.09%), and grade 10–12 (n = 28, 4.75%). This showed that more than half of single-subject reading intervention studies have been implemented at the elementary level. A total of 589 effect sizes were analyzed and presented in Figure 3 based on the grade level of participants.

Figure 3

Result of grade level.

The large effects were observed in following orders: grade 10–12 (ES = 0.86; CI, 0.76–0.90), mixed grades (ES = 0.81; CI, 0.78–0.84), preschooler (ES = 0.79; CI, 0.76–0.83), grade 1–3 (ES = 0.76; CI, 0.73–0.79), grade 4–6 (ES = 0.73; CI, 0.69–0.76), and grade 7–9 (ES = 0.71; CI, 0.65–0.78). Weighted mean effects of all group was higher than .70, which are large effects with no significant difference.

Disability type

Through the descriptive analysis of 84 studies, majority of struggling readers were identified with low achievement and/or learning disabilities. Other types of disabilities were identified as well; however, due to various types of disabilities, they were coded as one group. Therefore, participants' disability types were classified as follows: (a) low achievement (n = 140, 23.77%), (b) learning disabilities (159, 26.99%), (c) mixed (n = 12, 2.04%), (d) and others (n = 278, 47.20%). Studies including participants with learning disabilities along with other types of disorders were categorized as mixed. Disabilities not fitting into any of (a), (b), (c) criteria (e.g. ADHD, intellectual disabilities) were labeled as others.

According to the result shown in Figure 4 the others group seemed to have benefitted the most from the given reading interventions (ES = 0.82; CI, 0.80–0.84), followed by low achievement (ES = 0.74; CI, 0.70–0.77) and learning disabilities group (ES = 0.72; CI, 0.69–0.76). Also, a narrow width of CIs of “others” group showed significantly larger effects compared to other groups. Low achievement and learning disabilities groups also revealed to have large effects, significantly higher than mixed group. The weighted mean effect size of mixed group was 0.59(CI, 0.50–0.68) representing moderate effects.

Figure 4

Result of disability type.

Intervention-related

There is no single perfect way of implementing interventions that produce promising effects. Yet, implementation related variables are reliably important factors determining the effects of intervention. Thus, this study attempted to examine the additional influential factors such as intervention types, implementers, and session durations. Finding results can be seen in Figures 5, 6, and 7.

Figure 5

Result of intervention type.

Figure 6

Result of implementer.

Figure 7

Result of session duration.

Intervention type

Packaged intervention programs or individually developed interventions by researchers were observed during the coding process. However, a large number of studies did not provide specific information needed for categorization. Although various interventions are implemented to enhance students' reading skills, similar patterns of educational goals were found among these interventions which aligns with the reading components identified by the National Reading Panel (2000). Therefore, the intervention types were then categorized into (a) phonological awareness, (b) word recognition, (c) vocabulary, (d) reading fluency, (e) reading comprehension. In case of dealing with more than one reading construct, intervention types were coded for all. Based on descriptive analysis, reading interventions aiming at reading comprehension was the most frequent intervention type (n = 197, 33.45%), followed by reading fluency (n = 173, 29.37%), vocabulary (n = 93, 15.79%), phonological awareness (n = 80, 13.58%), word recognition (n = 46, 7.81%) intervention.

Intervention types were set as moderators for effect size analysis. The mean weighted effect size for word recognition revealed to be the largest (ES = 0.83; CI, 0.78–0.82), followed by reading comprehension (ES = 0.80; CI, 0.78–0.82), vocabulary acquisition (ES = 0.78; CI, 0.75–0.82), phonological awareness (ES = 0.77, CI, 0.73–0.81), and reading fluency (ES = 0.67; CI, 0.63–0.71). Mean effect sizes of all variables except reading fluency were very high and no significant differences were found among these group. However, reading fluency which had moderate effect was significantly low, compared to all other groups.

Implementer

To examine the different effects carried by various intervention implementers, groups were classified as (a) researcher, (b) teacher, (c) graduate student, and (d) others. Half of the interventions were delivered by the researchers who designed the studies (n = 299, 50.76%). Fair proportion of teachers (n = 80, 13.58%) and other (n = 16, 2.72%) groups also implemented reading interventions for struggling readers while only 2.72% graduate students delivered the intervention (n = 16). In “others” group, parents, peers, and therapists were included. 483 effect sizes were analyzed and 106 were excluded due to missing information.

The result indicated that interventions implemented by the teacher had the largest effects (ES = 0.83, CI, 0.80–0.86). Interventions delivered by the researchers also resulted in large effects, but no significant differences were observed between these two groups (ES = 0.79, CI, 0.77–0.81). Medium size effects were found in “others” and “graduate students” groups, having significantly lower effects (ES = 0.63, CI, 0.58–0.69; ES = 0.58, CI, 0.42–0.73) than those of teachers and researchers.

Session duration

The duration of interventions was divided into (a) 1–10 sessions, (b) 11–20 sessions, (c) 21–30 sessions, and (d) 31 and above. 589 effect sizes were examined. Based on descriptive analysis, the majority of reading intervention studies were implemented between 1 to 20 sessions (1–10 sessions, n = 259, 43.97%; 11–20 sessions, n = 208, 35.31%). 21–30 sessions (n = 75, 12.73%) and 31 and above (n = 47, 7.98%) showed relatively small proportion of effect sizes.

The mean weighted effect size for 21–30 sessions showed the largest effect of 0.82 (CI, 0.78–0.86), followed by 11–20 sessions (ES = 0.80, CI, 0.77–0.82), 31 and above (ES = 0.82, CI, 0.78–0.86), 1–10 sessions (ES = 0.68, CI, 0.65–0.71). Among these groups, 1–10 sessions had significantly small effects.

Discussion and implications

This meta-analysis study overall provides valuable information about effective intervention conditions for those who struggle with reading. It examined and evaluated 84 studies of single-subject reading intervention studies in the last 35 years for struggling readers through meta-analysis using the IRD index. By examining a total of 589 computed effect sizes, the study demonstrated clear effects of single-subject reading interventions on struggling readers across related variables. A summary on findings and implications can be drawn as follows.

First, the overall effect size of all reading interventions was 0.77, indicating a large effect size. This may support the idea of small-group intervention effectiveness for struggling readers for its increased instructional intensity, often naturally produced by small intervention group size (Foorman and Torgesen, 2001). It is also related to the point of Response to Intervention (RTI) model in special education. RTI is a model which has emerged accentuating the provision of intervention strategies in line with student's actual performance level from Tier 1 to Tier 3. By applying these tiers throughout the intervention stages, small-group or individualized instructions are provided to students based on their actual response levels (Vaughn and Fuchs, 2003). Thus, this result may support that RTI as the appropriate model to screen the students at-risk beforehand and to provide appropriately focused small-group instructions for struggling readers to produce overall positive effects.

Second, the analysis of different grade levels indicated an effect size of 0.70 or larger, and the inter-group differences were not found to be significant. Comparing the effect sizes among those groups showed that the effect size of the grades 7–9 was the lowest with 0.71 while the effect size of grades 10–12 was the largest with 0.86. However, it still can be concluded that the small-group instruction and individualized instruction types are both effective for all age groups, being all above the threshold of 0.70 for large effects. This may be linked to the notion that there should be more of instructional emphasis on the implementation of small-group instructions for struggling readers in upper grades. Many of included studies primarily conducted studies for struggling readers in the elementary school level while significantly lower attention has been drawn to secondary school graders whose academic and instructional demands became more complex. Even evidence-based research on RTI has not been implemented until recent years for grades 4 through 12 (Vaughn and Fletcher, 2012). Consistent reading interventions from early years are essential for its function of preventing incidence of significant failure in the growth of reading abilities (Zentall and Lee, 2012; Vaughn et al., 2009). However, repeated academic failures may lead to emotional and psychological problems; therefore, it is necessary to continuously provide systematic reading interventions to struggling students in all grades based on adequate policy and financial supports.

Third, the magnitude of effect sizes varied depending on the types of disability. Other types of disabilities (e.g., ADHD, intellectual disability) than learning disabilities showed significantly larger effects on reading improvement than low achievement, learning disabilities, or mixed groups. Students with disabilities are often emotionally and academically neglected in inclusive education settings due to their non-responsiveness to education (Fuchs et al., 1993; Imaniah and Fitria, 2018; Vaughn and Schumm, 1995), yet Afacan et al. (2017) suggested that for the children with intellectual disability, effective reading instruction should encompass not a single but multiple reading skills just like in general education. The findings from this study may indicate that reading interventions can be successfully applied to the populations of various disability types; providing instructionally well-designed reading interventions to students with disabilities in inclusive education environment may be more useful than once considered. Moreover, one interesting finding emerged from this analysis is that the weighted mean effect size was significantly smaller when students with learning difficulties and other students with disabilities received interventions together in a mixed group, suggesting that perhaps reading intervention should carefully be implemented to students with similar academic needs. All but the mixed group had relatively large effect within this variable analysis; thus, the learner's learning conditions should be accurately assessed and measured to form a homogeneous intervention group to better accommodate emotional and academic inclusiveness.

Fourth, the intervention type denotes that the word recognition showed the largest effect size (ES = 0.83), followed by reading comprehension (ES = 0.80), vocabulary acquisition (ES = 0.78), phonemic awareness (ES = 0.77), and reading fluency (ES = 0.67). The ability to obtain meanings from texts is highly dependent on the abilities of word recognition and fluency, whereas fluency is dependent on the development of word recognition (Chard et al., 2002; Snow et al., 1998). In this notion, the largest effect size shown in word recognition can be interpreted meaningfully, which supports the similar results found in Lee and Son (2010) and Shin et al. (2016) studies. Though a relatively large effect sizes were also observed in all areas, some professionals and practitioners steadily support the idea that effective reading instructions almost always include systematic and explicit instruction, quality environment (National Reading Panel, 2000).

Fifth, reading intervention implementer analysis indicated that when all implementer types conducted the intervention to the students, there was a significantly larger effect size for the teacher group (ES = 0.83), followed by the researcher group (ES = 0.79). This result may come as no surprise as the teachers, who continuously interact with the students and comprehensively understand them, contributed to most effectiveness among all intervention provider groups. Foorman and Torgesen (2001) also stated that when explicit instruction was given by the classroom teacher, reading failure rate was significantly reduced, according to the findings from evidence-based research. Teachers intervene at both the universal classroom level and at the individual level; they are the leaders for instruction and activities and also often provide extra guidance or coaching, overall indicating the large role effect of the teachers as a group. On the other hand, interventions conducted by university graduate students produced a relatively smaller effect size (ES = 0.58). Some studies have found that expertized general education teachers and reading specialists for Tier 2 level of students in schools had quite powerful impact on students' reading improvement (Fuchs and Deschler, 2007; Motiejunaite et al., 2014; Vogt and Shearer, 2016). Thus, this finding suggests that adequate training of teachers and placing them for the students with reading difficulties may be the critical conditional criterion that can increase the likelihood of students' success in reading.

Lastly, according to the analysis of the single-subject reading intervention session duration, intervention sessions of 21–30 and 31 times or above both had the equaling effect (ES = 0.82). Intervention sessions given 1–10 times in total significantly had less effect than other session groups (ES = 0.68). This result indicates that intervention instructions should strive for a long-term approach and be systematic rather than basing on a short-term approach. Intervention sessions more than 10 times were also found to be significantly effective than less number of sessions on struggling readers (Gresham et al., 2001; Wanzek and Vaughn, 2008).

This study attempted to examine the overall effectiveness of reading intervention programs and discern more effective variables contributing to reading improvement in students struggling with reading through meta-analysis on single-subject research studies. The study results indicated overall positive effects in reading outcomes of struggling readers in preschoolers to grade 12. Previous meta-analysis on reading interventions for students with reading difficulties has mainly conducted on studies with treatment-comparison or experimental or quasi-experimental design (e.g., Graham and Hebert, 2011; Wanzek et al., 2013). Students with reading difficulties have been identified by heterogenic traits, and their responses to interventions can vary as well. Considering that students with reading difficulties are supported in the form of individual or small group in RTI tier 2 and 3, the variables with moderating effect to reading outcomes revealed through this study can have great implications for the field. Therefore, it will be necessary to develop and implement reading interventions that reflect the effective variables identified in this study to effectively support struggling readers in the field of education.

As some of the limitations of the study, it should be underscored that although IRD index used here has its advantages as discussed earlier, it still has weaknesses such as a tendency for its high effect size discrimination (Kim et al., 2016; Parker et al., 2009); other indices such as PND, PEM-T, PAND, and NAP may have yielded a slightly different result. Also, future studies may further consider other influencing factors, such as examining identical types of single-subject design methods, inspecting different environmental factors of the intervention settings, or distinguishing similar or dissimilar reading intervention teaching methods.

Declarations

Author contribution statement

Dongil Kim, Yeji An: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data; Wrote the paper.

HyeYun Gladys Shin, Jaeho Lee, Soyoung Park: Performed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data; Wrote the paper.

Funding statement

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Competing interest statement

The authors declare no conflict of interest.

Additional information

No additional information is available for this paper.