The Effect of Illustration on Improving Text Comprehension in Dyslexic Adults.

This study analyses the effect of pictures in reading materials on the viewing patterns of dyslexic adults. By analysing viewing patterns using eye-tracking, we captured differences in eye movements between young adults with dyslexia and controls based on the influence of reading skill as a continuous variable of the total sample. Both types of participants were assigned randomly to view either text-only or a text + picture stimuli. The results show that the controls made an early global overview of the material and (when a picture was present) rapid transitions between text and picture. Having text illustrated with a picture decreased scores on questions about the learning material among participants with dyslexia. Controls spent 1.7% and dyslexic participants 1% of their time on the picture. Controls had 24% fewer total fixations; however, 29% more of the control group's fixations than the dyslexic group's fixations were on the picture. We also looked for effects of different types of pictures. Dyslexic subjects exhibited a comparable viewing pattern to controls when scenes were complex, but fewer fixations when scenes were neutral/simple. Individual scan paths are presented as examples of atypical viewing patterns for individuals with dyslexia as compared with controls.

RCT Entities:   Population Interventions Outcomes

Background

The primary symptoms of dyslexia are associated with severe difficulty acquiring basic reading skills (Everatt & Reid, 2009; Vellutino et al., 2004). Although the cause of dyslexia is still unknown, it appears to have a universal neurocognitive basis that transcends culture and language, and influences learning and teaching methods (Paulesu et al., 2001). The conceptualization of dyslexia has changed over the years, during which time it has been attributed variously to impairments in visual, linguistic, and sensory functions, as well as deficiencies in general learning abilities (Vellutino et al., 2004). More recent research indicates that dyslexia is a syndrome (Everatt & Reid, 2009; Paulesu et al., 2001; Snowling, 2000) with weak phonological coding as one important aspect (Ramus et al., 2003; Vellutino et al., 2004). This study is based on a multifactorial perspective of what causes dyslexia. A phonological weakness might theoretically be assumed to cause difficulties at an attentional level, so‐called ‘sluggish attentional shifting’ (SAS) in individuals with dyslexia (Hari & Renvall, 2001; Lallier et al., 2009). A simultaneous processing order (Lassus‐Sangosse et al., 2008) or a visual attention span deficit (Bosse et al., 2008; Prado et al., 2007) might also contribute to developmental dyslexia, the latter possibly reducing the number of visual elements processed in parallel. The significance of eye movements in dyslexic individuals has been widely debated: some have suggested that dyslexia might be caused by abnormal eye movements (Biscaldi et al., 1998); another hypothesis holds that the reverse may be true (Olulade et al., 2013). However, the cognitive processes associated with dyslexia are now widely accepted as not being a consequence of atypical patterns of eye movement during reading (Rayner, 1983). Liversedge and Findlay (2000) argue that eye movements are an excellent measure of the cognitive processes underlying reading, and describe how eye movements reflect cognitive processes:

We argue that deciding where and when to move the point of fixation are key aspects of eye‐movement control and that understanding the relationship between the two is necessary to understand fully the cognitive processes reflected by eye movement. (p. 12)

Although reading certainly is predicated on a number of lower‐level cognitive processes, dyslexia has also been linked to higher‐level cognitive processes, such as poor executive functioning (Baker & Ireland, 2007; Reiter et al., 2005). Executive functions involve the management of cognitive processes, including working memory, reasoning, task flexibility, problem solving, planning, and execution. Lallier et al. (2009) found that differences in visual SAS between individuals with dyslexia and controls were not present in children, and assumed that this was because of delayed maturation of visual SAS relative to auditory SAS, which is an argument for studying young adults. The validity of Lallier et al.'s (2009) finding was supported by the fact that results were consistent across task designs. In contrast, Liversedge and Findlay (2000) found that ‘readers make more fixations and fixate for longer when they experience processing difficulty’ (p. 10), which suggests that differences in task designs should have an effect on fixation, if the changes in task design also changed processing difficulty. According to Kirkby et al. (2008), it is unclear whether dyslexic readers and normal readers show the same eye movement patterns. Although some studies have found this to be the case, others have reported differences such as longer fixation duration, larger numbers of fixations, shorter saccade lengths, and more regressive movements. A thorough analysis of eye movement patterns contributes to our understanding of the differences between dyslexic individuals' and controls' cognitive strategies. Kirkby et al. (2008) note that: ‘In non‐reading tasks, evidence for differences in oculomotor control between dyslexic and normal readers is much less clear’ (p. 755).

Eye Movements when Viewing Pictures

Eye movements may be classified as either saccades or fixations. Saccades are rapid movements of the eye between points, and fixations are pauses during which the eye focuses. Fixations are thought to be related to cognitive processing of information and occur frequently during reading of text (Vidal et al., 2012; Foster et al., 2013; Holsanova, 2014; Mason et al., 2013). Eye‐tracking has long been used to investigate eye movement in reading and picture viewing (see Rayner, 1978; Rayner, 2009a).

Saccades and fixations also occur during observation of scenes, and the duration of the fixations and the length (or amplitude) of the saccades tend to be longer during scene observation than during reading (Bradley et al., 2011; Holsanova, 2014). However, because pictures tend to be more visually complex than text, they also tend to yield more complex patterns of eye movements. Eye movements are influenced by a number of features of pictures, such as edges and contrast, and fixations during viewing of pictures seem to be related to the extraction of such feature information, just as is done during reading (Bradley et al., 2011). For instance, Bradley et al. found fewer fixations on locations within scenes that had constant or uniform information (i.e., relatively fewer features). They also found interesting differences in the patterns of eye movements, or scan patterns, when participants viewed different types of compositions. They manipulated two compositional variables: neutral/emotional (content of the pictures was emotionally neutral or emotionally evocative) and figure‐ground/complex (pictures depicted a single object against a uniform background or depicted multiple objects on a varied background). They found that complex scenes resulted in more fixations and broader scanning than did simple figure/ground compositions, and that emotional pictures prompted more fixations and longer scan paths than did neutral pictures.

Two different mechanisms have been proposed for processing a visual scene: eye movements may reflect bottom‐up processing, which takes low‐level image features and builds them into a cohesive whole; versus top‐down processing, which is driven by high‐level cognitive factors such as previous knowledge, stored meaning and expectations, and integrates the details into this framework. In normal perception, both processes can occur concurrently: for example, faces contain both low‐level features of visual salience (such as eye colour or the presence of a scar or a mole) as well as high‐level features of potential semantic importance (as can occur, for example, when one mistakes a stranger for a friend; the mistake is only realized after the details of the stranger's face are taken into account). Boeriis and Holsanova (2012) have used eye‐tracking as a cue to learners' top‐down versus bottom‐up processes. Research on readers with low sensitivity in frequency doubling technology tests (a measure of spatial localization ability) shows greater impairment for reading whole texts than for reading single words (Pammer et al., 2004); this further supports the assumption of a more frequent use of bottom‐up processing.

Multimedia Studies

If reading and image viewing are considered cognitive problems to be solved through efficient use of both bottom‐up and top‐down strategies, the question remains whether images related to a text will help or hinder these processes, and whether the effect differs for readers with and those without dyslexia. Given that dyslexic readers display abnormal patterns of eye movement during reading, would the addition of pictures constitute a distraction from feature extraction or an aid to the more global, top‐down overview that normal readers are able to achieve naturally? Research has shown that saccades are shorter during reading than during scene perception, that words longer than eight characters or more are nearly always fixated, that most saccades for English readers proceed from left to right, and that the visual span is larger to the right of a word (Kirkby et al., 2008). The latter two findings should both affect the design of multi‐modal material.

Communication media are typically multi‐modal, including a combination of text, images, and sounds. Print media are often multi‐modal as well, with a potentially dizzying array of headlines, fact boxes, photographs, diagrams, graphs, and maps (Holsanova, 2014). Research has begun to investigate various types and combinations of materials to determine how these might affect learning processes (see reviews by Eitel et al., 2013; Mason et al., 2013; Schüler et al., 2012).

Many studies investigating text + picture integration have used expository texts and ‘pictures’ that included graphs or schematics (e.g., Hegarty & Just, 1993; Holsanova et al., 2009; Mayer, 2005; Schnotz et al., 1993; Schnotz, 2011). Various types of pictures can also play illustrative, explanatory, or augmenting roles. Such variety in picture types and roles is seldom considered in multimedia research (Scheiter et al., 2008). Levie and Lentz (1982) concluded from the results of 55 experiments that illustrations were somewhat more helpful to poor readers than to strong readers. A more recent literature review reported that the positive effects of pictures presented with text were found mainly among individuals with low literacy (Houts et al., 2006). However, some studies have shown that pictures distract the attention of low‐ability learners from the text, thereby hampering their reading comprehension (Harber, 1983; Rose, 1986). The multidimensionality of pictures, together with the various cognitive skills and aptitudes of learners, makes this a particularly difficult area in which to find converging evidence; therefore, studies that exercise some control over both learner and stimulus characteristics have the most promise for developing effective teaching and learning solutions for those with particular challenges.

There have been few studies on the effects of multimedia research on learning in individuals with severe reading disabilities (e.g., dyslexia). Among these studies, the results diverge; Lefton, Nagle, Johnson and Fischer (Lefton et al., 1979) claim that eye movement patterns for children with reading impairments are non‐systematic, while Beacham and Alty (2006) reported that text‐only material was beneficial to dyslexic readers; Paivio (1971), however, found that text + picture materials were more helpful. To enhance the effect of text–picture integration, much research has been directed toward designing materials that promote simultaneous, integrative processing, with the goal of enabling construction of richer mental models (Hegarty & Just, 1993; Mayer, 2005, 2009). Mason et al. (2013) used integrated text + picture materials and operationalized integrative processing as shifts of eye fixation from text to picture or vice versa. In a group of normal fourth‐grade readers, they found that ‘greater integrative processing of the illustrated text was associated with higher learning performance’ (p. 95). However, this facilitative effect of shifting attention within the same sensory modality (visual) was not obtained when inputs occurred simultaneously in two modalities (visual and audial).

Most multimedia designs for learning materials draw on Baddeley's model of working memory as two storage systems: the phonological loop and the visuospatial sketchpad (Baddeley, 1992, 2003). These two storage systems, or channels, are highly involved in reading and also have been found in readers with ADHD (Jacobson et al., 2011). Baddeley's model implies that information processed simultaneously in both channels allows individuals to construct more efficient mental models. For example, if one is viewing a slide presentation and hearing a speaker discuss the slides, this information will be learned more easily if the speaker's words and the visual content of the slides can be processed at the same time. If processing is not simultaneous, however (for example, if the slide sequence does not match the spoken topic), there is a risk that the two different modalities may split the learner's focus (the ‘split‐attention effect’) and thus hamper learning (Kalyuga et al., 1999). Schüler et al. (2012) argued that material with both written text and picture(s) creates a temporal split in attention.

Based on the findings presented above, the following hypotheses were drawn:

Understanding of written material is easier for readers with low reading skills/dyslexia if it is presented without images. This would mean that the decoding of the images involves visual processing that takes the focus away from the reading process for that group.

Understanding of written material is easier for readers with low reading skills/dyslexia if it is presented with images. This would mean that the decoding of the images informs the reading process for that group.

In both conditions, the controls with normal reading skills outperform the participants with lower reading skills (dyslexia).

In a text + illustration condition, the dwell time on the illustration is higher for readers with low reading skills/dyslexia than for readers with high reading skills/without dyslexia, meaning that the embedded information in the images requires more time for students with dyslexia to process.

The Current Study

In this study, we assume that both dyslexia and reading skill are continuous phenomenon closely linked to each other. As such, this study is not a comparison of two distinct participant groups. Instead, it is an investigation about the influence of reading skill when learning by the use of learning material consisting of text only or text + picture. Even so, we have chosen to label the groups ‘dyslexic’ and ‘controls’ as they have been recruited to the study based on whether they have been diagnosed as dyslexic or not during their school years. The current study focused on three research questions. The first pertained to what effect pictures have on patterns of fixation (i.e., visual attention) in individuals with dyslexia compared to controls. The second was whether pictures help or hinder learning; and the third was whether differences in picture styles (realistic, figure/ground, or complex) yielded different patterns of fixation (scan patterns). Such differences have been found by Bradley et al. (2011).

Methods

Participants

We recruited 46 participants in higher education through advertising on two university websites and enlistment of a teacher in a Swedish Folk High School to invite young adults with and without dyslexia, preferably 18 to 25 years old, who were native speakers of Swedish and had normal or corrected‐to‐normal vision. The decision to choose adults instead of children was based on the more developed binocular coordination among college‐aged adults, compared with that observed in children (Kirkby et al., 2008). In total, 19 participants with normal IQ who were classified as dyslexic in accordance with the guidelines of the ICD‐10 classification of Mental and Behavioural Disorders, and 31 non‐dyslexic (control) participants were recruited (Figure 1). Their ages ranged from 19 to 41 years, with a mean age of 23.8 (SD = 4.2) (Table 1).

Figure 1

Participants in the experiment.

Table 1

Characteristics of participants

Group	University students	Folkhögskola students	Total	Age (mean)	Age (SD)
Control group	23	8	31	23.48	4.47
	(8 male,	(3 male,	(11 male,
	15 female)	5 female)	20 female)
Dyslexic group	12	7	19	24.37	3.93
	(1 male,	(0 male,	(1 male,
	11 female)	7 female)	18 female)
Both groups	35	15	50	23.8	4.2
	(9 male,	(3 male,	(12 male,
	26 female)	12 female)	38 female)

Each participant from the two groups (control or dyslexic) was assigned randomly to one of two conditions: text only or text + picture. The participants with dyslexia had all been subjected to medical examinations for dyslexia and had received special support in compulsory school for their difficulties.

Instrument

We strived to avoid having participants in the dyslexic group whose difficulties with phonological awareness had decreased since their dyslexia diagnosis in the earlier school years. We also wished to avoid having participants in the control group who had undiagnosed impairment in reading abilities. The DUVAN Dyslexia Screening Battery (Lundberg & Wolff, 2003) was used to confirm the putative difference between participants assigned to the dyslexic and control groups. We have used DUVAN scores as it provides a more sensitive measure than the diagnosis. The DUVAN is based on a phonological theory of dyslexia, and is used in Scandinavian schools to screen students and adults for potential dyslexia (Green et al., 2009; Lindgrén & Laine, 2007). The test was standardized in Swedish in 2002 (Wolff & Lundberg, 2003) with 271 students in secondary school, of whom 47 suffered from dyslexia and the rest (224) were normal readers. The mean age was 22.5 years, and 70% were female. The standardization resulted in a mean of 186.5 (SD: 26.7) points for those without dyslexia. The critical limit was set to 140 and test–retest reliability was .83 (Wolff & Lundberg, 2003). The screening focuses on ‘phonological awareness, quality of phonological representations, as well as working memory and orthographic skill’ (Lindgrén & Laine, p. 419).

The first part of the screening contains 15 items related to difficulties caused by dyslexia, and 5 items regarding reading interest. Maximum score on the first part is 60 points.

The second part measures the phonological ability concerned with working memory (Green et al., 2009). Participants listen to a recorded voice that says one or more letters, then makes a statement followed by a question. For example, in the first sequence of trials, the voice might say, ‘letter K’, then the statement, ‘Dogs have four legs’, followed by the question, ‘Can dogs fly?’ Participants are asked to hold up a card that says either ‘Yes’ or ‘No’ to answer the question. (These responses are not scored.) Participants must then write down the letters they have heard on all trials to that point (to a maximum of four letters). A new trial begins 3 s later with a new letter, statement, and question. These four‐trial blocks are repeated 9 times, with a maximum score of 4 per block, giving an overall maximum score of 36 on this part of the screening.

The third part of the DUVAN, vocabulary, is a test without time constraints that mainly concerns phonological representations and some phonological working memory. Here, participants are asked to find the correct synonym from three different words that are phonologically similar. Each of the 14 items is worth one point, for a maximum score of 14 points.

The fourth part, spoonerisms, consists of a reversed spoonerism task in which phonological awareness and working memory are assessed. Participants listen to a recorded voice saying two words (e.g., brown car), and their task is to silently change the first letters in the words and decide which of three pictures shows the new word pair (in this case, crown bar). The test has a strict time limit and comprises 24 tasks. The maximum score is 24 points.

The fifth part of the screening, phonological choice, addresses phonological awareness and phonological representation during a 2‐min test. For each of the 60 test items, participants are asked to select one of three non‐words that would sound like a real word if read aloud. The maximum score is 60 points. Time limit is 2 min, and the respondent is not expected to solve all the items during this time.

The structure of the final test, orthographic choice, is the same as the previous one, but the word triplets are used to test orthographic knowledge. Participants select the correctly spelled word from the list of three, two of which are slightly misspelled. The maximum score is 100, and as with the phonological test, there is a 2‐min time limit, but the participant is not expected to complete all items in this time.

The maximum scores of the sub‐tests and the self‐reported questionnaire total 294. A value of 140 or lower means that the participant could suffer from dyslexia and should undergo further screening; therefore, we set a criterion of DUVAN score of 140 for excluding participants who had been designated as controls. We did not exclude the participants in the dyslexic group whose scores were above 140 because the comprehensive medical examination is more trustworthy than a screening. In the recruited control group (n = 31), four participants were excluded (n = 27 controls) for low scores. In the end, there was no overlap of DUVAN scores between controls and subjects with dyslexia; that is, the highest score for a dyslexic participant (157) was lower than the lowest score of a control participant (160). A summary of participants' total scores and subtest scores is offered in Table 2.

Table 2

Test scores for each section of the Duvan dyslexia screening battery

	Dyslexic			Non‐dyslexic
	n	Mean	SD	n	Mean	SD
Duvan A	19	28.05	4.97	31	49.90	7.78
Duvan B	19	28.63	6.30	31	32.77	5.56
Duvan C	19	8.74	3.05	31	10.87	2.71
Duvan D	19	13.37	4.07	31	17.61	4.72
Duvan E	19	17.63	4.74	31	30.45	9.95
Duvan F	19	28.89	9.82	31	55.32	16.53
Duvan total	19	125.32	17.92	31	196.94	36.29

Procedure

All participants signed a written informed consent before participating. Data were collected in two separate phases. In the first phase, all participants completed the DUVAN individually, with instructions given in small groups led by one researcher (the same person for all participants). Before the experiment began, participants were introduced individually to the setup and were shown a test item that was not included in the actual test. Next, participants underwent the eye‐tracking experiment individually with either the text‐only or text + picture materials.

Experiment

The experiment started with an initial five‐point calibration of the tracking of eye movements, followed by another four‐point calibration to validate the accuracy of the first calibration. The measured accuracy reported by iView X for all participants across the validation positions was 0.52 degrees (SD = 0.22) horizontally and 0.54 degrees (SD = 0.34) vertically. Eye movements were recorded during the entire experiment.

The stimuli comprised a set of displays on a computer equipped with an eye‐tracking device. The content of the material was information on six different genres of art (abstract art, impressionism, cubism, pop‐art, romanticism, and surrealism). Examples of the text‐only and the text + picture conditions are presented in Figures 2 and 3, respectively. The texts were like those frequently used in ordinary textbooks to describe what characterizes different genres of art. In the text and picture condition, the picture was placed to the right of the text to ensure maximum opportunity for discernment, according to the previously discussed finding that perceptual span is larger to the right of a fixation (Rayner, 2009b). The informational texts ranged from 75 to 91 words (mean = 81) and described the origin of the genre and the styles, colours, motifs, and techniques in the paintings. All text was in Swedish. All participants saw exactly the same six texts, and the only thing that differed between text‐only and text + picture conditions was the inclusion of the picture to the right of the text in the latter condition.

Figure 2

Comparison of picture fixations between participants (controls and dyslexia). Participants without dyslexia inspect the picture more than do those with dyslexia early after stimulus onset. In contrast, no such differences are observed during the last 40 fixations (corresponding roughly to 12–13 s of viewing).

Figure 3

Scan‐path for participant #182 (impressionism) (dyslexic).

Participants were presented with six different art genres, which allowed them to contrast the examples to discern how the pictures differed. Diversity of the scenes presented in the pictures (neutral, figure/ground, and complex art scenes) was balanced across the stimuli. The texts were constructed to give the reader an opportunity to discern the critical characteristics of a particular genre; this made it possible to separate the features that define paintings from a particular genre and to distinguish between the paintings, as such, and the different genres of paintings (Table 3). The design of the texts followed a template. Initially, the genre was anchored geographically (if possible) and in correct period of time. Then, the texts contain information about the development of the genre, what is tried to be expressed with the technique used by the artists, and finally main type of motifs. In both conditions, one or two sentences per art screen consist of statements which were directly linked to the provided example picture which was only presented at the text + picture condition. Those sentences described the motifs but also the art genre as such.

Table 3

Text presented at the screens in the experiment (Times New Roman 12p, double‐spaced, about 50 characters with spaces/line, M1.3 words/screen)

Art genre	Words	Sentences	Paragraph
Abstract	82	6	1
Impression	84	7	1
Cubism	77	5	1
Pop‐art	79	6	1
Romanticism	75	7	1
Surrealism	91	6	1

All text was in Swedish. All participants saw exactly the same six texts, and the only thing that differed between text‐only and text + picture conditions was the inclusion of the picture to the right of the text in the latter condition. The text was one region of interest, and the picture was one region of interest. In all, two regions of interest on each screen displayed for the participants. Each participant viewed a total of 43 screens, resulting in 1.978 scan paths (46 participants × 43 screens).

Participants' initial familiarity with the genres of art was assessed by a pre‐test in which the name of each genre and three pictures (one representing the named genre) were presented. Participants clicked the picture that they thought best represented the genre.

When the pre‐test was completed, the participants continued with the learning and assessment part of the experiment, which was divided into a number of consecutive steps. Participants were presented first with the name of a genre, then with information from either of the two experimental conditions (text only or text + picture). Their instructions in both conditions were simply to ‘study the information on the screens’, that is, no specific advice was offered about using the pictures as learning aids. When the participants had studied the information and were ready to answer questions about it, they clicked the mouse and an empty screen appeared. While instructed to keep looking towards the screen area, participants were asked, ‘What information about [genre] appeared on the last screen?’ in line with thinking‐aloud methodology (Fox, 2009). Their verbal answers were recorded with the SMI Experiment Center (v. 3.0.155; SensoMotoric Instruments Inc., Teltow, Germany). When they had no more information to provide, they were asked to click the mouse to proceed and to answer three multiple‐choice questions. The first multiple‐choice question was presented on the screen: What characterizes [genre]?, and the respondent was offered a choice of five text responses, one of which was correct. The second question asked participants to estimate how confident they were about their previous answer on a scale from 1 (very uncertain) to 7 (very certain). Finally, they answered the same questions that they had answered during the initial test, to assess whether learning had taken place during the experiment (post‐test). In the tests, the positions of the three pictures were ordered randomly.

The experiment was self‐paced and had no time limit. Participants left‐clicked the mouse button to continue to the next screen. The question on the final screen, ‘How familiar were you with these art genres earlier?’, was answered on a scale of 1 (very unfamiliar) to 4 (very familiar). All participants were rewarded with a cinema ticket. The screens in order of presentation are (1) name of art genre; (2) test or text + picture display; (3) blank screen (verbal question about what information was shown); (4) text question; (5) confidence rating; and (6) picture question (post‐test).

Apparatus for recording eye movements

Eye movements were recorded binocularly at 250 Hz with the SMI RED250 eye‐tracker and iView X (v. 2.7.13; SensoMotoric Instruments Inc., Teltow, Germany). The eye‐tracker was connected to a Dell laptop computer. Stimuli were shown on a 22‐inch Dell monitor with resolution 1680 × 1050 with a refresh rate of 60 Hz, and presented with SMI Experiment Center (v. 3.0.155). The inspection time (time spent on viewing the material) was measured. The participants adjusted their chairs until the distance between the visual stimuli and their eyes resulted in a comfortable viewing situation.

Data analysis

Data pre‐processing

Eye fixations were detected with the IDF Event Detector 3.0.14 (SensoMotoric Instruments, Teltow, Germany), using the default settings (a peak velocity threshold of 50°/s and a minimum fixation duration of 40 ms). All further data processing, as well as the definition of areas of interest, were performed with MATLAB R2010b (The MathWorks, Inc., Natick, MA, USA).

Statistical analysis

Each individual's DUVAN score was used as a predictor of dyslexia. For the sake of comparison, descriptive statistics are given separately for the controls (DUVAN > 159, n = 27) and dyslexic participants (DUVAN ≤ 159, n = 19). The DUVAN mean score for control participants was 196.9 (SD = 36.2), and mean score for those with dyslexia was 125.3 (SD = 17.9).

The dependent variables answers (correct/incorrect answers for picture and text questions), dwell time (sum of fixation times on the text or image), time to first fixation (TTFF) on the picture, and transition rate (rate of switching fixation between text and image) were analysed with linear mixed effect models. The independent variables were group (controls/dyslexic), condition (text only vs. text + picture), viewing time (time to view text only or text + picture before answering a comprehension question) and previous knowledge (measured during the pre‐test; one binary value for each stimulus) as fixed effects, and participant and stimulus as random effects. Viewing time and condition were used as predictors only when applicable. We used the binomial version of the linear mixed effects model for dichotomous response variables. When possible, 95% confidence intervals and p values were calculated through Markov chain Monte Carlo simulations. Some of the dependent variables were transformed to an approximate normal distribution; dwell time and fixation duration were log transformed; and TTFF was Box–Cox transformed (λ = 0.5). Only significant interactions are reported.

For the categorical outcome variables, linear mixed effects models were chosen over ANOVAs to easily account for random intercepts of participants and stimuli and to avoid spurious results associated with the ANOVAs (Jaeger, 2008). Interaction terms were included only if they improved the model fit compared to a less complex model without the interaction. Using the anova()‐function in R, interaction terms were included if the model comparison showed significant differences p ≤ 0.05.

To facilitate comparisons across all analyses, to increase ease of modelling random factors and to better deal with the slightly unbalanced design, the other dependent variables were analysed in a similar manner. All analyses were performed with R 2.13.0 (R Development Core Team, 2011) using the packages lme4 (Bates & Maechler, 2009) and languageR (Bayeen, 2011).

Qualitative analysis

In line with Creswell's (Creswell, 2013) suggestions to use different data analyses to better explain and understand data, a qualitative analysis was made of the scan paths at an individual level. The qualitative analyses aim to further explore and explain the quantitative data and to illustrate the way respondents in the different groups approach the material visually. After analysing the entire data set, viewing patterns from two cases, one from each group, were chosen to highlight differences in viewing strategies.

Results

The results are presented in the order that the three research questions were described: (1) What effect do pictures have on patterns of fixation (i.e. visual attention) in individuals with dyslexia compared with controls?; (2) What effect do pictures have on learning?; and finally (3) Do differences among picture styles (realistic, figure/ground, or complex) yield different patterns of fixation (scan patterns)? The three hypotheses were tested, and two of them (1 and 2) are a contradiction where the first is supported while the second is rejected as the results show that participants with dyslexia learn more from the material with text only. The third hypothesis states that the controls outperform the other groups no matter of condition, which was supported. Finally, the fourth hypothesis claims that the group with dyslexia will spend more dwell time at the picture is rejected as the participants in this group seem to neglect the picture while reading.

Pictures' Effect on Patterns of Fixation

Inspection time

Participants spent an average of 49.1 s (SD = 19.4) inspecting the text‐only screen and 50.0 s (SD = 21.5) viewing the text + picture screen. The text‐inspection time for participants with dyslexia increased from 58.3 s (SD = 19.4) for the text‐only screen to 61.2 s (SD = 20.4) for text + picture screen. The opposite trend was found for controls, who decreased inspection time from 40.6 s (SD = 15.5) for text‐only screen to 39.6 s (SD = 16.2) for text + picture screens.

In the text + picture condition, the picture was viewed (defined as having a dwell time of at least 60 ms) by 95.2% of the controls and 78.3% of those with dyslexia. Overall, participants spent 45.9 s (SD = 2.1) looking at the text, 2.2 s (SD = 2.3) looking at the picture, and 0.8 s (SD = 1.0) looking elsewhere.

Proportion of total dwell time

PA scores were associated with a significantly higher proportion of total dwell time on the picture, and a significantly lower dwell time on the text.

Table 4 summarizes the results from the two linear mixed effects models using dwell time on the text and dwell time on the picture as outcome variables.

Table 4

Means and standard deviations, M (SD), of the measures of interest, separated by group (Controls /Dyslexics), and condition (T—text only, TP—text and picture)

	Controls		Dyslexics
Measure	T	TP	T	TP
Text comprehension (%)	92.3 (26.8)	88.1 (32.6)	88.9 (31.7)	70.0 (46.2)
Learning (%)	60.2 (49.3)	59.5 (49.3)	64.8 (48.2)	53.3 (50.3)
Total inspection time (s)	41.9 (16.1)	40.5 (16.5)	59.6 (19.2)	63.2 (20.7)
Dwelltime on text (%)	97.7 (2.8)	91.7 (6.1)	97.5 (4.8)	95.9 (3.7)
Dwelltime on picture (%)	NA	6.2 (5.3)	NA	3.0 (3.6)
Time to first fixation on picture (s)	NA	16.3 (17.6)	NA	41.1 (33.4)

Time to first fixation on the picture

The most interesting result of the study concerning the strategy of obtaining a global overview (i.e., starting with an overall inspection of the information on the screen) is the TTFF on the picture. While controls tended to spend more time looking at the picture than did those with dyslexia, this seems to be explained by the first 10–15 fixations after stimulus onset. Figure 2 illustrates how the controls and subjects with dyslexia differed in their allocations of visual attention directly after stimulus onset and towards the end of a trial.

As shown in Table 5, TTFF became significantly smaller as the DUVAN score increased. To ensure that fixations were associated with sufficient visual input, only fixations longer than 200 ms were analysed (Holmqvist et al., 2011). Moreover, the first fixation after the stimulus onset was excluded because its position would be associated with the previous stimulus rather than the current stimulus.

Table 5

Full output from the LME models

DV	Predictors	Estimate	Std. error	z/t value	Pr(>|z/t|)
Inspection time	(Intercept)	285.6488	21.4016	13.347	<2e − 16	***
	DUVAN score	−0.4052	0.1140	−3.553	0.00094	***
	Text + picture	3.9552	10.7354	0.368	0.71437
	lmer(inspection_time ~ DUVAN_score + condition (text / text + picture) + (1 | participant_id) + (1 | stimulus), data = mydata)
Text comprehension
	(Intercept)	3.4438	0.6233	5.525	3.3e − 08	***
	DUVAN score	1.1300	0.3527	3.203	0.00136	**
	Text + picture	−1.3287	0.5954	−2.232	0.02565	*
	Viewing time	0.4599	0.2864	1.606	0.10834
	Priorknowledge	−0.7010	0.4532	−1.547	0.12187
	glmer(correct_answer ~ DUVAN_score + condition(text / text + picture) + viewing_time + prior_knowledge + (1 | participant_id) + (1 | stimulus), family = binomial, data = mydata)
Learning
	(Intercept)	0.5523	0.2712	2.036	0.0417	*
	DUVAN score	0.1094	0.1599	0.685	0.4936
	Text + picture	−0.2576	0.2888	−0.892	0.3724
	Viewing time	0.2494	0.1691	1.475	0.1403
	glmer(learning ~ DUVAN_score + condition(text / text + picture) + viewing_time + (1 | participant_id) + (1 | stimulus), family = binomial, data = mydata)
Prop. dwell time on text
	(Intercept)	3.590457	0.308893	11.624	2.29e − 11	***
	DUVAN score	−0.006267	0.001644	−3.811	0.00094	***
	Prior knowledge	0.077682	0.081402	0.954	0.34175
	lmer(logit(dwelltime_text) ~ DUVAN_score + prior_knowledge + (1 | participant_id) + (1 | stimulus), data = mydata)
Prop. dwell time on picture
	(Intercept)	3.590457	0.308893	11.624	2.83e − 11	***
	score	−0.006267	0.001644	−3.811	0.00248	**
	Prior knowledge	0.077682	0.081402	0.954	0.44336
	lmer(logit(dwelltime_picture) ~ DUVAN_score + prior_knowledge + (1 | participant_id) + (1 | stimulus), data = mydata)
Time to first fixation on picture
	(Intercept)	532.3272	83.9934	6.338	1.77e − 06	***
	score	−1.4679	0.4514	−3.252	0.00366	**
	Prior knowledge	10.1629	26.5527	0.383	0.70256
	lmer(bxcx(ttff,0.5) ~ DUVAN_score + prior_knowledge + (1 | participant_id) + (1 | stimulus), data = mydata)
Number of transitions
between text and picture	(Intercept)	−3.6735677	0.0430569	−85.32	<2e − 16	***
	preKnowledge	0.0170121	0.0110187	1.54	0.125
	score	0.0005649	0.0002275	2.48	0.021	*
	lmer(logit(nTransitions) ~ DUVAN_score + prior_knowledge + (1 | participant_id) + (1 | stimulus), data = mydata)

Signif. codes: 0

0.001

0.01

0.05 ‘.’ 0.1 ‘ ’ 1

Transition rate

On average, the dyslexic group made 10.5 transitions (SD = 10.2) per stimulus and the controls made 11.3 (SD = 7.0). Table 5 also shows that the transition rate between the text and the image or vice versa increased significantly as a function of the DUVAN score.

Pictures' effect on learning

The comprehension pre‐test showed that participants had no more previous knowledge than would be predicted by chance (1/3): M = 0.25 (SD = 0.43) for the dyslexic group, and M = 0.35 (SD = 0.47) for the control group. At the end of the test, participants rated their prior familiarity with the genres as 3.36 (SD = 0.69) (dyslexic subjects) and 3.70 (SD = 1.32) (controls) on a scale of 1 (no previous knowledge) to 7 (extensive knowledge).

Comprehension of the presented material

Each of the 46 participants answered six questions, for a total of 276 answers. Participants correctly answered an average of 85.51% of the questions (SD = 0.06) (dyslexic subjects: M = 78.95, SD = 0.15; controls: M = 90.12, SD = 0.15) with an average confidence rating of 4.75 (SD = 0.14) (dyslexic subjects: M = 4.20, SD = 0.17; controls: M = 5.13, SD = 0.19). The DUVAN score was a significant predictor of a correct answer. The addition of a picture to text seemed to decrease, rather than augment, the ability to provide a correct answer (cf. Table 2) in both groups. However, this trend appeared to be driven largely by results from participants with dyslexia. When analysed separately, the dyslexic group showed a marginal effect of fewer correct answers when pictures were present (p = 0.07). No significant difference was found for controls (p = 0.13). Prior knowledge tended to increase the number of correct answers, but the effect was not significant.

Learning of the presented material

In both conditions, correct answers in total were given for 63.77% (SD = 0.04) of the questions (dyslexic subjects: M = 63.16, SD = 0.08; controls: M = 64.20, SD = 0.05). Because the only significant predictor was prior knowledge (p < 0.001) (Table 4), a new dependent variable called learning was created by comparing the answers given during the initial test with those given after viewing the learning material. Zero (0) denoted a wrong answer and one (1) a correct answer, resulting in four possible combinations: 0 prior knowledge (pk) + 0 subsequent knowledge (sk); 0 pk + 1 sk; 1 pk + 0 sk; and 1 pk + 1 sk; only 0 pk + 1 sk implied learning. The variable learning was labelled with a 1 when a wrong answer in the initial test was corrected in the later test, and with a 0 otherwise. Table 4 shows that none of the predictors reached significance in the global analysis, nor when control and dyslexic participants were analysed separately.

The effect of picture styles on patterns of fixation

In addition to the quantitative data presented in Tables 4 and 5 (number of fixations, length of dwell time, TTFF, and correlations with the results of the DUVAN test), qualitative differences in eye movement patterns were found by studying the viewing patterns of each participant in the text + picture condition (n = 24; 14 controls and 10 with dyslexia). Analyses of the scan paths showed different viewing patterns in the two groups, as well as differences within the groups related to the different conditions, which have already been reported in the results section of the quantitative analysis. We have chosen to present one example from each group, which represents typical differences in viewing patterns. The scan paths in the dyslexia group differ from the viewing patterns of the controls. The patterns are linear for the control and non‐linear for the dyslexic.

The between‐group differences in first‐time fixation on the picture showed that the picture itself made a difference to the respondents' viewing time (for statistical data from all participants, see Figure 2). The differences found within and between the groups are in accordance with previous reports that difficulties attract longer fixations (Liversedge & Findlay, 2000), and as the participants with dyslexia find it more difficult to read, they tend to fixate more on the text than the picture (Table 5). The viewing pattern of the dyslexic group also supports observations of a different way of simultaneous processing offered by Lassus‐Sangosse et al. (2008).

The first presented comparison of scan paths is from a participant from the group with dyslexia, and is a typical scan path for the viewing patterns in this group. This person had more or less the same strategy for both screens; however, there were fixations only on the picture in the surrealism genre alone. Participant #182 scored 119 on the DUVAN screening (mean score for participants with dyslexia was 125.3 (SD = 17.9)). The scan paths from the eye‐tracking study of the controls show a pattern that it is different from that of the participants with dyslexia. First, the participants with dyslexia (Figure 3 for impressionism and Figure 4 for surrealism) exhibited difficulties following the lines of text: they jumped between the lines and went backwards and forwards in a more haphazard way than did the controls, a pattern also supported by the quantitative data in Table 6. However, there were more fixations on the picture in the surrealism genre, an art form that does not represent the world as one would expect and that might therefore be more difficult to interpret.

Figure 4

Scan‐path for participant #182 (surrealism) (dyslexic).

Table 6

Results of test scores and fixations (mean)

	Dyslexia (10)	Controls (14)
DUVAN score	122	214
Fixations total	2072.3	1581.2
Fixations/picture	20.6	26.6

Participant #208 from the controls scored 252 on the DUVAN screening [group mean was 196.9 (SD = 36.2)]. As shown in Figures 5 and 6, the viewing pattern of the controls was linear, but the more difficult picture also resulted in longer fixations, similar to what was seen for participant #182.

Figure 5

Scan‐path for participant #208 (impressionism) (control).

Figure 6

Scan‐path for participant #208 (surrealism) (control).

Differences also were observed in the participants' scores on the questions about the material. After reading the material, participant #208's score increased more than did that of participant #182. Participant #182 answered four of six text questions correctly, with confidence scores of 16–42 points, and matched two of six pictures correctly with the art genre. In contrast, participant #208 had six correct answers on each type of question, with confidence scores of 29–42 points.

View paths for different stimuli

The presented scan paths showed different viewing patterns between the participants, and similarities in scan paths for different scenes in the stimuli. The control group averaged a total of 1.581 fixations, of which 26.6 (1.7%) were on the picture (Table 5); and the group with dyslexia averaged 2.072 fixations, of which 20.6 (1%) were on the picture. Although the viewing patterns differed between the groups, both groups had the highest number of fixations on the surrealist paintings. Three paintings were used to represent realistic (impressionism), figure/ground (cubism), and complex (surrealism) art scenes. The dyslexic group spent 3.8% of their viewing time on the complex picture in the surrealist stimulus versus 2.2% on the figure/ground picture and 2.0% on the realistic picture. A similar pattern regarding longest viewing time was found in the control participants, who spent 9.43% of their total viewing time on the complex picture in the surrealist stimulus, 3.64% on the figure/ground picture, and 5.07% on the neutral picture.

Discussion

The aim of this study was to examine the effect of pictures on improving text comprehension in dyslexic and non‐dyslexic adults based on the influence of reading skill as a continuous variable of the total sample. The study sought to answer three research questions: (1) What effect do pictures have on patterns of fixation (i.e. visual attention) in individuals with dyslexia compared with controls?; (2) What effect do pictures have on learning?; and (3) Do differences in picture styles (realistic, figure/ground, or complex) yield different patterns of fixation and/or different scan patterns? Despite the fairly small sample size and the predominance of female respondents, the results indicate some interesting findings that are worthy of future study in a larger population. The analysis used a mixed‐method design (Creswell, 2013), and the main data set was analysed both quantitatively and qualitatively to enable deeper investigation of the phenomena of interest and to allow convergence of results to strengthen the conclusions drawn.

Concerning the first question, our results were consistent with findings reported by Bradley et al. (2011) and Vidal et al. (2012): Differences in impairments (dyslexic versus controls) produced different patterns of fixations and saccades. Overall, control participants in the present study spent proportionally more time viewing the pictures, and proportionally less time viewing the text than did participants with dyslexia.

As for the second question regarding the effect of pictures on learning, it is clear that the results show a significant difference between participants with dyslexia and controls, and that the different informational modalities (text or text + picture) affected their learning in different ways, namely, a decreased learning outcome for participants with dyslexia in the text + picture condition compared to the text‐only condition (Table 4). However, perhaps the most interesting finding regarding learning strategies was the difference in the participants' initial strategies for discerning the material, that is, their use of a global overview. The control participants started by obtaining an overview of the presented information as a kind of background. They seemed to automatically search for information that was meaningful in the picture and text simultaneously as a cohesive unit, by using a wider visual attention span. The text + picture combination affected the control participants in a more positive way than it did the participants with dyslexia. Because the more capable readers discerned the picture early on, they apparently took advantage of the simultaneous experience of it with the information in the text, which resulted in increased learning — an opportunity that was not used by the participants with dyslexia. This is consistent with findings about simultaneous processing disorder (Lassus‐Sangosse et al., 2008). The subsequent viewing patterns for each stimulus also showed fewer attentional shifts between text and pictures for the participants with dyslexia.

Regarding the third question, although differences among picture styles (realistic, figure/ground, or complex) yielded different patterns of fixation (scan patterns) for controls and dyslexic subjects, the two groups shared a pattern of fixating most on the complex pictures. Although the present results show impairment in the fixations and saccades of the dyslexic group, this did not affect their pattern of attention to different styles of the picture as stimuli. So although dyslexic participants fixated less on pictures overall, they displayed the same increase in focused attention to complex and unfamiliar motifs. This shared viewing pattern is similar to that found by Bradley et al. (2011), in which complex scenes prompted more fixations. Despite the similarity of the viewing pattern, qualitative analysis of the scan path patterns showed that participants with dyslexia showed a more randomized viewing pattern of the material than did the control group. Further research is needed to examine if this could hypothetically be because of weak executive function. A direct assessment of participants' executive function would be needed to strengthen this interpretation. Although our results revealed that participants with dyslexia paid relatively little viewing time at the pictures, the mere display of pictures seems to have influenced them somehow, because there was a marginal reduction in the proportion of correct answers (p = 0.07). A common visual behaviour, according to Rayner et al. (2001) and Underwood (2005), is that people tend first to inspect a picture briefly, read the text next, and then to attend more carefully to the picture. This suggests an attentional shift between the modalities of text and picture, and is exactly what controls did. The problem for participants with dyslexia seems to be less an issue of temporal splitting of attention (Schüler et al., 2012) and more a different approach to viewing information. People with reading difficulties need to focus on the text to decode it. The diversity of research results concerning multi‐modal stimuli in different groups of readers may therefore be explained by the different strategies used by participants with and without dyslexia to process the same information. The most interesting finding is that dyslexia seems to affect fixation on the picture early in the stimulus presentation, when initial impressions could be linked to existing knowledge to form a framework for new information in the text. This phenomenon should be investigated in a larger sample to see whether these tentative results can be confirmed.

Phonological awareness also significantly influenced the eye movement behaviour of learners, in that controls paid more and earlier attention to the pictorial information and made more transitions between the two modalities. This suggests a systematic strategy to discern different parts of the material. Significant differences between the groups were observed on the quantitative measures, which correlated with the PA scores. Higher DUVAN scores were associated with significantly higher total dwell time on the picture and significantly lower total dwell time on the text, indicating that participants with dyslexia tended not to fixate on the picture but to spend more time inspecting the text.

The present study has a number of limitations. The adults in the study who had been diagnosed with dyslexia had received interventions for this disability during their school days. Although their reading and writing impairments were validated by their DUVAN test scores, the participants who had a dyslexia diagnosis almost certainly experienced language differences in areas other than just phonological ability (Nation & Snowling, 2004). It is not clear how this difference in classifying participants may have affected the results. The sample used in the study was relatively small and was obtained by convenience sampling; both of these constraints limit the generalizability of the results. The small sample was uneven regarding gender and schooling, as we had some difficulties to engage participants, which is a limitation of the study. Furthermore, the limited power associated with the small sample was further diminished by splitting the control and dyslexic groups into two conditions (test only and text + picture); this also may have affected the results. The study results should therefore be taken as preliminary. Another limitation is the choice to use two areas if interest (text and picture) instead of breaking down the data into several smaller areas of interest within the two different stimuli.

Despite these limitations, the comparisons between the two groups within and across conditions showed interesting differences and similarities. Our assumption is that the initial global overview, which the participants with developed reading skills and high learning outcomes frequently used, is important for making sense of a piece of learning material and for simultaneous analytical information processing of the different stimuli within it, so that optimal learning results.

Given material containing both text and picture, participants with dyslexia mainly view the text.

Text‐inspection time for participants with dyslexia increased for text + picture displays, while the opposite effect was found for controls.

A global overview of learning material was less frequently used by participants with dyslexia.

Material that differed from what was expected drew participants' attention whether they had dyslexia or not.