An online multimedia resource in behavioral neuroscience.

The advance of web-based technology has stimulated innovation in education. This paper discusses the development and evaluation of an online multimedia resource for undergraduate-level behavioral neuroscience education. This resource surveys four major subject areas: language, attention and perception, thinking, and autism. It employs audio and video streaming, online demonstration experiments, computer simulation, and internet links. This online resource has two distinct advantages over a paper textbook. First, a considerable proportion of the content is conveyed using multimedia, thus making the learning experience more vivid and dynamic. Second, its interactive components provide opportunities for students to participate in the various experimental tasks introduced in the text and to compare their own performance with those of others. This hands-on experience not only enables students to gain in-depth procedural knowledge of the tasks but also has positive effects on their motivation. Feedback from three undergraduate classes that used this resource as supplementary material showed that students were highly positive about its pedagogical values. This free resource is available on the web at http://psych.rice.edu/mmtbn/.

The internet has become an increasingly important part of the infrastructure of the educational system. The unique contributions of the internet and the world wide web derive from the fact that they provide easy access to a knowledge base that is an order of magnitude larger than before. Moreover, to educators involved in the day-to-day practice of teaching, the internet presents a means of accommodating different types of instructional media.

Multimedia refers to a class of information delivery formats that includes text, graphics, audio, video, and hypertext. Different medium types used to require specific equipment for content delivery. Although at the present time, multimedia instructional content is delivered largely through the use of compact discs (CDs), the rapid development of web-based technology now allows the delivery of multimedia materials through the internet. This advance not only significantly lowers the threshold of multimedia information access, it also stimulates innovative ways in teaching and learning that have been made possible only with this new platform (Brooks et al., 2001).

Much of the strength of multimedia instruction relies on the opportunity it provides for students to interact with the program and to learn the materials in an active way (Schwier and Misanchuk, 1993). With this in mind, we developed a web-based multimedia resource for teaching undergraduate-level behavioral neuroscience. Rather than develop materials for an entire course, our goal was to explore the potential of web-based interactive multimedia instruction from both a technical and pedagogical perspective. Therefore, this online resource surveyed four subject areas including language, attention and perception, thinking and autism, and featured different web-based technologies for different content areas.

This online resource is based on the metaphor of a traditional textbook: each subject area constitutes a chapter, and each chapter is further broken down into a number of sections. Each section is contained on a single web page. The navigation scheme of the site allows a student to access chapters and sections either sequentially or in a customized order. A number of web-based technologies, including audio and video streaming, online demonstration experiments, computer simulation, and hypertext links, are employed to illustrate individual topics. These multimedia materials are integrated into the text through links so that once a link is clicked, the multimedia content shows in a new, pop-up window. In addition, at the end of each chapter, we include a “Multimedia” section to provide a complete list of all the interactive and/or multimedia components contained in that chapter.

Below we provide a survey of the four subject areas covered and introduce some examples of the technologies we used in developing this online resource.

CONTENTS

LANGUAGE

The chapter on language is the most fully developed among the four subject areas in this resource. Taking a cognitive neuropsychological approach, the chapter covers basic topics in human language processing and includes sections on speech perception, word comprehension, sentence comprehension, word production, and sentence production. The text provides an overview of the most influential theoretical thinking on each individual topic. Wherever possible, we supplied data from brain-damaged individuals to show the effects of brain damage in order to provide insight into how the unimpaired brain processes language. As an example, the section on sentence production is organized around the comparison between two classes of models, feed-forward models and interactive models. The text starts with the differentiation between two stages involved in sentence production, functional encoding and grammatical encoding, along with speech error evidence in support of such separation. Then it discusses the major differences between the feed-forward models (as represented by Garrett’s model and further elaborated by Levelt’s work) and the interactive models (represented by Dell’s proposal). The section proceeds to discuss that the two classes of models agree that phonological planning in speech production is incremental and that this view is supported by evidence from aphasic patients. The discussion is centered on video clips of two aphasic patients and a control performing the moving-pictures task (for a detailed description of these video clips, see the Streaming Audio and Video section). The clips show that the specific deficit of a patient has a distinct impact on his/her performance, thus providing evidence as to the kind of information involved in phonological planning.

ATTENTION AND PERCEPTION

The chapter on attention and perception consists of three sections: shape constancy, attention, and attention deficit disorder. Shape constancy is exemplified by the canonical Shepard Illusion demonstration (Shepard, 1981). This interactive demonstration displays two identical shapes at different angels so they seem very different. Once a student rotates and moves one of the shapes to make the two aligned, it becomes clear that they are exactly the same. The section on attention discusses spatial attention and selective attention with an emphasis on research methodology. Both behavioral (such as the visual search and visual cueing paradigms) and neurological (such as lesion, ERP, and neuro-imaging) methods are introduced in the text. Finally, the section on attention deficit disorder is organized around the experiences of an adult who suffered attention deficit hyperactivity disorder (ADHD) in his childhood. Three sets of video clips are included, each providing a unique perspective on this topic. The first set includes 11 segments from an interview with Dr. Deborah Pearson and provides background information such as the definition, cause, diagnosis, and treatment of ADHD. The second set consists of 18 clips from an interview with the adult (DG) who gave vivid descriptions of what life was like for him growing up with ADHD, as he struggled, coped, and finally succeeded through different stages of his life. The final set has nine segments from an interview with DG’s father, who emphasized how he and his wife, as parents, helped their young son cope with this disorder. Each clip in this section is accompanied with a text commentary informing the student of its central point. Together, these clips tell a fascinating story of how love, courage, and ingenuity have helped the family overcome the negative impact of ADHD.

THINKING

This chapter contains an engaging discussion of reasoning. Using a single study, it illuminates the difference between deductive and probabilistic reasoning and illustrates important methods in neuropsychology. The deduction task required distinguishing valid from invalid arguments. An example of a valid argument is: The probability task required judging whether the conclusion of an argument was more likely to be true than false given the premises. For example, All of the arguments in the probability task were (logically) invalid. The information in the premises was therefore insufficient to force the conclusion to have either high or low probability. The judgment was thus subjective in character, and required the reasoner to integrate background knowledge (e.g., about jobs and recreation) not explicitly presented in the argument.

If he is an electrician then he spent two years in night school.

He is an electrician and owns a computer.

He spent two years in night school.

If he is a heart specialist then he either bicycles to work or swims regularly.

He is a heart specialist.

He bicycles to work.

Positron emission tomography (PET) scanning produced images of the brain blood flow that occurred during the experimental tasks. The analysis of the data was designed to discover four things: (1) the locations of the active regions that are common to the two kinds of reasoning, (2) the regions that are distinctive to reasoning compared to linguistic processing, (3) whether the distinction between probabilistic and deductive reasoning interacts with left versus right hemispheric processing, and (4) the presence of specific contrasts between the brain sites active for probabilistic versus deductive reasoning. The results of the study including the images from the PET scans are presented and related to theoretical issues in reasoning.

AUTISM

The chapter on autism starts with a discussion of the general aspects of this disorder, such as symptoms, diagnosis, and treatments, and subsequently focuses on its neurological underpinning. It surveys four specific areas of the brain: the amygdala, the frontal lobe, the temporal lobe and the cerebellum, and provides an extensive literature review on how each area may be involved in this disorder.

In summary, the four chapters of this online resource cover basic and important topics in each subject area. They are further strengthened by the various interactive, multimedia components integrated into the content. Following is a description of the major web-based technologies that we used in developing this online resource.

INTERACTIVE/MULTIMEDIA TECHNOLOGIES

STREAMING AUDIO AND VIDEO

This online resource makes extensive use of streaming audio and video. To accommodate different network connection speeds, we developed three versions for each video segment (in QuickTime format). These are large-frame (320 × 240 pixels) video for broadband connections, small-frame (160 × 120 pixels) video for modem connections, and sound-only (streaming audio) for lower-speed modem connections. The default connection speed for the website is broadband, but a student can switch to a different setting at any time by clicking on a link that is conveniently located above the content page.

Streaming audio and video are most extensively used in the language chapter. As we took a cognitive neuropsychological perspective, many of the theories introduced in this chapter are based on behavioral patterns of patients with various forms of language deficit. Therefore, we decided to supplement the text with video and audio clips of patients’ performance on a variety of language tasks. As an example, Martin and her colleagues (e.g., Martin and Romani, 1994) have proposed a hypothesis on the role of working memory during planning of speech production. In their view, there are separate storage systems in memory for different types of information involved in language processing. Martin et al. distinguished between semantic and phonological retention capacities, and further between input and output buffers. In a study providing support for this hypothesis, Martin and Freedman (2001) adopted the moving-pictures task to show that aphasic patients who had different damaged brain areas exhibited distinctive behavioral patterns. In the moving-pictures task, subjects are asked to describe a scene starting with either a simple or a complex noun phrase, as in “The watch is below the duck and the gate” and “The saw and the faucet are below the wall.” One patient, ML, who suffered a left hemisphere stroke involving the left frontal and parietal operculum, has a semantic short-term memory (STM) deficit. He had normal comprehension and single word processing, but his speech was slow and effortful. In contrast, a second patient, EA, who suffered a left hemisphere stroke involving the left temporal and parietal lobes, has a phonological STM deficit. She demonstrated good comprehension and fluent speech, but had difficulty retaining phonological information. These two patients’ performances on the moving-pictures task, together with that of a third normal subject as the control, are illustrated with streaming video clips.

Figure 1 shows the moving picture on the top and the waveform of a subject’s speech on the bottom. If a student clicks on a segment of the waveform, the corresponding speech segment is played. The waveform is time-stamped and therefore it allows the student to view the time course of the speech. In their original research, Martin et al. argued that the onset difference for the two types of sentences as manifested in each patient’s performance provided evidence about the type of information involved in speech planning. The clips show that the first patient, ML, was much slower starting the complex phrase compared to the simple one (the onset difference was about 1000 ms.) This is because the semantic STM deficit ML suffered affected the retention of word-level semantic information before it was integrated into the meaning of the sentence. In contrast, patient EA’s onset difference (50 ms) was equivalent to that of the normal subject (50 ms). These results support the conclusion that semantic rather than phonological STM deficits impinge upon phrase planning that the moving-pictures task was designed to tap.

Figure 1.

Normal and aphasic patients’ performances on the moving-pictures task. If a student clicks on a segment of the waveform (the highlighted area), the corresponding speech segment is played.

The language chapter includes a total of 14 streaming audio and video clips. In addition to the moving-pictures task, we also provide video clips with patient ML performing a number of other language tasks, including word repetition, word and nonword reading, plausibility judgments, and picture naming. Taken together, these clips not only provide detailed information about the task procedures, but they also allow students to gain insight into the effects of specific deficit on one’s language ability. ML’s semantic deficit resulted in his low performance on plausibility-judgments, word and nonword reading, as well as picture-naming tasks. In particular, students may notice that ML had difficulty naming body parts even though he fared quite well with other categories such as fruits and furniture. Two additional clips of less-structured conversations also allow students to observe ML’s language deficits in a more general context. Here ML talked about a trip he had recently taken and how suffering a stroke had profoundly affected his life. His speech, though coherent, was slow and hesitant, as he had great difficulty turning the ideas into the right words. This symptom is typical of Broca’s aphasia. In contrast, in another audio clip, MS, a patient who has Wernicke’s aphasia, spoke fluently but his speech was unintelligible. Thus, the two clips provide a striking contrast between the two types of aphasia.

COMPUTER SIMULATION

One recent development in neuroscience is the use of computer models to simulate cognitive processes in both the normal and the damaged human brains. The increasing popularity of computer models is largely because these models are explicit and lead to testable predictions. To demonstrate the potential of this modeling approach, we included in the online resource a neural network model on word production.

A neural network is a modeling architecture that draws heavily on the mechanism of human neural system. It consists of large numbers of neuron-like information processing units, each unit representing a certain aspect of the information in the environment, such as conceptual objects (e.g., features, letters, words, etc.) or more abstract elements. The units are inter-connected so that they influence other aspects and at the same time are influenced by them. Information processing takes place through the interactions among these units. This is commonly referred to as parallel-distributed-processing (PDP).

The computer simulation we implemented was based on a PDP model of monosyllabic English word reading developed by Plaut et al. (1996). This model was proposed as an alternative to the dual-route approach to word production. Both the dual-route and the PDP approach provide explanations as to why skilled readers are able to read regular and irregular words (e.g., MINT and PINT) correctly and effortlessly. The dual-route model suggests that there are two different procedures involved when converting print to speech: if the reader has learned the word before, its pronunciation is retrieved by looking up an internal lexicon. If the reader encounters letter strings that have never been seen before, then a non-lexical route is taken so that the reader resorts to a system of rules specifying the relationship between letters and sounds. In contrast to this dual-route approach, Plaut et al. (1996) proposed a single, uniform procedure for computing a phonological representation from an orthographic representation regardless of whether the latter constitutes a regular word, an exception word or a nonword. As both the dual-route and the PDP approach have gained support from empirical research, it is not yet clear which one is a more valid account for this language phenomenon. However, the PDP approach possesses two important features that the dual-route model currently lacks: It is computational and it learns. Therefore, the PDP approach offers a useful way of thinking about the information processing underlying word production.

Plaut et al.’s PDP model consists of three layers of processing units (Figure 2). The input layer contains 105 orthographical units, each representing a grapheme. The output layer contains 61 phonological units, one for each phoneme. The phoneme units are grouped into mutually exclusive sets, and are ordered from left to right to reflect the left-to-right ordering constraints imposed within consonant clusters. Between the two layers there is an intermediate layer of 100 hidden units. The model adopts a simple feed-forward architecture so that all the input units feed into each hidden unit on the second level, and all the hidden units, in turn, feed into each output unit. The connection between two units carries a weight. As the network is exposed to a corpus of words (together with their pronunciations), the weights are adjusted gradually so that the network eventually captures the statistical property of grapheme-phoneme correspondence in the training set. Once trained, the network is tested on novel items and its performance is compared to human performance data.

Figure 2.

Specification of Plaut et al.’s simple feed-forward PDP model.

We implemented Plaut et al.’s model as a Java applet. As shown in Figure 3, the user interface of this computer simulation is roughly divided into three functional areas: (1) the text area in the middle displays the word corpus used for training or testing, (2) parameters related to network training are displayed on the left, and (3) functions related to network testing are displayed on the right. The two learning parameters, learning rate and momentum, have default values that allow for optimal learning. However, a student may also choose other values to see how the two parameters affect the learning process. The network is considered fully trained if it produces the correct pronunciations for all the words in the training corpus. The student then can test the model with novel items. In addition, the program allows the student to lesion the network and observe the effect on model performance. A lesion is simulated by randomly removing a certain percentage of the hidden units from the network and therefore is analogous to brain damage in human patients. Finally, the program displays the weight pattern within the entire network, from the input to the hidden layer, and again from the hidden to the output layer.

Figure 3.

User interface of the PDP model on word production

The PDP model was trained with a set of 2,998 monosyllabic words, and later tested with six additional sets: high- and low-frequency consistent words, high- and low-frequency inconsistent words, and consistent and inconsistent nonwords. The results reflected the major findings in the original Plaut et al.’s study: there was an interaction in naming latency between word frequency and word consistency. In addition, consistent nonwords were pronounced with a higher rate of accuracy than inconsistent nonwords. Overall, the simulation results are consistent with empirical research data from human subjects. However, due to the extensive calculations required during network training, running the simulation can take a considerable amount of time. To overcome this drawback, we provide the option of using a shorter training set of 200 words randomly selected from the original 2,998. With this condensed training set, it takes only a few minutes to train the network fully.

Because of the complexity of the subject matter, we provide written instructions along with the simulation to help students explore this program. The instructions lead a student, step by step, through various stages of the simulation, including model specification, network training, testing, and lesioning. At each stage, the student is required to set parameters for the program and pay close attention to certain aspects of the simulation results. We hope that, by doing so, this simulation program not only provides in-depth knowledge about the PDP approach on the specific topic of word production, but it also serves as an effective tool for teaching neural network modeling to students who have only minimal prior knowledge on this general topic.

ONLINE DEMONSTRATIONS

Interactive demonstrations are conducive to better learning than the simple presentation of facts. Moreover, they have a positive impact on students’ motivation. We developed 12 demonstration experiments illustrating various topics in language, attention, and perception. These programs demonstrate the tasks commonly employed in neuropsychological research, and require students be active participants in the learning process. As with the PDP model on word production, these demonstrations come with additional information, including theoretical background, instructions on how to carry out the task, as well as an explanation on the typical outcome. As a result, this self-contained style provides the necessary support for students’ learning. From a technical perspective, it also helps the programs gain independence from the text so that they can exist as “knowledge modules” that can be easily reused in a different context.

An example of online interactive demonstration is the visual search task illustrated below. This task is based on the paradigm developed by Treisman and her colleagues to study the mechanisms underlying visual attention (e.g., Treisman & Gelade, 1980). In this task, a subject is presented with arrays of multiple simple visual stimuli and is required to make a speeded response indicating whether or not a target is present. In the feature-search condition, the target differs from the distracters by a single feature such as color. In the conjunction-search condition, the target and distracters share some features so that the target can only be defined by a conjunction of features. The two search conditions result in different search functions. In the feature-search condition, search time shows little or no change as a function of the number of distracters. In the conjunction-search condition, however, search is hypothesized to be serial and subjects’ response time is a linear function of the number of distracters. Data from this visual search task and other behavioral methods anticipated the more recent neuroanatomical and neurophysiological work on the analytic aspects of visual perception.

The online visual search demonstration we developed presents two feature-search conditions (a red target among black distracters and an X target among O distracters) and one conjunction-search condition (a red X target with black Xs and red Os as distracters). The numbers of distracters in each condition are 4, 8, 16, and 32, respectively. The program randomly presents six blocks of stimuli, with two blocks in each search condition. Each block contains 32 trials with the target present on half of the trials. A student is asked to make a “yes” or “no” decision by pressing certain keys on the computer keyboard; the program records the response time. It takes a few minutes to complete the task. Once the student finishes the experiment, the program calculates the average response times across trials and plots the search function for each search condition (Figure 4, top). If the performance reaches a certain level of accuracy (i.e., 90%), the results from this individual are transferred to the server hosting this online resource and enter a database. The program then retrieves all the data in this database, collapses response times over subjects, and again plots the overall search functions (Figure 4, bottom), making it possible for the student to compare his/her own data against the overall performance.

Figure 4.

Results of the online visual search experiment

Based on the average performance of 75 subjects, the effects of search condition are clearly shown in Figure 4. In the two feature-search conditions, search time remains relatively constant as the number of distracters increases, and “yes” responses take essentially the same amount of time as “no” responses. In contrast, in the conjunction-search condition, search time increases linearly with the increase in number of distracters, and the slope for “yes” responses is only about half as steep as that for the “no” responses, suggesting that subjects employ a serial self-terminating search.

As evident in this visual search example, the collective performance on a task is more reliable than the results from an individual subject. Therefore, while students gain experiential knowledge by participating in an online demonstration, it is equally important for them to observe that the overall data pattern closely matches the descriptions in the text.

HYPERLINKS

Most of the interactive multimedia materials introduced above have been developed in our lab and we refer to them as internal links. While these materials are very effective for teaching, they require considerable resources, in both time and monetary terms, to develop. On the other hand, there are already many resources in behavioral neuroscience on the web. We experimented with incorporating these resources in our own writing, and refer to them as external links.

The chapter on autism illustrates the use of such external links. This chapter surveys four major aspects of this disorder, including its symptoms, diagnosis, treatments, and neurological underpinning. In the text, after the discussion of each topic, we included a block that displays the external links relevant to the materials just covered. Surrounded with borders, this link block is visually distinguished from the rest of the text, making the supplementary nature of these links evident. Inside each link block, the theme as well as the source of each link are clearly displayed so that the student has the necessary information for deciding whether to pursue a specific link. It is our hope that implementing links this way will give a student easy access to additional information without disrupting reading of the main text.

Because many of the interactive multimedia components we developed are stand-alone modules, they can be re-used in different contexts. Therefore, at the end of each chapter, we include a separate “Multimedia” section listing and linking to all the interactive/multimedia components.

USING THIS RESOURCE IN TEACHING

Although the current project was designed as a proof of concept on the viability of web-based instructional technologies, we soon realized that this online resource could be integrated into teaching practice in many different ways. One is to integrate the materials into lectures. For example, in a classroom setting, an instructor could use the Shepard’s Illusion program to give a quick and convincing demonstration of the power of this illusion. The streaming audio and video clips we developed for the language chapter are especially well suited to be included in a lecture. Second, the interactive multimedia materials can also be used outside of class to supplement the lectures and reading assignments. The visual search demonstration would be best used this way, as it takes several minutes to complete the task and so is not easily accommodated in class. Finally, these materials could be used as part of a distance-learning course. We believe that the interactivity provided by this online resource could help in maintaining a high level of motivation and partially compensate for the lack of in-person communication.

During the fall semester in 2002 this online multimedia resource was adopted in three undergraduate classes at Rice University. Students enrolled in Cognitive Psychology, Biopsychology, and Introductory Psychology classes read a chapter on specific topics (for the three classes, these were language, attention and perception, and autism, respectively) as supplementary material. Students evaluated the materials afterwards by filling out a survey. The critical question asked was how much the interactive multimedia content added beyond their paper textbook. The students rated the materials on seven-point scales ranging from “not at all” to “very much.” Table 1 summarizes the results from the three classes. It is evident that the vast majority of the students agreed that the activities added more than “somewhat” to their textbook.

Table 1

Proportions of students assigning each rating for the three chapters in the online multimedia resource

Rating	Language	Attention & Perception	Autism
Not at all 1	0%	0%	0%
2	0%	0%	0%
3	0%	16%	0%
Somewhat 4	8%	16%	14%
5	42%	26%	50%
6	33%	23%	22%
Very much	17%	19%	14%
# of Students	12	23	31

Comments from students concerning various individual applications corroborated the above results. The following quote is from a student enrolled in the Cognitive Psychology class: Another student from the same class wrote: It is encouraging to see that the students appreciated the values of the interactive multimedia elements in this resource.

“I find that video clips can do a good job explaining, especially in showing the experimental techniques. Just seeing how an experiment is done is so much easier than trying to figure it out by reading. Another benefit of the videos is that they are more memorable than most text. What’s more, they make it easier to stay focused and make progress, as opposed to text, which can be sleep inducing at times making it hard to make any progress. I found hyperlinks very useful. A lot of them would be good. Breaking up the text with headings, pictures, hyperlinks, or boxes makes the learning more ‘bite-sized’ and less daunting, as opposed to long spans of uninterrupted text.”

“I liked the structure of the lessons, and never before did I believe the McGurk effect until I tried it...neat project!”

CONCLUSIONS

The recent advance in web-based technology has stimulated innovative ways in teaching and learning. In this paper we demonstrated how streaming audio and video, computer simulations, online demonstrations, and internet links could be used to enhance undergraduate-level neuroscience education. The resource we introduce here emphasizes the role of the student as an active participant in the learning process, and uses interactive multimedia tools as a vehicle to reach that goal. The responses from students in three undergraduate classes have attested to the pedagogical values of this approach. Therefore, we encourage our colleagues to use this resource (available at http://psych.rice.edu/mmtbn) in their own teaching.

We currently plan to add more material to this online resource. Because we developed the contents as self-contained knowledge modules, the website is easily expansible. Therefore, we strongly encourage researchers and educators in this area to contribute their materials. Recently the Virtual Learning Lab project (http://virtuallearninglab.org) at the Stanford University Medical Media and Information Technologies (SUMMIT) has agreed to share two of their excellent interactive modules (cranial nerves and the visual system) with us. We are hopeful that others will share their materials as well. The readers of JUNE are clearly a great potential source of high-quality educational materials, and we encourage contributions.