Perceptual elaboration paradigm (PEP): A new approach for investigating mental representations of language.

To examine hemispheric differences in accessing a mental representation that embodies perceptual elements and their spatial relationships (i.e., perceptual elaboration and integration), we developed a cross-modal perceptual elaboration paradigm (PEP) in which an imagined percept, rather than a propositional concept, determined congruency. Three target image conditions allow researchers to test which mental representation is primarily accessed when the target is laterally presented. For example, the "Integrated" condition is congruent with either propositional or perceptual mental representations; therefore, results from both hemifield conditions (RVF/LH vs. LVF/RH) should be comparable. Similarly, the "Unrelated" condition is incongruent with either propositional or perceptual mental representations; therefore, results from both hemifield conditions should be comparable as well. However, the "Unintegrated" condition is congruent with the propositional mental representation but not the perceptual mental representation. Should either hemisphere access one representation initially, differences will be revealed in either behavioural or electroencephalography results. This paradigm:•is distinct from existing paired paradigms that emphasize semantic associations.•is important given increasing evidence that discourse comprehension involves accessing perceptual information.•allows researchers to examine the extent to which a mental representation of discourse can embody perceptual elaboration and integration.

Method details

Background

We developed a cross-modal perceptual elaboration paradigm (PEP). In this context, perceptual elaboration refers to the individual's active, imagined elaboration of lexical stimuli [9]. We presented participants with an auditory passage, followed by a visual target. Existing amodal and cross-modal paired paradigms have used word pairs (e.g., [5], [6]; Meyer and Schvaneveldt, 1971), sentence-word pairs (e.g., [3]), sentence-picture pairs (e.g., [4], [11], [13]), picture-picture pairs (e.g., [1], [2], [7]), or picture-sound pairs (e.g., [8], [10], [12]). These paradigms emphasize semantic congruency between prime(s) and target. We sought to develop a paradigm in which an imagined percept, rather than a propositional concept, determined congruency. To that end, we created auditory passages in which a meaningful construct could not be realized until all verbal instructions directing the production of a mental image were followed and (in most cases) a target image was revealed. Overwhelming reports from participants that a concept was often only realized after the match target appeared provides support for the latter. The PEP emphasizes perceptual congruity over semantic congruity. It is the first paired paradigm to do so and given increasing evidence of perceptual influence on discourse, it may be useful to researchers hoping to examine the role of perceptual inferencing during higher-order language comprehension.

Stimuli construction

A Sony Electret Condenser Microphone was used to record 168 auditory passages in an English female voice at a natural intonation (mean transmission=16.8 s). To ensure auditory stimulation was monophonic (unidirectional) we digitized passages (22.05 kHz, 16-bit resolution) using Cool Edit 2000 (Syntrillium Software Corp., Phoenix, AZ). Each passage varied in length and contained two to three sentences. Each sentence provided propositional information and instruction for the imagined construction and integration of perceptual components. For example, “Draw a long teardrop shape. Colour the teardrop in green. Now, draw a short brown vertical line extending down from the base of the teardrop.” Successful construction and integration would yield a meaningful mental representation (in this case, a green leaf on a brown stem). Each sentence within the passage represented a single piece of visual information (a single element).

There were three target image conditions: Integrated, Unrelated, and Unintegrated. The integrated image matches the correctly positioned (i.e., integrated) perceptual elements described in the preceding passage (e.g., a green leaf with a brown stem). Both perceptual and propositional representation are congruent. The unrelated image is a meaningful object but contains none of the elements described in the preceding passage; therefore, it is not congruent with either perceptual or propositional representations. Importantly, integrated and unrelated targets do not share elements. For example, a brown vertical line could not appear in an unrelated target image if a ‘brown vertical line’ was mentioned in any sentence within the preceding passage. For this reason, each passage has a unique set of target images that are not interchanged with other passages. The unintegrated image contains all the individual elements (e.g., a teardrop, the colour green, a brown vertical line) but omits spatial relationships such that the elements are not integrated into a meaningful whole. The perceptual representation is not congruent; however, the propositional representation may be because the individual elements (e.g., brown vertical line) are present.

We created one integrated, one unrelated, and one unintegrated target image for each auditory passage using GNU Image Manipulation Program 2.0. Target objects were centred within a grey (RGB 128, 128, 128) square, framed by a thick black border to distinguish it from the background.

Stimulus validation

To validate match representativeness nine volunteers were asked to read each passage and construct a mental representation of the described object. The target image was then revealed to them. Each volunteer rated the likeness between their mental representation and the image using a 4-point scale (0 = not a match, 3 = a match). Passages whose match targets were rated low in representativeness (2 or below) were removed, leaving 90 passages. In addition to this, in a different experiment, 34 English-speaking, right-handed participants (6 males), with a mean age of 25.4 years (± 8.8 years), completed a post-test questionnaire to indicate which objects had been described in the auditory passages. They were shown a total of 16 images (11 matches and 5 mismatches). Participants circled either “yes” or “no” for each image depending on whether they recalled constructing the image during the experiment. Mean accuracy was 90% (± 7%).

See Appendix A for the complete list of passages. WAV files can be provided upon request to the authors. Target images corresponding with the passages in Appendix A appear in Appendix B.

Procedure

During the experiment, targets appeared in either the Right Visual-Field (RVF) or Left Visual-Field (LVF), hence there were six conditions in the 2 × 3 repeated-measures design (RVF-Integrated, RVF-Unrelated, RVF-Unintegrated, LVF-Integrated, LVF-Unrelated, LVF-Unintegrated). Each experimental session contained 90 trials, lasting between 50 and 60 min. Trials were divided into blocks of 6. There were 15 blocks. A custom written E-Prime (Psychology Software Tools) script delivered the stimuli and randomized pair presentation such that each participant only heard each passage once. Each passage was paired with one of the three randomly selected target images (targets). The only constraint was that each block contained one trial from each condition. Each participant saw a target only once. Therefore, participants did not necessarily view the same set of targets although all participants heard the same auditory passages.

A chin rest stabilized each participants’ head 60 cm from the LCD computer monitor (24-in, 1920 × 1080 pixels, and 100 Hz refresh rate). From this distance, targets subtended 4° of visual angle. We asked participants to focus on the always-present white fixation cross. The length of each bar of the fixation cross was 0.6° of visual angle; the width was 0.03° of visual angle. The background was grey (RGB 128, 128, 128). In-ear-headphones delivered auditory passages binaurally.

The experiment began with verbal and written instruction (i.e., ‘On each trial you will listen to a few short sentences. During this time, you should focus your gaze on the central cross. A picture will then be presented briefly to either the left or right side of the cross. Make sure you don't move your eyes from the central cross; that is, don't try to look directly at the picture. Your task is to decide whether the picture matched the description in the sentences or not. Press the ‘1’ key if it matches; press the ‘2’ key if it does not. Please answer as quickly and accurately as possible. You will receive rest breaks throughout the session. If you understand the instructions, please press the Spacebar to begin.’). Thereafter, each trial commenced with a written message on the monitor, ‘Ready for the next trial? Press any key’. Participants initiated each trial by pressing any key on a high-speed Direct-In response box. After a short 500 ms interval, the auditory passage commenced. To allow participants enough time to correctly integrate all imagined elements, a 1500 ms inter-stimulus-interval (ISI) separated the end of the passage and the onset of the target.

Targets appeared for 300 ms and the inner boundary of the target was 2.5° of visual angle left or right (depending on the condition) from centre. Target offset initiated a 2000 ms response window and participants made a match or mismatch response (1 = match, 2 = mismatch) with either their left or right hand (counterbalanced across participants) using the response box. Immediately after a response or at the end of the response window, participants received visual written feedback on response accuracy (i.e., ‘Correct’, ‘Incorrect’, ‘Too Slow’). Conclusion of feedback initiated the next trial.

Fig. 1 depicts all six possible trial types given a single auditory passage. For illustrative purposes, all responses are the same (i.e., the participant selects match).

Fig. 1

Schematic illustration of the six possible trials within the perceptual elaboration paradigm (PEP), given a single auditory passage (italicized within the black rectangle). Each trial is initiated by the participant. Thereafter a white fixation cross appears. After a brief 500 ms interval, the auditory passage (symbolized by the black speaker) commences. The length of the passage varies between 14 and 30 s (auditory stimuli are transcribed in Appendix A). A 1500 ms inter-stimulus-interval (ISI) separates the offset of the passage and the onset of the target image. The target appears for 300 ms in either the right visual-field (RVF) or left visual-field (LVF) and it is either Integrated, Unrelated, or Unintegrated. A 2000 ms response time window begins from the offset of the target. Once a response is detected or at the end of the response window, participants receive immediate feedback on their performance. For illustrative purposes we assume the response for all trials was ‘match’, in which case the RVF-Unrelated (C), LVF-Unrelated (D), RVF-Unintegrated (E), and LVF-Unintegrated (F) feedback would be ‘Incorrect’; whereas, the RVF-Integrated (A) and LVF-Integrated (B) feedback would be ‘Correct’ For illustrative purposes, text, fixation cross, and target are not to scale.

Declaration of interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Specifications Table

Subject Area	• Psychology
More specific subject area:	Language, Perception
Method name:	Perceptual elaboration paradigm (PEP)
Name and reference of original method	As a novel paradigm, we draw links to existing priming paradigms that are similar (see background), but these paradigms were not the basis of the PEP.
Resource availability	Johnston, D. (2000). Cool edit 2000. Phoenix: Syntrillium Software Corporation. http://www.oldversion.com/windows/cool-edit-pro/
	Psychological Software Tools. (2018). E-Prime. Pittsburgh, PA: Electrical Geodesics Inc. https://pstnet.com/
	The GIMP Team. GNU Image Manipulation Program. The GIMP Website. https://www.gimp.org/