An annotation of cuts, depicted locations, and temporal progression in the motion picture "Forrest Gump".

Here we present an annotation of locations and temporal progression depicted in the movie "Forrest Gump", as an addition to a large public functional brain imaging dataset ( http://studyforrest.org). The annotation provides information about the exact timing of each of the 870 shots, and the depicted location after every cut with a high, medium, and low level of abstraction. Additionally, four classes are used to distinguish the differences of the depicted time between shots. Each shot is also annotated regarding the type of location (interior/exterior) and time of day. This annotation enables further studies of visual perception, memory of locations, and the perception of time under conditions of real-life complexity using the studyforrest dataset.

Introduction

Cognitive neuroimaging research is moving towards studying brain behavior under conditions of real-life-like complexity, and motion pictures are being utilized with increasing frequency as stimuli in “neurocinematics” studies [1]. What sets motion pictures apart from other dynamic naturalistic stimuli is that they are more likely to evoke time-locked response patterns in a larger portion of the brain while retaining synchrony across multiple individuals who are experiencing the same movie [2, 3]. One likely reason for this is the structure of movies. They are typically not prolonged, contiguous captures of an environment from a first person perspective, but rather they are carefully assembled, using “cuts”, from hundreds of short sequences shot from a variety of perspectives [4]. These cuts are sharp discontinuities in the sensory input that require all viewers to re-assess the depicted environment in order to perform a cognitive re-orientation in fictional space and time. This re-orientation can be complex and involve a large bandwidth of cognitive processes: interpretation of contextual cues for detection of familiar settings, retrieval of prior knowledge from memory, discovery of change in locales and depicted characters. Consequently movies, and their cuts in particular, offer an excellent instrument to study complex, concurrent, real-life cognition.

In this study, we focus on spatial and temporal viewer re-orientation, and, to this end, describe changes in depicted location and time for all cuts in the motion picture “Forrest Gump”. This movie is the core stimulus of the studyforrest project ( http://studyforrest.org). Two fMRI datasets are publicly available: 1) participants listening to an audio-movie version [5] and 2) a subset of the original participants watching the audio-visual movie with simultaneous eye tracking [6]. Additional imaging data and movie annotations are available [7, 8], including an individual localization of the parahippocampal place area [9] that has been implicated in spatial perception and scene processing [10].

This new annotation extends the available knowledge about the structure of this complex natural stimulus and enriches the overall studyforrest dataset. These data can be used to investigate the formation of a representation of viewer location and the perception of (speeded or negative) temporal progression in the movie stimulus. For any study focusing on other aspects of real-life cognition, these new data can serve as additional confound measures describing key properties of major building blocks of this movie stimulus.

Materials and methods

Stimulus

The annotated stimulus was a slightly shortened ( ≈2 h) version of the movie Forrest Gump (R. Zemeckis, Paramount Pictures, 1994) with dubbed German soundtrack that is identical to the audio-visual movie annotated in 8. Further details on this particular movie cut, and how to reproduce it from commercially available sources, are available in 6.

Annotation procedure

First, the movie was explored by two people, one of whom has an academic background in documentary film making, in order to generate a consistent list of labels for depicted and recurring locations.

Subsequently, the actual annotation was performed by the first author using a multi-pass strategy. The movie was manually inspected frame-by-frame to determine the location of cuts (using the video editor Shotcut v16.02.01). For each new shot (sequence between two cuts), a number of properties (described below) were discerned and entered into a table. A total of four passes were performed by the same observer in order to validate the annotation.

Data legend

The annotation table contains one line per shot and seven columns: 1) a shot’s start time, 2) a label for the shot’s major location, 3) a label for the setting within the location, 4) a label for the locale within the setting, 5) a flag indicating an interior or exterior setting, 6) a label for the type of temporal progression with respect to the previous shot, and 7) a label for the time of day. Further details are provided in the following sections. The respective column header labels are given in parenthesis.

Shot start time ( time)

A shot’s start time is defined as the onset time of the first video frame of a shot after a cut. Time stamps a provided in seconds of movie onset.

Location

Location was coded with three labels, each describing the depicted scenery with an increasing level of detail.

Major location ( major_location) provides a coarse identification at the level of a town, county, or region where the respective story is taking place. Examples are: “Greenbow” or “Vietnam”.

Setting ( setting) further details the location by distinguishing places at the same major location, but are not in direct sight of each other. For example, Forrest Gump’s elementary school and the high school’s football field are both in Greenbow, Alabama but are not part of the same setting. A switch from one setting to another is typically synonymous with a transition to a new scene in a cinematographic sense. If the camera switched settings within a scene, the annotation deviates from the screenplay to make explicit the switch to another setting.

Locale ( locale) subdivides settings into distinguishable locales. Indoors, a locale is congruent with a particular room enclosed by walls. For example, Forrest Gump’s bedroom, the corridor downstairs, and the corridor upstairs are three different rooms inside the Gumps’ house (setting) on the Gumps’ property (major location). Outdoors, locales were distinguished when they were separated by a logical boundary, substantial distance, or shared no discernible landmarks. For example, the glade at the river and the location of the wounded Bubba are two different locales in the embattled jungle (setting) in Vietnam (major location). A locale’s label is identical to its setting label when only one locale is depicted for that setting.

Interior or exterior ( int_or_ext)

This flag indicates whether a particular location is an open (“ ext”) or enclosed space (“ int”), such as a building or a vehicle.

Temporal progression ( flow_of_time)

This label indicates the depicted progression of time between the previous and the current shot. Four categories were distinguished: “ -” labels a flashback, or jump into the past, independent of the temporal distance; “ 0” indicates no noticeable break in the ongoing stream of time, for example a sole change of viewing perspective; “ +” represents noticeable jumps in time, ranging from several seconds to about one or two hours; and lastly “ ++” marks major time jumps from several hours (e.g. night vs. day) to several years.

Time of day ( time_of_day)

This flag indicates whether a scene is at least partially illuminated by sunlight. Consequently, daytime and twilight (early sunrises and late sun settings) are labeled as “ day”. If sunlight is entirely missing, the time of day is coded as “ night”.

Dataset content

The released annotation is a single, text-based, comma-separated-value (CSV) formatted table ( ).

Each row corresponds to a shot in the the movie

Click here for additional data file.

The source code for all descriptive statistics included in this paper is available in (Python script).

Click here for additional data file.

Dataset validation

To check for human error in the cut time annotation, timings were compared to the results of an automatic detection algorithm and any deviation was manually verified.

In summary, the shortened version of the movie comprises 870 shots (duration: min=0.48 s, max=151.08 s, median=4.92 s, SD=10.86 s). There are 612 shots depicting outdoor locations and 256 interior shots. Most shots take place during daytime (706 day vs. 162 night). The majority of cuts involve no noticeable discontinuities of depicted time (640), but there are 61 small and 135 large time jumps, as well as 32 flashbacks.

Table 2 provides information on the portrayal of unique locations in the movie.

Table 2.

Descriptive statistics for all three levels of location annotation.

Number of shots indicates the total number of shots in the movie for any particular location. Number of consecutive shots indicates how many shots are shown between two location changes at the respective level. Times revisited indicates how often a location reappears in the movie after it was depicted for the first time.

	major locations				settings				locales
number of unique	21				95				147
	min	med.	mean	max	min	med.	mean	max	min	med.	mean	max
number of shots	1	14	41.3	195	1	2	9.1	104	1	2	5.9	58
number of consecutive shots	1	2	7.3	70	1	2	4.7	69	1	1	3.2	32
times revisited	0	1	4.7	21	0	0	0.9	20	0	0	0.9	20

Data and software availability

F1000Research: Dataset 1. CSV table with of depicted locations, and temporal progression in the motion picture “Forrest Gump”, 10.5256/f1000research.9536.d134823 [11]

F1000Research: Dataset 2. Python script to compute all descriptive statistics presented in the Data Note manuscript from the released annotations, 10.5256/f1000research.9536.d134824 [12]

This data note labels cuts in the motion picture "Forrest Gump" with starting time, three levels of location information, indoor/outdoor information, temporal progression, and the time of day. An automatic detection algorithm was also used to validate the labeling. It is very interesting as it provides an important tool for anyone who may be interested to study perception, attention, and other cognitive functions. I would like to thank the authors for their efforts and sharing this wealth of data.

The introduction and data description including materials and methods are all clearly written. I approve this article.

Minor points:

It might be informative to provide to readers that among the total number of deviation of human labeling and automatic detection, how many were human errors and how many were computer errors.

Page 3, last sentence: typos?

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

This data note describes an annotation dataset consisting of time indices and annotations describing location and temporal information for each identified shot in the movie Forest Gump. This dataset of movie annotations is intended to be used with the associated functional neuroimaging datasets of participants viewing the movie Forrest Gump. All in all, I think this is an excellent addition to the growing empire that is StudyForrest, one that clearly represents an enormous amount of effort.

I have only minor comments:

The timing seems to be a bit off relative to the previously released dataset of scenes in the studyforrest GitHub repository. Presumably the annotation for shots and scenes should line up at scene starts but there appears to be consistent offset of about 12ms. For example, the last scene (“School bus stop”) starts at 6944.96 in the scenes.csv and 6944.84 in the attached dataset for shots in this paper. Moreover, the shots in this annotation don’t quite line up with the shots.csv on the github repo. The ~12ms offset is too large to be a single frame. It appears the authors switched from Advene to Shotcut for movie segmentation and annotation, perhaps therein lies the source of the mismatch?

Could the authors expatiate on their method of identifying shots and cuts? If memory serves, in previous datasets they used an automated method to identify shots that was subsequently edited by hand. In this dataset, it appears all shots were identified by hand.  Where all cuts identified? Or are there special cases were two cuts appearing in close succession were considered a part of one shot? For instance, in an action heavy scene you could presumably get an overabundance of cuts, but that level of granularity isn’t really useful (nothing changes) and potentially these could be combined into a single shot. If every cut was indeed identified and annotated, then my sincere condolences to the coder!

Although it is extremely generous of the authors to provide python code for generating descriptive data and associated figures, I’ve examined this code file and unfortunately this reviewer simply cannot support the premature use of Python 3… You can pry 2.7 from my cold dead hands. ;)

Finally, I would like to again thank the authors for openly sharing this wealth of data with the community. These annotations and the associated imaging data represent a generous sharing of valuable resources, one that I have no doubt will be useful to many researchers interested in the neuroscience of naturalistic cognition.

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

This Data Note presents a very useful (and labor intensive!) complement to the studyforrest dataset, providing additional annotations that can be used for data analysis: the timings of all the cuts in the movie, together with the depicted location and temporal progression for each transition. There are of course many, many other features that can be labeled in this movie, yet this particular set of features is useful on its own and will add to the bank of features already available. I have no major changes to suggest.

Minor comments:

would the authors consider publishing the code for the automated detection routine that they used to cross-check manual cut detection?

a few cuts are annotated as locale "unknown_...". Is there no label that can be used rather than unknown? (I don't currently have the movie available to check these four cuts, at time stamps 4544.32, 4554.96, 4559.32 and 4561.56).

Very minor comments:

the second sentence in the Introduction, "What sets motion pictures apart from other dynamic naturalistic stimuli is that they are more likely to evoke time-locked response patterns in a larger portion of the brain while retaining synchrony across multiple individuals who are experiencing the same movie", is rather unclear to me, in particular the use of "while" which indicates a contradiction -- why would time-locked responses in large portions of the brain not be synchronous across individuals?

typos:

- in Materials and Methods/Shot start time (time) : correct to "Time stamps *are* provided in seconds *from* movie onset".

- caption of Dataset 1: correct to "CSV table with depicted locations and temporal progression [...]"

table 1: for consistency, maybe the column headers should be the same as in the actual csv file?

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Table 1.

Example annotations for four shots at the beginning of the movie.

Note that table headers do not literally correspond to column headers, see Data legend (ToD: time of day).

time	major location	setting	locale	int/ext	flow of time	ToD
300.56	Greenbow Alabama	doctor’s office	doctor’s office	ext	0	day
311.96	Greenbow Alabama	main street	crossroads	ext	+	day
318.28	United States	flashback countryside	flashback countryside	ext	-	day
343.04	Greenbow Alabama	main street	in front of barbershop	ext	++	day