Musical Collaboration in Rhythmic Improvisation.

Despite our intimate relationship with music in every-day life, we know little about how people create music. A particularly elusive area of study entails the spontaneous collaborative musical creation in the absence of rehearsals or scripts. Toward this aim, we designed an experiment in which pairs of players collaboratively created music in rhythmic improvisation. Rhythmic patterns and collaborative processes were investigated through symbolic-recurrence quantification and information theory, applied to the time series of the sound created by the players. Working with real data on collaborative rhythmic improvisation, we identified features of improvised music and elucidated underlying processes of collaboration. Players preferred certain patterns over others, and their musical experience drove musical collaboration when rhythmic improvisation started. These results unfold prevailing rhythmic features in collaborative music creation while informing the complex dynamics of the underlying processes.

1. Introduction

Across cultures in history, music has always been a universal part of human life [1]. Whether pursued as a form of art to express ourselves or as a therapeutic tool to address emotional, cognitive, physical, and social needs, we are all familiar with the nature and value of music [2]. However, little is known about the process of creating music, even in simple rhythmic improvisation comprising a few notes.

Igor Stravinsky stated that a musical form is “far closer to mathematics than to literature—not perhaps to mathematics itself, but certainly to something like mathematical thinking and mathematical relationships” [3]. For example, Ernîo Lendvai identified the presence of Fibonacci numbers and golden ratios in many of Béla Bartók’s pieces [4]. Musical structures can be visualized and quantified by studying self-similarity over time from recurrent patterns [5,6,7] or constructing networks on the basis of the pitch and duration of notes [8]. Interestingly, recurrence-quantification analysis revealed that compositions of Bach’s inventions and sinfonias are more complex than a mere Markov process [9]. Predictably, the mathematical elements of music can be uncovered through machine learning, which could be used to detect the temporal structure of music [10], and even to compose music [11,12].

From mathematically principled analysis of musical structures within a single piece, one may attempt to compare pieces by different musicians. For example, a popular approach to the comparison of musical structures is to measure the distance between recurrence plots constructed on musical features [13]. Although the approach could, in principle, be extended to the study of musical collaboration, the literature in this field is scarce. To the best of our knowledge, the application of recurrence-quantification methods to musical collaboration is limited to Walton et al. [14], who evaluated the dependency of two acoustic signals in collaborative music creation through cross-recurrence quantification. The area of spontaneous synchronization of beats shares some similarities with musical creation [15,16], but music is generally more complex than synchronization on an emerging pattern.

A unique setting to experimentally study musical collaboration is improvisation, where music spontaneously emerges from unstructured dynamical interactions between players who embrace a sequence of decisions toward their sense of music [17]. Without rehearsals or scripts, music can be created through cognitive efforts that involve short- and long-term memory [18], and communication based on calls and responses [19]. In this context, understanding the processes and outcomes of musical improvisation may offer a deeper insight into human nature of musical cognition. However, little effort has been directed toward the application of mathematically principled approaches to elucidate how people interact during improvisation and what kind of music they create.

In this study, we investigated processes and outcomes of collaborative musical improvisation. We focused on situations where people without professional training create music together through rhythm, which constitutes a fundamental element of music that humans are wired to appreciate [20,21]. In the experiment, participants with various musical expertise were randomly paired and asked to freely create music without rehearsals or scripts. Facing against each other in a room, each participant was provided with a velocity-sensitive drum pad with only two marimba notes. In two improvisation sessions, participants were allowed to interact only through the music they heard and created, thereby eliminating visual cues that may otherwise contribute to musical collaboration [22,23]. In this sense, the outcomes of the collaboration were also the means that supported the processes of collaboration through the sharing and transfer of information.

By examining the sound data collected in the experiment, we studied the rhythmic patterns of improvised music through recurrence quantification, which offers a mathematically principled approach for studying musical structures [7,13]. From the percussive sound produced by the two players, we formed a symbolic time series where each symbol identified a specific ordinal pattern in the amplitude of consecutive sound samples. Each time series was examined through the lens of recurrence-quantification analysis to create symbolic-recurrence plots that captured the recurrences of different ordinal patterns in a multidimensional recurrence plot [24,25]. The more the points that populated the recurrence plot, the more repetitive the rhythm was and the higher the symbolic-recurrence rate was. Entropy on symbolic recurrences was used to quantify preference for specific musical patterns that emerged during collaboration. Hence, low entropy values indicated a preference of the players for specific rhythmic patterns, while larger entropy values pertained to a less marked preference for patterns over others. We hypothesize the emergence of recurring patterns with a potential preference for specific musical patterns, as found in human solo drumming [26].

To elucidate the interaction between the players, we performed multivariate recurrence analysis on the two time series of the sound amplitudes produced by the players within each pair. From these time series, we measured the amount of information that was shared and transferred between the two players through salient information-theoretic metrics on joint symbolic-recurrence plots [25,27]. Mutual information was used to quantify the association between the rhythmic patterns of the two players, and transfer entropy was employed to measure the responsiveness of the players to their partners. We hypothesize that the process of collaboratively and spontaneously creating music is supported by strong information sharing and transfer between the players.

To explain the variation in the degree of interaction between the players, we inspected the expertise of the participants in playing music, acquired through independent surveys. Following the mental model on teamwork that emphasizes the importance of individual experience and skills on the outcome of collaboration [28], we hypothesize that the extent of information sharing and transfer within a pair is explained by the musical expertise of the pair.

2. Materials and Methods

2.1. Experiment Setup

The instruments provided to the participants were MIDI controllers with pads (nanoPAD2, KORG, Melville, NY, USA), digitally programmed with samples of a marimba sourced from the public-domain library of the University of Iowa Electronic Music Studios. The MIDI controllers fed velocity-sensitive information directly to the recording interface via two sets of adjacent rubber pads. The recording interface was a standard digital-audio-production application (REAPER, Cockos Incorporated, New York, NY, USA). This platform was chosen because of the customizable nature of the interface, audio-routing capabilities, and compatibility with audio drivers and the MIDI instruments. The recordings were taken at 44,100 Hz on a Windows laptop augmented with an external USB sound card in addition to inbuilt audio capabilities.

Within the REAPER interface, incoming MIDI signals were rendered as the sampled marimba audio signal and sent to both participants’ sets of headphones (ATH-AVC200, Audio-Technica, Tokyo, Japan). The MIDI controller was configured to play a set of two notes. MIDI recordings were performed in REAPER and saved as REAPER project files. Data were exported as WAV audio files for listening and further analysis.

Two players controlled four notes in an F-major seventh chord (F, A, C, and E), one of the traditional chords in Western music that is frequently utilized within an improvisational context [29]. To promote collaboration, each player could only make a partial chord on their own, which would then be extended with the addition of their partners’ complementary notes. Specifically, one player was assigned the fifth and seventh (C and E) of the chord, while the other player was assigned the first and third (F and A). This selection was also helpful for players to discriminate their sound from that of their partners. Although F and E are dissonant, our participants showed preference for these keys over the middle ones (A and C; see Appendix A).

2.2. Data Collection

Participants were recruited from the New York University community in the Brooklyn campus, NY, USA. Each trial consisted of a tutorial followed by two experiment sessions. The tutorial was based off a classic one-note call-and-response exercise, toward introducing a standardized basis of collaboration and improvisation. Two participants sat in the same room facing away from each other (Figure 1). The headphones of the participants were connected to two distinct audio outputs, and audio information was isolated between participants during the tutorial. Through the headphone, the participants heard a short series of measure-long rhythms, each followed by two measures of rest, progressively increasing in complexity. The experimenter instructed them to mirror what they heard exactly by using their respective base notes. Then, the participants were exposed to the same series of rhythms through their headphones and instructed to improvise a response instead of merely repeating. The tutorial ended with a 30 s practice session where they could use both the notes while listening to a prerecorded drum backing track.

Figure 1

Study flow. (A) Two participants sit facing against each other in a same room to create music together by improvising using drum pads while acoustically communicating with each other through headphones. (B) Sound amplitudes extracted (2 min × 2 sessions, excluding first 15 s with a backing track from each session). (C) Music recurrence plots created from sound amplitudes by symbolizing following ordinal patterns. Colored areas of a recurrent plot indicate recurrence of a symbol at time t and s, with colors representing which symbol recurred. Portion of the recurrence plot shown for clarity.

During a short intermission after the tutorial, participants were asked to complete a survey regarding their musical expertise. Specifically, they filled out their experience in playing music with others on the Likert scale (0: never, 1: rarely, 2: sometimes, 3: frequently, and 4: always) and the duration of practicing music in years.

Upon completing the surveys, the participants began the experiment that consisted of two improvisation sessions, each of 2 min in duration. Before commencing the session, the experimenter instructed the participants as follows: “Now you will be playing together and collaborating in your improvisation. Feel free to experiment, but remember to collaborate”. Different from the tutorial, the participants’ headphones were connected to a single audio outport on the USB sound card by using an audio-splitter device so that they would hear the music they collaboratively created. The same drum backing track used in the tutorial was played for the first 15 s of each session, providing a starting tempo for the participants. After these initial 15 s, there was no accompaniment, and the participants improvised for the duration of each 2 min session. Between sessions, the notes controlled by each participant were swapped to randomize the key assignment. In total, we collected data from 30 pairs. The experiment was approved by the Institutional Review Board of the University (IRB-FY2017-898).

2.3. Symbolic-Recurrence Quantification

Given scalar time series of T samples, we constructed the symbolic time series of symbols on the basis of ordinal patterns of length m, , where , is the phase-space vector at time t, and is symbolization mapping. For example, if , we had an alphabet of six symbols, each identifying a specific pattern for three consecutive readings in the time series from a sequence of three numbers that continuously decreased to three that instead steadily increased. From the symbolic time series, we assembled a symbolic-recurrence plot [24] to encode the recurrence of each symbol of the alphabet in time (Figure 1), that is, The symbolic-recurrence rate of generic symbol was computed by counting the total fraction of recurring symbol, that is, This quantity estimated the probability of recurrence of . By summing these partial rates, we calculated symbolic-recurrence rate that measured the overall extent of recurrence without discriminating whether it pertained to few or many symbols that were repeating in time. For reference, an independent identically distributed time series would have a symbolic-recurrence rate of .

To afford further recurrence quantification in the phase space, we examined the entropy of the symbolic-recurrence plot [27]. By exclusively focusing on the portion of the recurrence plot that encoded recurrence, we estimated the probability of recurrence of generic symbol as the fraction of its recurrences over the total number of recurrences, that is, Hence, the entropy of time series upon symbolization is where we used the logarithm to base 2 so that we measured entropy in “bits.” Again, for an independent identically distributed time series, entropy should be .

When looking at two or more time series, we can study a multivariate form of the symbolic-recurrence plot in which we examine a phase-space vector in the higher dimensional space given by the Cartesian product of the original phase spaces. In this vein, the symbolic-recurrence plot of two time series with ordinal patterns of length m tracks symbol pairs. From this symbolic-recurrence plot, we computed the mutual information between time series and upon symbolization as where is the joint entropy of and upon symbolization. Similarly, we could examine the symbolic-recurrence plot of multivariate time series to compute transfer entropy on symbolic recurrences as where is the entropy of conditional to , and is the entropy of conditional to , upon symbolization. Analogous to transfer entropy on two univariate time series [30], transfer entropy on symbolic recurrences measures the reduction of uncertainty in predicting the future state of one time series given the current state of the other time series within a probability space constructed over symbolic recurrences [27]. With respect to focal time series , this value quantified the directional influence of other time series .

Throughout the study, we downsampled the time series at a rate of 150 ms, mirroring typical auditory reaction time [31]. This yielded a total of samples for each trial. To capture the complexity of the time series while balancing the limited length of the time series, we used for symbolization. In the appendix, we illustrate the robustness of these choices by examining the cases of downsampling at 100 ms with and downsampling at 150 ms with .

2.4. Analysis

To test whether musical improvisation brings about an emergence of recurring patterns with marked preference for certain patterns, we compared the and entropy of the music created by pairs against random values. To that end, we created a new dataset of 30 pairs by randomly pairing individual sound data of the 60 participants for each session. Hence, the new dataset represented music created by pairs where players within a pair could not acoustically communicate with each other. The observed mean of each variable was compared against the corresponding null distribution of the mean, which was generated by repeating the shuffling process 20,000 times. When the observed mean fell outside of a 95 percentile of null distribution (two-sided), we deemed that the variable was significantly different from a random one.

Similarly, we tested whether players exhibited a greater extent of information sharing and transfer within pairs by comparing the mean mutual information of 30 pairs and the transfer entropy of 60 players against random values. To that end, we generated the null distribution of the mean of each variable for each session by randomly pairing individual sound data and repeating the process 20,000 times in the same way as described above. When the observed mean fell outside of a 95 percentile of null distribution (one-sided), we deemed that the variable was significantly different from a random one.

Further, we investigated the difference between sessions and consistency within pairs in the musical characteristics and extents of information sharing and transfer. Specifically, , entropy, mutual information, and transfer entropy were compared between sessions by using a paired t-test. Similarly, within-pair consistency in these values were investigated using Pearson’s correlation.

Next, we investigated the musical expertise of each player as a possible factor for variation in the extent of information sharing and transfer among pairs. For information sharing, we characterized pair traits with the sum of experience in playing music with others (score 0–8) and the difference (score 0–4), as well as the sum and difference of music-practicing duration (in years). Mutual information was fitted into a generalized linear model with gamma error distribution and a log link. The interaction terms of the sum and difference were also included in the model. For information transfer, transfer entropy that focal players received from their partners was fitted into a generalized linear model, with the musical expertise of a focal player and their partner as explanatory variables. The model was specified with gamma error distribution and a log link. The interaction terms of focal players and their partners’ musical expertise were also included in the model.

All data analyses were performed using base R ver. 3.6.0 [32], R package ‘seewave’ ver. 2.1.4 [33], ‘car’ ver. 3.0-3 [34], and Python package ‘NumPy’ ver. 1.17.2 [35].

3. Results and Discussion

3.1. Symbolic-Recurrence Quantification of Music

For each of the two improvisation sessions, we characterized the music created by each pair in terms of symbolic-recurrence rate () and entropy. We observed an of (mean ± standard deviation, pairs) in the first session, and of in the second one. The mean of was significantly greater than chance (two-sided permutation test, for both sessions; Figure 2). Overall, pairs showed consistent values of between the sessions (Pearson’s correlation, ), but values were smaller in the second session (paired t-test, ). The recordings of the experiments with the lowest and the highest are available at https://github.com/shinn1/music.

Figure 2

Mean observed recurrence metrics of the music created by a pair against null distributions: music was characterized by rhythmic patterns, and players preferred some patterns over others. First session: (A) symbolic-recurrence rate and (B) entropy. Second session: (C) symbolic-recurrence rate and (D) entropy. Vertical red lines, observed means; grey areas, null distributions of means.

The entropy of the music was bits in the first session ( pairs), and bits in the second one. The mean of the entropy was significantly smaller than chance ( for both sessions; Figure 2). Entropy was correlated between sessions (), although pairs showed greater values in the second session ().

An empirical study demonstrated common structural regularities in rhythm when humans solo-play a drum [26]; our results revealed the emergence of such regularities in collaborative music creation. Improvised music collaboratively created by our participants was characterized by repetitive rhythmic patterns with marked preference for specific patterns over others, indicated by higher symbolic-recurrence rates and lower entropy. Considering that the origin of music is rooted in social activities [36,37], humans may have an innate inclination to rhythmic patterns that are easy to learn and memorize [26]. Indeed, people are more likely to perceive rhythmic patterns as a division of sound duration by small integers [38]. Cross-cultural similarities in rhythmic patterns [39,40] further support the possibility. Unlike solo music, however, musical collaboration through improvisation requires the social exchanges of musical motifs with dynamic responses and adjustments [17,19,41]. The need for such complex interaction might be the reason why children are incapable of performing collaborative improvisation in music until later in life [42]. Our results confirmed that adults are able to exchange musical motifs through acoustic cues toward collaboratively creating music.

3.2. Information Sharing and Transfer on Symbolic Recurrence

How players shared information with each other and how they responded to their partners were measured through mutual information and transfer entropy on symbolic recurrences, respectively. For each trial, we computed one value of mutual information and two values of transfer entropy (from partner to focal player, corresponding to the responsiveness of the focal player). We observed mutual information of bits in the first session ( pairs), and of bits in the second. The mean of mutual information was significantly greater than chance (permutation test, for both sessions; Figure 3). Mutual information was similar between sessions (), and pairs showed strong consistency across sessions (). The recordings of the experiments with the lowest and highest mutual information are available at https://github.com/shinn1/music.

Figure 3

Mean observed recurrence metrics of interaction within a pair against null distributions: process of musical collaboration is underpinned by information sharing and transfer between players. First session: (A) mutual information and (B) transfer entropy received from partners. Second session: (C) mutual information and (D) transfer entropy received from partners. Vertical red lines, observed means; grey areas, null distributions of means.

Transfer entropy was bits in the first session ( players), and bits in the second. Again, the mean was significantly greater than chance ( for both sessions; Figure 3), and values were correlated between sessions (). There was no change between sessions ().

3.3. Effects of Pair and Individual Traits on Information Sharing and Transfer

The survey revealed a wide range of musical expertise between participants, measured through two independent variables. With respect to experience in playing music with others, 12 participants answered “never” (score 0), 11 “rarely” (1), 17 “sometimes” (2), 13 “frequently” (3), and 7 “always” (4). The duration of practicing music ranged from 0 to 15 years (first quartile: 0, second: 3, and third: 7 years).

Delving into variations in information sharing and transfer across trials, we confirmed our hypothesis that players’ expertise in playing music is responsible for the processes of musical collaboration. Musical expertise explained variation in mutual information in pairs in the initial phase of the improvised musical collaboration (Figure 4). In the first experiment session, participants were found to share more information when playing music with partners that had a different level of experience in musical collaboration.

Figure 4

Effects of pairwise traits in musical expertise on mutual information: in the first session, information sharing was favored by differences in experience in playing with others and similarities in duration of practicing music. First session: (A) experience in playing music with others and (B) duration of practicing music. Second session: (C) experience in playing music with others and (D) duration of practicing music.

Specifically, in the first session, mutual information was associated with the interaction between the within-pair sum of experience in musical collaboration and within-pair difference (). It was also marginally explained by the interaction between the within-pair difference in duration of practicing music and within-pair difference () and by the within-pair difference (), but not by the within-pair sum (). These results indicated that pairing experts with novices in musical collaboration favored information sharing compared to pairing players with moderate experience in musical collaboration. By contrast, similarities within the pair in the duration of practicing music were conducive to information sharing, although pairing experts in musical instruments led to stronger information sharing than pairing novices.

In the second session, however, mutual information was not explained by experience in musical collaboration ( for the sum; for the difference; for the interaction). The duration of practicing music did not explain the variation in mutual information, either ( for the sum; for the difference; for the interaction).

Variation in information sharing was partly associated with how individuals musically responded to their partners, quantified through transfer entropy on symbolic recurrence. In the first session, transfer entropy was associated with the interaction between focal player and partner in their experience in musical collaboration (), and in the duration of practicing music (; Figure 5). These results indicated that, in the initial phase of the musical collaboration, players’ responses to their partners were explained by the musical expertise of both players. Predictably, novices to musical collaboration are influenced by partners who have experience in playing with others; these experienced partners can, in turn, adjust their rhythm more when playing with novices. The extent of this feedback depends on their relative training in music, whereby participants responded more strongly when partnered with others who practiced music for a similar duration. In this way, participants adjusted acoustic responses to their partner without knowing their musical expertise.

Figure 5

Effects of individual traits in musical expertise on transfer entropy (from partner to focal player): in the first session, information transfer was favored by differences in experience in playing with others and similarities in duration of practicing music. First session: (A) experience in playing music with others and (B) duration of practicing music. Second session: (C) experience in playing music with others and (D) duration of practicing music.

By contrast, in the second session, transfer entropy was not explained by experience in musical collaboration ( for focal players; for partners; for the interaction). The duration of practicing music did not contribute to the model fit ( for partners; for the interaction), with marginal significance for the focal players’ duration of practicing music ().

4. Conclusions

This is the first study that elucidated the processes and outcomes of collaborative musical improvisation through a mathematically principled approach. Pairs created music characterized by repetitive rhythmic patterns with marked preference for specific patterns over others, and the formation of such musical characteristics was underpinned by information sharing and transfer between players. Musical collaboration was established in the initial phase through players’ musical expertise, but the influence of musical expertise disappeared over time. These results unfolded prevailing rhythmic features in collaborative music creation while informing the complex dynamics of the underlying processes.

Music created by the pairs evolved over time, where rhythms became less repetitive, with more diverse patterns. These musical traits suggest that participants attempted to invent new rhythmic patterns once they established communication, resulting in the creation of music that was more unpredictable. Although the extent of information sharing and transfer in the second session was correlated with those in the first session, the musical expertise of the players no longer explained the variations. One possibility is that musical expertise played a role only until participants understood their partner’s rhythmic inclinations and responses through learning [19]. We also propose that, as time progressed, players gained confidence in their own musical expression, living a unique moment of inspiration independent of their musical expertise or that of their partner. Further study is needed to fully understand the dynamics of improvised music over time and the underlying factors that contribute to the dynamics.

In this study, we did not appraise the quality of the improvised music, as the notion of music is elusive [36]. Although most music entails common traits in rhythms, such as a use of isochronous beats and a metrical hierarchy in meters [39], music perception is largely shaped by enculturation [43,44,45,46]. Hence, people from different cultural backgrounds may exhibit disparate preferences [47,48,49,50]. For example, American infants prefer drum patterns with familiar Western meters (pulse duration ratio of 2:1:1) over unfamiliar Balkan meters (pulse duration ratio of 3:2:2), whereas Turkish infants who are familiar with both meters do not express preference [51]. Considering that participants in our study were from a student pool of a university that is home to students from diverse cultures, similarity in cultural backgrounds could also explain the extent of information sharing and transfer, in addition to their musical expertise.

In conclusion, we studied the processes and outcomes of musical collaboration in rhythmic improvisation through symbolic-recurrence quantification and information theory. In reality, musical collaboration could be achieved through other elements of music, such as melody, harmony, timber, and texture [52]. Further, there exist implicit rules that facilitate musical collaboration in jam sessions [53,54,55], including body gestures [23]. Nevertheless, our results shed light on the human ability of musical collaboration through rhythm, which constitutes a fundamental element of music from evolutionary and ethnomusicological perspectives [56,57].