Every musical creation starts with an idea, which can be a phrase, a sound object or a rhythmic pattern. Musicians are inspired by the individual sounds, bringing compositions to life and playing around the idea by improvising and adding parts to increase complexity. In the context of songwriting sessions of music groups, new compositions are often created through improvisational interactions between the musicians – this is called jams.
Spire Muse is a co-creative agent that supports musical brainstorming. The following is a brief explanation of how it works.
During jam sessions, basic musical ideas emerge, and the more one subsequently interacts with this idea, the more diverse the musical form develops. Creative interaction during improvisation is therefore very important for the emergence of further ideas. In general, musical interaction is a strategy built up from iterative phases.
The more one interacts with the idea, the more diverse the musical form can change. To make an open-ended creative interaction possible, improvisation is very important. A musical interaction is a strategy built from iterative phases. This means that improvising musicians decide whether they want to change something in their sequences, which can either be an initiative (new) change or a reaction. Reaction categories are adoption, augmentation and contrast.
Feedback during improvisation is essential; if it is positive, it can reinforce certain ideas; if it is negative, it can extinguish the spark of an idea or lead to new ideas different from the previous one.
Creativity cannot be defined in a uniform way, but is a process of multiple interactions that trigger unpredictable and undefined results. Through emergence of human-computer interaction (HCI), as well as artificial intelligence (AI), the perception of creativity is changing. With the help of AI, there are two ways that have an impact on the creativity process. Systems that have a supportive effect on human intelligence, so-called “creativity support tools” or those that generate results that are classified as creative from the perspective of an unbiased observer. The symbiosis of human and computer-assisted idea generation or creative process is then a co-creativity.
We have focused on designing a co-creative system that realizes the concept of a virtual jam partner.
by Notto J. W. Thelle and Philippe Pasquier
Spire Muse focuses on creating such co-creativity through a virulent jam partner. The strategy behind this is that the human or computer components interact with each other.
In a human interaction, action and decision interact, so the human is often the decision maker, which a computer cannot be. To avoid a dominant side in a co-creation, interactive behaviors are analyzed and defined in 4 categories, which are classified based on Reactive and Procative properties.
- Shadowing – user follows what he does synchronously
- Mirroring – Information or musical content is mirrored back in a novel way.
- Coupling – system, can clearly take the lead
- Negotiation – attempts to achieve goal through output modification.
When switching between the three behaviors – Shadowing, Mirroring, Coupling – Negotiation occurs, either autonomously or by the user.
The Spire Muse agent was generated on MASOM (Musical Agent based on Self-Organising Maps) and in MAX.
The learning process of the agent starts in the 1st stage by slicing the audio data in the source folder (the corpus) and then labelling the individual audio slices with feature vectors. This results in 55 dimensions of melodic and harmonic data.
Duration, loudness, fundamental frequency and chroma are used to encode the harmonic dynamics.
The feature vectors are then mapped onto a selforganizing map (SOM), an artificial neural network that employs unsupervised learning, so that the feature vectors can be displayed in a two-dimensional topological grid. Then, the tempo for each song is derived from the Python script via OSC to match the original tempo of the music piece. The last training part focuses on creating one sequence per song, so that repetitions and interaction dynamics can be shown.
The four influence parameters are rhythmic, spectral, melodic and harmonic.
The influences can be adjusted with sliders, so any combination of relative influences is possible.
Interactive modes
- Shadow mode: Agent is reactive here. It plays the most appropriate audio slice in the corpus for each registered onset in the input.
- Mirroring Mode: agent is reflexive interaction, respond with similar phrases after prolonged listening to single phrases.
- Coupling Mode: song is automatically selected from the corpus and chosen with respect to two criteria:
- Harmonic dynamics on the meso-time scale:.
- Tempo similarity:
The program starts in Shadow mode, which is the initial mode, as well as the evasive mode. The program returns to this mode if the requirements for activating Mirror or Coupling Mode are not met.
In addition to the automated behaviors of the agent, there is also a button with the functions Back, Pause/Continue, Change and Thumbs Up.
Thumbs Up indicates to the system that the current interaction is good and then maintains the current state for the next 30 seconds. All keys are operated with foot pedals.
“Throwback” is supportive of call-and-response interactions, but can become unpredictable in some SOM regions. These interfaces allow manipulation of behaviors and provide sufficient room for automated operations.
→ Spire Muse is not intended to enhance the agent’s musical performance, but rather to get the user to create ideas with a sense of shared exploration.
→ Ultimately, we believe that the most promising feature of Spire Muse is not the agent’s musical performance per se, but rather its ability to get users to explore ideas with a sense of shared ownership.
Spine Muse is expected to learn more algorithms in the future to reduce unpredictability through repeated use. By observing multiple sessions, the agent should be able to generate a profile that recognizes behavior and thereby play out different responses based on the situation.
Personal Comment:
Encouraging creativity is always a great approach, just by engaging with this new interface artists can already break out of their usual environment and be inspired. Such systems can be very useful in artistic development, especially when there are blockages or lack of creativity. Nevertheless, as an artist, one should not only rely on digital systems, but also be able to and be inspired by analog.
I would find a stronger integration of human-human interaction very exciting, in order to bring more emotions into the generated ideas. Maybe an implementation of Jam-with-Friends-Feature would be a good approach to bring together several artists with computer support in different places and to create a kind of community.
Source: Spire Muse: A Virtual Musical Partner for Creative Brainstorming – by Notto J. W. Thelle and Philippe Pasquier (https://nime.pubpub.org/pub/wcj8sjee/release/1)
