Category: Interaction Design

Every musical creation starts with an idea, which can be a phrase, a sound object or a rhythmic pattern. Musicians are inspired by the individual sounds, bringing compositions to life and playing around the idea by improvising and adding parts to increase complexity. In the context of songwriting sessions of music groups, new compositions are often created through improvisational interactions between the musicians – this is called jams.

Spire Muse is a co-creative agent that supports musical brainstorming. The following is a brief explanation of how it works.

During jam sessions, basic musical ideas emerge, and the more one subsequently interacts with this idea, the more diverse the musical form develops. Creative interaction during improvisation is therefore very important for the emergence of further ideas. In general, musical interaction is a strategy built up from iterative phases.

The more one interacts with the idea, the more diverse the musical form can change. To make an open-ended creative interaction possible, improvisation is very important. A musical interaction is a strategy built from iterative phases. This means that improvising musicians decide whether they want to change something in their sequences, which can either be an initiative (new) change or a reaction. Reaction categories are adoption, augmentation and contrast.

Feedback during improvisation is essential; if it is positive, it can reinforce certain ideas; if it is negative, it can extinguish the spark of an idea or lead to new ideas different from the previous one.

Creativity cannot be defined in a uniform way, but is a process of multiple interactions that trigger unpredictable and undefined results. Through emergence of human-computer interaction (HCI), as well as artificial intelligence (AI), the perception of creativity is changing. With the help of AI, there are two ways that have an impact on the creativity process. Systems that have a supportive effect on human intelligence, so-called “creativity support tools” or those that generate results that are classified as creative from the perspective of an unbiased observer. The symbiosis of human and computer-assisted idea generation or creative process is then a co-creativity.

We have focused on designing a co-creative system that realizes the concept of a virtual jam partner.

by Notto J. W. Thelle and Philippe Pasquier

Spire Muse focuses on creating such co-creativity through a virulent jam partner. The strategy behind this is that the human or computer components interact with each other.

In a human interaction, action and decision interact, so the human is often the decision maker, which a computer cannot be. To avoid a dominant side in a co-creation, interactive behaviors are analyzed and defined in 4 categories, which are classified based on Reactive and Procative properties.

• Shadowing – user follows what he does synchronously
• Mirroring – Information or musical content is mirrored back in a novel way.
• Coupling – system, can clearly take the lead
• Negotiation – attempts to achieve goal through output modification.

When switching between the three behaviors – Shadowing, Mirroring, Coupling – Negotiation occurs, either autonomously or by the user.

The Spire Muse agent was generated on MASOM (Musical Agent based on Self-Organising Maps) and in MAX.

The learning process of the agent starts in the 1st stage by slicing the audio data in the source folder (the corpus) and then labelling the individual audio slices with feature vectors. This results in 55 dimensions of melodic and harmonic data.

Duration, loudness, fundamental frequency and chroma are used to encode the harmonic dynamics.

The feature vectors are then mapped onto a selforganizing map (SOM), an artificial neural network that employs unsupervised learning, so that the feature vectors can be displayed in a two-dimensional topological grid. Then, the tempo for each song is derived from the Python script via OSC to match the original tempo of the music piece. The last training part focuses on creating one sequence per song, so that repetitions and interaction dynamics can be shown.

The four influence parameters are rhythmic, spectral, melodic and harmonic.

The influences can be adjusted with sliders, so any combination of relative influences is possible.

Interactive modes

• Shadow mode: Agent is reactive here. It plays the most appropriate audio slice in the corpus for each registered onset in the input.
• Mirroring Mode: agent is reflexive interaction, respond with similar phrases after prolonged listening to single phrases.
• Coupling Mode: song is automatically selected from the corpus and chosen with respect to two criteria:
  • Harmonic dynamics on the meso-time scale:.
  • Tempo similarity:
    

The program starts in Shadow mode, which is the initial mode, as well as the evasive mode. The program returns to this mode if the requirements for activating Mirror or Coupling Mode are not met.

In addition to the automated behaviors of the agent, there is also a button with the functions Back, Pause/Continue, Change and Thumbs Up.

Thumbs Up indicates to the system that the current interaction is good and then maintains the current state for the next 30 seconds. All keys are operated with foot pedals.

“Throwback” is supportive of call-and-response interactions, but can become unpredictable in some SOM regions. These interfaces allow manipulation of behaviors and provide sufficient room for automated operations.

→ Spire Muse is not intended to enhance the agent’s musical performance, but rather to get the user to create ideas with a sense of shared exploration.

→ Ultimately, we believe that the most promising feature of Spire Muse is not the agent’s musical performance per se, but rather its ability to get users to explore ideas with a sense of shared ownership.

Spine Muse is expected to learn more algorithms in the future to reduce unpredictability through repeated use. By observing multiple sessions, the agent should be able to generate a profile that recognizes behavior and thereby play out different responses based on the situation.

Personal Comment:

Encouraging creativity is always a great approach, just by engaging with this new interface artists can already break out of their usual environment and be inspired. Such systems can be very useful in artistic development, especially when there are blockages or lack of creativity. Nevertheless, as an artist, one should not only rely on digital systems, but also be able to and be inspired by analog.

I would find a stronger integration of human-human interaction very exciting, in order to bring more emotions into the generated ideas. Maybe an implementation of Jam-with-Friends-Feature would be a good approach to bring together several artists with computer support in different places and to create a kind of community.

Source: Spire Muse: A Virtual Musical Partner for Creative Brainstorming – by Notto J. W. Thelle and Philippe Pasquier (https://nime.pubpub.org/pub/wcj8sjee/release/1)

Background

Touch interfaces for musical applications have been first introduced in the early 90s by Schneiderman. They started out controlling vocal synthesis. Since then the Monome grid has developed into the standard of interfaces for musical equipment over the last 15 years.

The article I read explains how they made an effort to provide an other solution to it. Their suggested solution is a grid device with capacitive touch technology to use swipe gestures as a menu system, expanding the interface capabilities of the popularized Launchpads nowadays.

One of the reasons that might have made the launchpad keep it’s simplicity and popularity is the fact that most instruments rely on haptic feedback, but almost none rely on visual feedback.

These two grid controllers, Tenori-On and Monome Grid har pushed their interfaces into the music industry which have been the same for 15 years now.

It’s now in more recent times adopted by the more popularized and professional standards in the different Launchpads.

The great thing about the interface and allegedly why it’s been adopted is it’s generic layout which allows any artist to customize and play however they want.

Added functionality to the grid layout is the buttons around the “core” allowing the launchpad to control different hardware interfaces with specific software.

Their project

Touchgrid was developed to solve problems from limited resolution and restricted space provided by grids using touch interfaces. It keeps the generic button layout of the launchpad, but uses capacitive touch to extent it’s possibilities.

1st iteration

Their solution was the capacitive touch display consisting of 16*8 grid of touch areas. By using time-based date they recognized different interactions such as swipes and bezel interactions.

Problem: Due to their now huge processing requirements, the system ran on a slower maximum sample-rate than preferred.

2nd prototype

For their next prototype they used a Adafruit NeoTrellis M4 which is a readymade DIY kit. Using LEDs and silicone buttons to expand their Launchpad.

Chosing from a wide range of possible interactions they chose the “Drag from off-screen” and “Horizontal swipe” as their interactions as they are well known from smart devices adopted by the common public today.

with a more restricted but better hardware layout they manage to fix their problem with high sampling rates and hence performance was bettered.

In an innovative solution this group made interactions replace the buttons from the Launchpad, allowing more space for music, less for navigation. The buttons allocated to menu-changing actions from the launchpads were now replaced with swiping motions we learned using our phones.

Their new menu layout is made in a spatial context to reduce the learning curve and mental workload of learning their device. Using the swiping interactions mentioned earlier they make an intuitive mapping as shown below.

In the end they managed to gather user insight from 26 people with prior knowledge of how to use similar instruments. Answers from their survey revealed that the touch patterns are recognized and their mapping works. There is however worries that the suggested interaction system might add complexity and interferences with an already working product.

Conclusion

As they are saying themselves, they will make the argument of expanding the capabilities of tangible devices instead of making touch screens more tangible. Meaning that they will take their learning from touch screen interactions and implementing them on the grids and tangible devices for a haptic screen feel.

Being an interaction designer it’s refreshing seeing a well developed tangible interaction system stand its ground towards more “modern” screen touch-based ones. In a very human way this project combines what we know and adopted from before with another dear interaction to us from another system.

by Daniel Chin, Ian Zhang, and Gus Xia

Breath control 🤧🥱🥅

Breathing is becoming increasingly important for stress relief. However it is not only good for controlling the body, but also for controlling wind instruments for example the flute.

With the development of the Hyper hybrid flute, an attempt was made to integrate the profound role of breath control into a digital flute and to take it in advance it was successful. In principle, musicians can control not only the volume, but also articulation, octave, micro-tones, etc. through breathing techniques on wind instruments. However, most existing digital versions do not capture the various effects of breathing as is possible in analogue. Instead, they rely on additional interface elements. An interface was developed that converts real-time breath data into MIDI controls. The Hyper-hybrid Flute can be switched between its electronic and acoustic modes. In acoustic mode, the interface is identical to the regular six-hole recorder. In electronic mode, the interface recognizes the player’s fingering and breathing speed and converts them into MIDI commands.

 The Hyper-Hybrid Flute interface has three contributions in particular:

• It simulates that acoustic property of the flute where higher breathing speed leads to higher octaves and more micro-tonal pitch bending.
• By exaggerating the parameters, the interface is expanded into a hyper instrument.
• A simple toggle supports the change between electronic and acoustic mode.

To detect if the hole is covered by a finger when playing the flute, a ring-shaped capacitive sensor is placed on each of the six holes and the breathing rate is measured by a BMP085 air pressure sensor.

Changing state

To enter the electronic mode, the musician inserts the air pressure sensor into the mouthpiece outlet. This mutes the recorder and simultaneously exposes the sensor to the air pressure in the recorder, from which the breathing speed is calculated. To enter acoustic mode, the player releases the air pressure sensor from the exit port, so that the playing of the interface is acoustic and the air pressure sensor is not triggered. The picture below shows the prototype with the attached sensors.

Controlling Octave and Micro-tone via Breath

The influence of the breath on the micro-tone and the octave can be modeled as follows:

• Harder blowing at a pitch leads to an upward micro-tonal pitch bend.
• When the breathing speed exceeds a specific threshold, the pitch jumps up an octave.

Such threshold values for the breathing speed increase with rising pitch. This is shown perfectly in the picture above. The higher the velocity of the breath with holding for example D# the higher the jumps to another octave are.

Measuring the relationship between pitch bend and breath pressure using an acoustic recorder gives a pitch bend coefficient of 0.055. The micro-tone enables the musician to perceive his position relative to the thresholds. This interactive feedback allows them then to calibrate their breathing speed and avoid unexpected octave jumps. Under the bend coefficient > 0.055 the interface becomes a hyper instrument. The micro-tone as a musical device offers an additional dimension of expressiveness.

How does it become a MIDI controller?

To know what pitch the instrument should produce at any given time does not make it a MIDI controller per-se, because MIDI requires a discrete stream of note on and note off events. So the interface must be stateful.

The breath velocity is compared to a threshold to determine whether the instrument should be at rest or producing a note. A rising edge in that signal marks the excitation of the instrument, which fires a Note On event. Meanwhile, a differentiator listens to the pitch and fires its output line when the pitch changes value. The differentiator output, conditioned on whether the instrument is at rest, also fires a Note On event.

What tools are used?

The interface is wireless. All sensors are connected to an Arduino Nano, which communicates with a Processing 3 sketch via Bluetooth. The sketch uses the midi bus library for MIDI messaging. The recorder body is modeled in Fusion 360 and fabricated with MJF 3D printing.

Reflections

In the results of this research, it is very clear that there is still innovation in the field of wind controllers. With the ability to measure octaves, the multi modal music teaching system can be expanded to include breathing technique in the learning outcomes. The MIDI interface is accurate and allows for optimal communication through the musician’s breathing.

Therefore, the hyper-hybrid flute poses as an interesting solution on the path to the digitization of wind instrument and also new didactic concept and learning more immersive. Besides teaching, I especially see this hyper-hybrid flute applied in the context of arts and performance art but also possible in commercial productions where simulations of wind instruments might be useful. Moreover I want to mention the importance of interfaces as bridges from the analogue to the digital world, what this flute also represents. It is of high interest to combine these worlds to create even better and more comprehensive solutions and experiences. Analog and digital, these opposites both have their justification and are to a certain extent equally dependent on each other, and definitely can profit from each others strengths.

I want to close this post with Adrian Belew’s words: “Digital for storage and quickness. Analog for fatness and warmth.”

^{Source: https://nime.pubpub.org/pub/eshr/release/1}