My next experiment was to turn the Bela board into a guitar effect and integrate the scanner into it. Bela has both built-in audio input and output, which can be addressed with the [adc~] and [dac~] objects respectively. By assigning an [adc~] to the input of the [delwrite~] object and a [dac~] to the output of the delay taps, I could run the signal from my guitar through the patch and hear the effect in action on my headphones or in my guitar amp.
To upload and run a PD patch on Bela, all I had to do was access its web-based IDE by addressing bela.local in my web browser and make sure to name the file _main.pd. Since the patch is no longer editable in the IDE, I couldn’t change any of the effect’s parameters, so I set up a patch with default values to test if the sound comes through Bela and if the patch works.
To simulate Hammond’s vibrato scanner unit in Pure Data, I first had to find a way to replicate the delay line. Although there are many externals including all-pass and comb filters that would be suitable for this purpose, I had to keep in mind that BELA supports only Vanilla objects so I had to work with a limited library.
An obvious starting point was to use the [delwrite~] object, which allocates memory to a buffer and writes an audio signal into it. This signal is then read by the [delread~] object of the same name with a variable delay time, which can be defined in milliseconds. As it is possible to use more than one [delread~] object for the same buffer, I was able to set up multiple delay taps for the same signal, where each step is being delayed by 5 milliseconds relative to the previous one. Mixing these with the original signal results in amplification and cancellations at regular intervals, producing a comb filter in the end.
Next, I built a mechanism based on the scanner that could sample these delay taps. To do this, I first created a simple counter that increments by one with a specified velocity and then starts over when the given maximum value is reached. Its value is then passed to a select object, which sends a bang to the corresponding outlet when the value matches its argument. This results in a sequencer actually, which is ultimately what the scanner also is.
To connect the delay line to the sequencer, I placed a [vline~] object at the output of each tap. [Vline~] creates linear ramps whose levels and timing are determined by the messages you send. By creating a ramp-up and ramp-down message for each [vline~] object, I was able to toggle the volume of the taps on and off in a specific order so that each successive bang increases the volume of its assigned tap while turning down the previous one. To make sure that there is always a smooth transition between the taps, I made the duration of the ramps relative to the speed of the sequencer’s clock, achieving a perfect crossfade at any given rate.
To adjust the depth of the fades, I do multiplications on the ramp value that is sent to the [vline~] with a value between 1 and 5. The larger this number, the less accentuated the ramps are, and the smaller the depth is. This also means that the smaller the depth, the more the taps get mixed, which results in a chorus effect subsequently. Additionally, by mixing the dry signal with the scanner, a chorus-vibrato can be achieved, just as in the original Hammond design.
My first experiment was to simulate the delay line with Pure Data, based on the research carried out in the previous phase of this project. Originally, the analog delay line is part of the Hammond Vibrato Scanner unit and it incorporates a series of second-order audio filter stages where each stage is shifted in phase relative to the previous one. This results in an increasing time delay between every successive step of the line (about 50μs per filter stage). This is then fed into a scanner, which is a single-pole 16-throw air-dielectric capacitor switch connecting nine selected taps from the line to the output.
As the scanner gradually transitions between these taps, it alternates the phase shift applied to the sound, causing slight variations to the pitch that result in a vibrato effect. The depth of the vibrato depends on the width of the frequency shift fed into the scanner, which means that scanning about one-third of the delay line would produce a lighter vibrato effect while scanning the whole line would significantly increase its depth. In addition to the vibrato, a chorus effect can also be achieved by mixing the dry input signal with multiple outputs on the delay line, or a chorus-vibrato by mixing the dry signal with the output of the vibrato.
Initially, I proposed the development of an analog multi-effects chain unit that combines different types of vintage audio components with the capabilities of modern microcontrollers. The core element of this device would have been an analog delay line which, when sampled in different sequences, produces different modulation effects. The unit could be expanded with additional analog modules such as spring reverbs and tape delays daisy-chained in interchangeable order and controlled by the same central ESP32 microcontroller. The user interface would be divided between a minimalist control panel on the device and a mobile app that allows for detailed parameter settings.
Due to difficulties in designing an analog circuit that could sample the delay line, I decided to take my project to the digital domain and create a digital multi-effects chain instead. After gaining extensive experience with Pure Data and the Bela Board in the first part of this semester, I decided to use these technologies for the project. The main plan remained essentially the same as proposed in the first semester, only the hardware had been replaced.
My new plan is to establish a community-oriented product by incorporating various open-source technologies. I propose the development of a digital multi-effect unit that combines the flexibility of Pure Data with the ultra-low latency of Bela and the integrability of the open sound control protocol. The core element of the effect chain is a delay line that produces different modulation effects. By design, the unit can be expanded with additional effect modules that are controlled by the same open sound control interface.
The biggest change that I await to happen next semester is to receive the SOMI-1 kit from Instruments of Things. This new kit will only come out during summer, but basically entail exactly what I need from it: the wristband sensors look and can do pretty much the same as the ones from the 2.4SINK kit, but the receiver is just a small USB interface, meaning there is no analogue data to clean up anymore. This of course makes the time I spent cleaning this data feel a little redundant, but this also gets rid of the bottleneck, allowing me to read out data from more than one sensor simultaneously, too. Which also brings me to my next point: I will spend some time on planning the multi-sensor mapping – basically defining what influence each sensor has on the soundscape when more than one sensor is active.
Other than that, I will want to spend a lot of time on the actual composition. And furthermore, I would like to plan the installation, since I would like it to be a multi-sensory experience. Either way, I am very much looking forward to spending more time on this project.
While the composition should of course always be in the back of my mind while developing the logic and mappings, the part in Ableton is where the music finally comes into play. I have not found a great workflow for creating such a composition yet, since I mostly end up changing mappings around, but I think when the time comes that I know my mappings quite well, the workflow also becomes a little easier and more intuitive.
I think that Ableton is a great choice for exactly this type of application because it was developed for live settings – which this installation also belongs to. Ableton’s session view is arranged in clips and samples that can be looped and will always play in sync, without an ending time. I am mostly playing all tracks at once and use the MIDI parameters to turn up the volume of certain tracks at certain points.
Composing for this format reminds me a lot of what I know about game sound design, since the biggest challenge seems to be to create a composition that consists of parts that might play together in ways one can barely anticipate, while it should still always sound good and consistent.
For the proof-of-concept composition, I used some stereo atmosphere I recorded myself and adjusted the width of it depending on the dispersion of the particles. A pad with many voices was also always running in the back, with slight pitch adjustments depending on the particle average position, as well as a low pass filter that opens when one’s arm is lifted up and closes when it is pointing downwards. With each soft movement, a kind of whistling sound is played, that is also modulated in pitch, depending on the arms position (up/down). This whistling sound also gets sent into a delay with high feedback, making it one of the only sounds at that level that reveals the BPM. As soon as the second level is reached, a hi-hat with a phasor effect on it starts playing, bringing a bit more of a beat into the soundscape. The phasor is controlled by the average velocity of the particles. The pattern of the hi-hat changes with a five-percent chance on every hard movement. The third level slightly changes the pad to be more melodic and introduces a kick and different hi-hat patterns. All in all, the proof-of-concept composition was very fun to do and worked a lot better than expected. I am looking forward to spending more time composing next semester.
To be able to send MIDI data to Ableton, there is a need to run a virtual MIDI cable. For this project, I am using the freeware “loopMIDI”. This acts as a MIDI device that I can configure in Pure Data’s settings as MIDI output device and in Ableton as MIDI input device.
In Pure Data, I need to prepare data to be in the right format (0 to 127) to be sent via MIDI and then simply use a block that sends it to the MIDI device, specifying value, device, and channel numbers. I am mainly using the “ctlout” block, which sends control messages via MIDI. I did experiment with the “noteout” block as well, but I did not have a need for sending notes so far, since I rather adjust keys and notes dynamically in Ableton.
There is a mapping mode in Ableton, which lets one map almost every changeable parameter to a MIDI pendant. As soon as I turn on the mapping mode, I can just select any parameter that I want to map, then switch to Pure Data, click the parameter that I want to map to the Ableton variable, and it will immediately show up in my mapping table with the option to change the mapping range. While this approach is very flexible, it unfortunately is not very well structured since I cannot read or edit the mapping table in another context (which would be very useful to avoid mistakes while clicking).
The logic module (still in the same Pure Data patch) is responsible for the direct control of the soundscape. This comprised translating the processed sensor data into useful variables that can be mapped to parameters and functions in Ableton, ultimately leading to an entertaining and dynamically changing composition that reacts to its listeners’ movements. Over the course of this semester, I wrote down various functions that I thought might be adding value to the soundscape development – but I soon realized that most of the functions and mappings just need to be tried out with music for me to be able to grasp how well they work. The core concepts I have currently integrated are the following:
Soft movements
Great to use for a direct feel of control over the system since each movement could produce a sound, when mapped to e.g., the sound level of a synth. An example might be something like a wind sound (swoosh) that gets louder the quicker one moves their hand through space.
Hard movements
Good for mapping percussive sounds, however, depending on one’s movements sometimes a little off and therefore distracting. Good use for indirect applications, like counting hard movements to trigger events, etc.
Random trigger
Using hard movements as an input variable, a good way to introduce movement-based but still stochastic variety to the soundscape, is to output a trigger by chance with each hard movement. This could mean that with each hard movement, there is a 5% chance that such a trigger is sent. This is a great way to change a playing midi clip or sample to another.
Direct rotation mappings
The rotation values of the watch can be mapped very well onto parameters or effects in a direct fashion.
Levels
A tool that I see as very important for a composition that truly has the means to match its listeners’ energy, is the introduction of different composition levels, based on the visitors’ activity. How such a level is changed is quite straight forward:
I am recording all soft movement activity (if a soft movement is currently active or not) into a 30 second ring buffer and calculate the percentage of activity within the ring buffer. If the activity percentage within 30 seconds crosses a threshold (e.g., around 40%), the next level is reached. If the activity level is below a threshold for long enough, the previous level is active again.
While by using all those mechanisms I could reach a certain level of complexity in my soundscape already, my supervisor inspired me to go a step further. It was true that all the direct mappings – even if modulated over time – had the risk to get very monotonous and boring over time. So, he gave me the idea of using particle systems inside my core logic to introduce more variation and create more interesting mapping variables.
Particle System
I found an existing Pure Data library that allows me to work with particle systems which is based on the Graphics Environment for Multimedia (GEM). This did however mean that I had to actually visualize everything I was calculating, but I figured out that at least for my development phase this was very helpful, since it is difficult to imagine how a particle movement might look just based on a table of numbers. I set the particles’ three-dimensional spawn area to move depending on the rotation of the three axes of the wristband. At the same time, I am also moving an orbital point (a point that particles are pulled towards and orbit it) the same way. This has the great effect that there is a second velocity system involved that is not dependent directly upon the acceleration of the wristband (it however is dependent on how quickly the wristband gets rotated). The actual parameters that I use in a mapping are all of statistical nature: I calculate the average X, Y and Z position of each particle, the average velocity of all particles and the dispersion from the average position. Using those averages in mappings introduces an inertia to the movements, which makes the compositions’ direct reactions to one’s motion more complex and a bit arbitrary.
Before I write about the core logic part, I would like to lay out how the communication between Pure Data and Ableton looks like. Originally, I had planned to create a VST from my Pure Data patch, meaning that the VST would just run within Ableton and there is no need to open Pure Data itself anymore. This also gave me the idea to create a visualization that provides basic control and monitoring capabilities and lets me easily change thresholds and parameters. I spent some time researching possibilities and found out about the Camomile package, which wraps Pure Data patches into VSTs, with visualization capabilities based on the JUCE library.
However, it did not take long until I found several issues with my communication concept: First off, the inputs of the Expert Sleepers audio interface need to be used, while any other interface’s (or the computer’s sound card’s) outputs must be active, which is currently not possible natively in Ableton for Windows. There would be a workaround using ASIO4ALL, but that was not a preferred solution. Furthermore, a VST always implies that the audio signal flows through it, which I did not really want – I only needed to modulate parameters in Ableton with my Pure Data patch and not have an audio stream flowing through Pure Data, since that would give me even more audio interface issues. This led me to move away from the VST idea and investigate different communication methods. There were two more obvious possibilities, the OSC protocol and MIDI. The choice was made quite quickly since I figured that the default MIDI resolution of 128 was enough for my purpose and MIDI is much more easily integrated into Ableton. Now with that decision made, it was a lot clearer what exactly the core logic part needs to entail.
Following the signal flow, in Pure Data the wristband sensor data gets read in and processed; and modulators and control variables get created and adjusted. Those variables are used in Ableton (Session View) to modulate and change the soundscape. This means that during the installation, Ableton is where sound gets played and synthesized, whereas Pure Data handles the sensor input and converts it into logical modulator variables that control the soundscape over short- and long-term.
While I separated the different structural parts of the software into own “modules”, I only have one Pure Data patch with various sub-patches, meaning that the separation into modules was mainly done for organizational reasons.
Data Preparation
The first module’s goal was to clean and process the raw sensor data in a way for it to become usable for the further process steps, which I spent a very substantial amount of time with during this semester. It was a big challenge to be working with analogue data that is subject to various physical influences like gravity, electronic inaccuracies and interferences and simple losses or transport problems between the different modules, but at the same time having a need for quite accurate responses on movements. Additionally, the rotation data includes a jump from 1 to 0 each full rotation (like it would jump from 360 to 0 degrees) which also needed to be translated into a smooth signal without jumps (converting to a sine was very helpful here). Another issue was that the bounds (0 and 1) were rarely fully reached, meaning I had to find a solution how I could reliably achieve the same results with different sensors at different times. I developed a principle of using the min/max values of the raw data and stretching that range to 0 to 1. This meant that after switching the system on, each sensor needs to be rotated in all directions for the Pure Data patch to “learn” its bounds.
I don’t calculate how the sensor is currently oriented in space (which I possibly could do, using gravity) and I soon decided that there is no real advantage in using the acceleration values of the individual axis, but only the total acceleration (using Pythagorean triples). I process this total acceleration value further by detecting ramps, using the Pure Data threshold object. As of now, two different acceleration ramps are getting detected – one for hard and one for soft movements, where I defined a hard movement, like one would move a shaker or hit a drum with a stick, and the soft movement is rather activated continuously as soon as one’s hand moves through the air with a certain speed.
I originally imagined that it should be rather easy to let a percussion sound play on such a “hard movement”, however, I realized that such a hand movement is quite complex, and a lot of factors play a role in playing such a sound authentically. The peak in acceleration when stopping one’s hand is mostly bigger than the peak when starting a movement. But even using the stopping peak to trigger a sound, it doesn’t immediately sound authentic, since the band is mounted on one’s wrist and peaks at slightly different moments in its acceleration than one’s fingertip or fist might do.
