Author: Björn Bernreiter

Everything works so far! With the help of a very esteemed study colleague, the splitter and mixer were successfully soldered and started working. These achievements marked the start of many advances in the project. Now, the left and right hand setups could actually be tested completely, involving all necessary parts, soft- and hardware. Luckily, they worked as expected. The left hand setup is now able to switch between to FX loops according to fretting hand position and the right hand setup is able to control effect pedals via the expression pedal input. Finally, two boxes made from ABS plastic were bought at Neuhold to fit the ready-made circuits in a protected enclosure. Nine holes in total were drilled into the plastic for the audio jacks as well as the power supplies.

All applications need more testing and fine-tuning. The code of the left hand setup must be fine-tuned for example to work even more accurately at detecting frets and more pedals have to be tested in conjuncture with the right hand setup in order to find out what expressive effects are ideally controlled by the strumming hand. Additionally, some gimmicks may still be implemented at a later date. These include:

• Variable threshold potentiometer
• LED indicators (ON/OFF, etc.)
• Battery monitor system for ESP32-LH/RH
• One box
• Box design

With the left hand setup working so well, it was finally time to integrate the ESP-NOW code into that of the right hand setup. So far, the ESP-NOW code only involves communication between two ESP32s. However, in the end the ESP32-CU should receive data from both devices, the ESP32-LH and the ESP32-RH. This means that ESP-NOW communication between three devices must be established. However, only one-way communication capabilities are necessary since the ESP32-LH/RH are the only ones to send data to the ESP32-CU with the latter sending no information back. Surprisingly, this made the code necessary a lot easier than it was before. Following a tutorial, the ESP32-CU was set up as the receiver (slave) and the other two ESP32s for left and right hand were set up as senders (masters). Then ESP-NOW communication was tested by sending dummy values from ESP32-LH and RH to the ESP32-CU. By sending a sender ID together with the dummy values, the ESP32-CU was not only capable of receiving the values but could also distinguish between the values sent by ESP32-LH and those sent by ESP32-RH. This was great news for the project!

With the basic code working, it was then time to add this new ESP-NOW code to the current codes of the left and right hand setups. As always, the ESP-NOW code was first added to the left hand setup code and it worked as expected. Next, the code was added to the right hand setup and communication between all three devices was tested simultaneously with the right hand and the left hand setup sending values to the ESP32-CU. This worked as well.

To top things off, the left hand setup was again fully tested in conjuncture with the digipot following the test setup explained in the previous blog post together with the new ESP-NOW code. This test worked as well making the Christmas holidays a very successful time for the project with much progress being made!

Sources:

ESP-NOW: Receive Data from Multiple ESP32 Boards (many-to-one) | Random Nerd Tutorials

Unfortunately, no progress has been made with the hardware part of the project, namely the splitter and the mixer. Especially, the mixer evolved into an issue because the circuit built on the breadboard appears to be correct and even the project’s supervisor could find no error upon a brief inspection. However, despite all efforts, it does not seem to work properly and makes horrible noises when connected with its 9 Volts power supply… Nevertheless, as time pressure increased, it was decided to move forward and solder the mixer on a Veroboard instead of testing it on the breadboard – maybe it will start to work when all connection are firm. Consequently, two Veroboards were ordered.

In the meantime, it was decided to perform a major test of the left hand setup, involving all necessary components except said mixer. The idea was to have everything working so that when the mixer miraculously starts to functioning properly everything else already works and is ready to be tested in conjuncture with the mixer. The thorough the test of the left hand setup included several tasks.

Firstly, after some time off the headstock the ToF sensor was re-positioned again at the headstock of the test electric guitar and reconnect with the ESP32-LH (left hand setup). Then the ESP32-LH and the ESP32-CU (central unit) were re-paired using the ESP-NOW protocol. A change had to be made in the code of both ESP32s: so far, the ESP32-LH did all the processing of the ToF sensor values including computing the distances and the according frets, deciding if the Solo Mode has been activated or disactivated according to the fret threshold set and sending a corresponding message of 0 or 1 to the ESP32-CU. It was decided that the ESP32-LH should only be used to detect the frets and send the frets to the ESP32-CU who should then go on to decide over Solo Mode (dis-)activation based on fret threshold set and send 0 or 1. This comes with the advantage that if a flexible threshold can be set on the central hardware unit via a potentiometer, the ESP32-CU can receive this directly and the potentiometer values sending the fret threshold must not be send back to the ESP32-LH. This facilitates the code necessary because only a one-way communication is happening. After the relevant changes have been made in both codes, the setup was tested and it worked as expected. In the last step, the digipot was added. It was ordered to change between two resistance values depending on the status of the Solo Mode (ON = hand above 5^th fret or OFF = hand below 5^th fret). The resulting change in voltage was again detected by the trusted Arduino UNO whose serial monitor input was used as reference. This test worked as well: depending on the fretting hand position (below or above threshold of 5^th fret) the resistance values changed. This basically means that the whole left hand setup is working. In the final setup, the change of the resistance values will be used to control the WET/DRY mix of the two FX loops.

As mentioned before, it was decided that another design addition is to add an expression pedal output to the central hardware unit that can be used to control effect pedals featuring an expression pedal input. As things currently stand, the movements of the right, strumming hand will be the primary input for the expression pedal but, of course, also the left hand setup could be used as an control input. After the second semester, it was initially planned to control a virtual effect in Pure Data with the movements of the right hand. This would have necessitated the use of a laptop during live performances which is not really common among traditional guitarists. Consequently, the idea of using an expression pedal accords with the project’s goal to be easily implemented into a guitarists current live rig.

Developing the expression pedal was made possible with solving the problem of how to use the digital potentiometer MCP4251. As mentioned in the previous blog post, interfacing and using this digipot was made possible so, now, work on the expression pedal could commence. With the author not having too much experience with expression pedals and their functionality, some reading had to be done beforehand. Then, it was decided to firstly build a kind of conventional expression pedal using an analog potentiometer. By following the instructions of a blog post explaining how to build a DIY expression pedal from a Wah-Wah pedal, a circuit was made on a breadboard. This circuit involved a potentiometer which simulated the expression pedal (which is basically a potentiometer) that sent its values via a TRS line cable to an Arduino UNO that simulated the effect pedal controlled by the expression pedal. The Arduino UNO read the values coming in over the TRS cable using the analogRead() function. As the potentiometer was turned, the changing values sent over the TRS cable could be detected by the Arduino.

In the next step, the analog potentiometer was replaced by the digipot connected to an ESP32 and a code was written that toggled the resistance of the digipot between two values, changing every second. Thus, the ESP32 sent commands to change the resistance to the digipot. These voltage changes were then sent over the TRS cable to the Arduino UNO acting as the effect pedal and printed in the serial monitor of the Arduino. While this setup only simulated the workings of an expression pedal, it nevertheless proved that the components of the circuit worked together as desired. The next step here is to test the setup with actual IMU sensor input from the right hand while guitar playing AND to connect the setup not to the Arduino UNO but to an actual effect pedal with an expression pedal input.

On 1^st January, the digital potentiometer finally started working. As mentioned in previous posts, the idea is to use some sort of mixer to mix together two FX loops. The DRY/WET settings of this mixer should be controlled by the left hand setup. The mixer layout chosen for the project uses two analog potentiometers to mix the two guitar signals coming thru the FX loops. The issue is that analog potentiometers cannot be digitally controlled so a different solution had to be found. This solution is a digital potentiometer of the type MCP4251. This digipot has two digital potentiometers built in whose wipers can be set digitally by commands of a microcontroller. Getting this digipot to work however, was not that trivial. It works via SPI which is quite complicated, and no suitable library could be found at first. Additionally, it was not initially clear how to connect the MCP4251 with a microcontroller. After a lot of digging on the Internet, a project using this exact digipot was found. The author wrote a custom library that could be used, and he also explained how to wire up the MCP4252 to an Arduino UNO. Apparently, the Arduino UNO has specific SPI pins that need to be used for the SPI communication with the digipot. Following the instructions of the author’s blog post, the digipot could finally be used to control the brightness of two LEDs.

Next, a code was made to simulate the following mixing function that the MCP4251 will perform in the future:

• Normal rhythm guitar mode (= fretting hand position < 5^th fret)
  • FX loop 1 -> always 100% wet
  • FX loop 2 (containing delay etc.)  -> 20% wet

• If Solo Mode active (= fretting hand position >= 5^th fret)
  • FX loop 1 -> stays 100% wet
  • FX loop 2 (containing delay etc.) -> turned up to 50% wet

The WET/DRY status was simulated by changing the brightness of the two LEDs. This worked as well with the one LED’s brightness remaining unchanged while the other changed according to which fret was played on the guitar.

So far, the digipot was used in conjuncture with an Arduino UNO since this microcontroller was used in the tutorial. In the next step, it was tried to transfer the code to work on an ESP32. It was simpler than expected: the SPI pins of the ESP32 were determined and then everything worked as expected. Getting the digipot to work marked the achievement of another small milestone in the project as the digipot will serve as the main switching hardware controlling the WET/DRY ratio of the two FX loops via the input of the left hand setup and acting as an expression pedal via the input of the right hand setup.

.

While the attachment device for the left hand setup is largely finished, the final attachment for the right hand setup must still be built. The basic idea follows a prototype that has already been built. This prototype involves textile straps onto which the IMU sensor, the ESP32 and the battery supply are sewn. Unfortunately, this version resembled bandages and still has room for improvement so it was decided to build another, better version.

Firstly, the strap for the hand that secures the IMU sensor to the back of the hand was created. Therefore, a black rubber/textile band was sewn to a shape to make it wearable in a way that places the sensor in the correct position while ensuring that it does not interfere to much with the guitar playing motions of the right hand. Velcro was added to the rubber strap in specific places so it can be easily put on and fastened as desired.

Another challenge that arose during the development of said new attachment device, is that the rubber strap places the sensor in a different position than it was previously. As the sensor is an IMU sensor, its position is quite significant. In the new position, the values read by the sensor do not fit the mapping previously made when the sensor was fitted onto the wrist watch. Accordingly, the values needed to be re-mapped.

With the sensor in its new position, an electric guitar was played with the right hand performing typical strumming movements. Then, the orientation data in x, y and z direction coming in the Arduino IDE serial monitor was copied into a Word document and the values subsequently copied into an Excel sheet. There the values were put into a graph. The graph of the y values seemed to represent the strumming movements in the most accurate way. Initially, the x values appeared to be useful as well but when restarting the IMU sensor code, the x values suddenly changed a lot, and the values were in a totally different range. Consequently, the y values which were more consistent were chosen as the values to be used.

To further test the y values’ suitability, the old Pure Data sketch containing the Wah Wah effect was revisited and the y values used to control the effect. This worked fine. However, one issue remains: a way must be figured out how to use the IMU values as an expression pedal to control guitar effect pedals with an expression pedal input.

While working on the arm strap onto which the ESP32 will be fixed, another issue arose that was already mentioned in the Blog post #12: the battery supply. The AA battery pack proved to be too unwieldy to be worn on the arm so another possibility had to be found. The current solution is to simply use a small power bank to power the ESP32 of the right hand setup and possibly the one of the left hand setup as well. According to some articles on the Internet it is not the most efficient solution to power an ESP32 but considering the slight time pressure, this solution will be adopted for now.

Another issue raised in the meeting with the project’s supervisor was the energy supply for the left hand, right hand setups and central hardware unit. The central hardware unit must supply power for the splitter and mixer and, additionally, for the ESP32-CU (central unit). The solution the splitter and the mixer was quite straightforward: both need a 9 volt power supply to function and the typical guitar effect pedal is also supplied by a 9 Volt battery. Accordingly, a 9 Volt battery or power input will be used to power most of the central hardware unit.

The greater challenge was to find a suitable power supply for the ESP32s. Using a 9 Volt battery would have been a practicable and comparatively small solution but as an ESP32 only needs 3.3 Volts a 9 Volt battery would also be a very inefficient solution. An Internet research session yielded two better possibilities: using 3 AA batteries or using one 18650 battery. The latter are lithium-ion rechargeable batteries with a voltage of 3.7 Volts. They are actually an ideal solution but are a bit dangerous when carelessly used. That is why the solution using 3 AA batteries was looked at first. At Neuhold a battery case for 4 AA batteries was obtained. Consequently, one battery slot had to be connected by a cable soldered to the contacts. To start off, the battery pack was used to power a LED, and this worked out fine. Next, an ESP32 connected to a ToF sensor was tested. The ESP32 sent the sensor data to the computer using MIDI-BLE so the USB-cable could be disconnected to see if it was really the battery pack that was powering the microcontroller. This worked as well. This made the 3 AA battery pack a suitable option to power the ESP32s in this project. The big disadvantage of using the AA battery pack is its size and unwieldiness. While this is not an issue for the ESP32-CU, it is a great problem for the right hand setup for example where the whole setup should basically fit into the back of the hand and the forearm without interfering too much with the actual guitar playing. This is why using a 18650 battery is re-considered, at least for the right hand setup.

When using the ESP32s on battery supply, it was decided that it would be a good feature to have some kind of battery monitoring, indicating a low battery status to the user. Luckily, there was a good tutorial online explaining how to get an ESP32 to monitor its own battery level. It appeared to be fairly straightforward, involving just two 10k resistors and a few lines of code. The idea was to flash a LED if the battery voltage dropped to a certain level, just like the principle of the power supply of electro-acoustic guitars where a low battery is also indicated by a red LED. Unfortunately, all (or just my) ESP32s seem to have a problem with the analogRead() function and the monitoring did not work as expected. This issue is something to be solved in the future.

Sources:

Everything You Need to Know About the 18650 Battery (commonsensehome.com)

Getting the ESP32 to monitor its own battery level – XTronical

After achieving the major goal of finding a final-product-worthy attachment solution of the left hand setup, it was decided to find out how to communicate between two ESP32s. Although the splitter and the mixer were still not working properly, the question of how to communicate between two (or more) ESP32s was still an important issue to solve. The manner of communication at this time of the project was MIDI-BLE so accordingly, solutions involving MIDI-BLE were investigated first. After doing some research, it was determined that communication between two ESP32s was possible via MIDI-BLE but quite complicated. However, another, entirely different option presented itself: apparently there is a way to communicate between several ESP32s using Espressif’s (inventor of ESP32s) native communication protocol called ESP-NOW.

According to the Espressif website, ESP-NOW is yet another protocol developed by Espressif, which enables multiple devices to communicate with one another without using Wi-Fi. The protocol is similar to the low-power 2.4GHz wireless connectivity that is often deployed in wireless mouses. So, the pairing between devices is needed prior to their communication. After the pairing is done, the connection is secure and peer-to-peer, with no handshake being required. Its features include:

• Quick and automatic connectivity once two or more devices have been paired
• Fast transmission speed
• Low power-consumption
• Improved Range and Reception

These qualities make it ideal for the GPRO project.

Following an online tutorial, it was quickly determined how to connect two ESP32 boards and establish a two-way communication between them. As always, a prototype sketch was made, focusing on the basic functionality of the ESP-NOW protocol. After basic communication capabilities had been achieved, ESP-NOW was integrated into the existing code for the left hand setup and the ToF sensor. The test was successful: now, ON/OFF commands based on the fret detected can be sent via ESP-NOW from the ESP32 connected to the sensor to the ESP32 simulating the future central hardware unit. It was noted that the transmission rate is indeed very fast and appears to be greater than the one of MIDI-BLE.

In a meeting prior to Christmas, the project’s supervisor suggested computing the frets and sending corresponding ON/OFF commands could be done by the ESP32 in the central hardware unit (ESP32-CU) instead of the ESP32 at the headstock (ESP32-HS). This would eliminate the need for the ESP32-CU to send the selected threshold to the ESP32-HS and a one-way communication would be sufficient. This will be done.

The ESP-NOW protocol must still be implemented in the code for the right hand setup and IMU sensor. However, it will likely be just as suitable for the right hand setup as it was for the left hand setup. The code for the right hand setup must still be changed anyways because by using the new and final attachment device for the right hand setup, the IMU sensor will be placed in a different position than before, resulting in the need to adapt the sensor readings. But more of this, in the next blog post…

Sources:

ESP-Now Overview | Espressif Systems

ESP-NOW: Espressif’s Wireless Communication Protocol | Espressif Systems

Getting Started with ESP-NOW (ESP32 with Arduino IDE) | Random Nerd Tutorials

As mentioned in the post before, jumper wires were soldered to all audio jacks in order to test the splitter and mixer by simulating two FX loops. As stereo jacks had been bought, it was important to solder the wires to the correct parts on the audio jacks to make them work with mono TS guitar cables. With the audio jacks sorted, the FX loops were then simulated by leaving one signal dry and putting a booster in the second signal chain. If the signal got louder, the FX loops should work as expected. Unfortunately, the circuits were not perfect yet. While both circuits, that of the splitter and that of the mixer seem to work fine on their own, they produced a very loud humming noise leaving the guitar signal barely audible. Sometimes it did not work at all.

Before Christmas, the project’s supervisor was consulted, and he provided a picture of the same mixer circuit that he recreated for some other project. Using this picture as a guidance, the mixer circuit was tackled again. Additionally, a bigger breadboard was bought to better accommodate the circuit. Nevertheless, the mixer’s performance did not improve a lot. While it seems to work, the humming noise is still very present making it unsuitable for further use within the project. As of now, the splitter/mixer components of the hardware switch are not working and a solution for this major problem must be found.

Another challenge was to find a solution for the attachment of the ToF sensor of the left hand setup. As already mentioned, the attachment device should be flexible enough to fit onto both, Fender headstocks as well as Gibson headstocks, the two most common headstock shapes in the guitar world. The first idea was to use a swan-neck and some type of table clamp and said parts were ordered. Upon delivery however, it was clear that the table clamp was way too big and heavy for the comparatively fragile guitar headstock. Additionally, the swan-neck proved to be too short and too inflexible so the sensor could not be positioned correctly. Afterwards, a swan-neck lamp by IKEA was tested. It was more lightweight and flexible enough to position the sensor correctly. However, it was not stable enough and changed its alignment with every move of the guitar. This, of course, made rendered it impossible to be used as an attachment device. There was more luck to be found with the third possible solution, a device consisting of a clamp and a semi-flexible arm with joints originally made to hold a GoPro. The clamp was small enough to fit onto guitar headstocks. The arm was flexible enough to position the sensor in the way required for a Fender headstock and a Gibson-type headstock. At the same time, it proved sturdy enough to hold its position while moving. A small setback happened that could be sorted out quickly: for testing a different (but very similar) ToF sensor of the type VL53L0X was used that could not detect the frets as accurately as the original ToF sensor. Worried that the new attachment device required major changes in the Arduino code of the ToF sensor, the original VL53L1X sensor was used. Luckily, it worked fine and could detect the frets as accurately as with the old attachment device. Just to be sure, another VL53L1X sensor was ordered as a back-up.

While switching between two FX loops was now theoretically possible, a DRY/WET control still needed to be implemented. My supervisor suggested using the schematics of a splitter and a mixer found on the Internet. As the name already suggests, the splitter splits the incoming guitar signal into two separate signals. These signals each have their own FX loop and go through different effects. Then, they are reconciled in the mixer, where the DRY/WET ratio of the two signals is set and they are both unified into one signal that is then sent on to the amp. The unfortunate part was that by using this method, no bypass switch using relay, MOSFET or Darlington transistor was necessary anymore. Nevertheless, important things have been learnt and the research and tests than were not completely for nothing.

Having never worked with schematics before, basic knowledge about reading and understanding schematics and the components involved had to be acquired. Then, it was decided to start of with the splitter and the necessary parts were bought at Neuhold. Luckily, a sound designer colleague could help me out with a 1uf Elko transistor which was not available at Neuhold. With all parts gathered, it was tried to assemble the circuit. The circuit was assembled on a breadboard. The advantage here is that the parts are not soldered and, thus, fixed in their position. Consequently, mistakes can be more easily remedied. Following the instructions, the circuit was built. To great surprise, it seemed to work. The circuit was tested using two guitar amps and a guitar. The signal of the guitar could be split up to go into both amps at the same time. Thus, the test was deemed successful. However, the two split-up signals appeared to be weaker than the one signal. According to my supervisor, this could be because of the use of different transistors than recommended in the schematic. It will be tried to acquire the correct transistors in order to remedy this problem.

With the splitter working, the mixer had to be built. The mixer posed a greater challenge. Fortunately, my supervisor provided me with the parts necessary. The building of the circuit, however, proved to be quite complicated. The schematic was firstly recreated in the DIY layout creator that lets you create circuits in a graphical way. With the circuit drawn like that, the actual circuit was built on a breadboard. It was testes by plugging in two guitars at the inputs and the output was plugged into an amp. The test was successful: Both guitar signals went through the circuit and into the amp. Furthermore, their individual volume levels could be adjusted using the potentiometers at the input. Nevertheless, I am not entirely sure that everything was done correctly – it just seems too good to be true.

Unfortunately, both parts, splitter and mixer, could not be tested in conjuncture due to the lack of two (!) crocodile clamps that are used to connect the audio jacks to their inputs and outputs. Thus, I cannot simulate the whole signal flow and experiment with the two FX loops yet. However, I have decided to solder cables to the remaining audio jack to be able to test my setup.