Nerve Sensors in Inclusive Musical Performance

Methods and findings of a multi-day performance research lab that evaluated the efficacy of a novel nerve sensor in the context of a physically inclusive performance practice.

by Lloyd May and Peter Larsson

Making the musical performance more accessible is something that many artists, such as Atau Tanaka and Laetitia Sonami, as well as scientists, have been aiming for a while. Therefore many efforts go toward the “nerve-sensor” direction. With this kind of approach, the detection of signals from nerve fring is more likely to happen rather than the skeletal muscle movement, so performers with physical conditions have more control over the sensors.

Even though the variety of gestures wasn’t as broad as other gestural instruments offer, the affordance of communication of gestural effort was better as proved in the explorations made on different sound-practices like free improvisation and the development of a piece called Frustentions.

Thanks to the Electromyography, a technique used to measure the electrical activity of skeletal muscles and nerves through the use of non-invasive sensors placed directly on the skin, we have seen more and more people with muscle atrophy or compromised volitional control of skeletal muscles, having access to technologies, for example when it comes to gaming. But, as it usually happens, the broader the accessibility is the more potentially harmful lens can come with it. Therefore it is important to keep in mind that every individual is unique and be aware of the invisible boundaries that the technology can set around the people it’s supposed to serve.

The more people with different physical and mental abilities get involved in these sound-making explorations, the better and opener accessible the design of the interfaces will be.

For this specific exploration, there were 4 investigated parameters: sensor position, gesture types, minimal-movement gestures, as well as various sound-mapping parameters. The lab was structured into several sessions, each concluding with a performative exploration, as well as a structured public showcase and discussion at the end of the lab. Other research lines like minimal-movement “neural” gestures were also investigated but not much data could be gathered. The outcome of the session was the previous said composed piece: Frustentions. A fixed media composition developed during the workshop.

Three groups of gestures were determined during the sessions in order to record the needed data: Effort gestures, which were particularly suited to audio effects that are well-aligned to swelling such as distortion, delay, or reverb, and adjustment gestures, which often required full focus and were not necessarily accessible at all times during a performance; and trigger gestures.

The nerve sensor was compared with other interfaces like the MiMu glove, the Gestrument, and Soundbeam. Even though these other instruments allowed wider recognition of the number of gestures with better accuracy, it was more challenging to use with limited fine-motor capabilities. In addition, the wearable sensor afforded the performer greater opportunities to connect visually with ensemble members or the audience as there was no immediate requirement to view or interact directly with a screen.


Research aimed at making musical performance accessible to everyone is something that has multiple benefits, clearly on a physical level, but above all on a neural and psychological level. It is surprising how many things associated with leisure are out of reach for many people, simply because their physical condition does not meet the standards for which they are designed. The possibility that all these people can access activities of enjoyment represents a clear increase in the quality of life for them and for the people around them.

Nerve sensors are just one example, and thanks to this exploratory initiative we can get to know them and compare data with other instruments on the market. In more advanced stages of research, I would like to imagine that these interfaces are also used medically, to alleviate the effects of some diseases, improve physical conditions, and even reduce motor damage that originates in the brain by promoting nerve and muscle movement. Music is obviously a means of enjoyment, but together with science, it can be a means of healing.

A good image worth more than a 1000 words 

Radiology is the discipline that is responsible for deciphering the information contained in images of our body to diagnose or inform other specialists and even the patients themselves of how the pathologies suffered are developing

Radiologists are exposed to a workload of approximately 200 cases in a normal working day. 200 cases equal 200 people, with different illnesses, and different stories. With this workload, the person’s story is normally relegated to the background, but we all know that human closeness in a context like this is a necessary asset and it should always remain in the foreground. 

There are different software tools, as well as imaging standards, that radiologists use to do their work. Among them, we could speak of DICOM (Digital Imaging and Communication in Medicine), the standard for the communication of information related to images. Regarding software, we could talk about PACSonWEB, a portal where both hospital specialists or doctors from private practice, and patients themselves can easily access their image repositories, avoiding the bureaucracy and long waits involved in transferring from one source to another. 

Companies from all over the world are dedicated to the development and improvement of these systems, trying to reduce as much as possible the time that specialists spend examining each image so that they can dedicate that time to their own professional well-being or to dealing directly with patients, procuring that necessary human proximity for a longer time. 

How can design help in this specific specialty? Through the interfaces and the constant study of the behavior of professionals in the work context. The interfaces with which doctors interact daily should be clear and accessible, all the most used tools should be quickly identifiable and above all, the series of images should be able to be combined in the way that is necessary to obtain a 360 view. degrees of the pathology being analyzed. It must be taken into account that these people spend the day in a dark room, therefore, the interfaces must be designed so as not to overexpose the eyes of the professionals. Likewise, it is at that very moment of image analysis in that dark room when the diagnosis is made, not afterward. Typing would be a waste of time, therefore the design and development of dictation tools and good speech recognition are highly necessary

To sum up, the continuous interaction with radiology professionals to understand their needs and how they work, the design and conception of interfaces in which navigation is totally intuitive, and the reinforcement of tools that facilitate voice interaction within the software, are the steps that the UX Design and UI Design must imminently address in order to improve the quality of life of radiologists in their work environment



Interview with a cardiologist (C) 

In the course of my visit to Spain during the Easter holidays, I had the opportunity to have a short interview with one of the heads of the cardiology department of the Salamanca general hospital.

After listening to what the doctor told me, 3 things were clear to me:

  • The time spent on correcting minor technical errors or sharing information between specialists should be reduced to the maximum in order to be able to use it in dealing directly with patients.
  • Advances in technology are very effective, but we cannot forget that the main customers of health systems are the elderly people, and this is going to be the tendency at least in the near future.
  • Pedagogy is key in order to making patients stop seeing the hospital as a hostile environment. Time and tools are needed to provide this inclusion prior to treatment and, if possible, the availability of real people for those who do not feel so comfortable with virtual assistants.

Here is the full interview.

*This interview has been the first instance recorded in Spanish and afterward translated and transcripted to this document. 


Me: Hello! Thank you for taking some time for me, I know you have probably a very packed day. 

C: Yes, indeed. But no problem at all, thank you for coming. 

Me: My pleasure. Just a little information before we jump into the interview itself: This interview is for educational use. I want my master’s thesis to be related to healthcare, more specifically speaking I want to find out what are the design challenges healthcare is facing right now. Since healthcare is just a big umbrella that brings together many different disciplines, I have decided to delve into radiology. I think it is a field where many different types of software and hardware are used, and also there is an ongoing relationship with other specialists and patients if I am not mistaken. All of this makes the field very attractive for a designer, and that’s why I’m here. 

The information will be shared in a blog that students and teachers from my department can access as well as will be used to shape the final version of my master thesis. Only if you consent, I’ll use some of the comments made during this session as quotes within the master thesis, only then, by using the proper form of citation. And here ends the bureaucracy. 

C: hahaha! No problem, of course, you can use the information for your master thesis and I’ll be happy to be quoted within it. So go ahead! 

Me: Thank you so much. Let’s start then. 

Me: How long have you been working here? 

C: 15 years already! 

Me: That is time enough to gather many different experiences. Tell me a bit about your daily routine. How many patients do you see on a normal day? 

C: 15 to 30. Depending on the pathologies. Sometimes I meet on the same day with patients who are going through similar pathologies because it is easier for me to do a bit of pedagogy with them and also to easily access the repositories. There is a huge archive where all heart diseases are collected, and they are labeled in alphabetical order by the name of the pathology. Therefore, I spend less time looking for a specific record if I move around the same area all the time. But of course, I always give priority to urgent issues, and you never know when a new one is going to show up during your day. 

Me: Besides meeting your patients, do you have other important meetings that you have to attend in your day? 

C: Other cardiologists. Residents. Specialists from many different fields, like, for example, radiologists. We compare diagnoses, we talk about further steps within a patient’s treatment… Nurses, and sometimes administrative staff. 

Me: Between all of those meetings, do you have time to take a break every now ad then? 

C: Hmmm… I guess I do have breaks, but not really as many as I’d need. Sometimes I don’t have time to eat properly or I can’t take the time to clear my mind after giving bad news to a patient. Although we are used to dealing with bad scenarios and informing patients about them, in the end we are all human beings, so it is also a human need to have a space to digest that you “hurt” someone’s feelings before going on with your day. 

Me: I understand. The breaks are not only to have a physical rest but also to recompose yourself psychologically. 

Me: Is there something you think that will make you have more of that time you are seeking? 

C: Hay muchas veces que las reuniones se podrían hacer más rápido. Algunos trámites administrativos podrían hacerse más rápido. La transferencia de datos de médico a médico para un diagnóstico posterior debería ser más rápida. Incluso hay ocasiones en las que tenemos que repetir procedimientos por falta de instrucciones dadas al paciente, o alguna pérdida de información. Son cosas que no deberían estar pasando en un gran hospital como este, porque si hay algo que buscamos, en realidad, todos buscan, es tiempo. 

Me: Let’s go to the analysis of some of the devices and procedures that you daily use. 

Me: What kind of devices do you use more often? 

C: Cardiac ablation catheters, Cardiovascular angioplasty devices, Cardiac pacemakers, Implantable cardioverter defibrillators, Prosthetic (artificial) heart valves, Stents, Ventricular assist devices, “domestic” monitor devices, web and mobile applications or portals. 

Me: Can we focus on the last ones? As I understand, patients can get more involved in those. 

C: Definitely. 

Me: Within the use of “domestic devices or those that patients take home, is the data collected sufficient to provide a complete diagnosis? 

C: Yes and no. For many patients, it is enough because we only look for abnormalities. So we see them or we don’t. But answering your question, no. It is not enough to completely diagnose a pathology. If we find an abnormality through the monitor, we will have to perform further procedures to find the best way to proceed with that specific patient. And many of those procedures will need to be done inside the hospital. 

Me: How many times do you need to repeat monitoring procedures? Average per year. 

C: Around 15% of the whole amount of procedures.  

Me: What do you think is the main cause? 

C: Technical problems many times. But there are also times when patients don’t understand exactly what we need for them, so we have to explain again how to interact with the device, what they can do, what they can’t do, and redo the procedure. 

Me: Is there a specific demographic group where the repetitions occur more often? 

C: Elderly people. In reality, most cardiology patients are older people. Sometimes we have children or young people who have heart problems but… You know, we are in a very old society. And it is expected to be even greater in the coming years… Medicine has to see the elderly as its main client and adapt procedures to them. And that takes time. Exactly what we don’t have here. 

Me: These elderly people, is there something they complain about when they are required to wear this device on them? 

C: They feel insecure. They don’t see why they need this machine if I’m here. I’m the doctor, what a machine is going to the better than me? They have a lot of questions. Many don’t even understand the most basic things so… They feel uncomfortable, even though they have to wear the device for only one or two days. 

Me: Do you think these people feel involved when dealing with these devices? Do they feel like an active part of the procedure or a secondary actor? 

C: Honestly, I think they feel totally aside. They don’t know what is going on. They follow instructions. Everything is cold and aseptic. They don’t have a voice. But they do have a voice, but many times we don’t have the time to listen to them because we use all our available time discovering what is going on there.  

Me: What do you think could be done in order to improve patient engagement and involvement?  

C: Definitely avoid all of those technical errors that are grabbing our time for the face-to-face relationship with our patients and the pedagogy we could do with them. More time = more pedagogy = less repetition of procedures = more rate of success when finding the diseases in their early stages. 

Me: What do you think has been already done to improve this? 

C: Many of our applications and portals, as well as AIs, provide training to our patients previous to their procedures. We go back to the problem of getting the elderly people there. They rather speak with you than navigate an application looking for the information they need or listen to the instructions from a non-human character. This has been widely discussed among the medical staff, so the awareness is also a good point and I guess some steps will be done to address this. 

Me: Thank you so much for your time. I think I have all I need. Do you maybe want to talk about something that I didn’t ask you that you would like to share with me? 

C: I think I don’t have any further information I can share with you. I already talked too much! But if more questions come up to your mind, don’t hesitate in contacting me. Thank you for the interview! 

“A very long Wait”

#The Topic

“One month after my episode I finally got the device, I wore it for 2 days, and then I gave it back. My doctor called me 10 days after, I will have to go through the process again because she needed more data. So… almost two months will be passed by the time I get a diagnosis. It’s a very long wait”.

This is one comment I got from one of the patients I interviewed. The feeling of being involved in a process that takes a lot of time to give a result back creates a feeling of uncertainty in the patient while waiting for a diagnosis. This can lead either to anxiousness or tedium depending on the patient’s personality.

The old-school Holter devices method of data transmission relies on the patient returning the device, downloading the data to the hospital, and analyzing it by the doctor. In addition to time, the effectiveness of the collected data is questionable due to the fact that these devices record information for two days in which there may or may not be heart rhythm disturbances.

New devices are testing the use of advanced algorithms that allow data to be recorded 24/7, offering clear advantages when it comes to making a proper diagnosis. In addition to this, they allow the possibility of comparing with other data already recorded from patients with similar indications, which can facilitate the analysis process.

The use of this new technology is not yet fully widespread, but it is obvious that this is the way that must be taken to increase the speed of diagnosis, facilitate the doctors’ work, and care about the patient’s mental health. Maybe the use of this type of new generation holters, which can be easily used by patients at home, combined with specific and occasional medical tests (if necessary) carried out in the hospital by more complex devices that can record more interesting data, is the recipe that offers both better results and better user experiences. But, could domestic use devices reach the complexity of hospital devices without leaving aside their comfort of use for the patient? Probably, but when? Is it going to be a very long wait?

Pocket ECG Holter Monitor | New Generation of Holter Monitors

Improving medical interfaces for patients

#The Topic

The Holter Monitor Interface is quite dated, bulky, and unintelligible. Patients do not feel involved in the process and do not understand what parameters are being measured.

In a world where technology users are more educated and informed than ever before, leaving patients out of the loop feels paternalizing. Making users feel disconnected from their own condition poses risk to their own health.

In the case of the Holter Heart Monitor, it is a much more advanced and capable device than the health wearables available in the market for consumers (e.g. Apple Watch, Miliband, Fitbit). But the interface in these wearables is more user-friendly and clear to inexperienced eyes.

Marrying the accuracy and depth of the data from the Medical Grade device with a tad of UX from the consumerready devices can help the patient understand what is going on in their body and be proactive towards a solution.

A better interface is the start of a change: From “patient”, by definition, waiting and passive, to an active player in their health status.

Garmin/Apple Watch/Fitbit

Living with a Heart Rate Monitor

# The Topic

heart rate monitor (HRM) is a personal monitoring device that measures heart rate in real-time or records the heart rate for later study. It is commonly used to collect heart rate data while performing various types of activity which are part of the patient’s day-to-day life. Portable medical devices are referred to as Holter Monitor which is designed for everyday use and does not use wires to connect.

Modern heart rate monitors commonly use one of two different methods to record heart signals: electrical and optical. Both types of signals can provide the same basic heart rate data, using fully automated algorithms to measure heart rate.

_ Electrical Devices: The electrical monitors consist of two elements: a monitor/transmitter, which is worn on a chest strap, and a receiver. When a heartbeat is detected a radio signal is transmitted, which the receiver uses to display/determine the current heart rate. This signal can be a simple radio pulse or a unique coded signal from the chest strap.

_ Optical Devices: More recent devices use optics to measure heart rate by shining light from an LED through the skin and measuring how it scatters off blood vessels. Smartwatches and cell phones can be included within this category, but their use for medical purposes is limited even though the accuracy in detecting several diseases increased significantly in recent years. Many professionals recommend anyway their use as support in data collection processes.

# Holter Monitor

A Holter monitor is a small, wearable device that keeps track of your heart rhythm. The doctor may want the patient to wear a Holter monitor for one to two days. During that time, the device records all heartbeats. This procedure can be repeated several times if the medical practitioner requires it to accomplish the goal of the overall study.

A Holter monitor test may be done if a traditional electrocardiogram (ECG) doesn’t deliver enough information about the heart’s condition. A Holter monitor may be able to spot occasionally abnormal heart rhythms that an ECG missed due to the short time that the patient is hooked up to the machine. The medical practitioner uses information captured on the Holter monitor to figure out if the patient has e a heart rhythm problem or a heart condition that increases your risk of an abnormal heart rhythm.

# What Patients say:

After interviewing two patients who had to wear the Holter monitor due to different heart conditions, plus the information collected in different clinical studies, these would be the most significant insights referring to the patient experience:

_ Most of the patients found the device uncomfortable. Sometimes, some of their daily activities were difficult to carry out due to the device.

_ Some patients had difficulty putting the device back on after showering. Due to this, the collected data wasn’t accurate.

_ Even though the majority of patients expressed full trust in their doctors, many did not feel involved in the process. They simply had to “follow orders” but did not understand what kind of data was being measured.

_ Some patients experienced a feeling of vulnerability when they gave the device back to their medical facility. Although the device was uncomfortable to wear, it made them feel safe, assuming that their heart was being controlled. Once they removed it, that feeling of security disappeared.

_ Some patients experienced skin allergies due to the adhesive tape on the device.

# What Doctors say:

After interviewing a cardiology physician and a general practitioner, I was able to gather the following insights:

_ It is difficult to ensure successful measurements with elderly patients due to the technical requirements of the device. The majority of patients using this device are older than 60 and sometimes they need to repeat the procedure more than once.

_ Although the device collects valuable information, some conditions, which might be symptoms of more serious heart problems, go unnoticed. (ex. certain types of arrhythmias).

_ They get frustrated because they cannot find the time to explain all the details of the procedure to their patients and they feel patients’ dissatisfaction and uncertainty.

_ “It takes too long to get the data that we need”. Patients must return the device back to the medical center, then the technicians extract the data from the device by connecting it to their system, then the data are included within the patient’s records and saved in databases. Only then doctors are able to have a clear vision of what is going on with their patients.

# Compared with other Devices for Heartbeat Monitoring:

Despite the fact that the Holter monitor is one of the most used devices to measure the heartbeat even today, for a few years, other heartbeat monitors have been coming onto the market offering significant improvements for patients. This is the case of Zio Patch, from iRythm Tech.

But even though this new monitoring option is more manageable, less cumbersome, and has a higher level of data accuracy than the original Holter Monitor, it requires longer use than the Holter Monitor to detect the same number of conditions. This makes procedures a bit harder to perform with both children and the elderly.

As mentioned before, many smartwatches and mobile phones can provide a service similar to heartbeat monitors but are not capable of perceiving certain abnormalities that may indicate heart problems. They are a very good aid for home use because they measure many different parameters such as oxygen saturation, thanks to the use of sensors including accelerometers, gyroscopes, and GPS.

# Conclusions and Challenges

Finding the ideal device is not easy. Both patients and medical professionals have needs that should be met in order to succeed when going through these procedures, and it seems that if you try to improve one part, the other will get worse.

For me, the main challenge is to rethink the design of the device. It is clear to me, that it needs to be something free of technical assistance. The ease of use must be a basic requirement due to the characteristics of the patients who generally use this type of device.

On the other hand, it should offer some kind of interface that could deliver clear information to patients, making them understand what data is being collected. This is already offered on smartphones or smartwatches.

And last but not least, the data collection. The more accurate data collection in the shortest possible time, the medical professionals will be able to deliver better diagnoses in less time. In this way, they will be able to care about the psychological needs of patients before, during, and after the procedures. In my opinion, finding a way to shorten the process from data collection until professionals can access and evaluate this data, should be a priority.

# References:,patients%20than%20the%20Holter%20monitor.

Data visualization in health systems.

Knowledge is power. Nowadays all kinds of sensors are continuously gathering data about our behavior, our physical activity levels, and different health indicators like blood pressure, blood oxygen levels, and more. All of this can be monitored through phones, smartwatches and other IoT devices conceived specially for medical applications

What happens with all this data? Raw data is useless if we don’t know what we are looking for.

A good strategy for data visualization is the key to the proper use of data.

When trying to create a better user experience for both health professionals and patients, we need to distinguish which information is needed for each side and how it should be presented:

Medical Staff:  An overloaded system can be helped by synthesizing and simplifying the amount of data that the doctors need to analyze.

A good presentation of numbers and facts can help the doctors identify low-risk patients faster, so they can prioritize and leave more time for those who really need it.

Patient: Being a patient is already a stressful situation. Medical terms that patients are not familiar with and uncertainty don’t help make the situation more comfortable. Feeling lost and uneducated is not the feeling of reassurance that a medical visit should give.

Presenting facts and figures in a clear and didactic way will make the patient feel included in the process and learn about the condition, gaining implications for the treatment.

_ Literature

IoT devices for medical applications

Data is driving change in most industries in the past years. Enterprises gathering significant data from users can tailor their experience better to them.

Major tech players have entered the field of Health in recent years. Fitness promotion, health indicators monitoring, and emergency contact are some of the key features that companies like Apple, Samsung, Meta, Fitbit, or Garmin have in their offerings through tracking wearables in form of smartwatches with GPS, Barometer, Gyromenters and an array of medical-grade sensors, like Apple’s FDA Approved ECG sensor in the Apple Watch.

This ability to gather data from the user in a non-invasive and continuous way is great news for health services around the world. Some of the main Health indicators, like blood pressure, oxygen levels, and glucose levels that previously required a visit to the doctor now can be done in real-time and telematic, reducing pressure on a strained health system and reducing cost in an ever-increasing expense for governments around Europe.

Telehealth or remote doctor’s consultations are another great option for non-urgent questions that don’t require a physical appointment anymore and help prevent overcrowded hospitals.

A sensible way to collect data can help health administrators track a population’s overall health and implement policy to help their citizens lead a healthier life because prevention is always a better solution than treatment. An easier and most continuous way of following up with patients via Telehealth is also a better way to ensure that people are on track with their treatment in a more efficient manner.


eHealth as the way to Improve Patient Experience


As we have seen recently, the health systems of the countries can be subjected to great burdens of responsibility and work for medical staff who must monitor the well-being of patients and in turn, seek psychological wellness during medical procedures to which such patients are subjected.

Since 2005, the World Health Organization (WHO) recognized eHealth as an essential part of healthcare systems and since then it has not stopped growing, expanding in multiple directions such as mHealth (use of mobile technologies to shorten the distance between patients and medical services ), TeleHealth (as a way of offering medical services over a distance – about 80% of consultations not related to covid were made by telematic means in 2020), the integration of AI (to suggest treatments or clinical diagnoses – the so-called Clinical Decision Support) or Data visualization (to bring the understanding of patients concerning their own health and facilitate to medical staff the sharing of data about their patients). These are just some of the central topics in this growing field.


Within this infinite topic, I am especially interested in everything that is focused on the patient experience, meaning everything that in some way will improve the patients’ experience from when he/she decides to go to the consultation until he/she obtains the recommendations for treatment and subsequent follow-up.

There are already many things done in this regard, even not related to technology and making use of more analog processes, but I would like to see what are the possible improvements or development of new functions in the field of AI (for example, assistance to patients outside of consultation) or imaging/data visualization (focused on the understanding and engagement of the patient in their health status).

#Good practices

MRI Scan to improve childrens’ experience – by Doug Dietz from GE Healthcare

#Design Challenges

Health systems are an environment where designers encounter many pain points that could be addressed. Trying to find a solution to these pain points entails certain challenges such as:

_To try to focus on only one thing at a time: There are many issues that need to be addressed, but it is important for designers to choose one of them and keep the focus clear during the whole design process.

_ The field requires a lot of time spent on research, understanding, and empathizing with users’ needs. There are thousands of people, both medical professionals, and patients, who might be affected by the same issue. It is important to take the time to collect all information and insights related to the topic.

_ To keep aware of what is going on out there. Since healthcare is a worldwide topic, many companies, designers, and institutions may be way ahead in their discoveries regarding the same issue. To be part of the huge healthcare community is mandatory to move forward in making improvements.

_ To become familiar with the treatment of sensitive data, data visualization, and techniques/machines which are commonly used in medical practices to treat different medical conditions.

_ To be mentally prepared to face a long design process with possible significant unanticipated changes.

Virtual Nurse App – by