Reactive Video: Movement Sonification for Learning Physical Activity with Adaptive Video Playback


Paper by Doga Cavdir, Chris Clarke, Patrick Chiu, Laurent Denoue, and Don Kimber

https://nime.pubpub.org/pub/dzlsifz6/release/1

This project aims to improve physical learning, in this case Tai-Chi, by implementing audio along with visual feedback. As you do your Tai-Chi exercises music composed of bell, chime and wind sounds plays in the background and two parameters: tempo and alignment of the joints compared to a reference video, are measured. If you drift from the reference in either parameter you will be given an audio-cue that you are getting off course. When drifting in alignment a low pass filter is applied to the soundscape cutting off some of the ethereal richness in the music and if you drift in tempo the tempo of the music changes. I think this sort of feedback is very appropriate for physical exercises considering the fact that you often have to use your vision for balance or coordination. The way they implemented the feedback in such an unobtrusive and unannoying way is very fitting for the meditative act of Tai-Chi, however I think it is a great way to approach any sonic project. They did not mention it in the paper, but I guess you correct your posture by looking at a visual. However, I wonder if it would be possible to eliminate the visual component by adding more parameters. For an activity like Tai-Chi this might not be feasible because of all the factors at play, but for a simpler activity, for example biljards, it might be possible.


Leave a Reply

Your email address will not be published. Required fields are marked *