Since well before the earliest recorded history of humanity, music and dance have been inexorably intertwined. These two art forms probably developed together, each inspiring in the other the invention of new forms and modalities for expressing what words cannot. And yet, each discipline requires its own practitioners, who must learn their respective skills separately before they can collaborate to create a unified combination of sound and movement.
Thanks to modern electronic technology, the nature of this collaboration might well change. Joe Paradiso, a principal research scientist and director of the Responsive Environments Group at the Massachusetts Institute of Technology (MIT) Media Lab (www.media.mit.edu), recently won the 2000 Discover Magazine Award for Technical Innovation in Entertainment for his Expressive Footwear system.
In the current version of this system, sensors are embedded within a pair of Nike sneakers with Dr. Scholl's foam inserts (see Fig. 1). These sensors include force-sensitive resistors (FSRs) in the toe that measure continuous pressure at two points under the toes and one point above them and a piezoelectric strip that measures changes in pressure under the heel. A pair of back-to-back resistive bend sensors measures the bidirectional bending of the sole.
All other sensors are on a small circuit board mounted on the outside of the shoe. A dual-axis, micromechanical accelerometer measures slow rotations about the shoe's length and width axes. You can think of these as the x and y axes, both of which are parallel to the floor. A 3-axis piezoelectric accelerometer measures sharp impacts and fast kicks in three dimensions. The shoe's spatial orientation is detected by a magnetic sensor that acts like a 3-D compass within the Earth's magnetic field, and a vibrating-reed gyroscope measures the rate of rotation about the z axis, which is perpendicular to the floor.
The circuit board is also equipped with a detector that receives sonar "pings" from transmitters located around the dance floor and measures the shoe's relative position within the performance space. In addition, an electrode beneath the insole can measure how far the shoe is above a properly equipped floor.
All signals from the sensors are digitized by an onboard microprocessor and broadcast to an offstage base station from a small FM transmitter and antenna. Two shoes transmit on different frequencies to separate base stations. At a data rate of 20 Kbps from each shoe, all parameter values can be independently updated 50 times per second.
The base stations send the data over a serial connection to a Windows computer running ROGUS, a MIDI-processing program written at the Media Lab. This software maps the incoming data to any desired musical structure and sends the appropriate MIDI messages to various synths and other devices to generate music, which can be presequenced motifs and rhythmic loops as well as free-form notes and chords. Various controls modulate volume, pitch, panning, envelopes, tempo, effects, and any other musical parameters.
By measuring 16 different, continuous motion parameters in each shoe, this system offers great potential for generating highly expressive musical events. Of course, making good music still requires a musician to program the system, so the collaboration between musician and dancer will continue. But their roles are likely to change, and it should be very interesting to see how these changes affect the future of dance performance.