The Outer Limits

Over the years, innovative approaches to using physical gestures in electronic music have been highlighted in the pages of EM. Although some of the technologies
Publish date:
Social count:
Over the years, innovative approaches to using physical gestures in electronic music have been highlighted in the pages of EM. Although some of the technologies

Fig.1 : The MTC Express, from Tactex Controls, can track the Velocity andposition of several stimuli simultaneously.FIG.2: More than just a ribbon controller, the Kurzweil ExpressionMate cansend Note On and a variety of controllermessages.FIG.3: David Wessel’s custom Thunder interface. Notice that some ofthe controllers have two sliders: one for vertical finger position, theother for pressure.FIG.4: Instruments from Starr Labs can be customized with variouscontrollers to fit the performer’s needs. Shown here are theMini-ZXS and Mini-ZS, which include Velocity-sensitive stringtriggers.FIG.5: Laetitia Sonami’s Lady’s Glove includes sensors thatdetect the distance between the glove and thefloor.FIG.6: The BodySynth can track up to 12 EMG sensors simultaneously fortranslating muscle movement into MIDI data.FIG.7: All of the I-CubeXsensors come prewired and ready to use—no soldering is necessary.As many as 32 sensors can be used with the system at onetime.FIG.8:The BigBriar Ethervox theremin is a world-class performance instrument withextensive MIDI capabilities.FIG.9: The ultrasonic sensing capabilities of the Soundbeam 2 make thesystem ideal for situations that require noninvasivebiofeedback.FIG.10: The translucent, pressure-sensitive rubber pads of the Rhythm Treeare an inviting interface for sonic exploration in the MindForest.FIG.11: Tod Machover of MIT’s MediaLab performs in the Sensor Chair,which senses the position and movement of the performer’s armsand upper body.FIG.12: Boston Pops conductor Keith Lockhart wears the Conductor’sJacket during 1998’s Tech Night at thePops.

Over the years, innovative approaches to using physical gestures inelectronic music have been highlighted in the pages of EM.Although some of the technologies we’ve covered have hadmainstream commercial success, many technologies haveremained—either by design or by accident—on thefringes.

However, like artistic and musical works, the commercial success ofa new controller shouldn’t be the sole criterion for judging itsworth. Like hammers and chisels, controllers are merelytools—albeit, in some cases, rather sophisticated tools that mayrequire a major paradigm shift in order to understand their fullpotential. Ultimately, these new tools are only a means to a musicalend; a controller’s effectiveness at getting across musical ideaswill be the greatest factor in its success.

David Wessel, esteemed researcher of gestural controllers anddirector of the Center for New Music and Audio Technologies (CNMAT) atthe University of California at Berkeley, put it best when he recentlynoted, “We’re on the verge of a controllerrenaissance.” This is primarily due to the growing number ofmusicians and engineers fighting to keep electronic music a uniquemedium of expression rather than a means of mimicking establishedforms. Many composers and musicians are using input devices that retainelements of traditional instruments, while others are tracking gesturesin new ways by measuring motion, light, gravity, temperature, airpressure, proximity, and anything else you can imagine. Rarely in thehistory of music has there been so much work—and such variedresults—in instrument development.

With that in mind, we decided it was high time to survey the currentapproaches to alternative input devices for electronic music. Some ofthe approaches are of the most personal and intimate kind, whereasothers are geared for the mass market. This article will cover bothextremes, as well as the universe of approaches in between.Commercially available MIDI controllers based strictly on conventionalinstruments, such as percussion pads, guitar, bass, and windcontrollers, and variations on the standard piano keyboard (such asaccordion controllers) will not be covered.


Over the centuries, every culture has developed ways to expressitself musically. By contrast, the field of electronic music, just overa century old, is perhaps moving into its adolescent stage,metaphorically speaking. Few of the earliest performance-basedelectronic instruments have survived this short test of time, thenotable exceptions being the theremin and the Ondes Martenot. But untilrecently, technology imposed strict limitations on what a performercould do in real time.

These days, the tools needed to analyze and use multiple streams ofinput data in musical contexts are readily available and shrinking insize and price. In fact, everyday computer input devices, such asjoysticks and graphics tablets, are sophisticated and inexpensiveenough to work in musical contexts. And as musical-instrumenttransducer systems get smaller and more powerful, acoustic and gesturalsensors are becoming easier to use with traditional instruments.

Already, external cable connections are beginning to disappear oninstruments; external cables will disappear completely asmicrotechnologies become more available. Imagine the possibilitiespresented by miniature computers (including power supply and wirelesstransmitter) that are less than a cubic millimeter in size. Thistechnology is what Kris Pister at U.C. Berkeley’s Department ofElectrical Engineering and Computer Sciences terms “smartdust” ( that, perhaps within this decade, will have a majorimpact on real-time performance applications.


Even though technologies change rapidly, well-designed instrumentsnever become obsolete.

Recent technological progress has allowed designers to use powerfulprocessors in smaller spaces and to overcome difficulties in precisepitch extraction; however, they still come up against the bandwidthlimitations of MIDI. No matter how small and unobtrusive the sensorsare, MIDI still imposes a speed limit of 31.25 Kbps. And althoughfaster data-transmission schemes abound, none have gotten enoughpopular support to dethrone the MIDI 1.0 specification. Yet engineersand artists have worked around the problems presented by MIDI to createinput devices that allow new and exciting ways of making music.


Image placeholder title

MTC Express. One of the newest and most promising touchsurfaces being developed is a 3-D controller that uses smartfabric—a soft material developed by the Canadian Space Agencythat contains an array of sensors interconnected by a network offiber-optic cables. The MTC Express ($495), from Tactex Controls (, is asquare of smart fabric, measuring 5.75 by 3.75 inches, that is coveredwith a padded surface and housed in an anodized aluminum slab (seeFig. 1). The entire unit weighs just 17 ounces.

The MTC Express can track multiple contact points within a 2-D (x-y)field, with a sensitivity of around 100 dots per inch. It also measures256 levels of pressure. The surface of the smart fabric is covered withnumerous taxels, each connected to a pair of fiber-optic cables.Because of the sensor density, the MTC Express can track very subtlephysical gestures.

Light from an LCD is sent down one of the cables and is returnedover the other. Sensors are used to track the deformation of the fibersby measuring the amount of light sent back down the cables when thefabric is touched. As pressure is exerted on the surface of the MTCExpress, variations in light are returned to the sensors.

Image placeholder title

ExpressionMate. At first, the Kurzweil ExpressionMate ($549)looks like the ribbon controllers that were used with analog synths inthe ’60s and ’70s (see Fig. 2). But unlike olderribbons, this one can send an impressive array of MIDI messages. Theribbon surface is divided into three sections, each individuallyassignable. The unit also includes three 16-step arpeggiators that cansend and receive data on separate MIDI channels. The control box can beconveniently mounted on the keyboard or on a mic stand.

Jam Bass. The Jam Bass ($253.50), by Kellar Bass Systems, isa MIDI controller (with internal synth) that attaches to the neck of anelectric guitar or bass. The control surface is made up of two rows of14 pads that mimic the fret layout below the E and A guitar strings.The pads are played with the thumb while the other fingers are on thefretboard. Using a ribbon cable, the control surface is attached to theCircuit Pack, which contains the processor, synth, and audio and MIDIoutputs. You can change Synth Voice and Performance modes using thepads.

Image placeholder title

Thunder. Although it debuted more than a decade ago, theThunder ($1,990), by Buchla and Associates, gives the performer asophisticated touch-control surface. The thunderbird-style surfacedesign is the result of ergonomic considerations—the layout comesfrom tracings done around the designer’s hands, and the long,featherlike control strips sit nicely under the fingers. Each of thesecontrol strips can track two control dimensions: pressure and position(see Fig. 3). All ten fingers can send pressure and positiondata simultaneously. Although the Thunder is no longer manufactured, alimited number of the controllers are still available directly fromBuchla and Associates.

A more recent development by the company is the MarimbaLumina ($2,995), an instrument that combines the familiar design ofa 31/2-octave mallet controller with advanced electronic technology andBuchla-style ingenuity. Designed by Donald Buchla with input frompercussionists Mark Goldstein and Joel Davel, the Marimba Lumina ismanufactured by Nearfield Multimedia, specialists in precision antennatechnology.

The Marimba Lumina comes with four color-coded, foam-covered malletsthat contain tuned circuitry. Embedded in each bar, strip, and pad onthe surface of the instrument is a radio antenna that can track andidentify the mallets, allowing each mallet to have independent controlfunctions. Although cosmetically it bears some resemblance to a moreconventional mallet controller, the Marimba Lumina is a highlysophisticated instrument capable of mapping a variety of responses toperformance gestures over its various control surfaces.

A special “Gold Edition” Marimba Lumina ($8,000) is a41/2-octave version of the instrument that features gold-plated barsand a curved frame that allows players to reach the furthest noteseasily. (See “What’s New” in the October 1999 issueof EM.) Both instruments include a Yamaha DB51 XG synthesizer,so they can be used without an external sound source.

Finally, there is the Marimba Lumina 2.5 ($1,995), a new21/2-octave version of the instrument (see “What’sNew” in this issue).

Starr Labs controllers. Harvey Starr, who heads Starr Labs(, has designed a number of variations on the MIDIguitar controller theme. Avoiding the common pitch-to-MIDI schemes usedin guitar controllers, Starr’s devices are more user interfacesthan MIDI guitar controllers. Starr’s controllers often combineguitarlike fingerboards with keys on the neck instead of strings.Sometimes a breath controller and joystick are thrown in for goodmeasure.

The following examples by Starr Labs represent only a fragment ofthe company’s output. Any of the configurations can be customizedto fit the needs of the musician.

Image placeholder title

The Mini-Z ($1,395) is a Velocity- and Pressure-sensitive 24-fretfingerboard designed for tapping. Options include a 4-way joystick, abreath controller, and a programmable strip along the side of the neckthat can add Modulation, Pitch Bend, or crossfades between sounds. TheMini-ZS ($1,795) includes the optional joystick and a set of sixVelocity-sensitive string triggers. The Mini-ZX ($1,695) adds a set ofdrum machine– style trigger pads. The Mini-ZXS ($1,995) containsall of the above: joystick, trigger pads, and string triggers (seeFig. 4). Note that the prices quoted above are for basic modelsonly; any added options or customization costs extra.

Starr Labs’ MT-48DD ($2,195) was originally designed forbassist Billy Sheehan, who wanted pedals that were better suited forbass playing than the traditional pedal configuration. The unitconsists of a 4 5 12 array of 2-inch rubber mounds that are playablewith mallets as well as with feet. Each mound is individuallyprogrammable and can send a MIDI event or group of events—up toeight per mound—including notes, chords, or even SysEx messages.The floor unit connects to a stand-mountable programmer that holds upto 32 user programs.

A collaboration between Harvey Starr, Stephen Taylor, andmicrotonalist Ervin Wilson has produced the Wilson 990 GeneralizedKeyboard Controller ($7,500). Inspired by the Generalized Keyboard thatR.H.M. Bosanquet developed in 1875, it is designed to work both as amicrotonal performance controller (with the ability to map multipletonal systems) and as a more traditional controller offering multiplemini-keyboards on a single surface.

The Wilson 990 includes nine ranks of 90 keys. Each rank is assignedits own MIDI channel—in fact, each key can transmit multipleuser-defined MIDI messages. The instrument comes with tuning softwarethat is compatible with E-mu and Ensoniq sound modules, as well asSymbolic Sound’s Kyma system.

Groups of keys can be defined for different musical purposes, anddifferent setups can be edited, stored, and recalled using a providededitor/librarian program. In addition to the banks of keys, the Wilson990 has four assignable sliders and a 4-way programmable joystick. Asmaller 288-key version ($3,200) is also available.

Besides these commercially available products, there are a couple ofproprietary touch-sensitive controllers that offer interestingsolutions to specific performance requirements.

JamODrum. Developed by Tina “Bean” Blaine and TimPerkis at Interval Research in Palo Alto, California, the JamODrum ismeant to inspire people to engage in spontaneous, collaborative musicmaking. In fact, the community drum circle became a metaphor thatguided the form and content of the team’s work. The designersalso intended the JamODrum as a way for participants to explore therelationships between rhythm and graphics. The resultant object—a7-foot circular table that includes six MIDI drum pads and doubles as avideo-projection surface—was something that people could gatheraround for jamming.

Blaine and audio engineer Kris Force spent several months slicing,dicing, and processing thousands of samples to create a custom libraryfor the project. During the JamODrum’s development phase, severalinteraction methods were available. For example, in thecall-and-response method, the sequencer plays short rhythmic patternsthat trigger synchronized flashing of the “call” area inthe center of the screen. The call patterns are followed by space forplayers to copy the pattern, directed by response cues. “YourTurn” indicators allow everyone at the table to play together, tobe split into subgroups, or to support solo sections. Once the playerscatch on to the system of when to play and when to listen,opportunities emerge for more-experienced players to improvise withinthe form. Although some players have found this rhythmic learningapproach too structured to be entertaining, others have enjoyed its“Simon says” aspect.

Of the many interaction methods explored, the JamODrum designersfound that call-and-response was the most successful in bringing noviceand expert players together for musical collaboration.

A JamODrum installation that scales from 6 to 12 participants iscurrently on exhibit at the Experience Music Project in Seattle. Athree-person JamODrum was recently donated by Paul Allen/VulcanVentures to the Entertainment Technology Center at Carnegie MellonUniversity in Pittsburgh (

Talking Stick. Under the direction of Bob Adams, a smallgroup of musicians and technologists at IntervalResearch—including Geoff Smith, John Eichenseer, JesseDorogusker, Mark Goldstein, and guitarist/producer MichaelBrook—joined forces to create a touch-sensitive, tubularinstrument called the Stick (no relation to the Chapman Stick). Theproject combines customized circuitry with an array of force-sensitiveresistors (FSRs) that take multiple data measurements with one touch.Because Brook intended the Stick to be played in a way similar to anacoustic bass, a vertical strip of FSR linear potentiometers was usedin lieu of a fretboard. Location and velocity information is determinedby the amount of surface pressure applied by the player’sfingers. To independently measure force and position, the teamdeveloped a custom library of Max patches for the Stick.

With plans under way for her work Songs and Stories from MobyDick, Laurie Anderson, with the help of Bob Bielecki and incollaboration with Interval Research, extended the capabilities of theStick controller. Repurposed specifically for Anderson’shybridized approach to music, movement, and spoken word, the newTalking Stick features a linear potentiometer and a pressure-sensitiveactuator for the manipulation of sampled audio, as well as a wirelesstransmitter for sending control information.

Anderson uses the Talking Stick to evoke the clicking patterns inthe language of sperm whales and the creaks and groans of a ship. JohnEichenseer, Lukas Girling, and Dominic Robson used Cycling’74’s Max/MSP to create a variety of granular synthesispatches for the short sound fragments. During performance, thesefragments are modified and resequenced in real time.

Besides the four Talking Sticks featured in Moby Dick, threeother Sticks are in existence—one at Stanford University, one atthe Berklee College of Music in Boston, and the one in the possessionof Michael Brook.

MIDIBall. Fans rushing a stage at a concert in Tokyo inspiredCandice Pacheco, cofounder of D’Cückoo, to design a giganticbeach ball that creates music as the audience bats it around. TheMIDIBall, a wireless 5-foot sphere, converts radio signals into MIDIcommands that trigger audio samples and real-time 3-D graphics withevery blow. The biggest challenge was finding a plastic material thatwould be durable enough to protect embedded sensors and withstand heavyhitting, yet would appear to float.

The MIDIBall also required wireless technology with a thresholdsensitive enough to reliably interpret a range of raps, slaps, andpunches without double-triggering. After experimenting with severaldifferent sensors, Pacheco ended up inserting an RF transmitter in asleeve sewn into the middle of the MIDIBall. The MIDIBall debuted atthe Grateful Dead’s Mardi Gras show at the Oakland Coliseum in1992.


Research and development in the field of bowed-string controllershave been going on for some time; examples include IRCAM’s(Institute of Research and Coordination in Acoustics and Music)SuperPolm for violinist Suguru Goto, Tod Machover’s work with thehypercello, and Peter Beyls’s use of multiple infrared sensors onviolins at Brussels University in Belgium.

However, the international ambassador of the extended violin is JonRose. Over the years, the English-born inventor/performer hasimplemented a number of technological innovations for the instrumentand has also developed some unique “deconstructed”bowed-string instruments. Rose’s interactive setups, developedwith help from STEIM (STudio for Electro-Instrumental Music), combinean accelerometer on the bowing arm, an ultrasound sensor mounted on thebow, a bow-mounted sensor that measures bow pressure on the strings,and three MIDI footpedals (


Sophisticated glovelike controllers continue to be popular withperformers of electronic music. At the end of the ’80s, TomZimmerman and Jaron Lanier’s DataGlove had scored some notoriety.When Mattel used the technology in its own junior version of thecontroller—called the PowerGlove—musicians such as MarkTrayle were able to hack into the greater potential of the $79 toy.

The Hands. In 1984 in the Netherlands, Michel Waisvisz beganperforming with an instrument he helped develop called The Hands. Theoriginal controller was made up of a group of keys and sensors mountedunder his fingers and thumbs. The data collected by the sensors wastranslated into MIDI using a microcomputer and custom software thateventually became the SensorLab, marketed by STEIM. Over theyears—with engineering help from Frank Balde, Johan denBiggelaar, Bert Bongers, Peter Cost, Tom Demeijer, Wim Rijnsburger, andHans Venmans—Waisvisz has made incremental improvements to hisgestural hand controllers. But Waisvisz keeps technological upgrades toa minimum. This allows him to master the performance techniquesnecessary for exploring the limits presented by the controller.

The latest version, Hands II, is a more refined version of theoriginal controller—with upgraded wiring andcomponents—that measures finger, hand, and arm movements. Thedistance between the hands is also measured, using ultrasound sensors.Hands III is currently being developed at STEIM by Jorgen Brinkman.

Lady’s Glove. In 1991, Laetitia Sonami ( beganher work on the Lady’s Glove by attaching magnetic sensors to thefingers of latex gloves. Since then, the Lady’s Glove has gonethrough a series of radical design changes, most recently with the helpof engineer/designer Bert Bongers through a sponsorship from STEIM.

Image placeholder title

In its present implementation, the Lady’s Glove (see Fig.5) is a full-length Lycra glove with an accelerometer that measureshand speed; numerous motion and pressure sensors; and ultrasoundtransmitters and receivers that detect the distance between the gloveand the floor.

Sonami uses the Lady’s Glove in live performances to controlsound, mechanical devices, and lights via MIDI—mostly in solosituations but also in improvisations with other instrumentalists. Hercurrent setup includes a STEIM SensorLab processor and a Mac laptoprunning Max/MSP.


For decades, composers have been using electrical signals from thebody as a source for electronic music. Most of these systems were builtusing parts originally designed for medical or scientific purposes.Although artists working with biofeedback continue in this mannertoday, a number of companies are marketing systems that are directlyapplicable to music.

BioMuse. Researchers Hugh S. Lusted and R. Benjamin Knapphave created BioMuse (, an interface that analyzes andinterprets the signals from up to eight simultaneous bioelectricsensors and translates them into MIDI data.

BioMuse has been used primarily with electromyographic (EMG) sensorsthat measure the flexion and extension of muscles. However, othersensors can be used with the system, including those that read eyemovement (electrooculographic) and brain waves(electroencephalographic, or EEG).

Image placeholder title

BodySynth. The BodySynth ($1,499), developed by EdSeveringhaus and Chris Van Raalte, uses EMG sensors attached to thebody of the performer (see Fig. 6). The basic setup comes withfour EMG sensors, a two-position switch, the Body Unit, a wirelesstransmitter, and a remote processor that handles the necessary A/Dconversion and signal processing and includes a collection ofalgorithms for musical handling of the MIDI data.

Input from each of the four EMG sensors can be mapped to any of theremote processor’s eight MIDI output channels (or systemchannels, as the designers call them). The BodySynth can handle up to12 EMG electrodes simultaneously.

Once they are attached to the body, the EMG sensors are plugged intothe Body Unit, a device roughly the size of a cigarette pack that isworn by the performer. The Body Unit comes with four EMG amplifiers anda gain control for each of the four channels. The performer also wearsa wireless transmitter that sends the signals from the Body Unit to theremote processor. Each BodySynth is configurable to fit the needs ofthe user.

Image placeholder title

I-CubeX. The I-CubeX ($625), manufactured by Infusion Systems(, is a comprehensive system that can translateup to 32 analog voltages into MIDI data from a varied host of gesturaland environmental sensors (see Fig. 7). The I-CubeX Digitizerhandles the I/O; besides MIDI information, the unit can also output1-bit voltages (that is, 0 or 5 volts). In performance, the I-CubeX canbe used with or without a computer. It has editors for both the Mac OSand Windows 95 that allow you to store sensor setups, enabling it tofunction as a stand-alone unit.

Each I-CubeX system comes with a Turn sensor and a See actuator.Additional sensors are available that measure temperature, light,pressure, acceleration, distance, and proximity; all are specificallycreated for use with the I-CubeX. For example, the TapTile is a12-inch-square pad that measures pressure when it is stepped on ordanced on. Another device, the TouchGlove, contains sixsensors—one used in the palm and five on the fingertips.

GoFly/IRFly. Infernal Devices ( offers two environmentalsensors in its Sensopede series of products. The GoFly ($95) is a tinyheat sensor (1.25 by 1.5 inches) that can be used to detect the motionof people in a room. The IRFly Ranging Detector ($59) is a1.25-by-1.5-inch infrared detector that ignores other light and heatsources. Both devices work with processors made by Infusion Systems,with Beehive Technologies’ ADB I/O, and with STEIM’sSensorLab (see the sidebar “Preaching to theConverters”).

IBVA. IBVA Technologies (www.ibva .com) sells a system that couples EEGbrain-wave sensors with Mac software specifically tailored to musicalapplications. The single-channel IBVA (which stands for InteractiveBrainwave Visual Analyzer) Core system ($1,300) includes a headbandwith electrode sensors, a wireless transmitter and receiver, and a PinInput extension pack for connecting other types of biofeedbacksensors.

Software accompanying the Core system includes the Step 1 ExpansionPak (containing various control applications), Alps+ for brainwave–to–MIDI control, and a software synthesizer. In fact,the IBVA system complies with the General MIDI specification.

MIDIVox. An interesting technique for tracking the voicenonacoustically involves electroglottography (EGG). An EGG sensormeasures laryngeal behavior through changes in electrical impedanceacross the throat. The MIDIVox ($1,295) uses EGG sensors for convertingpitch and intensity data from the larynx as MIDI, binary, analog, andgate output. Invented by SynchroVoice, the MIDIVox is now availablefrom

The MIDIVox is made up of two components: a neck band (available inblue or black) with four biosensors, and a 1U rack-mount interfacemodule. The hypoallergenic neck band is wrapped around thesinger’s neck and affixed with Velcro so that two of the sensorsare on either side of the Adam’s apple. The neck band attaches tothe processor with a ribbon cable. Since its review in the July 1992issue of EM, the MIDIVox has been upgraded with new motherboardsand EPROMs, larger biosensors, and a new Velcro neck band.


Until the 20th century, the sense of touch has been one of the mostimportant elements in playing musical instruments. With few exceptions(primarily the voice and the aeolian harp), earlier instrumentsrequired physical contact to make them work. Recently, however, thisfundamental principle has changed.

Some gestural controllers operate by measuring the capacitance of anobject. The capacitance (or ability to hold an electrical charge)varies based on the object’s distance from an adjacent object.Once the measurements of these changes are recorded, they can beconverted to MIDI information with the assistance of A/Dconverters.

Other controllers use beams of light or ultrasound. For example,when an object is moved within a field of infrared light, opticalsensors measure the increasing and decreasing amount of reflected lightand generate MIDI data based on the values produced by thesemeasurements. (For a more technical description, see ScottWilkinson’s article “Interactive Light” in theDecember 1995 issue of EM.) This particular technology is not arecent development—several MIDI guitar controllers have beendeveloped that measure reflected light as an alternative topitch-to-MIDI conversion, where fingers placed on the fretboardinterrupt an infrared beam, and the altered length of the beam is thenconverted to a MIDI Note message.

Radio Drum/Radio Baton. Developed by Bob Boie and MaxMathews, the Radio Baton and the Radio Drum track the 3-D movement oftwo or more batons over a base unit. The ends of the batons are coveredwith copper tape and topped with large foam balls that resemblemallets. Each baton transmits a discrete frequency that is localized bymeasuring the electrical capacitance between the tip of the baton andthe array of receiving antennas embedded in the base unit.

The system allows performers to predefine behaviors of thebatons/mallets. For example, Mathews wrote a conductor program toprovide new ways of interpreting and performing traditional musicscores. This software, coupled with the Radio Baton, enables singersand soloists to create their own orchestral accompaniment, processtheir voice, or work with algorithmic compositions. A program writtenby Andrew Schloss lets a performer conduct Standard MIDI Files. In thismode, the baton sends motion information to a computer, which theninterprets the baton’s signals and sends MIDI commands to asynthesizer for playback.

An interesting example of a work created using the Radio Drum iscomposer David Jaffe’s pairing with Schloss in the duo Wildlife.With Jaffe playing a Zeta violin and Schloss using the Radio Drum, twocomputers interpret and respond to each player’s actions,superimposing the output of one instrument onto the other. For example,a glissando on the Zeta violin can change the pitch of notes played onthe Radio Drum, or the drum can modify the output of the Zeta.Performing in such situations requires the musicians to develop newinteractive skills, especially when their musical intentions arewrested away from them by other musicians.

Image placeholder title

Ethervox MIDI theremin. The theremin is arguably one of thefinest and best-known gestural controllers. Big Briar’s Ethervox( offers several enhancements to the traditional theremin,including the addition of more complex waveforms and a filter that canbe modulated by pitch (see Fig. 8). The Ethervox theremin cancontrol external sound sources and can be played from a MIDIsequencer.

In many ways, the Ethervox is fairly straightforward in its MIDIimplementation. Pitch Bend is transmitted, as is Control Change 7(volume). The device can also send and receive Note On, Note Off,Velocity, Program Change, and System Exclusive messages. As you wouldprobably guess, the Ethervox sends a constant stream of Pitch Bendmessages. Fortunately, the user can scale the updating of Pitch Bendand Volume information to thin out the MIDI data stream.

Big Briar’s most popular theremin, the Etherwave ($369assembled; $299 kit), doesn’t have MIDI capabilities but doesboast a 5-octave range as well as controls for waveshape andbrightness, and it comes with an instructional videotape.

Dimension Beam. Developed by Interactive Light, the DimensionBeam uses an infrared beam to track the position of an object movingwithin its field. This invisible beam is shaped like an elongated eggbalanced on one end, and it has 128 layers that can be assignedspecific MIDI values.

One of the interesting aspects of the Dimension Beam is that,although the device may have seemed a bit esoteric when it was firstreleased in the mid-’90s, it has since found commercialacceptance: Roland licenses the technology (now referred to as DBeam,and no longer sold to the public by Interactive Light) for use in manyof its products, including the SP-808 Groove Sampler, the HPD-15HandSonic drum controller, and the MC series of Groove Boxes.

Lightning II. Buchla and Associates’ Lightning II($1,990) is a spatial controller that features a pair of wirelesswands, a half-rackspace processor, and a stand-mountable remotereceiver. The Lightning II uses infrared trigonometry to track thevertical and horizontal position as well as the velocity of each wand.It divides the performance space into eight Zones, configured in a 4 52 array. A Stimulus from either wand can be assigned to each of theZones. Stimuli include movement within a Zone; entering or exiting aZone; and clicking, double-clicking, or releasing the buttons. You canalso use footswitches with the Lightning II, for yet another level ofcontrol.

The processor contains a 32-voice, 18-bit Kurzweil MASSsample-playback chip that provides a General MIDI sound bank. Eachpostage stamp–size memory card can store 30 presets. A preset canhold up to 40 patches that contain user-definable mapping of gesturesto responses.

Image placeholder title

Soundbeam. Developed more than a decade ago by EMS, theEnglish company famous for its popular Synthi and VCS3 analogsynthesizers, the Soundbeam ( is an ultrasonic MIDI-controlsystem favored in special-education situations where noninvasivebiofeedback is needed (see Fig. 9). With the recent release ofthe Soundbeam 2 ($2,777), EMS has enhanced the feature set of theoriginal model to include the ability to track the speed of a movingobject within the ultrasonic field, as well as proximity and on/offstatus. The Soundbeam 2 also allows you to divide each ultrasonic beaminto 64 sections, each of which can be assigned its own notes, chords,or MIDI control parameter.

Up to four ultrasound beams can be used simultaneously, each withits own MIDI channel. The Switchbox ($250) sends data from eightadditional controllers (such as switches and joysticks) to theSoundbeam 2 controller, and it works simultaneously with the beams. Thebeams have a variable range from 2 to 20 feet, whereas sensors can beplaced more than 150 feet away from the controller. The Soundbeam 2comes with presets containing pitch sequences, as well as a large bankfor user-defined sequences.

OptiMusic. The OptiMusic system ( reflected, visible light beams as controllers and is also usefulin physical-therapy situations. The system comes in two versions: thesingle-beam OM-L1 and the 7-beam OM-L7. Both versions are controlled bythe OM-PCI light–to–MIDI control software.

The lights sense the reflection from performers moving within thebeams. The color, shape, angle, sensitivity, and distance between thelights is customizable. The OptiMusic system allows up to 99 notes perbeam and can work with up to 32 light beams. It will be available inSeptember.


One thing changing the alternative-controller landscape is thatstandard control devices from the graphics and gaming worlds arebecoming increasingly powerful, plentiful, and lower in price. And,more and more of them feature Universal Serial Bus inputs. “Infact,” notes CNMAT’s David Wessel, “USB is just nowbeginning to mature as we speak.”

Among the devices popular with electronic musicians are the Wacomgraphics tablets. For example, Wacom’s UD-1212-R offers, amongother things, 32,000 points along an x-y axis and pressure and anglesensitivity. In addition, each UltraPen—the little stylus that isused as an input device—can have its own ID.

Matt Wright, musical-systems designer at CNMAT, uses a Wacom tabletwhen performing. In recent performances with Wessel and Pakistanisinger Shafqat Ali Khan, Wright used the tablet to playadditive-synthesis sound descriptions (using MSP) of Khan’svoice. Wright uses templates to indicate how the sound descriptions aremapped horizontally across the tablet. This allows him both to scrubthrough the sound in either direction (with the pitch kept independentof the scrubbing speed) and to visually locate particular musical partsso he can set the UltraPen directly onto them.

“Joysticks have also matured over the years,” commentsWessel, “and some include enough control options—buttons,triggers, movement direction—for sophisticated musicmaking.” After some research into the subject, composer BobOstertag suggested to Wessel and CNMAT that the Cyborg 3D USB by Saitek( one such controller worthy of investigation in this area.


Perhaps the most prolific source of research into alternativecontrollers is MIT’s MediaLab, under the direction of composerTod Machover and Joseph Paradiso. The MediaLab’s manydevelopments have moved beyond the ivory tower and are now used inprestigious venues around the globe.

Paradiso is the technology director of the Things That Thinkconsortium, a group captivated with the infusion of intelligence intoeveryday objects. He also leads MIT’s Responsive Environmentsgroup, which has developed controllers used in musical/graphical boardgames, as well as “smart sneakers” that, when worn duringperformances, endow dancers with the ability to produce continuousmusical output.

Since the mid-1980s, Machover has been passionate about augmentingthe expressivity of traditional instruments—keyboards, strings,and percussion—with computer systems that measure and interprethuman expression and feeling. These “hyperinstruments,”which are tailored specifically for professional musicians, expand thecapabilities of existing instruments and redefine the ways in whichpeople interact with objects that may or may not typically beassociated with making music. As sensors, software, and signalprocessing have become more sophisticated, the development ofhyperinstruments has evolved to include interactive musical instrumentsfor amateur musicians as well.

In 1996, Machover created a work that blends livehyperinstrumentalists with audience participation—both live andvia the Internet—in the polymorphous production known as the“Brain Opera”( .edu). After twoyears of touring, the Brain Opera and its associated gadgetry are stillundergoing constant revisions and upgrades. Many of the controllersdescribed here will become permanent installations in Vienna’sHouse of Music when it opens this summer.

For some attendees of the Brain Opera, the most engaging aspect isnot the actual performance but the opportunities for hands-on musicalinteraction. This takes place within an eclectic assortment ofsculptures known collectively as the Mind Forest.

Image placeholder title

Rhythm Trees. From the gigantic pods of the Rhythm Trees hang300 translucent rubber drum pads that trigger vocal samples when struck(see Fig. 10). Implanted in each pad is a pressure-sensitivepiezo-electric strip that can accommodate a wide range of strikevelocities. Networks of 30 pads are routed to a PC running customsoftware under Windows NT. As performers strike the pads, the softwarelooks for patterns between players and generates new rhythms inresponse, creating a kaleidoscopic array of sounds, lights, and imagesin the process.

Singing and Speaking Trees. The Singing Trees respond withaudiovisual feedback to the tonal quality of notes sung next to them. Awell-sung note results in a calm, atmospheric response from the system,while badly sung notes receive a more complex response. Speaking Trees,on the other hand, allow people to record stories, memoirs, lyricalphrases—in other words, to speak on any topic at all. SpeakingTrees automatically edit and process the recordings and incorporate thesound bytes into each Brain Opera performance.

Harmonic Driving. One sculpture that closely approximates anarcade-style video game is the Harmonic Driving controller. Respondingto video generated by Rolf Rando’s 3-D rendering system on an IBMRS/6000 computer, participants use a steering wheel to navigate througha multicolored course complete with bends, turns, and potholes.Melodies and harmonies are created that correspond to the path taken;icons placed in the course signify “hot” or“cool” musical tracks. The driver determines whether theride is a rhythmic journey or a wandering detour of ambient sound. Inthe true spirit of a video game, the player is given a skill rating atthe end.

Image placeholder title

Sensor Chair. Normally, one would think of a chair assomething you sit on while making music with an instrument. But theSensor Chair captures movement of the arms and upper body and convertsthose motions into music while the performer is sitting in a chair (seeFig. 11).

Using the Sensor Chair, the performer becomes an extension of atransmitting antenna embedded in the chair’s cushion. Fourreceiving antennas mounted on poles around the Sensor Chair allow bodygestures to control sound. Joe Paradiso, Neil Gershenfeld, and EdHammond developed the chair’s hardware, and Pete Rice and EranEgozy devised the interpretive software. The mystical nature of thisnew controller even attracted the magicians Penn & Teller, who usedit in a routine to highlight the innovations of music, magic, andmachines.

Gesture Wall. Movement from a participant’s entire bodymodifies the on-going sounds and projected images as they stand nearthe Gesture Walls. Beneath the floor are plates that transmit alow-voltage electrical signal to the participant’s body throughhis or her shoes. Sensors mounted around the projection screen abovethe walls receive these electrical signals as they leave the body. Thisallows the instrument to follow precisely even the subtlest ofgestures.

Future Music Blender. After wandering through the MindForest, visitors to the House of Music will have an opportunity toexplore a new addition to the Brain Opera—the Future MusicBlender. Chris Dodge, Alex Westner, Peter Colao, and Ed Hammond workedwith Tod Machover to create a sonic sculpture designed to collect musicsamples that could easily be incorporated into a performance medley.While some of the sounds are activated from a prepared database, othersamples will be generated by visitors to the Mind Forest. Using aSensor Chair retrofitted with a “multimodal mixer” forcontrol, people will be able to access, select, and play samples.Simply waving a hand in the air will enable a visitor to“blend” sounds that are then complemented by musicalaccompaniment (generated through custom algorithmic software) to createlarger compositions.

Music toys. The latest round of controllers is disguised as“music toys.” With names like Simple Things, the Big Thing,and Music Shapers, they are made from as many squeezable, stretchable,and reconfigurable parts and materials as possible. (Even Play-Doh isused as a conductive material to manipulate sounds.) The ultimate goalbehind the creation of these controllers is to distribute them tomusic-education programs in five host cities (New York, Boston, London,Berlin, and Tokyo). The culmination of the project will be a series ofToy Symphony compositions and performances with local symphonyorchestras.

As the name implies, Simple Things make simple sounds, such asindividual notes or sample playback, and are intended to be handheldstand-alone devices. Josh Strickon, Abie Flaxman, Tristan Jehan, andDiana Young use infrared links to exchange sounds between Simple Thingsand to synchronize groups of Simple Things.

Music Shapers are fabrics, furniture, balls, and spatial instrumentsthat offer fun, tactile ways to explore different aspects of music.Designed by Maggie Orth and Gili Weinberg, these malleable instrumentsprovide new ways to physically shape and manipulate musical sounds andtextures by measuring the exertion of force and pressure on stretchableobjects. This tangible approach to playing music is inspired by thedesire to create flexible interfaces to complement the multidimensionalaspects of synthesizers and computer-generated music.

The Big Thing—a multitiered structure designed by Orth,Weinberg, Dan Overholt, and Mary Farbood—is intended to be thebrain that controls the network of interconnected music toys. The BigThing enables young people to compose, arrange, and perform music in a3-D construction kit, letting them experience musical expression bychanging the physical relationships between a variety of sensingobjects, connectors, and “chunks.” Each chunk containssounds or commands that can be reorganized by moving, twisting, andinterconnecting with other chunks to create “islands” ofsound. The sensor objects can be used to add touch sensitivity orgesture-controller functions. Obviously, children will need a bit ofcoaching before they can fully understand the capabilities of thesegiant Lego-like controllers and music toys, but the prospects forcollaboration are encouraging.

There are many more controllers—too many to list in thisarticle—that have also been an outgrowth of developmentthroughout MIT’s MediaLab. Among these are “musicalbottles” that control different sounds and patterns of coloredlight when the stoppers are removed; musical threads that are sewn intoa denim jacket and emit sounds; and gesture-tracking digital batonsthat provide position, orientation, and surface-pressureinformation.

Conductor’s Jacket. Wearable computers are not anessential part of most people’s wardrobe—unless you happento be hanging around Teresa Marrin Nakra. Interested in finding newavenues for musical and emotional expression while studying at theMediaLab, Marrin Nakra mounted a collection of EMG sensors to her upperbody, with the wires strapped inside a “conductor’sjacket” to collect a variety of physiological measurements.

Image placeholder title

After taking readings on heart rate, respiration, skin conductivity,temperature, and muscle tension, Marrin Nakra hoped to find acorrelation by mapping these measurements to the gestures and rhythmictiming of conducting. She was able to convince a number of well-knownconductors, including Keith Lockhart of the Boston Pops, to try out theConductor’s Jacket in performance so that data pertaining tothese unknown relationships could be collected (see Fig. 12). Toher surprise, Lockhart was more than willing to get wired for hisaudiences; he even requested a flashier jacket than the one that wasoriginally provided.


As a research center dedicated to the performing arts, STEIM (STudiofor Electro-Instrumental Music) has been a hotbed ofalternative-controller research ( Notable developments include TheHands, developed with Michel Waisvisz, as well as the Sweatstick andthe MIDI Conductor. Important computer applications for use in liveperformance have also come from STEIM, such as LiSa, for live sampling;BigEye, which converts video into MIDI data; and Image/ine, forreal-time video manipulation.

A number of prominent performers using live electronics havedeveloped highly personalized instruments at STEIM, including Jon Rose,Bob Ostertag, Laetitia Sonami, Miya Masaoka, and Kaffe Matthews.

STEIM also features an exhibition called the Electro Squeek Club,where visitors can experience firsthand a collection of audio and videopieces. Some are completed installations, while others are indevelopment at the center. Visitors are encouraged to explore theindividual properties of each of the exhibits, which include CrackleBoxes, Babblephones, the Electronic Baby Mirror, and the BeBop Table.The artists represented include Michel Waisvisz, Bert Bongers, JorgenBrinkman, and Tom Demeyer, among many others.


The research center in Paris known as IRCAM (Institute of Researchand Coordination in Acoustics and Music) has been at the forefront ofmusic and technology for decades. Research into musical inputstructures has led to the development of several controller prototypes,including the SuperPolm and an extension of the computer mouse, dubbedJerry.

This and other information has been assembled for a CD-ROM titledTrends in Gestural Control of Music, by Marc Battier and MarceloWanderly, and is available from the Electronic Music Foundation (www.cdemusic .org).Sections of the disc are dedicated to performance issues, definitionsof gesture, and gestural analysis. It also includes a roundtablediscussing the “Present and Future of Gestural Control inMusic,” with contributions by such luminaries in the field asDonald Buchla, Mark Goldstein, Joel Chadabe, Tod Machover, TeresaMarrin Nakra, Robert Moog, Jean-Claude Risset, Laetitia Sonami, andMichel Waisvisz.


It is interesting to note that many of the controllers surveyed hereare continually being refined—some could even be considered worksin progress. Some readers may feel that there is a gratuitous use ofnew technologies behind some of these new musical-input devices, butdeeper investigation reveals that musical results are the chiefmotivating force behind most of the controllers.

Although some of the input devices in this survey are theatrical,the ultimate goal is for the results to transcend the novelty of thevisual aspects that a device presents. In most cases, the artisticsuccess of a particular controller will depend on its transparency inthe creation of the music: the main purpose behind all of these devicesis the natural and immediate translation of physical gestures intomusic.

Some of the controllers we’ve examined here are based on thehighly personal goals of their developers, whereas others are createdwith mainstream applications in mind. Performers who want to developtheir own controllers need to define the gestures they wish to use andthen find the best way to measure these gestures and translate theresults into a useful data format, such as MIDI. Sometimes a series ofgestures may require more than one type of sensor.

A number of companies have already developed the various componentsneeded in an interactive control system (such as sensors, signalconverters, and software). All that inquiring musicians, dancers, orartists need to do is spend some time configuring their own systems andlearning how to use them.

Ever since Marty Cutler started as assistant editor forEM, he has referred to his banjo as a “real-time5-string arpeggiator.” Armchair thereministGinoRobairis an associate editor forEM.Bean’smusic-making methods include sneakinginto schools around the Bay Area with her group, RhythMix. Specialthanks to David Wessel, David Jaffe, Donald Buchla, Alex Artaud, PeterElsie, Joel Chadabe, Mary Gallardo, Miya Masaoka, Pamela Z, Bela Fleck,Bob Applebaum, and Jimi Tunnell.




Big BriarEthervox


Buchla andAssociates Lightning II


Buchla andAssociates Thunder


Cycling ’74MSP


Interactive LightDimension Beam




Nearfield MultimediaMarimba Lumina








CNMAT/Wacomtablet/Buchla Thunder








MIDIBall/Interactive stageshow




Tech Page:“Interactive Light”



One of the most popular commercially available software applicationsfor sound artists who need flexibility in mapping physical gestures toMIDI is an object-oriented programming language called Max (Mac; $395)available from Cycling ‘74 ( Created in 1987 by MillerPuckette at IRCAM and later developed into a commercial product byDavid Zicarelli, Max provides a graphical user interface for combiningthe basic building blocks used in an object-oriented environment. AWindows version is on the horizon.

Cycling ’74’s MSP lets you create, analyze, and processaudio and is designed for use with Max. The full version of MSP ($295)requires Max 3.5.8 or higher. A free runtime version of MSP, whichcomes with the free MaxPlay application, allows you to play, but notcreate, MSP patches.


A number of converters are available for translating analog signalsto digital ones.

The SensorLab (www.steim .nl/sensor.html) is a voltage-to-MIDIconverter developed at STEIM for use with any type of interactivecontroller. Although popular with many artists, the SensorLab iscurrently out of production. Keep an eye on the developer’s Website for further details.

The ADB I/O ($199) by Beehive Technologies ( you to use up to eight input sensors with a Mac. You can connectup to four units, for a total of 32 I/O channels. The device acceptssensors from Infernal Devices and Infusion Systems and can work withHypercard, Supercard, AppleScript, Macromedia Director, Symantec C/C++,and Cycling ’74’s Max. A beta version of a Max objectcreated by David Zicarelli for the ADB I/O can be downloaded for freefrom the Beehive downloads page.

To accommodate different types of sensors (both gestural andenvironmental), IRCAM has begun marketing its AtoMIC Pro ($665). TheAtoMIC Pro translates sensor information (electrical voltages) intoMIDI data. The device was designed to be open-ended, so it can workwith any type of controller or signal. It has 32 analog inputs (usingtwo multipin connectors), eight digital inputs and outputs on multipinconnectors, one MIDI input, and four MIDI outputs.

CNMAT has developed a low-latency, high-quality multichannelinterface for use with laptop computers in live performance situations.Code-named the Rimas Box (after its primary developer, researcher RimasAvizienis), the interface uses the 100BaseT Ethernet port tocommunicate with the computer, allowing the device to simultaneouslyhandle 64 channels of sample-synchronous control-rate gesture data, 10time-stamped MIDI I/O streams, and up to 10 channels of 24-bit audio.“Latency measurements show that we can get signals into and backout of Max/ MSP in less than 7 milliseconds,” says CNMAT directorDavid Wessel. “One of the most important features of ourinterface is that data from sensors is treated as signals that aresynchronized at the sample level with the audio. This provides the userwith a very high degree of control intimacy.”

The Rimas Box, which is designed to sit neatly under a laptopcomputer, is currently being manufactured in a limited quantity forfinal beta testing.