Search Gear
 

Using Nintendo Wiimote for Music Production

March 1, 2009
share
Nintendo Wii

The Wii Remote — or “Wiimote” to its millions of fans — has radically altered the video-game world by offering players an unprecedented level of physical participation. Nintendo has packed into a $40, palm-size package control features that a half-dozen years ago would have cost 50 to 100 times as much (see Fig. 1). Instead of pushing buttons and moving joysticks, virtual golfers and bowlers now swing their arms and pump their legs, and virtual battlefield commanders literally point and shoot at their targets. If you're an electronic musician with a hankering for something new, the fun really starts when you add a Wiimote.

The Wiimote is a squared-off white cylinder about the size of a clave or a large hot dog, with 11 buttons and a dark area at one end that has something to do with infrared light. Inside are an Analog Devices ADXL 330 3-axis 3G accelerometer; a highly accurate, 2-axis infrared tracking system; a cell-phone-style vibrator called a “rumble generator”; a tiny speaker; and a Bluetooth transmitter and receiver. For another $20, you can almost double the Wiimote's power by adding a Nunchuk, a pickle-shaped device that plugs into the Wiimote with a cable and gives you a second ADXL 330, a two-dimensional joystick, and a couple of more buttons.

FIG. 1: The author holds a Wiimote in front of a MacBook running the piece “Imaginary Dialogues.”

FIG. 1: The author holds a Wiimote in front of a MacBook running the piece “Imaginary Dialogues.”

The ADXL 330 (which, if purchased separately, would cost nearly twice as much as the Wiimote) is a remarkable little chip. It senses motion in the x (left to right), y (up and down), and z (forward and back) planes and generates three different analog voltages in response. But its 3G (3 Gravities) rating means that it is sensitive enough to respond to the earth's gravitational field, even when it's not moving. So it not only measures acceleration, it can also measure static position relative to the earth's pull — in other words, tilt. At all times, therefore, the unit reports its rotational position in three dimensions: what airplane pilots call pitch (vertical plane), yaw (horizontal plane), and roll (twist).

The infrared system is just as remarkable. It uses a Pixart Multiple Object Tracking CMOS optical sensor, a technology that is also used in cameras. If you're one of the millions who have bought one of Nintendo's Sensor Bars for the Wiimote, you may be surprised to learn that all of the infrared tracking is done within the Wiimote. “Sensor Bar” is a complete misnomer: all that's in it are a handful of infrared LEDs, which the Wiimote's sensors track as you move the Sensor Bar. (This is why the Sensor Bar costs only $10.) The Sensor Bar has to connect to a Wii console not to pass data, but to draw power from the console. If you don't have a Wii, then you can opt for a “wireless Sensor Bar,” which is nothing more than a set of battery-powered infrared LEDs, available from several companies.

If you've inferred from the last paragraph that you don't need a Wii console to get information from the Wiimote, you're right. The Wiimote communicates with the console, and anything else in range, using Bluetooth technology. Therefore any computer or other device with Bluetooth capability can be taught to respond to Wiimote-generated data.

Of course, getting the Wiimote to connect to your Mac or PC is one thing, but figuring out what it's sending is a lot more complex. Fortunately, there are a number of free or very inexpensive programs for both platforms that will read Wiimote movements, tracking data, and button pushes and translate them into useful information. The Wiimote-hacking community is huge and truly international, with contributions from almost every corner of the globe.

Some Wiimote-reading utilities use the data to simulate what one normally does in front of a keyboard: move the mouse, pull down menus, and press keys. For example, WiinRemote, from Japan, is a free utility for Windows that lets you move the computer cursor using either the accelerometer, the IR sensors, or the Nunchuk's joystick and lets you assign different keys and combinations to the various buttons. Similar applications for the Macintosh are DarwiinRemote (free, from Japan) and Remote Buddy (19.99 euros, or approximately $27 at this writing), which is made by a German company called IOSpirit. These utilities can be used in conjunction with almost any program: for instance, with Ableton Live you can use the Wiimote's buttons, joystick, and the rest to enable loops, mix tracks, change tempos, play with synth parameters, or trigger notes or samples on a virtual keyboard. Many of these utilities can also work with other Bluetooth-enabled devices, such as EyeTV remotes, cell phones, and the instruments in the Rock Band and Guitar Hero games.

Other programs are more specifically oriented toward music and generate MIDI or Open Sound Control commands, which can then be passed on to music applications using the Windows MIDI Mapper or Apple's Audio MIDI Setup. (For more on OSC, the media processing protocol developed at the University of California, Berkeley, see “Square One: Open Sound Control” in the November 2008 issue, available at http://emusician.com/tutorials/square-one-open-sound-control/.)

FIG. 2: The Max Object called aka.wiiremote that brings Wiimote data directly into Max/MSP.

FIG. 2: The Max Object called aka.wiiremote that brings Wiimote data directly into Max/MSP.

For Windows users, there's a comprehensive tool called GlovePIE (donationware), developed by Australian programmer Carl Kenner. Originally created for use with virtual reality gloves (as the name would indicate; the second part stands for “Programmable Input Emulator”), GlovePIE is a scripting language that supports a wide range of wired and wireless controllers, including the Wiimote. It's capable of working in many environments, and the downloadable package includes both OSC and MIDI commands.

Wiinstrument is a free, simple, open-source, cross-platform (Mac, Windows, and Linux) application from Germany that lets you generate MIDI notes and controller messages. In one mode, it provides a virtual keyboard, and in its Percussion mode, you can use the Wiimote like a drumstick. Acceleration in the vertical plane is mapped to Velocity sensing, and the buttons select various drum sounds.

If you're a Cycling '74 Max/MSP user, you can get Wiimote data into your creations with a free Max Object called aka.wiiremote (see Fig. 2). It allows you to use all the different Wiimote data streams in your Max patch and lets you send data to the Wiimote, activating its lights and rumble generator. Aka.wiiremote has a unique feature: you can throttle back the speed of the incoming data to cut down on noise and to avoid overwhelming your MIDI stream. (The Japanese developer, Masayuki Akamatsu, also has a very cool Object that uses a Macintosh laptop's Sudden Motion Sensor to tell you when you're tilting the computer in any direction. It may not be all that useful for music, but you can use it to turn your MacBook into a seismograph.)

My favorite way of getting Wiimote data into my computer is a neat little application called OSCulator, which comes from France (donationware, minimum $19). Its developer, Camille Troillard, is a power user of Symbolic Sound's Kyma, and he wrote the program so he could choose from all sorts of external controllers — graphics tablets, 3-D mice, JazzMutant's Lemur, and a variety of wireless gadgets, including Wii-Fit Balance Boards and iPhones — to play his Kyma. In the process, he put in hooks for OSC and MIDI and designed a very simple and elegant, yet highly versatile, interface (see Fig. 3). Unfortunately for Windows users, OSCulator runs only on Macs.

FIG. 3: The main screen for assigning MIDI commands to Wiimote parameters in OSCulator.

FIG. 3: The main screen for assigning MIDI commands to Wiimote parameters in OSCulator.

OSCulator lets you assign any Wiimote parameter to any MIDI note, controller message, or other system message as single events or toggles. A special Note w/Parameters command allows you to use continuous parameters to control the characteristics of a note. For example, when you use a button to trigger a note, you can use the vertical tilt of the Wiimote to specify the note number and the horizontal position of the Nunchuk joystick to set the Velocity. Wiimote functions can be mapped to different MIDI channels, and the software can accommodate up to four Wiimotes with their Nunchuks at the same time. You get separate smoothing functions for all the continuous sensors to cut down on noise, as well as a prominent indicator for the battery level, which goes down amazingly fast. OSCulator also does a heroic job of “discovering” multiple Wiimotes — and does so far more efficiently than Apple's Bluetooth Preference Panel.

Any of these programs will let you do some cool musical things with a Wiimote. Adding a Nunchuk gives you 12 different real-time parameters you can control, and you can split these among as many MIDI channels as you want. Imagine a virtual drum set in which you can select which drums you play by pointing at them and then flicking your wrist to make it sound; or a loop with a steep resonant filter that opens up and growls as you raise your arm; or a vocal patch that responds to one arm's position for pitch, the other arm's position for timbre, and the twist of a wrist for vibrato. The more continuous controllers a synth patch has available, the more you can mangle the sound with your physical gestures.

You can really go crazy with a Wiimote if you're willing to do a little programming in a processing language like Max/MSP or the open-source PureData (Pd). With these tools, the action of any parameter can be interdependent with the action of any other, and you can do tricks like reassign all the controls on the fly.

FIG. 4: One of the display screens from “Imaginary Dialogues,” showing which parameters are being controlled and their current state.

FIG. 4: One of the display screens from “Imaginary Dialogues,” showing which parameters are being controlled and their current state.

A piece, “Imaginary Dialogues,” that graduate student Phil Acimovic and I recently wrote and performed for two Wiimotes, Nunchuks, and Max/MSP had 12 scenes. When I raised my left hand and shook the Nunchuk hard, it sent a message to Max (and also a visual message to Phil) to change scenes — and the actions of the controllers instantly changed completely.

The output of Max/MSP was sent to two Propellerhead Reason racks, each with a bundle of synthesis and processing modules, and different combinations of modules were used in different scenes. In one scene, pressing the B button played and held a random note, but the randomness was restricted by the middle three buttons: whichever of those was last pressed determined the scale the note would fall in — major, harsmonic minor, or pentatonic. Raising the pitch angle of the Wiimote raised the volume, while changing the yaw angle moved the sound within the stereo field, and moving the Nunchuk's joystick altered the filter envelope. When used with a slow pad from a Maelström module, it was very effective.

A test patch in Max that lets you look at the data from the Wiimote as it is being converted into MIDI.

A test patch in Max that lets you look at the data from the Wiimote as it is being converted into MIDI.

In another scene, holding the B button spit out a string of random short notes, again restricted by the scale selection, with the yaw controlling the speed of the notes and the pitch controlling the range. The Nunchuk's joystick and motion sensors changed their character within the scene depending on which buttons were pressed: they might be used for LFO rate, filter frequency, vocoder mix, reverb wet/dry mix, distortion, or the degree of feedback in a flanger (see Fig. 4).

Phil did some very fancy programming that showed us, on two screens, exactly which parameters were being controlled at any moment and what their current values were. You can look at some of the Max patches at emusician.com (see Web Clips 1, 2, and 3) and watch our performance at tuftsemid.com.

This is just the beginning. With the economies of scale the video-game industry offers, we can look forward to devices with even more sophisticated capabilities in the future. And because Wiimotes are small, they can be attached to other objects or to people's clothing and can give them the power of motion detection and tracking. It won't be long before the technology is put to use by anyone or anything that moves, from dancers and circus performers to model-car makers and pet owners. Right now, though, these simple and inexpensive toys will let you get creative, onstage and in the studio, in ways you've probably never thought of before.


Paul D. Lehrman is the author of The Insider Audio Bathroom Reader, a collection of 11 years of his columns for EM's sister publication Mix. He is also coordinator of music technology for Tufts University and has been known to do strange things with player pianos and robots.

SOFTWARE FOR WIIMOTE CONTROL

Product Operating System Price URL
aka.wiiremote Mac OS X free (requires Max/MSP) www.iamas.ac.jp/~aka/max/
DarwiinRemote Mac OS X free sourceforge.net/projects/darwiin-remote/
GlovePIE Windows 98, 2000, or XP (for all features) donationware carl.kenner.googlepages.com/glovepie
OSCulator Mac OS X donationware (minimum $19) osculator.net/wiki/
Remote Buddy Mac OS X approx. $27; Premium Edition, approx. $40 iospirit.com
WiinRemote Windows free onakasuita.org/wii/index-e.html
Wiinstrument Windows XP or Vista, Mac OS X, Linux free sourceforge.net/projects/wiinstrument/

Show Comments

These are my comments.

Featured

Reader Poll

Do you spend more time producing or playing?


See results without voting »