With the d-touch tangible user interface, you place markers in the interactive area, a Webcam records an ongoing image of where they are, and that information is used to control a software sequencer or drum machine. Pictured here is the interactive area for the sequencer. Notice that the marker patterns are very similar, but they differ in the arrangement and orientation of the elements within each pattern. These differences allow the software to distinguish between them and associate different sounds with each one.
Photo: Courtesy D-Touch
Have you ever wanted to participate in cutting-edge computer-music research? Well, here's your chance, and it's free. Several scientists in Switzerland and the Netherlands are studying human-computer interactions by enlisting as many people as possible to use a system they call d-touch (d-touch.org). All you need to join in is an Internet-connected computer (Windows or Mac OS X) with audio I/O, a separate Webcam (not the one built into many computers these days) and a printer.
The d-touch system is based on what's called a tangible user interface, in which physical objects are used to control and represent digital information. In this case, you place objects called markers on a printed surface called the interactive area and move them around. The Webcam captures an ongoing image of the markers in the interactive area, and the associated software interprets their positions to control a loop-based software sequencer or drum machine.
All you do is download the software and image files and mount the Webcam so that it points downward from a certain height. (It needs to see the entire interactive area.) Print the markers and interactive area on A4 or letter-sized paper, cut out the markers and then place them on the interactive area, which is directly below the Webcam. Everything, including complete instructions, is available for free from the d-touch Website.
Once you have everything assembled, the Webcam aligned with the interactive area and the software running, it's time to have some fun. The sequencer and drum machine use different interactive areas; let's take a look at the sequencer first (see the image above). You start by placing a marker in the surface's Record area and playing something into the computer's microphone. The sound you play is linked to the printed pattern on that marker, and different sounds can be linked to different patterns.
Next, place the marker in one of the two playback areas, and the sequencer plays the associated sound in a loop. The marker's horizontal position determines the sound's timing within the loop, the vertical position determines the sound's volume and the marker's rotation determines the sound's playback speed.
In addition, two special markers let you define a start and end point within the loop, and a tempo marker determines the length of the loop. Once you've created a loop, you can place a marker in the Store Pattern area, which links the entire loop to that marker. In this way, you can build ever more complex musical structures.
The drum machine's interactive area includes 11 rows, each corresponding to a different prerecorded sound. As with the sequencer, the horizontal positions of the markers in each row determine the timing of the corresponding sound.
While you're playing with the system, the position of each marker is recorded and sent to the d-touch team to aid in its research into tangible user interfaces. The Website emphasizes that no personal information is ever recorded, and if your data is published, the reference will be completely anonymous. The d-touch team says this is the first time such an experiment has been conducted with a large group of people, which should help determine, for example, if such an interface is useful in everyday activities beyond museum exhibits.
The d-touch Website has several video clips of live performances that feature the system. The resulting music is somewhat minimalist but very interesting nonetheless, and the process looks like a lot of fun. In fact, I might give it a try myself — all in the name of science, of course!