Tips and techniques for composing interactive music. Music has always been a linear art form. No matter what the venue, style, instrumentation, or medium,
Publish date:
Social count:
Tips and techniques for composing interactive music. Music has always been a linear art form. No matter what the venue, style, instrumentation, or medium,

Tips and techniques for composing interactive music. Music hasalways been a linear art form. No matter what the venue, style,instrumentation, or medium, a piece of music always has a beginning,middle, and end, performed by musicians and listened to by an audienceon an immutably sequential timeline.

Interactive media have changed the paradigm by letting the audiencecontrol the musical experience. User actions modify the music incomputer games and Web sites. How can a composer create compellingmusic for a realm in which unpredictability and constant change reignsupreme?

I spoke with several prolific high-profile composers in thegame-music business to compare their Internet audio ideas and writingstyles with my own. While most thought that good music makes for goodsoundtracks, they also agreed that composing interactive music presentschallenges different from those you find when writing in moretraditional genres. From games to the World Wide Web, thinking of aninteractive musical score as a two-dimensional map or network seems tobe the most useful approach.

The explosion of Internet-audio technologies has made the Web ahotbed of new interactive music programs. Macromedia's Flash andShockwave are ubiquitous, and they allow cartoon-style soundtracks,sound effects triggered by mouse clicks and rollovers, and mixing ofmultiple audio streams. Although these browser plug-ins contain ahigh-quality MPEG audio-compression utility, their interactive musiccontrols can be somewhat rudimentary. This isn't really surprising,because they focus mainly on low-bandwidth graphics.

On the other hand, the Beatnik technology ( wasspecifically designed for interactive applications. The proprietaryRich Music Format (RMF) contains a wide variety of controls that youcan access from an HTML page through JavaScript. This enables the userto mute and unmute tracks; change volumes, instrument patches, andtempos; and switch between multiple files. (For an introduction toBeatnik and its interactive capabilities, see this month's "DesktopMusician: The Beatnik Player.")

One of the most popular Beatnik programs is Mixman's eMix (formerlyGrooveGram), an online interactive remixer. Basically, eMix runs alarge number of tracks simultaneously and lets the user mute and unmutethem. Code records when the various tracks are turned on or off, so youcan save your version of the mix and even e-mail it to a friend.


Last year, I was contracted by Beatnik to produce RMF content fortwo remixes: Queen Latifah's rap song "Latifah's Had It Up 2 Here" anda Weird Al Yankovic parody called "Pretty Fly for a Rabbi." I wasprovided with samples from the original multitrack recordings (vocals,drums, guitar, and so on), which I had to edit and arrange so theycould function together in appropriate and interesting ways while stillretaining the flavor of the original linear recording. That meantcreating up to 16 tracks of looped MIDI data that triggered the guitarand vocal samples and played bass, drum, and keyboard lines using thebuilt-in Beatnik bank instruments, all at the appropriate times.

The challenges of adding interactivity was different for each song.The Latifah tune had a fairly simple structure: a bass line; astraightforward hip-hop beat; a few vocal, chorus, and string-ensemblelines; and the rap itself. The eMix format pairs two tracks in an A/Bmanner; you can turn on one of the tracks or turn off both. The limitedsource material went on the A track, and I wrote a new tune for the Btrack, which was intended to run in contrasting parallel to theoriginal song. The tune consisted of a new bass line, a double-timedrumbeat, a bluesy piano groove, and some additional percussioneffects. The trickiest part was interleaving the rap and other vocalsso they wouldn't interfere with each other if both tracks were turnedon. You can hear the results (Mac usersrunning Microsoft's Internet Explorer cannot currently access this Webpage.)

The Weird Al Yankovic song had a driving rock 'n' roll feel, offsetby absurd lyrics peppered with Yiddish cliches. While it didn't have alot of harmonic movement (basically just I-IV-V-I), one section did gointo a half-time backbeat, and I used it to populate track B. I alsoadded a klezmerlike clarinet to track A and accordion (Weird Al'ssignature instrument) riffs to track B. For the vocals, I had the basicverse lyrics on one track and created a novelty conglomeration fromsampled bits and pieces to place on the other track. The result was anawful lot of bizarrely mismatched material, but because comedicjuxtaposition was the name of the game, this remix ended up being quiteamusing. Check it out for yourself at (As with the QueenLatifah remix, Mac users running Internet Explorer will not be able toaccess the site.)


The most difficult part of putting together the eMix files waswrestling the linear parts into 16- and 32-bar loops that would work inharmony. I have always believed that it would be much easier to writemusic specifically designed to remix in interesting ways than it wouldbe to "interactivize" linear songs. Drawing inspiration from a sectionof M. C. Escher's Metamorphosis II - which depicts bees that morph intofish that morph into birds - I wrote my own remixer, which you can findat

The piece contains three different songs in three different styles."Bees" is modal with a Latin beat, "Fish" boasts an up-tempo pop feel,and "Birds" is a bluesy/funky number in 6/8. Each song has similartrack layouts (ostinato, melody, bass, chords, and percussion), so youcan mix and match the various parts. For example, you could play thebass line from "Bees" with the "Fish" melody to the rhythm of "Birds."An author's mix provides a mechanism for cycling through 24 differenttrack combinations that gradually morph the music from one song to thenext. Users can also toggle tracks on and off to create their own mixversions.

JavaScript commands contained in the music-object.js file controlinteractivity. When the page finishes loading into the browser, the RMFfile begins playing in a loop with all tracks muted. The various partscan then be turned on or off in response to mouse clicks - I set upthat option using the setTracksMute() function. This simple buteffective method of creating interactive music experiences is usefulfor a wide range of web applications, including soundtracks that changewhen visitors go from page to page within a site, audio interfaces thatrespond to user input, and online gaming.


I designed my remixer to provide as many musical possibilities as itcould without creating complete cacophony. The most important part ofthis process was choosing key areas and chord progressions that wouldcontrast without clashing. Keeping everything in a single key with thesame chords is the easiest way to do this, and it works well withcertain kinds of grooves. However, I wanted the piece to surprise thelistener with unexpected harmonic and rhythmic movement.

I started working on "Bees" first, using a simple F maj/G maj/Amchord progression with a standard Latin-style bass line. Next, tocreate contrast with the first section's somewhat moody feel, I wrotethe peppy, happy "Fish" tune in the key of C major with a II-Vturnaround and a flowing flute solo. Then came the funky "Birds"section, which I put in the key of D minor, adding a bluesy Hammond B-3riff and a fast, edgy synth-guitar solo. Thus, by putting all threesongs in related "white note" keys, I minimized harmonic conflict,because a melody in C major or A minor usually functions over a Dmchord.

I also wanted three different rhythmic styles. The first section'sLatin percussion ensemble plays a standard "2-3 clave" mambo beat. Therock backbeat of the second section can alternate between single anddouble time, easing the rhythmic transition from the mambo to the thirdsection's 6/8 shuffle. Because a 150 bpm tempo continues throughout thepiece, the bars of the 6/8 section are 112 times longer than those ofthe 4/4 sections. That means the bar lines don't line up all the time(in fact, they line up only every other time), which creates someinteresting temporal shifts in the melodic and chordal parts when usersswitch back and forth between the sections.


Being limited to related keys or simple loops does not have to becreatively stifling; in fact, it's quite the opposite when viewed as anaesthetic challenge. In his book Poetics of Music, Igor Stravinsky saysthe "sea of possibility" is so huge and daunting that it's unwritable.But if you narrow your focus to, say, a single pebble on the shore,that you can write. "My freedom will be so much the greater and moremeaningful the more narrowly I limit my field of action and the more Isurround myself with obstacles," writes Stravinsky.

Letting the user click tracks on and off raises the issue of how tolimit the choices to prevent too dense a mix, because turningeverything on at once results in a tumultuous racket. Though I defineda few limits in "Metamorphosis" (for example, you can have only onebass line at a time), I usually leave the options open so the listenerscan produce a "serendipitous cacophony" if they wish. Choosing what andwhen to play is one of the most important aspects of creatingmusic.

Jeff Lipton of the Internet-audio company Sonicopia takes adifferent approach when producing "sonified" web sites, which areintended for a mostly nonmusician audience. The Sonicopia site( is a good example: a one-chord groove plays asbackground music on most of the pages, while various menu items androllovers produce different musical notes. Users play along with thesong by moving the mouse, and the notes are constrained to a pentatonicscale to prevent dissonances.

Lipton also extracts phrases from the background composition andapplies them to different links. That way, mouse clicks can triggerflourishes in the song's key, and they can also trigger melodic motifsassociated with the graphic elements. "It all works together and givesthe user a sense of control over the composition, but within a narrowlydefined set of parameters that you've set out," he explains.


Musical scores for computer games present different kinds ofchallenges and limitations for composers, though many of the issues ofhow and when a musical transition takes place are similar to thosesurrounding Web-based applications. These days, many popular computergames are less like Space Invaders or Tetris and more like interactivemovies. This style of "digital entertainment" frequently uses puzzles,characters, and shoot-'em-up scenes to move a story line forward.

In these situations, game music composers usually adopt a cinematicapproach, producing scores intended to set up and sustain a mood.However, they must throw away any notion of locking to picture withSMPTE time code or employing film-style timing. During game play, theplayer can stay on a level for an indeterminate period of time and thenbranch to one of several environments. How then can a composer arrangefor the music to flow appropriately from one section to the next?

Many game composers think of the score as a map or network, asopposed to the more traditional linear film-score form. Each level orenvironment is blocked out, and connections between the blocksdelineate all the possible ways the player can go. The result - whichcan end up looking like a New York City transit system map - describesall the music that needs to be composed and helps the composer decidehow to deal with the transitions between the pieces.

THE UNPREDICTABLE Composer Clint Bajakian calls this process"preparing for the unpredictable." To pull off this seeminglyimpossible task, composers create an ambient background score followedby a big flourish when the transition occurs. When properly executed,this tactic sends emotional information to the player and also smoothsover any harmonic or rhythmic inconsistencies, creating the illusion ofa seamless but responsive piece of music.

Tommy Tallarico, one of the industry's most prolific composers,describes the process. "Let's say you've got a Tomb Raider-type gamewhere you're trapped in a cave, looking for an exit," Tallarico says."I might have a 2-minute looping ambience in the background until thecharacter pulls a lever to open a stone door. Then I'll quickly fadeout the ambience and play a 4-second music sting, like a harp glissandoor string crescendo, then quickly fade in a 30-second loop of suspensemusic while the character walks down the corridor behind the door. Whenthe hallway opens into a big room - where 30 guys are waiting to attack- I'll hit with a big orchestral cue."

Tallarico isn't all that concerned with making the transitionalflourishes happen on the beat or even in the same key. He avoidsspending a lot of time and energy trying to get the pieces to fittogether perfectly. Because the transitions are in response to the gameplay and are frequently intended to alert or surprise the user, adegree of abruptness can be a good thing. Also, the human ear wants tohear a musical flow and therefore tries to make the connection. "I'm ahuge believer in just writing a great tune," Tallarico says. "It maynot even match exactly what's going on at any given time in the game,but that's okay. If the player is really excited about what he'slistening to, he's going to want to reach the next level just to hearthe next tune."


Bajakian takes an almost three-dimensional approach to interactivegame soundtracks. He believes that aesthetically they are more akin tosculpture and architecture than to symphonies or film. Unlike anaudience member who passively watches a movie from beginning to end, aplayer views a game from multiple angles while wandering around avirtual environment. The music must enhance that immersiveexperience.

Using multiple versions of the same tune is one technique forproducing that feeling of immersion. In Peter McConnell's film noirscore for the LucasArts game Grim Fandango, a song featuring a BennyGoodman-style clarinet plays in a secret casino hidden behind a cafe.As the Manny Calavera character moves through the game environment, theplayer hears the song in a variety of ways, not only in the casino, butalso filtering through the cafe's walls, coming up from the floor ofManny's office, and broadcasting over an intercom system. Thiseffective use of sound design makes the music seem to fit into areal-world setting.

Composer Michael Land also used multiple song versions in his scorefor Monkey Island II: LeChuck's Revenge. One of the environments is ashop-lined street, and a 32-bar song loop is heard as the player goesfrom place to place. However, each shop has its own version of thesong, with different instrumentation and moods appropriate for thecharacters and atmosphere. The transitions from version to version takeplace on the bar lines. The subtle and effective technique makes eachchange practically unnoticeable, weaving a coherent musical structurewhile introducing sonic variety.

VARIETY OF STYLES The biggest challenge a game composer faces is theloop. "Repetition is the curse of interactive audio, and music is thefirst thing you will notice repeating," says game-music composer GeorgeSanger. Just do the math: if a puzzle or level contains a 2-minutemusic loop and finding the solution or killing all the bad guys takes20 minutes, the player will have to listen to the same tune ten timesbefore moving on. (no wonder so many players reach for the Turn MusicOff switch.) The average game can take as long as 40 hours to finish,which translates into an enormous amount of music, much more than afilm needs.

Fortunately, a long-standing model for dealing with large-scaledramatic musical scores exists: Wagnerian opera. The Wagnerian conceptof the leitmotiv (a musical phrase or chord progression associated witha character, place, or object) can help organize the huge amount ofmusic required for a game. Orchestral music often works well for thisapplication, thanks to its broad and flexible palette of tone colors.The percussive qualities of many of the instruments can also produce awide range of textures and effects. Bajakian believes that the symphonyorchestra lends a greater credibility to the game's subject matter.McConnell says that you can "practically cut from anything to anythingelse" in an orchestral score, an important advantage when he's editinga John Williams score for a LucasArts Star Wars game.

The game industry's other popular musical genre is techno/industrialelectronic music, such as the score to the hit movie The Matrix, withits hard-driving drumbeat, dark textures, and heavy use of guitars.Although this overused style is on the verge of becoming passe, itspowerful beat neatly solves some transitional problems. A strongrhythmic pulse can easily carry a player from one scene to the next,even if there's a momentary timing glitch or change in tempo. However,variety becomes even more important when the beat is so constant.

LOOP THE LOOP It's a real challenge to create a looping piece thatdoesn't sound like it's looping. One technique in Land's score for TheDig employs key changes and chord progressions in measures that are notmultiples of 4. "When you combine odd phrase lengths with modulation,you can really hide a loop," Land says. "There's an art to writingambient music where you create the effect of sparseness and use randombubblings of ideas that don't really have a coherent thrust to them tocreate a mood." McConnell expresses a similar sentiment, which echoesthe old movie cliche that good scores aren't heard. "With puzzle games,it's good to learn how to graze," McConnell says, "so that after thestinger, the music fades into the background and becomes a sort ofnoncommittal audio wallpaper."

Tallarico has developed a technique for more energetic soundtracks."I don't use a song structure," he says. "If you have only two minutes,there's no need to go to a chorus. I'll never play the same part twiceif I'm writing a song that's going to loop for a while. The entire twominutes is your verse, and I try to fill it with unique parts."Tallarico's approach brings to mind composer Aaron Copland'sexperiences as a student of Nadia Boulanger. Copland writes, "She hadbut one all-embracing principle: the desirability of creating what shecalled la grande ligne - the long line in music. Much was included inthat phrase, the sense of forward motion, of flow and continuity in themusical discourse, a feeling for inevitability."

Sanger also has a "more is better" attitude. "I get much moremileage out of just composing more music," he says. "The changesbetween two different musical phrases are much more satisfactory to methan changing the mix or muting tracks. In games, where so littlethematic material is budgeted, it's best not to repeat at all."

THE INTERACTIVE FUTURE Writing music for interactive media is afledgling art form. The production tools and delivery platforms are sonew compared with those of traditional musical styles that it'sdifficult to predict what forms interactive music might take down theroad. Nonetheless, a few of the current trends point the way towardfuture developments.

On the Web, it's all about bandwidth. Ubiquitous broadband accesswill eventually eliminate the need for highly compressed,audio-compromised files and allow for a much wider instrumentalpalette. Ultimately, a silent web site will resemble a silent movie.There will be many more opportunities to produce a broader range ofmusic, sound effects, and voice-overs in the interactive medium.However, until standards evolve for delivery format and SMPTE-likesynchronization, the Web will continue to be a Wild West-type frontier,with competing audio technologies shooting it out for marketdominance.

In the game industry, scores produced by wavetable synthesis andMIDI are giving way to digital audio soundtracks. (this may happen onthe Web as well, but not anytime soon.) As the speed and data-storagecapabilities of game platforms increase, multiple-synched streams ofCD-quality stereo audio will give composers and sound designers muchgreater fidelity and flexibility. But the most important developmentcould be a system of standards that smooths the integration of musicand sound effects into an interactive context - without involving theprogrammers.

New forms of music will certainly evolve to take advantage of newmedia capabilities. It will be interesting to hear what happensnext.

Clint Bajakian composed scores for Outlaws, Indiana Jones and the-Infernal Machine, and Star Wars: Dark Forces, among many other games.He recently started C. B. Studios, an audio-production facility.

Michael Land and Peter McConnell were composers for LucasArts forten years, where they scored The Dig, Grim Fandango, the Monkey Islandseries, and numerous Star Wars titles. They recently and are working on a better way to make digital mediafunction interactively.

Jeff Lipton is a composer and engineer who has produced web-basedinteractive audio for MGM, McDonald's, KFC (PepsiCo), and PBS. He'svice president of sonification at Sonicopia, an Internet-audio startuplocated at San Francisco's DubeyTunes Studios.

George Sanger, aka the Fat Man, is a legend in the game-audioindustry. He has produced soundtracks for Wing Commander, The 7thGuest, and Putt Putt Saves the Zoo and hosts Project Bar-B-Q, aTexas-style audio technology think tank.

Tommy Tallarico, of Tommy Tallarico Studios, is one of the mostprolific composers in the game industry, with more than 120 titles tohis name. He has released five CDs of his game music and won numerousindustry awards. He hosts the weekly video-game television show TheElectric Playground.