“Orchestras are having a hard time,” Smith says. “I don''t necessarily love them as institutions, but as musical instruments they still can''t be beat.”
Photo: Bruce Gilbert
The conductor lifts his hands and nods to the others on the stage. He lowers his right hand, and at the bottom of its arc, the orchestra begins to play the first movement of Beethoven's Symphony No. 1. The sound is lush and lovely. The blend between the first violins on the left, the cellos on the right and the winds upstage and in the center is perfect. The conductor is wearing an embroidered formal jacket that would not look out of place on the cover of Sgt. Pepper's. He is softly lit from beneath, giving him a bit of a mad-scientist aura. As he swings his arms and leans to the left and right, the tempo changes, the balances shift and the music comes to life.
But there is no orchestra — no violins, cellos or winds. The conductor is alone on the stage except for a computer operator, a computer display and five Bang & Olufsen BeoLab 5 towers arranged in a rectangle with one in the center. The sounds originate from a quad-core Mac Pro running Apple's Logic Pro, which is selecting from more than a million samples from Vienna Symphonic Library's Vienna Instruments. The conductor is wielding not a baton, but a copper-painted Nintendo Wii Remote (or Wiimote). Another Nintendo controller, a Balance Board, is picking up his body's movements.
I'm at the world-premiere concert of the Fauxharmonic Orchestra, a six-year-old project by Paul Henry Smith, a conductor, composer and expert in MIDI orchestration (for more about Smith, read it here at emusician.com/online_exclusive/fauxharmonic_bonus_material). Along with about 150 other curious audience members, I'm in the huge Holy Name Church in a residential section of Boston. The reverberation would be overpowering for any acoustic instrument louder than a lute, but it happens to work very well tonight.
Before the Beethoven symphony, Smith and his machines do a short demonstration of how the computer responds to the Wiimote by playing a trifle by Johann II and Josef Strauss, the “Pizzicato Polka.” The music follows his motions, speeding up and slowing down as real orchestral players would. We hear a loud pop before the music starts, and the flutes emit a few strange peeps during pauses. Hindemith's Trauermusik with violinist Noralee Walker is next, followed by two arias from a modern opera and another from a Handel oratorio, each featuring mezzo-soprano Tynan Davis. The blend between the live and sampled sound sources is quite convincing.
The Beethoven opens with the music stopping and starting again several times. In the last movement, the orchestra decides to skip a few bars. Smith is understandably a bit rattled but forges ahead, and the rest of the movement goes fine. Even if the orchestra and the conductor lose each other, the virtual musicians are always perfectly in sync with one another — which any conductor will tell you is a definite advantage over a real orchestra.
A lot is going on here, and not just inside the Mac. In an age of dwindling resources for arts organizations, is the Fauxharmonic a harbinger of the future? Will virtual-instrument technology supplant the real thing in live performance the way it has revolutionized recording? And is simulating a 19th-century musical institution the best use we can make of the brilliant and cheap new technologies that interactive games are making available to us?
Nuts and Bolts
On a technical level, what Smith is doing is relatively straightforward. He uses a shareware program called OSCulator (see the March 2009 EM for more on using this software with the Wiimote) to interpret the Wiimote's accelerometer data and button presses. Information about how quickly he changes the Wiimote's direction is sent directly to Logic Pro's Tempo fader, which controls the sequence he has constructed for each movement or piece. “OSCulator also lets you send application control data, like to move the cursor to the beginning of the file or a specific marker,” he says.
Another way to use the Wiimote is to perform beat detection using Cycling '74 Max and an object called aka.wiimote to generate MIDI timing clocks. Smith did not do that in this concert, however. “In the latest version, Logic no longer responds to external tempo changes through MIDI clocks,” he says. “So the only way to get it to change tempo in real time is through that fader.”
The Balance Board that Smith stands on has four pressure sensors that send data reflecting the amount of force on each one. The data changes when Smith moves his feet or shifts his weight by leaning, for example. “This also goes through OSCulator. I assign the sensors to Transformer functions in Logic and scale them,” he says, “and then use them to control velocity values in different sections of the orchestra.
“Sometimes the hall has too much bass resonance so you lean toward the first violins to increase them. But I have to scale the response differently for different sections, so that leaning toward the violins might affect them to one degree while leaning toward the basses affects them differently. And each piece has to be set up differently. I start with a good balance that doesn't need much input, and then I tweak the numbers for each instrument section and each piece to give me just the right amount of flexibility.”
Paul Henry Smith—accompanied by a Mac Pro, a Wii Remote and an arsenal of software—performs classical works before a live audience, either alone or with one or two soloists.
Photo: Paul D. Lehrman
In some aspects, this setup is a conductor's dream — an orchestra that always plays the parts correctly and always follows — but for the audience, maybe not so much. “I'm not that self-absorbed that I think it's enough for the audience just to see me,” Smith says with a laugh. “The interaction between me and the two soloists in the concert was good for the audience. One person told me she preferred seeing my solo conducting, but that was my mom. For the Beethoven symphonies, it might make sense to have two or three other people to help me; they could be concerned with the balance while I'm working on the tempo. I wouldn't divide the orchestra along timbral lines, but along other factors, like articulation and energy — for example, making the harsher samples play in VSL when I want to give it more edge.”
At the very least, the Fauxharmonic Orchestra is a proof of concept. We now know that a virtual orchestra works with a live conductor, even if it doesn't result in an explosion of similar “ensembles.” And it's a great advertisement for the capabilities of Vienna Instruments. In many passages, if you close your eyes — and I did try this — you wouldn't know you're not listening to a real orchestra. Step back a level and imagine you are listening to a recording of a real orchestra, and it's even harder to tell that it wasn't made on a concert hall stage with 80 instruments.
Yet little things do give it away, such as one observation I jotted down during the concert, and that Smith agreed with: When the woodwinds get louder, they also seem to get closer. “VSL switches sample layers to get dynamics,” he explains. “To get smooth crescendos between notes that are in different layers, you can tweak the volume of each note, but because the next note is at a different level, unless you scale back the volume at the exact start of the second note, you'll get a staggered crescendo. That's an awful lot of tweaking.
“They also have samples of crescendos, but unless you can use them just as they are, you'll have problems. For instance, what if I need a 3.8-second crescendo? Do I have it? No, I just have 3.0 and 6.0. So what do I do? It's especially important when I'm performing not to make those notes too long or too short, or I may reach the end of the sample before the end of the note, or else cut it off too soon. You can't have every possible dynamic change on every possible note in the library — it would be impossible to control.”
Live in the Future
Smith sees room for improvement in virtual orchestras — both in live performance and in the studio — but he thinks the next major step will not involve sample libraries. “I think we have a short-lived moment in history with these huge sound libraries,” he says. “We are always dealing with how to do things now, and so the sample libraries are the solution of the moment, but I think it won't last. The future is an algorithmic approach such as the Synful Orchestra, which creates instruments in real time using additive synthesis and reconstructive phrase modeling. When the people at Synful, or whoever else does this, get better at their algorithms, there will be a tipping point at which composers will say, ‘Yes that's the way.'' With an algorithmic orchestra, I don't need to look in the library to see if I have the right crescendo — I can create it on the fly.
“But the sequencing software needs to get better, too. We're stuck in this paradigm in which you have a track with MIDI or audio on it, and you're applying a transform to it. But there are no ‘select by rule'' or condition commands [like] they have in every other kind of software, like Microsoft Office. You should be able to tell the software to, say, look for all the dominant chords and change them so they're in Dorian mode. When you select a group of notes in Logic, it will tell you what the chord is, but it doesn't let you do anything with that information.
“And for live performance, why can't the software anticipate gestures so that a performer can set things up, like [changes in] tempo and articulation? A real musician knows what's coming next and adjusts his playing to it. The software knows the score and knows when a note is going to end, so why can't it look ahead and deal with some of these issues, too?”
Going back to my earlier questions about whether technology could someday replace real orchestras and whether we're optimizing current technology, the answers are probably all no. But the answers aren't really the point. The fact that we can ask the questions seriously is more important, and Paul Henry Smith's work is not only making that happen, it's also already leading to a lot more questions. And as the community of musicians creating sophisticated interactive performance systems using tools like the Wiimote grows, what questions are asked should lead to a pretty interesting future for live music.
Paul Lehrman, the former “Insider Audio” columnist for Mix magazine, has been doing live musical performances with computers for more than 25 years. He is coordinator of music technology at Tufts University.