If you can wrap your brain around the colossal impact that electronics and software have had on live music, then you might have an indication of what's
Publish date:
Updated on

Click here to watch some clips of Kasumi's performance with Grandmaster Flash and other artists

If you can wrap your brain around the colossal impact that electronics and software have had on live music, then you might have an indication of what's going down in the still-young genre of live video art performance, otherwise known as VJing. In the past year or so, VJ software/hardware has crossed the border from producing a workable (if sometimes cheesy) accompaniment to live music to being deep and fundamental components of a performance. Things have come so far in the field in such a short amount of time that entirely new means of collaborative performance are now taking shape.

Sound and image have been used in combination ever since the first primitive people grunted and pantomimed stories. Even before the “talkie” movies, live musicians jammed to the screen. Filmmakers, animators, music-video directors and crazed electronic musicians have all experimented for years creating visual work based on sound. But what's new today is a VJ responding to and interacting with music live, just as a DJ spins records in a set to lead the audience in a certain direction. New York — based filmmaker and music-video director Kitao Sakurai says “As computer-processor speeds grow faster and increasingly sophisticated software is developed, the ability to construct — in real time — a motion picture that flows and is responsive to the feel of the audience and the musician is now possible.”

But all the latest technology and expensive equipment won't necessarily produce a better show. The key to a solid collaboration between musicians, DJs and VJs is communication — developing a common vocabulary with which you can express ideas.

Traditional composers indicate tempo and emotion with specific words. As the one playing your own music, there's no need to have such instructions, but when you team up with a VJ, you must be able to relay that information so that the visuals exist as an integral component of the music, not just something slapped on. You might describe your work metaphorically (a part sounding like a diesel engine), literally (by illustrating lyrics of a song), abstractly (red, fast, sad, etc.) or symbolically (an ambulance siren to create a feeling of emergency).

When working with a video artist, however, a balance must be struck between the need to articulate the meaning of the music and the need to give the visual artist freedom to interpret the music in the way that he or she sees fit. One way to maintain such a balance is to use audio samples extracted from the video clips that your VJ has chosen, thus unifying and intensifying the performance. Conversely, your audio peaks can drive certain animation-effects controls in VJ software such as GarageCube Modul8. If you and the VJ riff on the same sample simultaneously, the potential is scary.

There's no formula for success — it's definitely a learn-as-you-go process — but communicating well with your VJ is a critical start. In addition, if you understand the unique conditions, available equipment and audience expectations of each performance, you can produce some wonderfully memorable shows. Knowing more about a VJ's needs will help you collaborate with one, and you can learn how different circumstances affect a performance by taking lessons from the following real-life experiences.


You may spend a great deal of time preplanning a show with a VJ, but you must also find out and prepare for the particulars of the performance venue. Earlier this year, I toured Germany with experimental saxophonist/electronic musician Nikola Lutz. We performed the day after I arrived, and although we had talked for months in advance about the program and sent material back and forth, we hadn't discussed the actual physical setups. The video was produced down to the frame, so she improvised to the video rather than the other way around. The images needed to have the audience's full attention, but Nikola also needed to face the screen in order to improvise. A tiny video Walkman that held the video was too small for Nikola to monitor the picture, so in venues without two projectors, we ended up behind the audience. A typical club setup has the video next to or behind the performers. Unless there's a second screen or monitor (like the visual equivalent of a DJ's booth monitors or a band's stage monitors), not only does the musician not see the video, but if the screen is too small — in the 10- to 12-foot range — the musicians and the screen start to compete visually.

The thing to remember is that you can never have enough information about the venue layout, projection systems, other available equipment (such as long video cables) and the lines of sight for performers and audience alike. Adding video projection to a venue adds another layer of complexity to the physical setup. With music, sound waves bounce off surfaces and can be heard without seeing the source. Not so with images — if something is blocking the view, you're out of luck. Ambient lighting can also ruin a show by making the video washed out and flat.


Any VJ can string together some psychedelic clips to the beat of your music, but it won't be very original or meaningful. To truly collaborate with a VJ, you have to respect the amount of effort it takes for a VJ to prepare and start planning early.

Last year I performed video alongside the New York Philharmonic at Lincoln Center. My video feed was toggled with a live video feed of a miniature puppet theater performance that was simultaneously taking place onstage. Monitoring was great. A massive screen faced the audience, and the stage was also lined with monitors. But there were other hurdles to overcome. Due to delays involving music copyrights, I received the go-ahead for the shows only one month before opening night. Normally, a month is enough time for a VJ to prep, but the music and the high-end nature of the venue and musicians demanded something new and spectacular that told a story, but not too much of a story. It had to be played live so it could follow the conductor's pace, and it had to be created fast. The music was rhythmically insane, so I had to memorize the piece to complement it. I made clips that faded in and out to give me wiggle room if the corresponding movements were played faster or slower. After a month of slaving over movements and colors that expressed the feelings inherent in the music, I thought there was plenty of material. However, after the first rehearsal, it was clear that I needed more video, and it took four all-nighters in render-purgatory to create the new content.

The lesson is that quality VJ content is not made by loading software and pressing buttons. It's always a creative challenge that requires time for the incubation of ideas and their execution. If you want something deep, creative and interesting to go with your music, allow enough time and find out from the VJ how much time is needed. Video is a much more cumbersome medium than sound. File sizes are huge compared with audio, and the video usually has to be rendered — sometimes taking hours — and exported to another format in order to play in real time in most VJ software.


Sometimes the VJ will just have to wing it. You may be touring or recording, without any time to collaborate with a VJ ahead of time. Or you may add video to your show at the last minute. All is not lost in these cases, and most of the time, it's still better to have the video than not. Still, even a minimal amount of preshow communication and then interaction during the performance can help tremendously.

This year I performed with Grandmaster Flash at the Museum of Fine Arts in Houston for the Beats of Basquiat series. I didn't have a playlist and didn't have a chance to talk with Flash before the show, so I had to create a story live — what I call video freestyling.

I brought lots of tiny loops instead of larger, more structured ones and used them like elements of a stream-of-consciousness narrative. I got ideas from recent Grandmaster Flash shows and also studied Basquiat's paintings, packed with symbols of youth culture, money, hype, excess, self-destruction and marginalized urban black culture.

I maxed out GarageCube Modul8 v. 2 software with 80 clips of gritty urban backdrops resplendent with sinister barbed wire, breakdancers alive with emotion, TVs, robots, radar antennae, surreal-looking faces, angry white guys, marching soldiers, old clips of jitterbuggers, protesters, policemen, turntables spinning and so on — all to be scratched and manipulated live along with Flash. Arranging the order and layering of the loops spontaneously altered the insinuations behind the images.

Even if video is a last-minute add-on, try to take a minute on the phone or in a short email to convey your desires to VJs so that they have a better idea of what to pack in the hard drive. Always provide your VJ with your music and a playlist beforehand if possible. I didn't have a playlist for Grandmaster Flash, but it helped that I knew his style and history. If your VJ doesn't know you, at least let him or her know what your music is like. During the performance, treat the VJ as if he or she were a member of your band. Communicate with your VJ through eye contact when beats or song sections are about to change. If you're a DJ, make sure the VJ can see your hands for simultaneous scratching and effects manipulation.


There may not be a standard way to collaborate with a VJ, but an almost ideal situation would include meticulous planning, ambitious ideas and a prior, preferably longstanding working relationship with a particular VJ.

At last summer's Ingenuity Cleveland Festival, I collaborated with video producer and musician VideoPsycotherapy to create a multiscreen setup with multiple audio sources. VideoPsycotherapy and I worked together on music for about five years, and during our rehearsal and recording sessions, we always played video — whether it was my works-in-progress, outtakes from other projects, clips from his TV show or chunks of public-domain film samples.

Therefore, when the opportunity to work together arose, we already had tons of material and had a solid feel for what visual material looked good with our sound. We planned a configuration of the four-screen setup in 3-D software. Ultimately, the four screens surrounded the audience, and the video cascaded around the screens four beats apart — all while the actual footage was timed to the music's beat. We mixed live video and four preproduced DVDs of related work with audio scrubbing and scratching from two laptops loaded with material. With the addition of a live sax player, some audience members sprawled out on the floor to get the full impact.

Although there was no official rehearsal for that performance, it was a case in which every aesthetic element was inherently understood and discussed beforehand, and every possible technical problem was solved before it happened. We had enough contingency plans to run a small army. Only a power failure could have stopped the show.

The lesson is that rehearsing with some kind of video source while you create your music will make you feel more accustomed to the process. Ultimately, however, rehearsing with a VJ and developing a working relationship is the best plan of attack.


Fusing sound and image deepens and widens the perceptual range of each element by revealing nuances to the extent that if either element were removed from the mix, it would cause a fundamental change. Collaborations between musicians and visual artists that interpret the feeling and tone of music are special, and every collaboration is its own unique animal. You can't foresee every technical glitch, but if you clearly express to the VJ the feeling that you're trying to evoke with your music, allow for a good chunk of preproduction time and research the details about the venue in advance, then you're likely to have a cohesive, meaningful show.

To watch a clip of Kasumi's performance with Grandmaster Flash, go


GarageCube Modul8 ($375; can do in real time what usually takes hours to render in a motion graphics program such as Adobe After Effects. Modul8 lets you move, rotate, scratch, scale and modulate RGBA channels and change contrast, saturation and brightness in both positive and negative. Its Trace setting allows you to apply and smear media like painting on a canvas. You can fashion complex spatial and time-related compositions, and instead of being stuck with presets, you can do your own scripting. Incoming sound peaks can control 3-D extrusion and displacement mapping — all with great resolution and in real time. You can change several parameters simultaneously using a MIDI controller.


Just as DJs are on the constant lookout for music to mix, scratch and sample, VJs constantly seek out cool, symbolically rich material that serves their artistic needs. Public domain material is great if you plan on reselling your work on DVD or the Web.

There's a colossal amount of VJ technical information, software, hardware, techniques, downloads, advice, forums, links and message boards.