Web Bonus! Watch a video of this entire process HERE.
THE PROCESS of scoring music to picture is an exciting challenge. Whether you’re using a broad approach based on mood or underscoring every available movement, you are defining your new work within the formal restrictions of the movie itself. It becomes your timeline—your beginning, middle, and end.
Fig. 1. First steps. One can view this process as either constraining or liberating. However, I find it to be liberating in the sense that, once a formal structure is defined, there is one less thing to worry about. The timeline ahead of you can help inspire musical ideas or themes. You can use the onscreen action to dictate tempo and activity, or not. It all depends on the mood you���re trying to create or underscore.
One of the main advantages of scoring to video within a DAW is the ability to accurately match a musical cue’s tempo to the action. For example, it allows you to place the musical downbeat on the exact frame of a new scene. Even if the music you compose ends up being played by live musicians, they can record to the click track (or metronome) within the session, and then you can line up the tracks with your video reference as closely as desired. This process is now an industry standard for film, commercials, animation, and video games, and it is the best way to work quickly and efficiently.
Of course, nothing can replace a sharpened No. 2 pencil and manuscript paper for some things—notes, ideas, condensed, or even full scores. And using those tools as a starting point can often be the best approach. But the usefulness of a DAW is undeniable.
In this article, I will show you how to begin a project in Pro Tools, sketch out the musical parts to fit the action using MIDI, then create a score in Sibelius from the MIDI data, which I use when I replace the virtual instruments with live musicians. As a demonstration, I will use a video of Voodoo Vince, the creation of game designer Clayton Kauzlaric for Microsoft’s original Xbox. This scene shows Vince the Voodoo Doll as a young man, drawn in the style of early 20th century animation. (Watch a video demonstrating these steps at emusician.com.)
Fig. 2. Importing video.Building Your Project Begin by creating a new Pro Tools session. I typically run my sessions at 96kHz/24-bit, but it is possible that for video compatibility concerns a 48kHz/24- bit session might be used. If you are creating content for clients, it’s always best to check with them and export your final bounced movie at the sample rate and bit depth they request (see first steps, Figure 1). Next, import the video you intend to score. Use the pull-down menu File>Import >Video (see Figure 2).
Then, locate and select the video. A pop-up window will ask you where you want to locate the imported video. Choose Session Start to have it begin at the beginning of the session timeline (see Figure 3).
At this point, the video you’ve selected will appear both as a track on your edit window and as a moveable, floating video window. The frame rate for this video is at 29.97 frames per second. (The frame rate shows up in the movie track window after it is imported; see Figure 4.)
I will change the frame rate from 30 fps to 27.97 to match the video by going to Setup>Session, and then selecting the proper frame rate (see Figure 5).
Fig. 3. Session Start.
Once the movie has been imported and the session is playing the movie at the correct frame rate, the image in the movie window will reflect where the cursor is set in the timeline of the video when you drag the cursor over the movie track (see Figure 6).
Tempos and Hits Now I’m ready to develop an overview for the tempos that I’ll use for various sections of the video, by creating a timeline with markers for quick reference. I will also insert MIDI instruments into the session in order to sketch out some musical themes and ideas that underscore the action and enhance the mood and intent of the video.
Fig. 4. Frame rate after import.Fig. 5. Setting frame rate.Fig. 6. Image at cursor.Fig. 7. Changing the tempo.Fig. 8. Checking the Tempo window.
For this part, I made sure to have the conductor icon lit in the Transport window so that the session’s tempo will follow the tempo track in the ruler above the edit window.
Starting with the default tempo of 120 BPM, I view the opening credits with an audible click track. Once the main character enters, I change the tempo in order to match his footsteps, by opening the Tempo Change dialog box on the top of the track window of the Edit Window (see Figure 7).
Depending on your setup preferences from previous sessions, this dialog window may or may not show up. If not, you’ll need to select the drop-down window next to the Bars/Beats header and check the tempo window (see Figure 8).
Clicking on the arrow sign next to Tempo will open up the Tempo Window timeline window, which allows a variety of ways to manipulate tempo; using the pencil tool for accelerandos and decelerandos, using the trim tool to select and change the tempo of an entire region, etc. (see Figure 9).
Clicking the plus sign to the right of the tempo bar will open a dialog box that will allow you to change tempo from any selected point in your timeline (see Figure 10).
Fig. 9. Tempo window timeline.Fig. 10. Changing tempo from timeline. I type in various BPM values until I find the one that works best.
In this case, I want to have the downbeat hit when Vince takes his first step on the ground. To do this, I click on the Meter Bar in the ruler above the Edit window’s timeline to open the Meter Change dialog box. Instead of changing the meter, I deselect the box that says Snap to Bar. This creates a new measure exactly where I want the music to begin, which in this case is where Vince takes his first step. The time signature—in this case, 4/4— changes to italics because you are not in Snap to Grid mode. From here on out, the other measures follow from that starting point until I institute another change.
Now I can try different musical ideas against the tempo. In this case, to match his gait, I build a piano part that has the feel of a Raymond Scott piece.
As Vince slips on a banana peel, I determine that a bar of 7/16 in a slightly slower tempo covers his somersault. From there, I return to the original time signature and tempo once his foot hits the ground again. To make this transition work, I insert a 7/16 measure using the Meter Change dialog box within the Tempo Ruler, immediately followed by a change back to 4/4. To change the tempo, I select across the 7/16 section and use the Trimmer tool to slightly lower the level of the tempo line (see Figure 11).
Throughout the 4/4 part, I have added a jazzy swing to the piano part.
As I develop the rough piano part, I decide to add 3-note chordal stabs and melodic fills to underscore the humorous onscreen action (in this case, calamities taking place with the folks Vince passes on the street). To keep track of the hit points, I place Memory Location markers by placing the cursor when I want to set a marker, then hitting the Enter key on my numeric keypad. A dialog box opens, letting me name the marker. The name will appear next to the yellow triangluar Memory Location marker above the timeline.
Fig. 11. Trimming tempo. Once I have placed and named the Memory Locations, I open the Memory Location window (under the Window tab or by typing command/control-5 on the numeric keypad). Now I can click on the name of a Memory Location in that window and the curser will jump to that marker so that I can play the session from that point.
After placing markers to determine the exact hit points, I can complete the sketch using MIDI sounds for the piano and percussion.
Ochestration Time Now it’s time to expand the orchestration using virtual instruments, which I will later replace with live musicians. For this project, I used Propellerhead Reason as the source of the instruments for my virtual orchestra—violin, trumpet, alto saxophone, and trombone—using ReWire to slave the sounds to my Pro Tools session. In this case, a benefit to using Reason is that it isn’t very CPU intensive, so my session plays back easily as I add musical parts to my sketch within Pro Tools.
Now I can take the 3-note chord stabs from my original MIDI piano part and re-distribute them among my virtual instruments. I begin by giving the top notes to the violin, and copying and pasting them to the violin’s instrument track. Then I do the same with the middle notes of the chords, which go to the trumpet track. The bottom notes are copied to the alto sax part. Next, I re-quantize the orchestral part with 77% swing to match the piano part, and then delete those pitches from the piano part so that there are no doubled notes.
Fig. 12. The Tracks window.Fig. 13. Selecting Notation Display Track Settings. I decided to double the piano’s left-hand part with an upright bass patch; to do this, I simply select the entire MIDI track from that part of the piano roll and paste it into the bass track. Then, I use the Trim tool to lengthen the MIDI notes of the bass so that they are more audible when combined with the percussive piano part. This adds to the Raymond Scott-like sound I’m looking for.
Export the MIDI Data Once the musical sketch is complete, I export the MIDI parts to Sibelius, the full-featured notation software where I can convert the raw notation into readable charts for the musicians.
Select the tracks you want to convert to notation, then open the Score Editor window (accessed by choosing Window>Score Editor). In this case, I have selected the violin and winds. By default, each MIDI track is displayed as a grand staff, as if it were a keyboard part, so I need to reconfigure the staves to fit each instrument: On the left hand side of your Score Window, go under the Tracks window (see Figure 12) and select Notation Display Track Settings (see Figure 13).
Select the proper clef and staff for each instrument, which will be reflected in the score. The score will reflect concert pitch for all instruments, and once the score is exported to Sibelius, instruments that need to be transposed will be transposed automatically.
Now I can export the data to Sibelius and clean it up further to make it presentable. Under File, I choose Export>Sibelius, then select a place to save the document where you will be able to easily find it.
Cleaning up the Score Because I deselected Snap to Bar in the Meter Change dialog earlier in the project, the exported score will have some truncated measures that don’t accurately reflect what I want the musicians to see. To correct this, I create a duplicate score with accurate measures and time signatures, and then copy the data from the exported score.
I copy the first measure up to the point where a truncated measure appears. Once I paste it into the new document, I will change the rhythmic values of the last bar to fit into the time frame that I want for the musicians. I do this for each section of the score as needed.
Once that is done, I can place the tempo markings. To do this, go to Create>Text> Tempo, then control-click and select the tempo for each section. For the sections where I used a swing feel in the Pro Tools session, I need to alter the notation in order for the rhythmic values to be accurately portrayed. In Sibelius, this is referred to as a metric modulation.
Start by going to the Create menu>Text> Other System Text>Metric modulation. Now control-click to get the window of modulation options. In this case, I select the one showing a pair of barred eighth notes next to a quarter-note- plus-eighth-note triplet. Once that is positioned in the appropriate place, I manually clean up the notation. (An easier way to do this would be to duplicate the MIDI tracks with swing eighths in Pro Tools and straighten out the timings so that they appear as straight eighths when you export them.)
From there, I can check out the individual parts and see how they look before printing them.
Recording Parts, Mixing, and Exporting a Quicktime File I prepared and printed the parts that I’ve designated for recording with live musicians, and hired a violinist (Alisa Rose), a trumpet player (Chris Grady), an alto saxophonist (Sheldon Brown), a trombonist (Andy Strain), and a drummer (John Hanes) to replace their equivalent MIDI counterparts.
I had the horn players play most of their parts together; it really helps the final result come off as a well-rehearsed ensemble when the musicians can practice articulations and dynamics together. Alisa and John came in separately to record their parts.
Microphones are up, levels are set, and once I’ve played the arrangement for the musicians, we begin tracking. Sometimes players will want to hear their MIDI part in the headphones, but most of the time, they don’t. Once they know what to play, they are usually better off listening to themselves and other live players. It’s more musical, and that’s the reason that I decided to use live musicians in the first place. For me, some MIDI instruments—the upright bass, the piano, etc.— will sound fine, and when combined with the magic touch of skilled human interpretation, these MIDI tracks take on a new life.
When everyone was done tracking, I began mixing. At this point, I made the decision to add sound effects to underscore some of the action of the video. Because I wanted to have a clean session with just the music, I mixed the audio first and opened up a new session with the final stereo mix and some extra tracks for sound design.
File>Import Audio (see Figure 14) will open up a browser window where you can navigate to your sound effects libraries and copy them to your session. (Do not just “add” them to your session unless you don’t want them to show up the next time you open up your session.)
Then you can move your sound effects along the timeline until they sync with the action you wish to underscore (see Figures 15 and 16).
Once you’re done mixing the sound design with the track, the last step is exporting the movie to a QuickTime (.mov) file by selecting the region in your edit window that you wish to create your movie from, and going to File>BounceTo>QuickTime Movie. You may want to use QuickTime Pro to adjust the resolution of the movie to make it web ready, but in most cases, YouTube prefers the high-res version, and will compress it on that end. When delivering to clients, find out their requirements and accommodate them appropriately. You can check out the final result, in its entirety, at emusician.com.
This was a really fun scoring project, and the musicians had a great time working on it. Adding real performers to music created in a MIDI environment can really breath life into your score. Now you’re armed with the tools to try this yourself, so take the leap!
Bonus! Watch a video of this entire process HERE.
Steve Kirk is a composer whose work has been featured in TV, film, and video games. He owns and operates Steve Kirk Pop Studios in Oakland, CA, and teaches guitar and composition both privately and at the Blue Bear School of Music and Community Music Center, both in San Francisco.