Foundations of a Visual Music

Author(s): Brian Evans

Source: Computer Music Journal , Winter, 2005, Vol. 29, No. 4, Visual Music (Winter,
2005), pp. 11-24
Published by: The MIT Press

Stable URL: https://www.jstor.org/stable/3681478

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide
range of content in a trusted digital archive. We use information technology and tools to increase productivity and
facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at
https://about.jstor.org/terms

The MIT Press is collaborating with JSTOR to digitize, preserve and extend access to Computer
Music Journal

This content downloaded from

145.97.177.214 on Mon, 03 Apr 2023 13:59:58 UTC
All use subject to https://about.jstor.org/terms
Brian Evans
Foundations of a Visual
University of Alabama
Department of Art and Art History
103 Garland Hall, Box 870270
Music
Tuscaloosa, Alabama 35487 USA
brian.evans@ua.edu

In the moving image we have a synthesis-the

design, a tentative but useable grammar for structur-
spatial counterpoint of graphic art, and the tem-
ing time-based visual art can be developed and even-
poral counterpoint of music. tually applied to the integration of sound and image.
-Serge Eisenstein For this article, I created some simple video ex-
amples as illustrations. Where useful, I will also re-
There is a form of film that is trying to evolve
fer to the works of some visual music pioneers and
that area of thinking which I call "moving vi-
other filmmakers. (Pieces were selected based on
sual thinking." And it is intrinsically a visual
music ... relevance and availability, and the list is by no
means exhaustive.)
-Stan Brakhage
Abstract animation, often called visual music, can
have a structural base similar to that of absolute Visual Music and Its History
music. In this article, I propose a groundwork for a
practical theory of visual music composition. The Visual music can be defined as time-based visual
approach builds from the simple premise that the imagery that establishes a temporal architecture in
resolution of tension moves us through time. The a way similar to absolute music. It is typically non-
article will review some fundamentals from art and narrative and non-representational (although it need
design and apply them to temporal structure. To es- not be either). Visual music can be accompanied by
tablish the idea of visual consonance, the design sound but can also be silent.
principle of proportion will be presented. From this, One can imagine the origins of visual music going
I will develop and codify the idea that it is possible back to the discovery of fire and dancing shadows
to resolve visual dissonance to consonance, and so on cave walls, or perhaps reflections of clouds when
move a viewer through time in a way similar to first seen on the surface of rippling water. In more
tonal harmony in music. We will then briefly con- recent history, color organs (instruments that pro-
sider problems of color. The article will review jected colored light) were seen by the public as early
some traditional approaches to color harmony and as the 18th century with Castel's Ocular Harpsi-
suggest a simple hierarchical approach to working chord. In the 1920s Thomas Wilfred toured the
with color in time. United States and Europe performing on his Clav-
Expanding to a more comprehensive system of ilux, an early electrical instrument that created
time-based design, the writings of Russian film- clouds and streams of continuous color. This his-
maker Serge Eisenstein will be mined for raw mate- tory intensified over the last century, parallel with
rial (Eisenstein 1942, 1949). Eisenstein believed thatthe development of cinema.
"art is always conflict" (Eisenstein 1949, p. 46). For While the Hollywood-style narrative dominated
this article, conflict is defined as the opposition of (and dominates) cinema, pioneering filmmakers
forces that motivates and shapes action. The op- were working to develop a non-narrative language
posed forces are dissonance/consonance or tension/of light. Typically non-academics, these pioneers
release. Eisenstein's writings contain many ideas worked in an experimental tradition similar to that
that are useful in the development of a theory of vi- of music composers such as Eric Satie, John Cage,
sual music-especially his definition of cinematic and Karlheinz Stockhausen. Among the early film-
montage. From the concepts of montage and time- makers were Germans Walter Ruttman, Viking
Eggeling, and Hans Richter. Oskar Fischinger, also
Computer Music Journal, 29:4, pp. 11-24, Winter 2005 from Germany and moving later to the United
? 2005 Massachusetts Institute of Technology. States, worked over thirty years creating abstract

Evans 11

This content downloaded from

145.97.177.214 on Mon, 03 Apr 2023 13:59:58 UTC
All use subject to https://about.jstor.org/terms
animations (Mowitz 2004). The career of New A simple definition of absolute music is the
Zealander Len Lye, working mainly in London, structuring of time with the materials of sound pat-
spanned decades. American pioneers included John terns (excluding again program music and literary
and James Whitney, Mary Ellen Butte, Stan Brakhage, forms such as song, opera, and theater). Stravinsky's
and Jordan Belson. Norman McClaren, Evelyn Lam- comment refers to music composition as the mak-
bart, and their colleagues at the National Film ing of "art for art's sake." A musical work seeks
Board of Canada also created significant works. from the listener an aesthetic response to the per-
There were many others (Russet and Starr 1976). ception of sonic pattern-the appreciation of "sig-
They all struggled against financial, institutional, nificant form." This was a primary focus of
and technical barriers, yet they left a body of work modernism and its formalist leanings (Bell 1914).
that provides a base from which to construct a the- When visual artists discuss composition, they are
ory of visual music. This work is finding renewed in- generally referring to static design, the formal distri-
terest as evidenced by recent exhibitions. The 2003 bution of objects in the picture plane or in three-
Sonic Light Festival in Amsterdam featured many dimensional space. (This article will deal
of the pioneering animations, as did Sons & Lu- exclusively with two-dimensional space.) In the
mieres, Une histoire du son dans P'art du 20e siecle 20th century, artists sought to express design in the
at the Centre Pompidou in Paris, closing in January abstract-significant form in visual space. They be-
2005 (Cruse 2004; see also www.sonicacts.com/03). gan to consider seriously their work in a musical
Also in 2005, the Los Angeles Museum of Contem- way, and the abstract form of the work was the con-
porary Art opened the exhibit Visual Music: Synaes- tent. Kandinsky, considered by many to be the fa-
thesia in Art and Music Since 1900, which then ther of abstract painting, said:
moved to the Hirshorn Museum in Washington, D.C.
A painter ... in his longing to express his inner
(Brougher et al. 2005). Much of this work is becom-
life cannot but envy the ease with which mu-
ing easier to find on video and DVD at online sources
sic, the most non-material of the arts today,
such as the Iota Center (www.iotacenter.org), the
achieves this end. He naturally seeks to apply
Center for Visual Music (centerforvisualmusic.org), the methods of music to his own art. And from
and even Amazon.com.
these results that modern desire for rhythm in
painting, mathematical, abstract construction,
for repeated notes of color, for setting color in
Visual Music/Absolute Music
motion. (Kandinsky 1914)
The concept of absolute music can be constructedUsing music as a basis for visual work was a con-
from Igor Stravinsky's statement, "Music means cept explored by many artists in the early 20th cen-
nothing outside itself" (Stravinsky 1956). This illus-tury. Notable is the work by Serge Eisenstein, a
trates a common mindset held by many composers Russian avant-garde filmmaker who put these ideas
for centuries. Form and content were often one in into practice and, fortunately for us, wrote down
instrumental music. (This of course does not include many of his theories of cinema. These ideas still
programmatic music and especially the occasional have potency and are especially relevant in discus-
attempts to imitate real-world sounds, such as bird sions of visual music, time design, and the combin-
calls, thunderstorms, etc.) Today, acoustic and elec- ing of sound and image.
tronic instruments can be used as generators of ab-
stract sounds without a referent in the real world
beyond the instruments themselves. These non- Visual Consonance
referential sounds are used to create abstract temporal
structures, from traditional musical forms to those For the visual artist, composition is "the arrange-
of a more experimental nature. (Many of these new ment of elements and characteristics within a de-
forms also find correspondences in visual music.) fined area..,. a grouping of related components that

12 Computer Music Journal

This content downloaded from

145.97.177.214 on Mon, 03 Apr 2023 13:59:58 UTC
All use subject to https://about.jstor.org/terms
 Figure 1. My harmonic
analysis of Hokusai's The
Great Wave off Kanagawa
(ca. 1831).

make sense together ... balanced by an overall ap-

,:X

. .. .. .....: .

pearance of continuity" (Bowers 1999). Good vi- ili~il!.4,

sual composition achieves harmony. Art theorist NW::

Rudolph Arnheim said, "One of the basic visual ex- PB~ lii~i '??'. . . . . . . . . . . . . . . . .'

periences is that of right and wrong" (1966). This

idea forms the basis of design foundations as taught .. ........

...........
. .

..
.. ..

.
..

in most art schools. First-year art students study

two-dimensional design and learn the principle of
. . . . . . . . . . . . .'

visual "rightness" and how to achieve it with the

conceptual visual elements of point, line, plane, and
volume. These elements manifest themselves as ob- -----

.. .. .. .. ... .. ..
-

jects in a defined space, expressed as size, shape,

color, and texture, establishing dynamic relation-
ships as these elements interact.
Visual "rightness" is visual consonance. This be-
comes an axiom from which we can build a gram- zonta
mar of visual music. From this premise, composing comp
visual music is a simple process. If rightness is codi- tenti
fied and understood, wrongness is easily defined by is pla
not being right. We might call this wrongness visual tal, a
dissonance, that is, visually active moments of ten-tal and vertical.
sion in a temporal design. Progressing from visuallyThe visual composition has a dynamic balance or
wrong to visually right moves us from dissonancesymmetry
to in the picture plane. We see elements of
consonance or tension to release, just as in music a
repetition. (Note the wave at left-middle matching
dominant harmony resolves to a tonic (e.g., V-I). Asthe shape of Mount Fuji, and the foreground shape
in music, much of the interest and energy is in the of the big arching wave mirrored with the back-
moments of tension. (In tonal music theory texts,ground sky.) We are not concerned here with the
how many more pages are devoted to the dominant subject matter, but rather the underlying principle
chord and all its variations versus the tonic chord?)
of composition or design.
Using the tension/release construction, we can Figure 2 shows my analysis of a drawing of an elk
from the walls of the Caves of Lascaux in France.
move dynamically-some might say musically--
through time with visual materials. We can estab-(Dating from approximately 13,000 BCE, it is doubt-
lish visual cadence points. With cadences, we can ful that the golden section was applied consciously,
articulate units of time and so develop larger tem-
although the drawing of the rectangle on the wall is
poral units such as motifs, periods, and phrases. curious as it predates Euclid by over 10,000 years.
Even more unusual is the fact that it is a golden rec-
tangle!) The animal seems to balance on the lower
Proportion left intersection of phi as it divides the bounding
rectangle on both the horizontal and vertical.
Consider a well-known design principle, the golden Whether by instinct or design, use of universal
section (usually symbolized as the Greek letter < or proportions is a common and effective formal de-
phi). A proportion of roughly 1:1.618, it is prevalent vice in visual and music composition. Proceeding
in the visual arts worldwide and throughout history from simple symmetry to a balanced but active
(Huntley 1970). Figure 1 shows my brief analysis composition is a useful skill in creating visual mu-
(what is sometimes called an harmonic analysis) of sic. Phi is just one of many proportions that has
Hokusai's famous The Great Wave off Kanagawa been explored over the centuries. Jay Hambidge's
(ca. 1831). The cut of the golden section both hori- studies in dynamic symmetry showed the applica-

Evans 13

This content downloaded from

145.97.177.214 on Mon, 03 Apr 2023 13:59:58 UTC
All use subject to https://about.jstor.org/terms
Figure 2. My harmonic Figure 3. From a harmonic
analysis of a drawing of an analysis of Botticelli's
elk from the walls of the Birth of Venus by Charles
Caves of Lascaux in France Bouleau (1963).
(ca. 13,000 BCE).

9 12 16

.????-, ????????r??????
? .

, I , !
.. . .. :,
..?1I ip

"
'I
, r . . I
.,

:X:2

i i !I 3*

" -: 1 : "
.. .. . 7'. , .. . . . , ??
16 12 9

tion of proportio
structions as wel
phi are well docu
1983). Stockhause
(a number series
several of his ear
tion of many mathematical constants, of including
phi 42, can even b
I3, and various other geometric constructions, which illus- should not
trating their application in everythingand from Grecian
the divine pr
urns to furniture to typography (Hambidge "secret
1919, knowledg
1932). The "secret" of the golden sectionIt was re-
should be stres
vealed during the Renaissance in Luca Pacioli's
are not a panacea
book Da Divina Proportione, which was tionwidely or visual con
read by artists of the time. doubtedly useful
Also at this time, Leon Battista Alberti composition
published requ
his writings on painting and architecture. the He de-picture plane.
scribed the system of one-point perspective visualand the music sho
application of the Pythagorean proportions in of allmusicits various a
to visual construction. Figure 3 is taken foundations
from a har- cour
monic analysis of Botticelli's Birth of Venus From by an underst
Charles Bouleau (1963). Here, we see an application in the pictu
sition
of phi on the cut of the horizon line along with a
variations, and lar
composition based on the simple harmonic propor- of design el
tions
tions of 9:12:16. Botticelli here used Alberti's musi- lesson in a book b
cal consonances. For Alberti, proportion was a visual a century ago. Th
music. In Botticelli's application of these propor- single design, wh
tions, we see the interval of the perfect fourth (9:12,"Little can be exp
or 3:4) and what Alberti called the duple diatessaron, Space [which] I s
or double fourth (9:16, or 32:42) (Alberti 1955). SITION ... all bein
Simple-number proportions are a foundation of principle, PROP
Western music intervals, but we can find applica- (Dow 1899).

14 Computer Music Journal

This content downloaded from

145.97.177.214 on Mon, 03 Apr 2023 13:59:58 UTC
All use subject to https://about.jstor.org/terms
Figure 4. From a design les-
son in a book by Arthur
Wesley Dow (1899).

and color. We see repetition. The two examples are

of equal duration of 8 seconds. They zoom in at
roughly the same rate, with a central focus of two
flowers moving to a single flower, focused left of
center. The examples feel similar, but there is varia-
tion. They contrast in their vertical motion, angles
of rotation, and colors; there is repetition, contrast,
and variation. We have the beginnings of a syntax
from which to construct larger forms and compose
motif visual music.
with 4 variations
The construction of a pattern of tension/release
moves a viewer through time, and so we seek to
build tension. The desire to resolve the tension nat-
urally follows. Time passes dynamically. Video Ex-
ample 3 is an exercise illustrating a single shot with
Temporal Design elements entering and exiting the picture plane in
an overall rounded binary form (AABA). The ex-
In temporal design, we move the viewer from visualample has four phrases that land on well-composed,
dissonance to cadences of visually balanced, well- visually "right" moments. Phrase 1 introduces
composed moments. Time passes musically through Mondrian-like graphic materials with red and blue
patterns of tension/release. Filmmaker Ken Burns's squares, shifting in color and moving into a com-
use of slow pan and zoom of a camera over old still posed cadence, seen as a small yellow square spin-
photographs is a simple technique that allows ex- ning to combine with two thick black lines. Phrase
ploration of this basic approach to time design. Us- 2 follows with a repetition of blue and red squares
ing only camera motion, we can see this phrasing inspinning in from the opposite direction, cadencing
films such as The Civil War-A Film by Ken Burns by coming to rest on a balanced design. A new, con-
(Burns 1990). Now a common technique in histori-trasting phrase follows with thin lines rotating
cal documentaries, camera motion over still images around an axis and a return of the small yellow
can also tell stories. See for example French film- square and thicker black lines. This overlaps into a
maker Chris Marker's 1962 short film La Jetie, an final phrase that again repeats the spinning red and
early and effective application of this device along blue squares while echoing the rotating lines that fi-
with montage and visual composition (Marker 1962).nally articulate closure, coming to rest on another
I created simple illustrations of camera move- complete and balanced design. The example illus-
ment with still images, included here as Video Ex- trates how visual time can be musical, using motifs
amples 1 and 2. [Editor's note: video figures for thisand phrases with the "camera" locked in place and
article can be found on the DVD accompanying the whole work unfolding in an single "shot."
this issue.] A virtual camera moves over digitally Developing a visual music piece in a single shot
processed photos of flowers. Key frames are seen inwith little or no editing is a common practice in vi-
Figures 5 and 6. Each example starts in balance withsual music composition; many of John Whitney's
a dynamic or simple symmetry. Once motion starts,films do this. His multi-movement work Arabesque
a tension is established that seeks resolution in is built on the development of 360 points of light
stillness and a return to a balanced composition. moving in the picture plane (Whitney 1975). The
Each example can be considered as a phrase. Wepoints move independently through various numeric
can eventually combine phrases, seeking points of processes, in and out of visual resonances where the
commonality and difference in the visual dimen- points resolve into circles or symmetries in simple
sions of motion (direction and speed), composition,numeric proportions. Whitney called this differential

Evans 15

This content downloaded from

145.97.177.214 on Mon, 03 Apr 2023 13:59:58 UTC
All use subject to https://about.jstor.org/terms
Figure 5. Key frames from Figure 6. Key frames from
Video Example 1. Dots Video Example 2, illustrat-
show points of focus, and ing points of focus and ba-
lines indicate the basic tra- sic camera trajectory. The
jectory of camera motion. camera is moving down
The camera moves up and and rotates clockwise.
rotates counterclockwise.

. ..... .......................... ..................

ri~snr~i.i iiiiiiiiiiiiiuiijiis.iliii;%iijil :eo

'!-'iiiilii~iiiill Ii"+."Iii!!!i'=.I i

~i
?iiii11"+1iiiiii1i
i i ir~liiiiiili~ lii'!?iiiiii ili:~ ? i?
3:: MMii::;;:

ii ~ ~ ~ ~ ~ ~ i =i~sialb~~~iiii " i=li~iiliiiiiiiiilii"+i;i"I I

ii~i!.i iliiiilii~i~iiiii%!i~i~iiiiiiII!
....i~iil ,,,, ....... .m,,aa".iil~iiii,:,?ii~.'iiielisi,!il'= ili :,II, IIiiiii=i!!!!
.... =! ili~iii,:;..r ? !31ii'i1
Wtill~ Iiii:ii =+..= "^::ii:: ^'=ii-i.;....li~i~i!iiii.=.iii ! .. .=iiliiili;.'"lil !!! iii'i !ii~ i'ii' . .... :<I iii~!:iiilj I

. ........

c::e -e

i .. ............ ... ...

i
elliit~i~liiiiie~i l
'2?::?:i
?? ?;:??? ~ ?; i~l'iiii'i~~ii'M oM:

liiiiiilii~i.. ............ ......i~

ill~i 8Ilr.. . ... ..........::
::';:::"on
....... ...f~ii t~i~ iiilii iliiiii

FIX -3-c' ?ol I~~ii~i: " fl

"HI;

olliliitiiliiiiilriii
..........: ?::?:: ? ???::
. .. ........ . ... .......... ~ i'iiftiili'i~iil'iiiiiii

ee e?''"
... . .... ... ......:: ??iiii

c?'en il8iflttilii~~ii8;ili:~~:

..... . ... . . .....'... . . . ......... . == == :' ... .. ; ....

171!i iiiiiiiii.. .. ... .. . . -i~r:

sz?:: ? illiliiMiii 8iiii;iil

??~i. . ............ ..:

~::.:W::
......... ........... .... .. .... .... ..? . : ::: .: .?

zw l .........

Figure 6

dynamics or harmonic procession (Whitney 1980). movements of the dancers resolve into single fig-
These moments of resonance provide points of vi- ures, reminiscent of a musical canon at the cadence
sual consonance. Moving away from and back to the when all voices come together.
consonance creates tension and resolution. Oskar Fischinger's Study No. 10 (1932), in black
Norman McClaren's Pas De Deux (1968) showsand white, is a wonderful example of this single-
mostly single-shot action using time-lapse filmingshot approach with movement through moments of
visual consonance. In this piece, Fischinger creates
of two dancers. The piece presents another approach
abstract materials that directly follow and express
to Whitney's idea of differential dynamics. Definite
the phrases and feeling of the music.
points of resolution occur when all the time-lapsed

16 Computer Music Journal

This content downloaded from

145.97.177.214 on Mon, 03 Apr 2023 13:59:58 UTC
All use subject to https://about.jstor.org/terms
Figure 7. Two color circles:
(a) Itten's ("painter's")
RYB color circle; (b) addi-
tive ("computer") RGB
color circle.

(a) (b)

Red Red

violet orange magenta yellow

ue Ye ll ow Blue Green

green cyan

Considering Color
tion of color, especially hue, can vary widely based
on genetics and experience (Livingstone 2002).
Color can Atalso the very least, color inbe art is confusing.useTake
resolution, for examplealthough
the two color circles seen in Figure 7.
Artists Both are used to determine
have develop complementary colors.
the past A basic tenet
centuries. of color theory is that complementary A
is harmony. colors (those directlyIn opposite on the circle) create
discu
De Grandis color harmony. said,
The circle on the left shows the "A
assumption standard color primaries
that as understood by artists
the
cord of over the centuries: red, yellow,
parts of and blue ("RYB").
a Jo- w
the relationships hannes Itten formalized this color circle in develop-
and
components" ing the color theories he taught (De at the Bauhaus. G
The problem This approach to color is still taught in many art in
lies
nents should schools today (Itten 1973).
be orga
versal The circle on the right is the one used in com-
categorization
Professor puterofgraphics displays as they project light. It uses
Neurob
studies how the additive primaries
the of red, green, and eyeblue (RGB).
notes that "much of what has been written about When added together, these primaries will make
color in art is nonsense." She points out that the white light. The complementary colors in this circle
brain processes hue and luminance (also called are cyan, magenta, and yellow (CMY), also called
value) separately. Some aspects of vision, such as the subtractive primaries, which theoretically pro-
depth perception and the separation of figure and duce black (the absence of light) when combined. As
ground, are colorblind, meaning that we get our vi- one mixes them, more and more light is absorbed or
sual cues from value alone. And the human percep- subtracted. Commercial and digital printing pro-

Evans 17

This content downloaded from

145.97.177.214 on Mon, 03 Apr 2023 13:59:58 UTC
All use subject to https://about.jstor.org/terms
cesses use the subtractive primaries with black ink an RGB space designed to create grayscale when the
added (denoted K), as real-world pigments cannot RGB component values are equal.
create a true black, but rather a somewhat muddy With digital images, it is possible to calculate
brown. sums of the amounts of red, green, and blue in an
In Itten's system, red and green are complemen- image. The closer the RGB sums are to being equal,
tary and concordant in combination. In additive the more "balanced" the image is with respect to
color, red and green are both primary and so are dis- color, as equal means gray. Color can move us
cordant. The two systems will create contradictory through time by resolving an emphasis of a single
harmonies. What is the measure we use to decide hue to a coloring that balances or equalizes RGB
which is right? Color continues to be problematic in summation values (Evans 1990). Tension/release
seeking visual harmony. Furthermore, the percep-with color can be established in this way, but it re-
tion of color is inexact, culturally influenced, and quires a detailed measurement of digital color that
personal. I might use a variation of green that re- may not be practical. What is more practical is a
minds me of the zest of a key lime. The same color general application of the perceptual hierarchies of
might remind someone else of rotting meat, trigger- brightness resolving to dark, and high saturation re-
ing a very different response. One must be carefulsolving to low saturation or gray.
with color. A familiar example is mentioned in Bordwell and
There are some generalizations we can draw, Thompson's analysis of the 1939 film version of The
however. First, color space is a three-dimensional Wizard of Oz (Baum 1939; Bordwell and Thompson
construct. Familiar color spaces, all hue-based, in- 2001). The larger structure of the film is an ABA
clude RGB, RYB, and CMY. Some color spaces in- form based on the settings Kansas-Oz-Kansas.
clude other visual dimensions. One artistically Dorothy is home in Kansas in the beginning, filmed
useful space maps a hue-only approach to one wherein grayscale. Through her adventures in Oz, where
color is defined as hue, saturation, and value (HSV) conflict builds, the film is in full color. Conflict re-
(Smith 1978). There is no perceptual hierarchy of solves back to grayscale when she returns home.
hue, but there are of value (bright to dark) and satu- Shifting from color to grayscale is now a common
ration (full chroma to grayscale). Movement through device on television, used to articulate or simply
these hierarchies provides areas of exploration. end the action. A transition to grayscale or simple
Chemist and colorist Maurice Chevreul in the desaturation often takes us to a commercial.
19th century suggested that the eyes seek the bal- Working with color in HSV space, it is useful to
ance of grayscale, as evidenced through the visual think of brightness as visual amplitude and satura-
phenomena of simultaneous and successive con-tion as spectral intensity. Bright, full, saturated col-
trast (Chevreul 1854). A sensation of harmony or ors can be tiring to a viewer when used in excess,
"chromatic equilibrium" should occur with com- just as loud, full-spectrum sounds can become tiring
plementary colors, as they would mix to create to a listener.
grayscale, a sort of visual tonic. Gray is balanced, Oskar Fischinger's Motion Painting 1 (1947) is a
whereas emphasis on one hue or another createsgood example of a general approach to using satura-
tension. Chevreul also points out the difficulty intion and brightness structurally. In the first section
practice using pigments or dyes as "we know of no especially, the movement of color is important as it
substance which represents a primary color." builds large phrases. In the film, we see a single
Newton showed us that we see white when all painting develop layer upon layer over eleven min-
colors of the spectrum of visible light are combined;utes. We feel tensions resolve in short phrases as we
lower the value or brightness and we see gray. Digi- see shapes developing as motives, begun and then
tal technology simulates this with the additive RGB brought to completion. Spirals coil and unfold; mo-
color space. We can measure the equilibrium of saics of diamonds appear and cover what is under-
color combinations without concern for color circles neath. Larger structures are perceived as repetition
or complementarity. Computer monitors work in and contrast of hue choices. Tension resolves as

18 Computer Music Journal

This content downloaded from

145.97.177.214 on Mon, 03 Apr 2023 13:59:58 UTC
All use subject to https://about.jstor.org/terms
high saturation, and bright colors give way to darker metric, rhythmic, tonal, overtonal, and intellectual.
colors and lower saturations. In discussing the five types, we will focus here on
the first three.

Montage and the Musical Phrase

Metric Montage
Cinematographer Joseph Mascelli lists the "Five C's"
of cinematography as camera angles, close-ups, cut-In defining metric montage, Eisenstein writes that
ting, continuity, and composition (Mascelli 1968)."pieces are joined together according to their
All but the cutting require some degree of pre- lengths in a formula scheme corresponding to a
production planning. Through the practice of story- measure of music" (1949, p. 72). We are only con-
boarding, we see this planning in action (Begleiter cerned with the duration of each shot, cutting each
2001). With storyboards, filmmakers illustrate andshot to a beat without regard for the content of the
articulate many of the decisions to be made when shot. We establish meter by making each cell the
shooting a scene. The principles of composition same length-the same number of frames. Each cell
used in support of the story are key to this process.is like a phrase of music. With shots of short dura-
A director plans a scene as a series of moments, tion, cells are like bars of music-sometimes even
composed as a series of drawings. The drawings beats within a bar.
guide the eventual motion, framing, and design of A time signature can be loosely established
the picture plane over time. Continuity, camera through accent of some visual dimension. Occa-
angle, and composition in a scene delineate phrases sionally, the filmmaker will actually put an all-
that cadence with moments of visual rightness. black or all-white frame at the end of each cell,
Time passes musically. making the visual meter obvious. Sometimes cells
With visual time passing under compositional are edited to match musical phrasing-a common
device in music videos. Because the frame rate of
control, a larger temporal structure can be built. Vi-
sual phrases are combined in film in a process called
film or video is constant, we can accelerate or slow
montage, the cutting or editing of camera shots into down the meter by changing the length of the cells
a sequence. Eisenstein's theory of montage can beas they sequence.
valuable to our developing grammar of visual music. Metric montage follows the "law of simple num-
He believed that it was montage that placed film atbers," and it is applied for clarity when based on a
the "pinnacle of the arts." The unit of montage, musical time passage (3/4, 2/4, 4/4, etc.), where it
what Eisenstein called the cell, combines with other can "bring into unison the 'pulsing' of the film with
units and creates a gestalt, expressing more than just
the 'pulsing' of the audience" (Eisenstein 1949,
the sequence of images. The power for Eisenstein p. 73). Using an irregular measure, "16/17, 22/57,
lay in the heights of expression possible. "To deter-
etc. ... brings a degeneration of the method" (Eisen-
mine the nature of montage is to solve the specificstein 1949, p. 72). From the extremes of simple to
problem of cinema" (1949, p. 48). More simply, he irregular is a middle ground where lengths are var-
states "montage is conflict" (1949, p. 38). Montage ied, alternating between two different contexts, jux-
is a multi-tiered construct of tension and release. taposed temporally back and forth. What he refers
We can think of each cell as a musical phrase. to here are irregular-to-metered durations, such as
From good visual composition, we can build phrases rubato to a tempo, which is itself a basic process of
and expand into a larger musical form. Temporal de- tension/release.
sign at the level of the shot moves us forward to ca- Norman McClaren and Evelyn Lambart's film Be-
dence points. Through montage, we construct gone Dull Care is an enjoyable example of metric
relationships from shot to shot, building a larger montage. Strips of clear celluloid were painted,
form and expressing our ideas, formally or other- scratched, textured, and processed in a variety of
wise. Eisenstein lists five basic types of montage: ways. They cut the strips into cell lengths to match

Evans 19

This content downloaded from

145.97.177.214 on Mon, 03 Apr 2023 13:59:58 UTC
All use subject to https://about.jstor.org/terms
with music by the Oscar Peterson Trio (Lambart Camera motion and zoom come to rest at the end
and McLaren 1949). of each cell. Cells ease-in and ease-out at endpoints
in the larger phrases. Cells enter with cross-fades,
the cross-fade itself a visual equivalent to musical
Rhythmic Montage elision.

In rhythmic montage, the length of the cell is deter-

mined by what we see. "Abstract determination of Tonal Montage
the piece-lengths gives way to a flexible relation-
ship of the actual lengths" (Eisenstein 1949, p. 74). Eisenstein calls his third montage type tonal mon-
Durations of phrases support the content of the tage. Continuing his borrowing of musical function,
cells, and most specifically "movement within the he describes tonal montage as "melodic-emotive."
frame impels movement from frame to frame..., We must be careful here to avoid misunderstanding
objects in motion, or of the spectator's eye directed his use of the terms tonal and dominant, as they do
along the lines of some immobile object" (Eisen- not relate very well to the musical definitions of the
stein 1949, p. 75). This is apparent in basic continu- terms. The tone of a work is its expressive tone.
ity editing in which a long shot, usually containing The dominant is the primary emotional feel of the
lots of image detail, lasts longer than a close-up. shots. He here begins to bring the thematic content
The logic is that it takes longer to see all the details of the film and the meaning of the images into the
in a long shot. The passage of time becomes more discussion. He attempts to maintain an objective
psychological than strictly measured; experiential measure by focusing on the graphic or plastic aspects
time is a consideration. Events and images within of the moving image: color and shape. He calls it
the picture frame determine duration. "graphic tonality"-the design elements of color,
Video Example 4 takes the simple camera motion line, and shape, and what they can represent.
with the flowers seen earlier and extends it into The time design begins to consider the subject
four phrases. The camera motion zooms in with matter and its signification. As Eisenstein explains
cells 1, 2, and 8, slowly panning and rotating from his ideas, he presages developments in semiotics and
and to a balanced composition. Consider these cellsstructuralist thought. This goes beyond our discus-
as phrases, cadencing with a similar design struc- sion, as we are focusing on the plastic elements alone
ture and a focus left-of-center. in composing visual music. The aim is to develop a
Cells 3-7 contrast this and together make up a simple design syntax built from a basic design idea.
contrasting phrase. They are much shorter in dura- We are sidestepping issues of content that go beyond
tion, with centered images and the camera zooming formal concerns. Bordwell and Thompson (2001)
in slightly. We see a basic rounded binary form: A write, "Since a camera movement consumes time
(cell 1), A (cell 2), B (cells 3-7), A (cell 8). The dura- on screen, it can create an arc of expectation and
tion of each cell reflects the amount of camera fulfillment." This restates the premise: as camera
movement within the cell. movement alone fulfills an expectation, the viewer
Metric montage is illustrated with cells 1 and 2 is moved through time to a resolution of tension.
(each 8 sec long) and cells 3 through 7 (each 2.4 sec Thus, while in tonal montage, color and shape do
long). As the camera moves over more space in cell begin a move into connotation, we can reduce an
8, it is consequently slightly longer in duration than image-in-motion to its design elements. From this
cells 1 and 2, yet similar in camera motion and final distilled level of montage, we articulate phrases
composition. This keeps the perception of speed with movement in color (saturate to neutral, bright
more or less constant in these cells. Rhythmic mon- to dark) and movement from and to a harmonious
tage is developed so the durational feel of cells 1, 2,distribution of materials on the picture plane.
and 8 is similar, and the total duration of the B sec- In the flower example, the graphic tonality is seen
tion (cells 3-7), feels in balance. in the repetition of points of visual focus and in rep-

20 Computer Music Journal

This content downloaded from

145.97.177.214 on Mon, 03 Apr 2023 13:59:58 UTC
All use subject to https://about.jstor.org/terms
etition or contrast of direction, rotation, and speed Returning to our simple, syntactical point of
of camera zoom and pan. Centered images of the B view, we can think about montage working at the
section, with little motion, are shorter, and they set three basic levels when building phrases of moving
up expectations for a return to rotation and a longer images. First is metric montage, which deals with
phrase. This is another iteration of the basic the simple duration of shots. This is expanded to
rounded binary structure, this time at the level of rhythmic montage, in which the duration of cells
tonal montage. supports the plastic elements in motion within the
Many of Stan Brakhage's works, usually with no shot. Lastly, the graphic elements such as color,
sound score, are good examples of tonal montage. line, and shape and the dominant expression of
Mothlight (1963), for example, is an interesting the shot build phrases in tonal montage. Each shot
work that uses dead moths and other organic debris or phrase articulates cadences through a construct
he collected from light fixtures. He used no camera of tension/release. Phrases are combined and grow
but instead collaged twigs, blades of grass, dust, and to a larger form through repetition, contrast, and
moth parts onto sticky tape, and then he printed it variation.
to celluloid for viewing through a film projector.
Cells were then cut to create a fast-paced and
haunting montage. Visual Counterpoint
Chris Marker's La Jetee and Ken Burns's A Civil
War are good studies of tonal montage with mini- An artwork can be defined as a complex relation-
mal sources. For example, battle scenes in A Civil ship of elements; it is multi-dimensional. In devel-
War may have shots of short duration, in contrast to oping the elements particular to an image in motion,
longer duration pans over battlefields in the after- it is possible to build the unfolding of these elements
math. Both films show how much can be done in in a contrapuntal structure (Evans 1992). Various
developing large temporal structures with simple graphic elements can work independently in sup-
composition and effective editing. port of a larger structure. Eisenstein calls this visual
counterpoint.
Montage is conflict. Cinematic conflicts unfold in
Overtonal and Intellectual Montage the picture plane. These conflicts work within the
cell or shot and in "colliding" cells or montage.
The last two types of montage are not really mon- Eisenstein lists the conflicts (which deal mainly
tage in the strict sense. Eisenstein calls his fourth with the compositional elements) as follows (the
type of montage overtonal montage. He begins to italics are his):
develop the relationship between the previous three
conflict of graphic directions (lines either static
types and the cinematic work as a whole. He sees
or dynamic); conflict of scales; conflict of vol-
the elements of montage metaphorically as har-
umes; conflict of masses (volumes filled with
monic partials of a musical timbre. Conflict devel-
various intensities of light); and conflict of
ops and resolves as these elements wrestle and then
depth. With a bit more "intensification," there
agree. Eisenstein himself admits that overtonal
can also be conflicts of close shots and long
montage is "not an altogether exact term" (1942,
shots; pieces of graphically varied directions
p. 86), but he is concerned with the larger structures
(pieces resolved in volume with pieces resolved
of the film and its impact on the audience. From
in area); pieces of darkness and pieces of light-
this, he takes us to his fifth level, intellectual mon-
ness; and conflicts between an object and its
tage, which will eventually build a "completely
dimension (and conflicts between an event and
new form of cinematography" (1949, p. 83). In later
its duration).
writing, he calls this intellectual cinema and is re-
ally considering the movement of thought rather [In this final conflict, he refers to optical distortion,
than moving light. stop-motion, or slow-motion.]

Evans 21

This content downloaded from

145.97.177.214 on Mon, 03 Apr 2023 13:59:58 UTC
All use subject to https://about.jstor.org/terms
 What we have here is a comprehensive list of the movement" (1942, p. 81). This inner connection is
design materials. With an understanding of two- among the images, the sounds, and the meaning of
dimensional design and tension/release, we can de- the work. For the audience it is the gestalt of the
velop all of these materials independently within a work as a whole, with all elements contributing to
shot and integrate them through many shots into a the overall meaning. In moving beyond the visual
"unified system" of montage, the basis of a "tenta- music foundations, much can be learned from the
tive film syntax" (Eisenstein 1949, p. 55). techniques of opera, musical theater, film scoring,
Again, consider the first two video examples. and sound design.
They repeat the horizontal direction of focus from When we use digital tools to create time-based
left to right. They contrast in vertical direction (one work, layers of material are usually visualized on a
with the camera going up, the other with the cam- timeline. Events are placed graphically like notes on
era going down). They repeat the relative scale of staves in a musical score. Events unfold horizon-
zoom, and they repeat the duration. They contrast tally, but vertical relationships are also important-
in rotation angle (clockwise vs. counter-clockwise). multiple lines moving independently yet in
Different elements with different temporal struc- harmony. Eisenstein's image of the orchestral score
tures-some in agreement and some opposed-cre- is realized in the user interface of current graphics
ate a counterpoint. software designed for time-based work. (Good ex-
amples include Adobe After Effects, Digidesign Pro
Tools, Macromedia Flash, and even Apple's iMovie.)
Vertical Montage (and the Sound Image) Horizontal layers represent video, audio, still im-
ages, camera motion, interpolation paths, and more.
Eisenstein expanded his counterpoint to combine These lines can interact and mix both horizontally
the "visual image with the sound image, in the and vertically.
process of creating a single, unifying sound-picture
image" (1942, p. 73). He considered this a new
kind of montage, naming it vertical montage. Toward a Visual Music Theory
Again, he uses the term loosely, referring to the in-
teractions of various elements as they are synchro- Eisenstein believed that the basis of every artwork
nized in time. is conflict according to its social mission, according
He saw vertical montage as an orchestral score to its nature, and/or according to its methodology.
with several staves of activity unfolding on the hori-The methodology of cinema is montage. At the
zontal yet with all elements interrelating on the graphical level, montage can move us in time
vertical, all moving forward with "intricate har- through the visual "rightness" one learns in two-
monic musical movement" (1942, p. 74). The mon- dimensional design. With experience and sensitiv-
tage structure of the visuals is just one more line on ity to design foundations, the artist, filmmaker, or
the musical score. The varied techniques of music visual music composer can explore the higher levels
composition now can be applied to build a sound of artistic conflict, its nature and social mission.
score that fits the overall contrapuntal texture of Even with studies in composition and movement
the work. using static images, a design sense can be exercised.
For Eisenstein, it was important that this be a Sensitivity to musical time can be developed and
true, polyphonic texture. He was not interested in practiced with visual materials. A sound score can
direct synchronization of sound and image, but be added and musical sensibilities applied to a uni-
rather the "inner synchronization between the tan- fied sound-image expression.
gible picture and the differently perceived sounds... Literacy these days means speaking the language
the 'hidden' inner synchronization where the plas- of new media, which is in many ways the language
tic and tonal elements will find complete fusion... of cinema. As it is a temporal medium, we can find
[where] we find a natural language common to both- at the foundations of cinema the concept of music.

22 Computer Music Journal

This content downloaded from

145.97.177.214 on Mon, 03 Apr 2023 13:59:58 UTC
All use subject to https://about.jstor.org/terms
The tenets of musical expression give us a method Burns, K. 1990. A Civil War-A Film by Ken Burns.
for moving the viewer/listener through time. With Alexandria, Virginia: PBS Home Video.
visual composition, we create consonance and dis- Cheuvreul, M. E. 1854. The Principles of Harmony and
sonance. The ability to resolve dissonance to conso- Contrast of Colors and their Applications to the Artist.
West Chester, Pennsylvania: Schiffer Publishing.
nance provides a syntactical basis for temporal
Cruse, V. 2004. Sons & Lumieres, Une histoire du son
design and the composition of visual music. Our
dans l'art du 20e siecle. Paris: Centre Pompidou.
compositions can be a single shot moving through
De Grandis, L. 1984. Theory and Use of Color. New York:
phrases. We build phrases through repetition, con- Harry N. Abrams.
trast, and variation of the many visual dimensions Dow, A. W. 1899. Composition. Garden City, New York:
including direction, speed, shape, size, and color. Doubleday.
Eisenstein's montage provides a conceptual frame- Eisenstein, S. 1942. Film Sense. New York: Harcourt.
work for putting the phrases together into larger Eisenstein, S. 1949. Film Form. New York: Harcourt.
forms. New media may require a new literacy, yet Evans, B. 1990. "Temporal Coherence with Digital
traditional ideas still have much to offer. Color." Digital Image-Digital Cinema, SIGGRAPH
'90, Art Show Catalog, published in Leonardo supple-
mental issue. Oxford: Pergamon Press, pp. 43-49.
Evans, B. 1992. "Elemental Counterpoint with Digital
Acknowledgments
Imagery." Leonardo Music Journal 25(4):13-18.
Evans, B. 1993. "Number as Form and Content: A Com-
This work was done with the support of a Research posers Path of Inquiry." In M. Emmer, ed. The Visual
Advisory Council Grant, University of Alabama. Mind. Cambridge, Massachusetts: MIT Press.
Fischinger, 0. 1947. Motion Painting No. 1. Recorded on
Oskar Fischinger: Pioneer of Abstract and Advertising
References Animation. LaserDisc. Carson, California: LaserDisc
Corporation (1986) Pioneer Special Interests, PSI-88-012.
Arnheim, R. 1966. "A Review of Proportion." In G. Fischinger, 0. 1932. Study No. 10. Recorded on Oskar
Kepes, ed. Module, Proportion, Symmetry, Rhythm. Fischinger: Pioneer of Abstract and Advertising Ani-
New York: George Braziller, pp 218-230. mation. LaserDisc. Carson, California: LaserDisc Cor-
Alberti, L. B. 1955. Ten Books on Architecture. London: poration (1986) Pioneer Special Interests, PSI-88-012.
Tiranti. Griffiths, Paul. 1981. Modern Music, the Avante Garde
Baum, L. F., director. 1939. The Wizard of Oz. 2004. (1939) Since 1945. New York: Braziller. p. 85.
Motion picture. Hollywood, California: Warner Stu- Hambidge, J. 1919. The Elements of Dynamic Symmetry.
dios, ASIN: B00000JS62. New York: Dover.
Bell, C. 1914. Art. London: Chatto and Windus. Hambidge, J. 1932. Practical Applications of Dynamic
Begleiter, M. 2001. From Word to Image: Storyboarding Symmetry. New Haven, Connecticut: Yale University
and the Filmmaking Process. Studio City, California: Press.
Michael Wiese Productions. Howat, R. 1983. Debussy in Proportion: A Musical Anal-
Bordwell, D., and K. Thompson. 2001. Film Art: An Intro-ysis. Cambridge: Cambridge University Press.
duction, 6th ed. New York: McGraw-Hill. Huntley, H. E. 1970. The Divine Proportion. New York:
Bouleau, C. 1963. The Painter's Secret Geometry. New Dover.
York: Harcourt, Brace, and World. Itten, J. 1973. The Art of Color. New York: Van Nostrand
Bowers, J. 1999. Introduction to Two-Dimensional De- Reinhold.
sign: Understanding Form and Function. New York: Kandinsky, W. 1914. Concerning the Spiritual in Art.
Wiley. New York: Dover.
Brakhage, S. 1963. Mothlight. Recorded on By Brakhage-- Lambart, E., and N. McClaren. 1949. Begone Dull Care.
Anthology-The Criterion Edition. DVD. New York: Recorded on Norman McClaren: The Collector's Edi-
Criterion DVD CC1590D (2003). tion. DVD. Harrington Park, NJ: Milestone Film and
Brougher, K., et al. 2005. Visual Music: Synaesthesia in Video (2003) ASIN: B0000AOV4F.
Art and Music Since 1900. New York: Thames and Lendvai, E. 1971. Bela Bartdk: An Analysis of His Music.
Hudson. London: Kahn and Averill.

Evans 23

This content downloaded from

145.97.177.214 on Mon, 03 Apr 2023 13:59:58 UTC
All use subject to https://about.jstor.org/terms
Livingstone, M. 2002. Vision and Art: The Biology of See- Russet R., and C. Starr. 1976. Experimental Animation:
ing. New York: Harry Abrams. an Illustrated Anthology. New York: Van Nostrand
Marker, C. 1962. La Jetie. Recorded on Short 2-Dreams. Reinhold.
DVD. Hollywood: Warner Studios (1997) ASIN: Smith, A. R. 1978. "Color Gamut Transform Pairs."
B00003 I VPS. Computer Graphics 12(3):12-18.
Mascelli, J. 1968. The Five C's of Cinematography. Holly- Stravinsky, I. 1956. The Poetics of Music in the Form of
wood, California: Cine/Grafic Publications. Six Lessons. New York: Vintage Books.
McClaren, N. 1968. Pas De Deux. Recorded on Norman Whitney, J. 1975. Arabesque. Recorded on The World of
McClaren: The Collector's Edition. DVD. HarringtonJohn Whitney. LaserDisc. Carson, California: LaserDisc
Park, New Jersey: Milestone Film & Video (2003) Corporation (1984) Pioneer Special Interests, PSI-88-003.
ASIN: BO00OOOAOV4F. Whitney, J. 1980. Digital Harmony: On the Complementar-
Mowitz, W. 2004. Optical Poetry: The Life and Work of ity of Music and Visual Art. New York: McGraw-Hill.
Oskar Fischinger. Eastleigh, UK: John Libbey Publishing.

24 Computer Music Journal

This content downloaded from

145.97.177.214 on Mon, 03 Apr 2023 13:59:58 UTC
All use subject to https://about.jstor.org/terms