Syllabus

Applications of Motion Tracking to Dance and Theater
2-week Workshop

Introduction

The past ten years have seen an explosion of interest in motion tracking technology; particularly in dance, theater and contemporary music (electroacoustic music). It has moved from being the trick pony of a handful of experimental high-tech groups (Troika Ranch, Palindrome, Ventura Dance) to a cause célèbre of high-profile companies in dance (Trisha Brown, Krisztina de Chatel, Merce Cunningham), theater (Robert Lepage), opera (André Werner, Robert Wilson), installation art (Bill Viola, Armando Menicacci) and pop entertainment (Stomp, Blue Man Group).

Many performing artists today use computers in their creative work. Common examples are:

  • to assist in musical or choreographic composition
  • to generate new kinds of graphics, animations, sounds or music
  • theaters use them to control lights, projections and special effects
  • internet-based events where participants hear, see and communicate from geographically disparate locations
  • to generate interaction

This workshop concerns the last: Interaction. While interactivity has always been part of a theater, music and dance digital technologies are giving it new meaning. Used with care and cleverness, sensors and computer systems can add subtlety spontaneity and expressive power to live performances. And yet, both in its theory and practice, it is often misunderstood and its application has had mixed results.

Computer technology, however, presents artists with a dilemma. While it can enhance artistic expression it can just as easily distract and actually compromise artistic integrity. No one questions that it is fun and catchy, but the "art" in interactive art is not self-evident. It is not something one can learn from books, nor is it something computer engineers can solve for us. It requires testing, thoughtful watching and listening -- in other words, the sensibilities of an artist.

This two-week intensive course covers a variety of systems and approaches highlighting their inherent advantages and weakness. It is comprised of both theoretical and practical sessions.

Practical Training

1 / how to focus a motion tracking camera
2 / software vs. hardware video image controls, using a graphic canvas
3 / creating and manipulating basic EyeCon elements (touchlines and dynamic fields)
4 / internal and external MIDI. how to locate and control MIDI synthesizers, and map sounds to movement
5 / how to record and capture sounds, text and music from the world around us
6 / how to edit sound samples and map them to movement
7 / how to modify sounds with movement (not merely trigger them)
8 / first creation
9 / how use a scene sequencer, using movement to control scene changes
10 / feature fields (size, shape, direction of travel, symmetry) motion tracking features
11 / how to use and control images (bitmaps) and video (avi's)
12 / how to use electrode-based sensors with EyeCon
13 / second creation
14 / optimizing patch performance (use filters, sensitivity and de-bounce controls in EyeCon)
15 / use EyeCon to export MIDI and OSC/UDP to external devices
16 / operate MAX/msp patches and control them with human motion (via EyeCon)
17 / use EyeCon to control stage lighting (via DMX protocol)

Theoretical Issues

Not every topic is covered in depth -- indeed, most are not. This is a list of possible topics.

Topic / Lecture Number / reading
(resource: page nos.)
performance theory
history of interactive performance / 1 / A:all, B:2-8,
the psychology of interaction / 1 / B: 10-13, D: all
mapping / 1,2
collaborative creation paradigms / 2 / D: all, E: all
how artists collaborate with engineers / 2
interactive perf. scenography and dramaturgy / 2 / F: 1-12
digital media
digital audio / G:all, H:all
what is sound / 3
sample rate and sample size / 3
Fourier analysis (FFT) / 3
video1 - light and cameras / I:all, J:all, K:all
what is a camera / 4
what is a lens / 4
photo-electric effect / 5
focal plane and focal length / 5
3D into 2D -- distortion issues / 5
video2 - film and video / 6
brief history of movie technology
progressive and interlaced video
when and how to de-interlace
video3 - compression / 7 / L:all, M:all
types (lossy, lossless, mpeg standards)
uses - for web sites, archiving, projection, dvds
ripping, editing, youtube
video4 - advanced themes / 8 / N:all, O:24-42
CCD architecture
interline frame transfer and other options
CCD vs CMOS
Bayer CFA - mosaicing and de-mosaicing
Pixel number vs. video resolution
video noise, color bleeding, other issues
motion tracking technology / 9 / A:all
body-oriented
electrodes (EMG,EKG,EEG,Touching)
WII technology (wireless accelerometers)
space-oriented
video-based motion tracking systems
sound and music control (electroacoustic music)
stage lighting control
controlling projections
infrared and other filtering methods
projections - video LCD and LED / 10 / A:all, P:all
projection techniques
inherent problems with video projections
interactive music and theater lighting