CTIN 499: Immersive Moviemaking - USC Interactive Media Division

USC School of Cinematic Arts | Interactive Media Division
CTIN 499: Immersive Moviemaking: Gestural Interface for
Cinematic Design
Instructor: John Underkoffler, Alex McDowell
Contact Info:
Semester: Fall 2009
Course units: 2
Prerequisites: Permission of instructors with portfolio review
Course Description:
Summary:
An intensive lab-based project course in which students will construct a prototype of next-generation film
production practice atop a gestural interface platform.
Course Context and Description:
The coming generation of media production and manipulation tools should propose as large a shift from
today's as today's digital editing machinery did from the Steenbeck, Moviola, and Kem. This class will directly
undertake the earliest design and implementation of some of those tools. The underlying thesis -- that a
profound advance in production methodology is possible as a matter of human-machine interface -- will be
exercised in the context of the Spatial Operating Environment, a new application development and execution
platform that combines gestural input, rewindable networking, and a paradigm of real-world pixels. (USC's g-
speak SOE installation is, almost literally, the technology and manipulatory approach recalled from scenes of
gestural interface in the film Minority Report.)
The course will consider present-day methodologies for art direction and production design, visualization,
production scheduling and workflow management, montage, sound mixing and design, and compositing and
other VFX techniques; but will also place a special emphasis on higher-level organizational structures that
hint at a new kind of integrated creative workflow. Interest in such new-practice workflow is emerging in
industry discourse, and to some extent the course will parallel (in an implementation-focused way) the
concerns of the "Virtual Production Group", a new consortium of the ADG, the VES, the ASC, the Previz
Society, the ACE, the DGA, and the PGA. These and other groups have begun to recognize the deep
implications of, e.g., using digital techniques to make editorial choices while still in preproduction, of deploying
digital asset management systems as a means of unifying and normalizing the production pipeline, and so on.
Crucially, the course will encourage an expansive approach to finding form for future production practice,
embracing contributions to interface and technique from mechanical production devices old and new,
electronic games, optical technologies, nonlinear editing tools, and all manner of recent alternative interface
systems.
The course is intended to appeal simultaneously to students from Production, Interactive Media, and
Computer Science & Engineering who are interested in bringing their skills and experience to bear in
collectively brainstorming, designing, and prototyping these new cinematic tools.
Course Objectives:
Participants will design and implement elements of next-generation production in the context of a gestural
interface platform. Visiting experts from particular domains -- design, editorial, production management, VFX,
animation, etc. -- will help to frame and deepen understanding of such tools' requirements. Simultaneously,
students will gain familiarity with the g-speak platform and gain basic proficiency in building g-speak-based
applications. Three tool-prototyping projects will constitute the bulk of the coursework. Each will be executed
in a different medium, the progression designed to parallel the present’s most forward-looking design
practices. The first project will be undertaken ‘on paper’ – as an expanded storyboard, essentially -- and
CTIN 499: Immersive Moviemaking
Interactive Media Division
USC School of Cinematic Arts 1

within the limitations of appropriate static media will serve as a blue-sky concept document describing the
transformation of some particular segment of production workflow. The second will be executed as a video
that demonstrates the interactions central to the transformed workflow segment. The third project will
comprise a nuanced and mature working implementation of a subset of these workflow tasks, and will exploit
the sophisticated HMI capabilities of the g-speak SOE. It is expected that by semester's end the projects,
taken together, will constitute a mature and compelling preview of near-future production methodologies and
an attendant creative workspace.
Curriculum:
This is a course to do with designing and then building things. These activities will occupy the lion's share of
students' overall effort. In-class time will shade from lecture-based (as foundational context is established)
through discussion- and critique-inflected (as proposals for system development are produced and evaluated)
to atelier-style (as construction of final systems proceed). Roughly two-thirds of the class meetings will offer
discipline-specific expertise in the form of guest presentations by exemplary industry practitioners. Readings
will serve an important supplementary role. Readings may be assigned opportunistically.
Team Structure:
After a short series of initial exercises undertaken by students working alone, the course’s major projects will
be executed by three-student teams. It is intended that teams may self-organize to exploit heterogeneous
skills -- separate principal experience in, for example, interaction design, production experience, and graphics
programming -- but other arrangements are fruitfully possible.
Aspects of Reality:
The full breadth of the production pipeline is the domain of this course, and early weeks will be spent partly in
survey of production disciplines. It is not however required (or even really probable) that the collection of team
projects address the pipeline's full span. Participants may elect, for example, to concentrate the teams'
projects in one area in order to experiment with a local ecology of tools. Or the vigorous schedule may
encourage a concentration of tools around some topic especially suited to rapid application development. Or
indeed the class may develop an intuition that, with the right toolset, conditions of self-similarity suddenly
apply: so that a focus on early visualization and planning in a virtual production space may accurately reflect
the full arc of production.
Evaluation of student performance:
Participation 15
Written & Graphical
15
Documents
First Project (System A) 20
Second Project (System B) 20
Final Project (System C) 30
Total: 100
Course content (summarized by class meeting)
Week 1: Primer and Overview
Major interlocking production domains. Visual research and locations; production design; modeling & virtual
construction; visualization; physical construction, prop manufacturing, set decoration; scheduling and
production management; lighting design, cinematography, color grading & timing; script supervision; editorial;
sound design and mixing; VFX.
Introductory look at the Spatial Operating Environment. Basic principles: gestural i/o; real-world pixels;
recombinant networking. The smallest program for the simplest g-speak application. Hands-on exposure to
basic gestural interaction.
CTIN 499: Immersive Moviemaking
Interactive Media Division
USC School of Cinematic Arts 2

Week 2: Uniting Production Workflow
Domain Expert: Ron Frankel, founder, Proof Inc.;
David Morin, Media & Entertainment Division, Autodesk
The social, financial, organizational, and technological forces leading to a more integrated workflow. Virtual
production space as venue for early-schedule migration of creative decision-making.
Critical analysis of 'TAMPER' installation as model of future rapid-assembly (compositing) workstation.
Introduction to character-based GRIPE symbolism for hand-pose representation.
due: Per student: fictional and non-fictional depictions of gestural & other alternative interface,
identified and collected from film, television, literature, graphic novels, video games; and from the
interwebs. Concise textual summary, evaluation of same.
Week 3: Animation as Analogy
Domain Expert: Michael “Wave” Johnson, Director, Moving Pictures Group, Pixar
Analysis of usefulness, technical plausibility, complexity, cognitive suitability of fictional interfaces previously
identified.
Emerging animation workflows as model for live-action methodologies.
Scripting language-based application prototyping in g-speak.
Presentation of Project A proposals. Critique of same. Formation of project teams. Selection of a single,
possibly hybridized, Project A (per team) for further development.
due: Per student: “ten questions” assignment. Proposal for “Project A” domain (particular production
activity) and initial concepts for gestural / new-HMI treatment.
Week 4: Linking Parts
Domain Expert: Tim Squyres, editor (Crouching Tiger, Hidden Dragon; Syriana; The Hulk; etc.)
Physically vs. virtually predicated editing systems. Spatial representations for organized multiples. Mapping
temporal structures onto spatial structures. Hybrid physical / digital systems for executing montage.
Flat displays that nonetheless occupy three-dimensional space. Simple vector geometry. A room coordinate
of x, y, and z for each pixel.
due: Per team: (in progress) Project A sketches, specification of gestural vocabulary, high-level
flowchart / block diagram of system logic. Initial 2D (static) images in support of same.
Week 5: Knowing the Machine That Knows Space
Domain Expert: John Nelson, VFX supervisor (Iron Man, etc.)
Further discussion of integrated workflow. VFX workflow as beneficiary thereof, and of coordinated, end-toend
asset management.
Scene graph representation of structured graphics. Event delivery mechanisms for accurate interaction.
Presentation and critique of completed Project A work.
due: Per team: (completed) Project A.
CTIN 499: Immersive Moviemaking
Interactive Media Division
USC School of Cinematic Arts 3

Week 6: Loosening Reality
Domain Expert: virtual production representative (i.e. Giant, WETA)
Performance capture workflow. Hybrid physical-digital tools on set. Preproduction as production. Production
as early postproduction.
Spatial events. Gesture as pose, orientation, and sustained motion. Absolute vs. relative motion in gestural
events. Gestural continuum: freespace, hover, touch.
Presentation and critique of Project B proposals.
due: Per team: proposals for Project B.
Week 7: World’s Most Serious Interface Design
Discussion and analysis of nominated videogame interface mechanisms. Effects of temporal disparity (lag)
between physical input and graphical output. Thresholds of perceptual causality.
Simple communication between distinct execution modules. Graphical applications as ecologies of looselycoupled
programs.
due: Per team: (continuing) progress report: Project B. Greenscreen elements, compositing plan.
Week 8: Human/Computer = 1/1: No Longer
Domain Expert: Kwindla Kramer, CEO & platform architect, Oblong
Distributed workflow. Application-level telecollaboration. Cognitively coherent representation of non-local
effectors. Problems of distributed causality. Possibility and benefits of explicitly depicting network lag.
Reusable coding idioms.
due: Per team: Project B: early subsequence composite work; any greenscreen reshoots completed.
Week 9: Tool Synthesis
Intensive look at the way in which individual digital tools have transformed isolated elements of production
workflow. Social / organizational forces that impede same from transforming flow between pipeline segments.
due: Per team: compositing and other postproduction for Project B nearing maturity.
Week 10: Organization Is Information
Domain Expert: production manager / line producer (tba)
Asset management systems as connective tissue between departments. Two- and three-dimensional
schemes for depicting large asset multiples. Nonlinear scaling of same for context-preserving local inspection.
Use of g-speak's SoftFloat, SoftVect, and SoftColor facilities for algorithmic animation.
Presentation and critique of Project B work.
due: Per team: finished Project B videos. Proposals for Project C: interactive system implemented
atop g-speak.
Week 11: Getting the Hands Right
Motion-based information. Subtle animation as subliminal cue. Information-bearing transitions and
transformations.
CTIN 499: Immersive Moviemaking
Interactive Media Division
USC School of Cinematic Arts 4

Presentation of in-progess gestural interaction framework. Design critique of same.
due: Per team: initial functioning code sketches for limited elements of Project C.
Week 12: Visual Feedback
Domain Expert: Ben Fry, information designer
Critical discussion of typography in interactive systems. Textual vs. graphical literacy, legibility.
Presentation of in-progress on-screen information design and visual interaction feedback. Design critique of
same.
due: Per team: major graphical design elements for Project C interaction modes.
Week 13: Two Point Five D
Transparency and layering in active interface. Z-stackng in service of history depiction. Z-stacking for
alternate-future selection.
All-hands coding session. Identification of speed & interaction bottlenecks. Simple optimization tactics.
due: Per team: full top-level framework for Project C systems complete and functioning.
Week 14: Finishing Touches
Working session. Optional code review. Fine tuning of interactions.
due: Per team: Project C largely stable.
Week 15: A View of Future Production
Presentation of completed production processes, comprising Projects A, B, and C.
due: Per team: code for completed System C projects. Per student: system summary, documentation,
& analysis document.
Final Exam: none.
Missing a Project, Incompletes:
The only acceptable excuses for missing a project or taking an incomplete in the course are personal illness
or a family emergency. Students must inform the professor before the exam and present verifiable evidence
in order for a make-up to be scheduled. Students who wish to take incompletes must also present
documentation of the problem to the instructor or teaching assistant before final grades are due. Incompletes
are only available after the 12th week Withdrawal deadline.
Attendance:
Attendance at all classes is mandatory. Two unexcused absences will result in your grade being lowered by
one full grade point. Three unexcused absences will result in your grade being lowered two full grade points.
If you have four or more unexcused absences you may be requested to withdraw from the course.
Note for students with disabilities:
Any student requesting academic accommodations based on a disability is required to register with Disability
Services and Programs (DSP) each semester. A letter of verification for approved accommodations can be
obtained from DSP. Please be sure the letter is delivered to us as early in the semester as possible. DSP is
located in STU 301, and is open 8:30am5:00pm Monday through Friday. The phone number for DSP is (213)
740-0776.
CTIN 499: Immersive Moviemaking
Interactive Media Division
USC School of Cinematic Arts 5

Academic Integrity:
USC seeks to maintain an optimal learning environment. General principles of academic honesty include the
concept of respect for the intellectual property of others, the expectation that individual work will be submitted
unless otherwise allowed by an instructor, and the obligations both to protect one’s own academic work from
misuse by others as well as to avoid using another’s work as one’s own. All students are expected to
understand and abide by these principles. Scampus, the Student Guidebook, contains the Student Conduct
Code in Section 11.00, while the recommended sanctions are located in Appendix A:
http://www.usc.edu/dept/publications/SCAMPUS/gov/. Students will be referred to the Office of Student
Judicial Affairs and Community Standards for further review, should there be any suspicion of academic
dishonesty. The Review process can be found at: http://www.usc.edu/student-affairs/SJACS/.
Biographies:
John Underkoffler
John Underkoffler is founder and Chief Scientist of Oblong Industries, developer of the g-speak Spatial
Operating Environment. Earlier, during fifteen years at MIT's Media Laboratory, he was responsible for
innovations in optical and electronic holography, novel animation systems, several large-scale visualization
techniques, and the I/O Bulb and Luminous Room Systems. His applications and installations are in use at
commercial and educational facilities, have been shown in galleries and museums and at festivals, and have
received a very small number of awards. Interstitially, John has served as science & technology advisor to
films including Minority Report, The Hulk (A.Lee), Aeon Flux, Stranger Than Fiction, and Iron Man. The
human-machine interfaces he has devised over the past two decades remain widely influential.
Alex McDowell
Alex McDowell, RDI is one of the world's preeminent production designers, with credits including The Crow,
Fear and Loathing in Las Vegas, Fight Club, Minority Report, The Terminal, Corpse Bride, Watchmen, and
Fantastic Mr. Fox. He has emerged at the forefront of thought about modern production workflow and about
the cross-disciplinary effects and applicability of design & design methodologies. He collected a large
community around these ideas by founding (in 2008) the 5D Conference, but also puts such approaches into
regular practice, designing across a wide array of media including opera, videogames, virtual architecture,
and live, immersive environments. He serves as chair of the Art Directors Guild's Technology Committee, on
the board of the University Art Museum at California State University, Long Beach, and as Visiting Artist at
MIT's Media Lab; he was named Royal Designer for Industry in 2006 by Britain's RSA. McDowell is currently
working on a digital stereo animated feature at DreamWorks Animation.
CTIN 499: Immersive Moviemaking
Interactive Media Division
USC School of Cinematic Arts 6