Virtual Production: Mixed Reality Filmmaking

Summer 2022

Non-Credit-MRF

Request Information

This course will explore the fundamentals of emerging mixed reality cinematic techniques that are quickly becoming integral production tools in both the film and gaming industries. Practical camera and video fundamentals will first be covered and how they translate into Unreal. From there we will quickly build full cinematic scenes in engine using the Marketplace for assets and animated virtual cameras and lighting on the Sequencer.

After establishing an initial cinematic workflow in Unreal, we will then learn how to dynamically control a virtual camera in engine using a mobile device and prep shots for export and integration into Adobe Premiere. Then, in the last week, more advanced forms of camera tracking will be explored, including virtual production green screen techniques that integrate live video into Unreal using a mobile device or DSLR combined with a Vive tracking system.


Key Takeaways

  • Basic camera principles and production terminology
  • Unreal as a cinematic production tool
  • Virtual camera control with a mobile device
  • Realtime chroma key and video compositing
  • Tracking and integrating a DSLR camera into Unreal using Vive components

Who Should Attend

Independent Filmmakers: Directors, Cinematographers, Producers, Executive Producers, Technical Directors, VFX Artists/Supervisors, Post-Production Supervisors

Experiential Creators: Immersive Storytellers, Game Engine Developers, Creative Technologists, New Media Artists

Networks & Media Companies: Content Development Executives, Directors of Video, Interactive Designers, Producers, Directors of Emerging Technology and Innovation

Creative Studios & Experience Design Firms: Producers, Technical Directors, Content Directors, Creative Technologists, Technical Producers

Corporate/Brand - Producers, Executive Producers, Creative Technologists, Chief Content Officers, Creative Production Managers, Emerging Technology and Innovation Directors

Agency - Producers, Executive Producers, Directors of Content Production, Integrated Production Managers, Emerging Technology and Innovation Directors

Educators - Teachers, Instructional Designers in Media Production


Technical Requirements

  • An Epic Games Creators account (register here)
  • A computer that runs Unreal Engine 4.27 (recommended specs)
  • At least 60 GB of free SSD hard drive space
  • A three-button mouse
  • ARKit enabled iPhone or iPad (iOS 11.0 or later and A9 or later processor)
  • Green Screen Backdrop (tabletop or larger setup)
  • Adobe Creative Suite (30-Day free trials are available)
  • OPTIONAL
    • DLSR with HDMI output
    • HDMI Adapter (ex: Elgato Cam Link 4K)
    • Vive Base Stations + Vive Tracker

SCHEDULE

Week 1

  • 1.1 - Mixed Reality Filmmaking Primer
    • [LECTURE]
      • Brief history of cinematic VFX
      • Emerging possibilities of Unreal as a film production tool
      • Current mixed reality cinematic practices
    • [ASSIGNMENT]
      • Watch introductory Unreal videos
  • 1.2 - Camera Basics
    • [LECTURE]
      • Focal length, aperture, and the exposure triangle
      • “Seeing as Practice” - photographic composition
    • [DEMO]
      • Importing marketplace assets into Unreal
      • Setting up a virtual camera in Unreal
      • Exporting images from Unreal
    • [ASSIGNMENT]
      • Unreal scouting photos
      • Watch introductory Unreal lighting videos
  • 1.3 - Lights, Camera, Capture!
    • [LECTURE]
      • Light is everything
      • Unreal lighting basics
      • Color theory and palettes
    • [DEMO]
      • Setting the Unreal color space
      • Cinematic lighting in Unreal
    • [ASSIGNMENT]
      • Reshoot your Unreal scouting photos with new lighting
      • Watch introductory Unreal sequencer videos
      • Watch introductory Adobe Premiere videos
  • LIVE SESSION - Live Workshop

 

Week 2

  • Motivated Camera Movement (2.1)
    • [LECTURE]
      • Cinematic shots and angles
      • Genres, Bayhem, and other cinema tropes
    • [DEMO]
      • Unreal sequencer overview
      • Keyframing virtual cameras
      • Unreal camera crane and tracks
      • Exporting videos for editing in Adobe Premiere
    • [ASSIGNMENT]
      • Create a color palette and deliver a short edited sequence
  • Doing It Live (2.2)
    • [LECTURE]
      • Codecs, bit depth, color space, bit rate, and frame rate
      • The art of the chroma key
    • [DEMO]
      • Virtual camera motion control using an iPhone/iPad
        • Unreal Take Recorder
      • Live video from iPhone in Unreal over NDI
        • Live keying in Unreal with a custom Composure material
    • [ASSIGNMENT]
      • Take a screenshot of a live green screen video composited into Unreal 
  • LIVE SESSION - Workshop and Office hours

 

Week 3

  • Mixing in More Reality (3.1)
    • [LECTURE]
      • The layers of mixed reality
      • Mixed reality production design fundamentals
    • [DEMO]
      • HDMI input with a DSLR and Vive tracking system
      • Syncing footage in Adobe Premiere
    • [ASSIGNMENT]
      • Deliver a mixed reality shot

 

  • Project Breakdown (3.2)
    • [LECTURE]
      • American Ballet Theatre production breakdown
      • Andra Day production breakdown
    • [DEMO]
      • Reducing key frames in Unreal sequence
      • Pulling a key in Adobe After Effects
    • [ASSIGNMENT]
      • Upload complete cinematic sequence
  • LIVE SESSION - Project Showcase

Faculty

Matthew

Matthew Niederhauser

Artist, Photojournalist, and Educator

Matthew Niederhauser is an artist, photojournalist, and educator. He studied anthropology at Columbia University, and upon graduating, worked for eight years on documentary projects across China. Upon returning to New York, Matthew earned his MFA in Art Practice from SVA while also acting as a Visiting Artist at MIT CAST and a member of NEW INC.

At NEW INC he cofounded Sensorium, an experiential studio working at the forefront of immersive storytelling. When not focusing on projects that have premiered at Sundance New Frontier, Tribeca Storyscapes, and IDFA DocLab, he teaches at NYU ITP and Johns Hopkins' ISET. Most recently, Matthew became a Studio Fellow at ONX, a new workspace and gallery for XR art created in partnership with the Onassis Foundation and the New Museum.