Instructor(s):

Gergely Vass
Weeks
1-14
Contact hours
4 hours every week
Credit
4 credits

Short Description of the Course:
The course focuses on current issues in digital post-production and the corresponding solutions utilizing computer vision algorithms. Most of the challenges in focus are related to merging real and computer generated images.

Aim of the Course:
The primary aim of the course is to educate and inspire at the same time. By understanding how essential mathematical techniques – such as solving a set of linear equations – are utilized in the high-end filmmaking process, one soon develops a new appreciation for software research and development. The “true”, practical value of mathematics and computer science introduced in this and other courses is revealed through numerous exciting examples from real projects and innovative software solutions.

Prerequisites:
A knowledge of linear algebra foundations and programming experience (in any language such as C, C++, objective C, Java, Pascal) are required.

In addition students should have a passion for movies and science.

Detailed Program and Class Schedule:
The first studio visit will be a guided tour in the Colorfront facilities, introducing the digital workflows and custom softwares used in the studio. Students will see the most advanced digital projection rooms with advanced color grading equipment, high resolution display devices, sound mixing boards and digital cinema cameras, to name a few things. Not only the full digital post-production pipeline will be introduced, but also the adventurous story of developing the first software color corrector for which the team won the Oscar award for Scientific and Technical Achievement. 

The academic material begins with image processing basics. The computational costs, advantages and limitations, and application of different classes of image filters will be discussed. Students will learn how to smooth and sharpen images, and how to compute the image gradients. Based on these rather simple algorithms robust feature detectors can be built, that identify interest points in images. Most high-level computer vision tools are based on these interest points, which encode all necessary information from the original image.

Students will receive the skeleton code for the programming assignments as well. Complete C++ classes are implemented with many useful helper functions, making the development process very similar to what Colorfront developers do on a daily basis. Students customize the code and implement image processing functions to get familiar with the concepts learnt in class.

To lay the mathematical foundations needed for the class one session is dedicated to an overview of linear algebra foundations from a software developer’s point of view: a “practical guide” to analyzing and solving large sets of linear equations. The majority of parameter estimation problems in computer vision are based on such linear constructions. Examples include aligning multiple overlapping photographs to form a single panoramic image, or tracking a moving object in a sequence of frames.

While not essential to post-production, object and class recognition is a very important field of computer vision. Understanding the various approaches also helps students get familiar with general concepts such as distance metrics, principal component analysis and linear subspaces. By focusing on real problems and their solutions – instead of the often overwhelming theory – students will grasp the power of such mathematical tools. 

During the second part of the course feature tracking and parameter estimation are the main topic. Students will understand and implement feature detection and tracking methods, from which basic motion information is reconstructed. This is the main idea behind the most powerful post-production tool: camera tracking, or matchmoving. This process is matching the motion of a virtual camera to the real camera used to capture the image sequence, which allows artists to seamlessly insert computer graphics into live-action footage with correct position, scale and orientation. While this very complex problem may seem unique to computer vision, the challenges are common in computer science and it is worth looking at the innovative solutions used in the field. 

To compare the implementation of feature tracking in the homework assignments and a real software product there is a last studio visit focusing on the application of the theory presented throughout the course. The deep walk-though of the stereoscopic feature set of the Emmy Award winning product will put the course’s material in perspective, and students will understand how these techniques are utilized to achieve certain artistic goals in modern S3D movies.

Method of Instruction:
The course is lecture-oriented with assigned readings and team-oriented programming and image manipulation projects as homework. There will be two visits to the Colorfront studio, one at the beginning and one at the end of the course.

Assessment:
Students will be evaluated based on assigned homework and final written exam.

Textbooks and resources:
Richard Hartley, Andrew Zisserman: Multiple View Geometry in Computer Vision, Cambridge Univ. Press, 2003. 

Richard Szeliski: Computer Vision: Algorithms and Applications, Springer, 2010.
http://research.microsoft.com/en-us/um/people/szeliski/Book/

Instructors' bio:

Gergely Vass (born 1978) is a researcher and developer at Colorfront Ltd., one of Europe's leading digital film post-production facilities. He got involved with 3D computer graphics at the age of 14 and soon became a 3D animator and instructor. As the technical director at the post production studio of the Hungarian Film Laboratories, he worked on the visual effects of several feature films and commercials. Upon receiving his M.Sc. at the Faculty of Electrical Engineering and Informatics of Budapest University of Technology and Economics, his interest shifted towards research and development in image processing and computer vision. As a member of Colorfront Ltd. from 2000, he got involved with motion tracking techniques. Following the acquisition of Colorfont technologies by the leading 3D software company Autodesk, Vass joined Autodesk’s Image Science Team in Montreal, Canada. He developed various tools and algorithms for the world-renowned high-end visual effects product line of Autodesk, including camera tracking, shape tracking, image warping and video stabilizing. Vass is a regular contributor to Computer Graphics World magazine.