Name: Justus Thies
Position: Post Doctor
Phone: +49-89-289-18456
Room No: 02.13.042


Justus Thies is working as a postdoctoral researcher at the Technical University of Munich. In September 2017 he joined the Visual Computing Lab of Prof.Dr. Matthias Nießner. Previous, he was a PhD student at the University of Erlangen-Nuremberg under the supervision of Gunther Greiner. He started his PhD studies in 2014 after receiving his Master of Science degree from the University of Erlangen-Nuremberg. During the time as a PhD student he collaborated with other institutes and did internships at Stanford University and the Max-Planck Institut Informatik. His research focuses on real-time facial performance capturing and expression transfer using commodity hardware. Thus, he is interested in Computer Vision and Computer Graphics, as well as in efficient implementations of optimization techniques, especially on graphics hardware. His publications opened up a new research field - real-time facial reenactment. The achieved quality, efficiency and the reduced hardware requirements of his developed methods raised a lot of attention in academia, industry and media. The dissertation "Face2Face: Real-time Facial Reenactment" of Justus Thies summarizes these publications and discusses the implications of the demonstrated technologies. Beside computer science, he has a strong interest in mechanical engineering and numerical controlled machines (CNC machines). His hobbies include building CNC machines, remote controlled quad-copters, planes and cars.
[Personal Webpage]



Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction
Guy Gafni, Justus Thies, Michael Zollhöfer, Matthias Nießner
CVPR 2021 (Oral)
Given a monocular portrait video sequence of a person, we reconstruct a dynamic neural radiance field representing a 4D facial avatar. The radiance field is conditioned on blendshape expressions, which allow us to then photorealistically synthesize novel head poses as well as novel facial expressions of the person. This can be used for self-reenactment, novel view synthesis and cross-subject reenactment.
[video][code][bibtex][project page]

SPSG: Self-Supervised Photometric Scene Generation from RGB-D Scans
Angela Dai, Yawar Siddiqui, Justus Thies, Julien Valentin, Matthias Nießner
CVPR 2021
We present SPSG, a novel approach to generate high-quality, colored 3D models of scenes from RGB-D scan observations by learning to infer unobserved scene geometry and color in a self-supervised fashion. Rather than relying on 3D reconstruction losses to inform our 3D geometry and color reconstruction, we propose adversarial and perceptual losses operating on 2D renderings in order to achieve high-resolution, high-quality colored reconstructions of scenes.
[video][code][bibtex][project page]

Neural Deformation Graphs for Globally-consistent Non-rigid Reconstruction
Aljaž Božič, Pablo Palafox, Michael Zollhöfer, Justus Thies, Angela Dai, Matthias Nießner
CVPR 2021 (Oral)
We introduce Neural Deformation Graphs for globally-consistent deformation tracking and 3D reconstruction of non-rigid objects. Specifically, we implicitly model a deformation graph via a deep neural network and empose per-frame viewpoint consistency as well as inter-frame graph and surface consistency constraints in a self-supervised fashion.
[video][bibtex][project page]


Neural Non Rigid Tracking
Aljaž Božič, Pablo Palafox, Michael Zollhöfer, Angela Dai, Justus Thies, Matthias Nießner
NeurIPS 2020
We introduce a novel, end-to-end learnable, differentiable non-rigid tracker that enables state-of-the-art non-rigid reconstruction. By enabling gradient back-propagation through a non-rigid as-rigid-as-possible optimization solver, we are able to learn correspondences in an end-to-end manner such that they are optimal for the task of non-rigid tracking
[video][bibtex][project page]

Egocentric Videoconferencing
Mohamed Elgharib, Mohit Mendiratta, Justus Thies, Matthias Nießner, Hans-Peter Seidel, Ayush Tewari, Vladislav Golyanik, Christian Theobalt
Siggraph Asia 2020
We introduce a method for egocentric videoconferencing that enables hands-free video calls, for instance by people wearing smart glasses or other mixed-reality devices.
[bibtex][project page]

Neural Voice Puppetry: Audio-driven Facial Reenactment
Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, Matthias Nießner
ECCV 2020
Given an audio sequence of a source person or digital assistant, we generate a photo-realistic output video of a target person that is in sync with the audio of the source input.
[video][code][bibtex][project page]

State of the Art on Neural Rendering
Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, Rohit K Pandey, Sean Fanello, Gordon Wetzstein, Jun-Yan Zhu, Christian Theobalt, Maneesh Agrawala, Eli Shechtman, Dan B Goldman, Michael Zollhöfer
EG 2020
Neural rendering is a new and rapidly emerging field that combines generative machine learning techniques with physical knowledge from computer graphics, e.g., by the integration of differentiable rendering into network training. This state-of-the-art report summarizes the recent trends and applications of neural rendering.
[bibtex][project page]

Adversarial Texture Optimization from RGB-D Scans
Jingwei Huang, Justus Thies, Angela Dai, Abhijit Kundu, Chiyu 'Max' Jiang, Leonidas Guibas, Matthias Nießner, Thomas Funkhouser
CVPR 2020
We present a novel approach for color texture generation using a conditional adversarial loss obtained from weakly-supervised views. Specifically, we propose an approach to produce photorealistic textures for approximate surfaces, even from misaligned images, by learning an objective function that is robust to these errors.
[video][bibtex][project page]

Image-guided Neural Object Rendering
Justus Thies, Michael Zollhöfer, Christian Theobalt, Marc Stamminger, Matthias Nießner
ICLR 2020
We propose a new learning-based novel view synthesis approach for scanned objects that is trained based on a set of multi-view images, where we directly train a deep neural network to synthesize a view-dependent image of an object.
[video][bibtex][project page]


FaceForensics++: Learning to Detect Manipulated Facial Images
Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, Matthias Nießner
ICCV 2019
In this paper, we examine the realism of state-of-the-art facial image manipulation methods, and how difficult it is to detect them - either automatically or by humans. In particular, we create a datasets that is focused on DeepFakes, Face2Face, FaceSwap, and Neural Textures as prominent representatives for facial manipulations.
[video][code][bibtex][project page]

Deferred Neural Rendering: Image Synthesis using Neural Textures
Justus Thies, Michael Zollhöfer, Matthias Nießner
ACM Transactions on Graphics 2019 (TOG)
We introduce Deferred Neural Rendering, a new paradigm for image synthesis that combines the traditional graphics pipeline with learnable components. Specifically, we propose Neural Textures, which are learned feature maps that are trained as part of the scene capture process. Similar to traditional textures, neural textures are stored as maps on top of 3D mesh proxies; however, the high-dimensional feature maps contain significantly more information, which can be interpreted by our new deferred neural rendering pipeline. Both neural textures and deferred neural renderer are trained end-to-end, enabling us to synthesize photo-realistic images even when the original 3D content was imperfect.
[video][bibtex][project page]

Face2Face: Real-time Face Capture and Reenactment of RGB Videos
Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, Matthias Nießner
CACM 2019 (Research Highlight)
Research highlight of the Face2Face approach featured on the cover of Communications of the ACM in January 2019. Face2Face is an approach for real-time facial reenactment of a monocular target video. The method had significant impact in the research community and far beyond; it won several wards, e.g., Siggraph ETech Best in Show Award, it was featured in countless media articles, e.g., NYT, WSJ, Spiegel, etc., and it had a massive reach on social media with millions of views. The work was arguably started bringing attention to manipulations of facial videos.
[video][bibtex][project page]

DeepVoxels: Learning Persistent 3D Feature Embeddings
Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, Michael Zollhöfer
CVPR 2019 (Oral)
In this work, we address the lack of 3D understanding of generative neural networks by introducing a persistent 3D feature embedding for view synthesis. To this end, we propose DeepVoxels, a learned representation that encodes the view-dependent appearance of a 3D object without having to explicitly model its geometry.
[video][bibtex][project page]


ForensicTransfer: Weakly-supervised Domain Adaptation for Forgery Detection
Davide Cozzolino, Justus Thies, Andreas Rössler, Christian Riess, Matthias Nießner, Luisa Verdoliva
arXiv 2018
ForensicTransfer tackles two challenges in multimedia forensics. First, we devise a learning-based forensic detector which adapts well to new domains, i.e., novel manipulation methods. Second we handle scenarios where only a handful of fake examples are available during training.
[bibtex][project page]

FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces
Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, Matthias Nießner
arXiv 2018
In this paper, we introduce FaceForensics, a large scale video dataset consisting of 1004 videos with more than 500000 frames, altered with Face2Face, that can be used for forgery detection and to train generative refinement methods.
[video][bibtex][project page]

HeadOn: Real-time Reenactment of Human Portrait Videos
Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, Matthias Nießner
ACM Transactions on Graphics 2018 (TOG)
We propose HeadOn, the first real-time source-to-target reenactment approach for complete human portrait videos that enables transfer of torso and head motion, face expression, and eye gaze. Given a short RGB-D video of the target actor, we automatically construct a personalized geometry proxy that embeds a parametric head, eye, and kinematic torso model. A novel real-time reenactment algorithm employs this proxy to photo-realistically map the captured motion from the source actor to the target actor.
[video][bibtex][project page]

FaceVR: Real-Time Facial Reenactment and Eye Gaze Control in Virtual Reality
Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, Matthias Nießner
ACM Transactions on Graphics 2018 (TOG)
We propose FaceVR, a novel image-based method that enables video teleconferencing in VR based on self-reenactment. FaceVR enables VR teleconferencing using an image-based technique that results in nearly photo-realistic outputs. The key component of FaceVR is a robust algorithm to perform real-time facial motion capture of an actor who is wearing a head-mounted display (HMD).
[video][bibtex][project page]

Deep Video Portaits
Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Nießner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, Christian Theobalt
ACM Transactions on Graphics 2018 (TOG)
Our novel approach enables photo-realistic re-animation of portrait videos using only an input video. The core of our approach is a generative neural network with a novel space-time architecture. The network takes as input synthetic renderings of a parametric face model, based on which it predicts photo-realistic video frames for a given target actor.
[video][bibtex][project page]

InverseFaceNet: Deep Monocular Inverse Face Rendering
Hyeongwoo Kim, Michael Zollhöfer, Ayush Tewari, Justus Thies, Christian Richardt, Christian Theobalt
CVPR 2018
We introduce InverseFaceNet, a deep convolutional inverse rendering framework for faces that jointly estimates facial pose, shape, expression, reflectance and illumination from a single input image. By estimating all parameters from just a single image, advanced editing possibilities on a single face image, such as appearance editing and relighting, become feasible in real time.
[video][bibtex][project page]

State of the Art on Monocular 3D Face Reconstruction, Tracking, and Applications
Michael Zollhöfer, Justus Thies, Derek Bradley, Pablo Garrido, Thabo Beeler, Patrick Pérez, Marc Stamminger, Matthias Nießner, Christian Theobalt
Eurographics 2018
This state-of-the-art report summarizes recent trends in monocular facial performance capture and discusses its applications, which range from performance-based animation to real-time facial reenactment. We focus our discussion on methods where the central task is to recover and track a three dimensional model of the human face using optimization-based reconstruction algorithms.
[bibtex][project page]


FaceForge: Markerless Non-Rigid Face Multi-Projection Mapping
Christian Siegl, Vanessa Lange, Marc Stamminger, Frank Bauer, Justus Thies
ISMAR 2017
In this paper, we introduce FaceForge, a multi-projection mapping system that is able to alter the appearance of a non-rigidly moving human face in real time.
[bibtex][project page]


Face2Face: Real-time Face Capture and Reenactment of RGB Videos
Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, Matthias Nießner
CVPR 2016 (Oral)
We present a novel approach for real-time facial reenactment of a monocular target video sequence (e.g., Youtube video). The source sequence is also a monocular video stream, captured live with a commodity webcam. Our goal is to animate the facial expressions of the target video by a source actor and re-render the manipulated output video in a photo-realistic fashion.
[video][bibtex][supplemental][project page]


Real-time Expression Transfer for Facial Reenactment
Justus Thies, Michael Zollhöfer, Matthias Nießner, Levi Valgaerts, Marc Stamminger, Christian Theobalt
ACM Transactions on Graphics 2015 (TOG)
We present a method for the real-time transfer of facial expressions from an actor in a source video to an actor in a target video, thus enabling the ad-hoc control of the facial expressions of the target actor.
[video][bibtex][project page]

Real-Time Pixel Luminance Optimization for Dynamic Multi-Projection Mapping
Christian Siegl, Matteo Colaianni, Lucas Thies, Justus Thies, Michael Zollhöfer, Shahram Izadi, Marc Stamminger, Frank Bauer
ACM Transactions on Graphics 2015 (TOG)
Using projection mapping enables us to bring virtual worlds into shared physical spaces. In this paper, we present a novel, adaptable and real-time projection mapping system, which supports multiple projectors and high quality rendering of dynamic content on surfaces of complex geometrical shape. Our system allows for smooth blending across multiple projectors using a new optimization framework that simulates the diffuse direct light transport of the physical world to continuously adapt the color output of each projector pixel.
[video][bibtex][project page]