2020
| Neural Non Rigid Tracking |
|---|
| Aljaž Božič, Pablo Palafox, Michael Zollhöfer, Angela Dai, Justus Thies, Matthias Nießner |
| NeurIPS 2020 |
| We introduce a novel, end-to-end learnable, differentiable non-rigid tracker that enables state-of-the-art non-rigid reconstruction. By enabling gradient back-propagation through a non-rigid as-rigid-as-possible optimization solver, we are able to learn correspondences in an end-to-end manner such that they are optimal for the task of non-rigid tracking |
| [paper][video][bibtex][project page] |
| Egocentric Videoconferencing |
|---|
| Mohamed Elgharib, Mohit Mendiratta, Justus Thies, Matthias Nießner, Hans-Peter Seidel, Ayush Tewari, Vladislav Golyanik, Christian Theobalt |
| Siggraph Asia 2020 |
| We introduce a method for egocentric videoconferencing that enables hands-free video calls, for instance by people wearing smart glasses or other mixed-reality devices. |
| [bibtex][project page] |
| Modeling 3D Shapes by Reinforcement Learning |
|---|
| Cheng Lin, Tingxiang Fan, Wenping Wang, Matthias Nießner |
| ECCV 2020 |
| We explore how to enable machines to model 3D shapes like human modelers using deep reinforcement learning (RL). |
| [paper][video][code][bibtex][project page] |
| SceneCAD: Predicting Object Alignments and Layouts in RGB-D Scans |
|---|
| Armen Avetisyan, Tatiana Khanova, Christopher Choy, Denver Dash, Angela Dai, Matthias Nießner |
| ECCV 2020 |
| We present a novel approach to reconstructing lightweight, CAD-based representations of scanned 3D environments from commodity RGB-D sensors. |
| [paper][code][bibtex][project page] |
| CAD-Deform: Deformable Fitting of CAD Models to 3D Scans |
|---|
| Vladislav Ishimtsev, Alexey Bokhovkin, Alexey Artemov, Savva Ignatiev, Matthias Nießner, Denis Zorin, Evgeny Burnaev |
| ECCV 2020 |
| We propose CAD-Deform, a method which obtains more accurate CAD-to-scan fits by non-rigidly deforming retrieved CAD models. |
| [paper][code][bibtex][project page] |
| ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language |
|---|
| Dave Zhenyu Chen, Angel X. Chang, Matthias Nießner |
| ECCV 2020 |
| We propose ScanRefer, a method that learns a fused descriptor from 3D object proposals and encoded sentence embeddings, to address the newly introduced task of 3D object localization in RGB-D scans using natural language descriptions. Along with the method we release a large-scale dataset of 51,583 descriptions of 11,046 objects from 800 ScanNet scenes. |
| [paper][video][code][bibtex][project page] |
| Neural Voice Puppetry: Audio-driven Facial Reenactment |
|---|
| Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, Matthias Nießner |
| ECCV 2020 |
| Given an audio sequence of a source person or digital assistant, we generate a photo-realistic output video of a target person that is in sync with the audio of the source input. |
| [paper][video][code][bibtex][project page] |
| State of the Art on Neural Rendering |
|---|
| Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, Rohit K Pandey, Sean Fanello, Gordon Wetzstein, Jun-Yan Zhu, Christian Theobalt, Maneesh Agrawala, Eli Shechtman, Dan B Goldman, Michael Zollhöfer |
| EG 2020 |
| Neural rendering is a new and rapidly emerging field that combines generative machine learning techniques with physical knowledge from computer graphics, e.g., by the integration of differentiable rendering into network training. This state-of-the-art report summarizes the recent trends and applications of neural rendering. |
| [paper][bibtex][project page] |
| Learning to Optimize Non-Rigid Tracking |
|---|
| Yang Li, Aljaž Božič, Tianwei Zhang, Yanli Ji, Tatsuya Harada, Matthias Nießner |
| CVPR 2020 (Oral) |
| We learn the tracking of non-rigid objects by differentiating through the underlying non-rigid solver. Specifically, we propose ConditionNet which learns to generate a problem-specific preconditioner using a large number of training samples from the Gauss-Newton update equation. The learned preconditioner increases PCG’s convergence speed by a significant margin. |
| [paper][bibtex][project page] |
| 3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation |
|---|
| Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, Matthias Nießner |
| CVPR 2020 |
| We present 3D-MPA, a method for instance segmentation on 3D point clouds. We show that grouping proposals improves over NMS and outperforms previous state-of-the-art methods on the tasks of 3D object detection and semantic instance segmentation on the ScanNetV2 benchmark and the S3DIS dataset. |
| [paper][video][bibtex][project page] |
| DeepDeform: Learning Non-rigid RGB-D Reconstruction with Semi-supervised Data |
|---|
| Aljaž Božič, Michael Zollhöfer, Christian Theobalt, Matthias Nießner |
| CVPR 2020 |
| We present a large dataset of 400 scenes, over 390,000 RGB-D frames, and 5,533 densely aligned frame pairs, and introduce a data-driven non-rigid RGB-D reconstruction approach using learned heatmap correspondences, achieving state-of-the-art reconstruction results on a newly established quantitative benchmark. |
| [paper][video][code][bibtex][project page] |
| Local Implicit Grid Representations for 3D Scenes |
|---|
| Chiyu 'Max' Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, Thomas Funkhouser |
| CVPR 2020 |
| We learned implicit representations for large 3D environments anchored in regular grids, which facilitates high-quality surface reconstruction from unstructured input point clouds. |
| [paper][video][code][bibtex][project page] |
| Adversarial Texture Optimization from RGB-D Scans |
|---|
| Jingwei Huang, Justus Thies, Angela Dai, Abhijit Kundu, Chiyu 'Max' Jiang, Leonidas Guibas, Matthias Nießner, Thomas Funkhouser |
| CVPR 2020 |
| We present a novel approach for color texture generation using a conditional adversarial loss obtained from weakly-supervised views. Specifically, we propose an approach to produce photorealistic textures for approximate surfaces, even from misaligned images, by learning an objective function that is robust to these errors. |
| [paper][video][bibtex][project page] |
| ViewAL: Active Learning with Viewpoint Entropy for Semantic Segmentation |
|---|
| Yawar Siddiqui, Julien Valentin, Matthias Nießner |
| CVPR 2020 |
| We propose ViewAL, a novel active learning strategy for semantic segmentation that exploits viewpoint consistency in multi-view datasets. |
| [paper][video][code][bibtex][project page] |
| SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans |
|---|
| Angela Dai, Christian Diller, Matthias Nießner |
| CVPR 2020 |
| We present a novel approach that converts partial and noisy RGB-D scans into high-quality 3D scene reconstructions by inferring unobserved scene geometry. Our approach is fully self-supervised and can hence be trained solely on real-world, incomplete scans. |
| [paper][video][code][bibtex][project page] |
| RevealNet: Seeing Behind Objects in RGB-D Scans |
|---|
| Ji Hou, Angela Dai, Matthias Nießner |
| CVPR 2020 |
| This paper introduces the task of semantic instance completion: from an incomplete, RGB-D scan of a scene, we detect the individual object instances comprising the scene and jointly infer their complete object geometry. |
| [paper][bibtex][project page] |
| Image-guided Neural Object Rendering |
|---|
| Justus Thies, Michael Zollhöfer, Christian Theobalt, Marc Stamminger, Matthias Nießner |
| ICLR 2020 |
| We propose a new learning-based novel view synthesis approach for scanned objects that is trained based on a set of multi-view images, where we directly train a deep neural network to synthesize a view-dependent image of an object. |
| [paper][video][bibtex][project page] |
2019
| RIO: 3D Object Instance Re-Localization in Changing Indoor Environments |
|---|
| Johanna Wald, Armen Avetisyan, Nassir Navab, Federico Tombari, Matthias Nießner |
| ICCV 2019 (Oral) |
| In this work, we explore the task of 3D object instance re-localization (RIO): given one or multiple objects in an RGB-D scan, we want to estimate their corresponding 6DoF poses in another 3D scan of the same environment taken at a later point in time. |
| [paper][video][bibtex][project page] |
| FaceForensics++: Learning to Detect Manipulated Facial Images |
|---|
| Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, Matthias Nießner |
| ICCV 2019 |
| In this paper, we examine the realism of state-of-the-art facial image manipulation methods, and how difficult it is to detect them - either automatically or by humans. In particular, we create a datasets that is focused on DeepFakes, Face2Face, FaceSwap, and Neural Textures as prominent representatives for facial manipulations. |
| [paper][video][code][bibtex][project page] |
| End-to-End CAD Model Retrieval and 9DoF Alignment in 3D Scans |
|---|
| Armen Avetisyan, Angela Dai, Matthias Nießner |
| ICCV 2019 |
| We present a novel, end-to-end approach to align CAD models to an 3D scan of a scene, enabling transformation of a noisy, incomplete 3D scan to a compact, CAD reconstruction with clean, complete object geometry. |
| [paper][bibtex][project page] |
| Joint Embedding of 3D Scan and CAD Objects |
|---|
| Manuel Dahnert, Angela Dai, Leonidas Guibas, Matthias Nießner |
| ICCV 2019 |
| In this paper, we address the problem of cross-domain retrieval between partial, incomplete 3D scan objects and complete CAD models. To this end, we learn a joint embedding where semantically similar objects from both domains lie close together regardless of low-level differences, such as clutter or noise. To enable fine-grained evaluation of scan-CAD model retrieval we additionally present a new dataset of scan-CAD object similarity annotations. |
| [paper][video][bibtex][project page] |
| DDSL: Deep Differentiable Simplex Layer for Learning Geometric Signals |
|---|
| Chiyu 'Max' Jiang, Dana Lansigan, Philip Marcus, Matthias Nießner |
| ICCV 2019 |
| We present a Deep Differentiable Simplex Layer (DDSL) for neural networks for geometric deep learning. The DDSLis a differentiable layer compatible with deep neural net-works for bridging simplex mesh-based geometry represen-tations (point clouds, line mesh, triangular mesh, tetrahe-dral mesh) with raster images (e.g., 2D/3D grids). |
| [bibtex][project page] |
| Deferred Neural Rendering: Image Synthesis using Neural Textures |
|---|
| Justus Thies, Michael Zollhöfer, Matthias Nießner |
| ACM Transactions on Graphics 2019 (TOG) |
| We introduce Deferred Neural Rendering, a new paradigm for image synthesis that combines the traditional graphics pipeline with learnable components. Specifically, we propose Neural Textures, which are learned feature maps that are trained as part of the scene capture process. Similar to traditional textures, neural textures are stored as maps on top of 3D mesh proxies; however, the high-dimensional feature maps contain significantly more information, which can be interpreted by our new deferred neural rendering pipeline. Both neural textures and deferred neural renderer are trained end-to-end, enabling us to synthesize photo-realistic images even when the original 3D content was imperfect. |
| [paper][video][bibtex][project page] |
| Multi-Robot Collaborative Dense Scene Reconstruction |
|---|
| Siyan Dong, Kai Xu, Qiang Zhou, Andrea Tagliasacchi, Shiqing Xin, Matthias Nießner, Baoquan Chen |
| ACM Transactions on Graphics 2019 (TOG) |
| We present an autonomous scanning approach which allows multiple robots to perform collaborative scanning for dense 3D reconstruction of unknown indoor scenes. Our method plans scanning paths for several robots, allowing them to efficiently coordinate with each other such that the collective scanning coverage and reconstruction quality is maximized while the overall scanning effort is minimized. |
| [video][bibtex][project page] |
| Face2Face: Real-time Face Capture and Reenactment of RGB Videos |
|---|
| Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, Matthias Nießner |
| CACM 2019 (Research Highlight) |
| Research highlight of the Face2Face approach featured on the cover of Communications of the ACM in January 2019. Face2Face is an approach for real-time facial reenactment of a monocular target video. The method had significant impact in the research community and far beyond; it won several wards, e.g., Siggraph ETech Best in Show Award, it was featured in countless media articles, e.g., NYT, WSJ, Spiegel, etc., and it had a massive reach on social media with millions of views. The work was arguably started bringing attention to manipulations of facial videos. |
| [paper][video][bibtex][project page] |
| Inverse Path Tracing for Joint Material and Lighting Estimation |
|---|
| Dejan Azinović, Tzu-Mao Li, Anton Kaplanyan, Matthias Nießner |
| CVPR 2019 (Oral) |
| We introduce Inverse Path Tracing, a novel approach to jointly estimate the material properties of objects and light sources in indoor scenes by using an invertible light transport simulation. |
| [paper][video][bibtex][project page] |
| 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans |
|---|
| Ji Hou, Angela Dai, Matthias Nießner |
| CVPR 2019 (Oral) |
| We introduce 3D-SIS, a novel neural network architecture for 3D semantic instance segmentation in commodity RGB-D scans. |
| [paper][video][code][bibtex][project page] |
| Scan2CAD: Learning CAD Model Alignment in RGB-D Scans |
|---|
| Armen Avetisyan, Manuel Dahnert, Angela Dai, Angel X. Chang, Manolis Savva, Matthias Nießner |
| CVPR 2019 (Oral) |
| We present Scan2CAD, a novel data-driven method that learns to align clean 3D CAD models from a shape database to the noisy and incomplete geometry of a commodity RGB-D scan. |
| [paper][video][code][bibtex][project page] |
| Scan2Mesh: From Unstructured Range Scans to 3D Meshes |
|---|
| Angela Dai, Matthias Nießner |
| CVPR 2019 |
| We introduce Scan2Mesh, a novel data-driven approach which introduces a generative neural network architecture for creating 3D meshes as indexed face sets, conditioned on an input partial scan. |
| [paper][bibtex][project page] |
| DeepVoxels: Learning Persistent 3D Feature Embeddings |
|---|
| Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, Michael Zollhöfer |
| CVPR 2019 (Oral) |
| In this work, we address the lack of 3D understanding of generative neural networks by introducing a persistent 3D feature embedding for view synthesis. To this end, we propose DeepVoxels, a learned representation that encodes the view-dependent appearance of a 3D object without having to explicitly model its geometry. |
| [paper][video][bibtex][project page] |
| TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes |
|---|
| Jingwei Huang, Haotian Zhang, Li Yi, Thomas Funkhouser, Matthias Nießner, Leonidas Guibas |
| CVPR 2019 (Oral) |
| We introduce, TextureNet, a neural network architecture designed to extract features from high-resolution signals associated with 3D surface meshes (e.g., color texture maps). The key idea is to utilize a 4-rotational symmetric (4-RoSy) field to define a domain for convolution on a surface. |
| [paper][bibtex][project page] |
| Spherical CNNs on Unstructured Grids |
|---|
| Chiyu 'Max' Jiang, Jingwei Huang, Karthik Kashinath, Prabhat, Philip Marcus, Matthias Nießner |
| ICLR 2019 |
| We present an efficient convolution kernel for Convolutional Neural Networks (CNNs) on unstructured grids using parameterized differential operators while focusing on spherical signals such as panorama images or planetary signals. To this end, we replace conventional convolution kernels with linear combinations of differential operators that are weighted by learnable parameters. |
| [paper][code][bibtex][project page] |
| Convolutional Neural Networks on non-uniform geometrical signals using Euclidean spectral transformation |
|---|
| Chiyu 'Max' Jiang, Dequan Wang, Jingwei Huang, Philip Marcus, Matthias Nießner |
| ICLR 2019 |
| We develop mathematical formulations for Non-Uniform Fourier Transforms (NUFT) to directly, and optimally, sample nonuniform data signals of different topologies defined on a simplex mesh into the spectral domain with no spatial sampling error. The spectral transform is performed in the Euclidean space, which removes the translation ambiguity from works on the graph spectrum. |
| [paper][bibtex][project page] |
2018
| RegNet: Learning the Optimization of Direct Image-to-Image Pose Registration |
|---|
| Lei Han, Mengqi Ji, Lu Fang, Matthias Nießner |
| arXiv 2018 |
| In this paper, we demonstrate that the inaccurate numerical Jacobian limits the convergence range which could be improved greatly using learned approaches. Based on this observation, we propose a novel end-to-end network, RegNet, to learn the optimization of image-to-image pose registration. By jointly learning feature representation for each pixel and partial derivatives that replace handcrafted ones (e.g., numerical differentiation) in the optimization step, the neural network facilitates end-to-end optimization. |
| [paper][bibtex][project page] |
| ForensicTransfer: Weakly-supervised Domain Adaptation for Forgery Detection |
|---|
| Davide Cozzolino, Justus Thies, Andreas Rössler, Christian Riess, Matthias Nießner, Luisa Verdoliva |
| arXiv 2018 |
| ForensicTransfer tackles two challenges in multimedia forensics. First, we devise a learning-based forensic detector which adapts well to new domains, i.e., novel manipulation methods. Second we handle scenarios where only a handful of fake examples are available during training. |
| [paper][bibtex][project page] |
| FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces |
|---|
| Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, Matthias Nießner |
| arXiv 2018 |
| In this paper, we introduce FaceForensics, a large scale video dataset consisting of 1004 videos with more than 500000 frames, altered with Face2Face, that can be used for forgery detection and to train generative refinement methods. |
| [paper][video][bibtex][project page] |
| Calipso: Physics-based Image and Video Editing through CAD Model Proxies |
|---|
| Nazim Haouchine, Frederick Roy, Hadrien Courtecuisse, Matthias Nießner, Stephane Cotin |
| Visual Computer 2018 |
| We present Calipso, an interactive method for editing images and videos in a physically-coherent manner. Our main idea is to realize physics-based manipulations by running a full physics simulation on proxy geometries given by non-rigidly aligned CAD models. |
| [paper][video][bibtex][project page] |
| Plan3D: Viewpoint and Trajectory Optimization for Aerial Multi-View Stereo Reconstruction |
|---|
| Benjamin Hepp, Matthias Nießner, Otmar Hilliges |
| ACM Transactions on Graphics 2018 (TOG) |
| We introduce a new method that efficiently computes a set of viewpoints and trajectories for high-quality 3D reconstructions in outdoor environments. Our goal is to automatically explore an unknown area, and obtain a complete 3D scan of a region of interest (e.g., a large building). |
| [paper][bibtex][project page] |
| Parsing Geometry Using Structure-Aware Shape Templates |
|---|
| Vignesh Ganapathi-Subramanian, Olga Diamanti, Soeren Pirk, Chengcheng Tang, Matthias Nießner, Leonidas Guibas |
| 3DV 2018 |
| In this paper, we organize large shape collections into parameterized shape templates to capture the underlying structure of the objects. The templates allow us to transfer the structural information onto new objects and incomplete scans. |
| [paper][video][bibtex][project page] |
| QuadriFlow: A Scalable and Robust Method for Quadrangulation |
|---|
| Jingwei Huang, Yichao Zhou, Matthias Nießner, Jonathan Richard Shewchuk, Leonidas Guibas |
| SGP 2018 (Best Paper Award) |
| QuadriFlow is a scalable algorithm for generating quadrilateral surface meshes based on the Instant Field-Aligned Meshes of Jakob et al.. We modify the original algorithm such that it efficiently produces meshes with many fewer singularities. Singularities in quadrilateral meshes cause problems for many applications, includ- ing parametrization and rendering with Catmull-Clark subdivision surfaces. |
| [paper][code][bibtex][supplemental][project page] |
| 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation |
|---|
| Angela Dai, Matthias Nießner |
| ECCV 2018 |
| We present 3DMV, a novel method for 3D semantic scene segmentation of RGB-D scans in indoor environments using a joint 3D-multi-view prediction network. In contrast to existing methods that either use geometry or RGB data as input for this task, we combine both data modalities in a joint, end-to-end network architecture. |
| [paper][code][bibtex][project page] |
| PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction |
|---|
| Yifei Shi, Kai Xu, Matthias Nießner, Szymon Rusinkiewicz, Thomas Funkhouser |
| ECCV 2018 (Oral) |
| We introduce a novel RGB-D patch descriptor designed for detecting coplanar surfaces in SLAM reconstruction. The core of our method is a deep convolutional neural net that takes in RGB, depth, and normal information of a planar patch in an image and outputs a descriptor that can be used to find coplanar patches from other images. |
| [paper][bibtex][project page] |
| HeadOn: Real-time Reenactment of Human Portrait Videos |
|---|
| Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, Matthias Nießner |
| ACM Transactions on Graphics 2018 (TOG) |
| We propose HeadOn, the first real-time source-to-target reenactment approach for complete human portrait videos that enables transfer of torso and head motion, face expression, and eye gaze. Given a short RGB-D video of the target actor, we automatically construct a personalized geometry proxy that embeds a parametric head, eye, and kinematic torso model. A novel real-time reenactment algorithm employs this proxy to photo-realistically map the captured motion from the source actor to the target actor. |
| [paper][video][bibtex][project page] |
| FaceVR: Real-Time Facial Reenactment and Eye Gaze Control in Virtual Reality |
|---|
| Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, Matthias Nießner |
| ACM Transactions on Graphics 2018 (TOG) |
| We propose FaceVR, a novel image-based method that enables video teleconferencing in VR based on self-reenactment. FaceVR enables VR teleconferencing using an image-based technique that results in nearly photo-realistic outputs. The key component of FaceVR is a robust algorithm to perform real-time facial motion capture of an actor who is wearing a head-mounted display (HMD). |
| [paper][video][bibtex][project page] |
| Deep Video Portaits |
|---|
| Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Nießner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, Christian Theobalt |
| ACM Transactions on Graphics 2018 (TOG) |
| Our novel approach enables photo-realistic re-animation of portrait videos using only an input video. The core of our approach is a generative neural network with a novel space-time architecture. The network takes as input synthetic renderings of a parametric face model, based on which it predicts photo-realistic video frames for a given target actor. |
| [paper][video][bibtex][project page] |
| InverseFaceNet: Deep Monocular Inverse Face Rendering |
|---|
| Hyeongwoo Kim, Michael Zollhöfer, Ayush Tewari, Justus Thies, Christian Richardt, Christian Theobalt |
| CVPR 2018 |
| We introduce InverseFaceNet, a deep convolutional inverse rendering framework for faces that jointly estimates facial pose, shape, expression, reflectance and illumination from a single input image. By estimating all parameters from just a single image, advanced editing possibilities on a single face image, such as appearance editing and relighting, become feasible in real time. |
| [paper][video][bibtex][project page] |
| ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans |
|---|
| Angela Dai, Daniel Ritchie, Martin Bokeloh, Scott Reed, Jürgen Sturm, Matthias Nießner |
| CVPR 2018 |
| We introduce ScanComplete, a novel data-driven approach for taking an incomplete 3D scan of a scene as input and predicting a complete 3D model along with per-voxel semantic labels. The key contribution of our method is its ability to handle large scenes with varying spatial extent, managing the cubic growth in data size as scene size increases. |
| [paper][video][code][bibtex][project page] |
| State of the Art on Monocular 3D Face Reconstruction, Tracking, and Applications |
|---|
| Michael Zollhöfer, Justus Thies, Derek Bradley, Pablo Garrido, Thabo Beeler, Patrick Pérez, Marc Stamminger, Matthias Nießner, Christian Theobalt |
| Eurographics 2018 |
| This state-of-the-art report summarizes recent trends in monocular facial performance capture and discusses its applications, which range from performance-based animation to real-time facial reenactment. We focus our discussion on methods where the central task is to recover and track a three dimensional model of the human face using optimization-based reconstruction algorithms. |
| [bibtex][project page] |
| State of the Art on 3D Reconstruction with RGB-D Cameras |
|---|
| Michael Zollhöfer, Patrick Stotko, Andreas Görlitz, Christian Theobalt, Matthias Nießner, Reinhard Klein, Andreas Kolb |
| Eurographics 2018 |
| In this state-of-the-art report, we analyze these recent developments in RGB-D scene reconstruction in detail and review essential related work. We explain, compare, and critically analyze the common underlying algorithmic concepts that enabled these recent advancements. Furthermore, we show how algorithms are designed to best exploit the benefits of RGB-D data while suppressing their often non-trivial datadistortions. |
| [bibtex][project page] |
2017
| FaceForge: Markerless Non-Rigid Face Multi-Projection Mapping |
|---|
| Christian Siegl, Vanessa Lange, Marc Stamminger, Frank Bauer, Justus Thies |
| ISMAR 2017 |
| In this paper, we introduce FaceForge, a multi-projection mapping system that is able to alter the appearance of a non-rigidly moving human face in real time. |
| [paper][bibtex][project page] |
| Matterport3D: Learning from RGB-D Data in Indoor Environments |
|---|
| Angel X. Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Nießner, Manolis Savva, Shuran Song, Andy Zeng, Yinda Zhang |
| 3DV 2017 |
| In this paper, we introduce Matterport3D, a large-scale RGB-D dataset containing 10,800 panoramic views from 194,400 RGB-D images of 90 building-scale scenes. |
| [paper][bibtex][project page] |
| Multiframe Scene Flow with Piecewise Rigid Motion |
|---|
| Vladislav Golyanik, Kihwan Kim, Robert Maier, Matthias Nießner, Didier Stricker, Jan Kautz |
| 3DV 2017 |
| We introduce a novel multi-frame scene flow approach that jointly optimizes the consistency of the patch appearances and their local rigid motions from RGB-D image sequences. |
| [paper][bibtex][project page] |
| 3DLite: Towards Commodity 3D Scanning for Content Creation |
|---|
| Jingwei Huang, Angela Dai, Leonidas Guibas, Matthias Nießner |
| ACM Transactions on Graphics 2017 (TOG) |
| We present 3DLite, a novel approach to reconstruct 3D environments using consumer RGB-D sensors, making a step towards directly utilizing captured 3D content in graphics applications, such as video games, VR, or AR. Rather than reconstructing an accurate one-to-one representation of the real world, our method computes a lightweight, low-polygonal geometric abstraction of the scanned geometry. |
| [paper][video][bibtex][supplemental][project page] |
| Autonomous Reconstruction of Unknown Indoor Scenes Guided by Time-varying Tensor Fields |
|---|
| Kai Xu, Lintao Zheng, Zihao Yan, Guohang Yan, Eugene Zhang, Matthias Nießner, Oliver Deussen, Daniel Cohen-Or, Hui Huang |
| ACM Transactions on Graphics 2017 (TOG) |
| We present a navigation-by-reconstruction approach to address this question where moving paths of the robot are planned to account for both global efficiency for fast exploration and local smoothness to obtain high-quality scans. |
| [paper][video][code][bibtex][project page] |
| Opt: A Domain Specific Language for Non-linear Least Squares Optimization in Graphics and Imaging |
|---|
| Zachary DeVito, Michael Mara, Michael Zollhöfer, Gilbert Bernstein, Jonathan Ragan-Kelley, Christian Theobalt, Pat Hanrahan, Matthew Fisher, Matthias Nießner |
| ACM Transactions on Graphics 2017 (TOG) |
| We propose a new language, Opt, in which a user simply writes energy functions over image- or graph-structured unknowns, and a compiler automatically generates state-of-the-art GPU optimization kernels. |
| [paper][bibtex][project page] |
| BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integration |
|---|
| Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, Christian Theobalt |
| ACM Transactions on Graphics 2017 (TOG) |
| We introduce a novel, real-time, end-to-end 3D reconstruction framework, with a robust pose optimization strategy based on sparse feature matches and dense geometric and photometric alignment. One main contribution is the ability to update the reconstructed model on-the-fly as new (global) pose optimization results become available. |
| [paper][video][bibtex][project page] |
| Intrinsic3D: High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting |
|---|
| Robert Maier, Kihwan Kim, Daniel Cremers, Jan Kautz, Matthias Nießner |
| ICCV 2017 |
| We introduce a novel method to obtain high-quality 3D reconstructions from consumer RGB-D sensors. Our core idea is to simultaneously optimize for geometry encoded in a signed distance field (SDF), textures from automaticallyselected keyframes, and their camera poses along with material and scene lighting. |
| [paper][code][bibtex][supplemental][project page] |
| A Lightweight Approach for On-the-Fly Reflectance Estimation |
|---|
| Kihwan Kim, Jinwei Gu, Stephen Tyree, Pavlo Molchanov, Matthias Nießner, Jan Kautz |
| ICCV 2017 (Oral) |
| We propose a lightweight, learning-based approach for surface reflectance estimation directly from 8-bit RGB images in real-time, which can be easily plugged into any 3D scanning-and-fusion system with a commodity RGBD sensor. |
| [paper][bibtex][project page] |
| ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes |
|---|
| Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner |
| CVPR 2017 (Spotlight) |
| We introduce ScanNet, an RGB-D video dataset containing 2.5M views in 1513 scenes annotated with 3D camera poses, surface reconstructions, and semantic segmentations. |
| [paper][video][bibtex][project page] |
| Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis |
|---|
| Angela Dai, Charles Ruizhongtai Qi, Matthias Nießner |
| CVPR 2017 (Spotlight) |
| We introduce a data-driven approach to complete partial 3D shapes through a combination of volumetric deep neural networks and 3D shape synthesis. From a partially-scanned input shape, our method first infers a low-resolution - but complete - output. |
| [paper][bibtex][project page] |
| 3DMatch: Learning the Matching of Local 3D Geometry in Range Scans |
|---|
| Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao |
| CVPR 2017 (Oral) |
| In this paper, we introduce 3DMatch, a data-driven local feature learner that jointly learns a geometric feature representation and an associated metric function from a large collection of real-world scanning data. |
| [paper][video][bibtex][project page] |
2016
| Learning to Navigate the Energy Landscape |
|---|
| Julien Valentin, Angela Dai, Matthias Nießner, Pushmeet Kohli, Philip H. S. Torr, Shahram Izadi, Cem Keskin |
| 3DV 2016 |
| In this paper, we present a novel, general, and efficient architecture for addressing computer vision problems that are approached from an `Analysis by Synthesis' standpoint. |
| [paper][video][bibtex][project page] |
| VolumeDeform: Real-time Volumetric Non-rigid Reconstruction |
|---|
| Matthias Innmann, Michael Zollhöfer, Matthias Nießner, Christian Theobalt, Marc Stamminger |
| ECCV 2016 |
| We present a novel approach for the reconstruction of dynamic geometric shapes using a single hand-held consumer-grade RGB-D sensor at real-time rates. Our method does not require a pre-defined shape template to start with and builds up the scene model from scratch during the scanning process. |
| [paper][video][bibtex][supplemental][project page] |
| Efficient GPU Rendering of Subdivision Surfaces using Adaptive Quadtrees |
|---|
| Wade Brainerd, Tim Foley, Manuel Kraemer, Henry Moreton, Matthias Nießner |
| ACM Transactions on Graphics 2016 (TOG) |
| We present a novel method for real-time rendering of subdivision surfaces whose goal is to make subdivision faces as easy to render as triangles, points, or lines. Our approach uses the GPU tessellation hardware and processes each face of a base mesh independently and in a streaming fashion, thus allowing an entire model to be rendered in a single pass. |
| [paper][video][bibtex][project page] |
| PiGraphs: Learning Interaction Snapshots from Observations |
|---|
| Manolis Savva, Angel X. Chang, Pat Hanrahan, Matthew Fisher, Matthias Nießner |
| ACM Transactions on Graphics 2016 (TOG) |
| We learn a probabilistic model connecting human poses and arrangements of objects from observations of interactions collected with commodity RGB-D sensors. This model is encoded as a set of Prototypical Interaction Graphs (PiGraphs): a human-centric representation capturing physical contact and attention linkages between geometry and the human body. |
| [paper][video][bibtex][project page] |
| ProxImaL: Efficient Image Optimization using Proximal Algorithms |
|---|
| Felix Heide, Steven Diamond, Matthias Nießner, Jonathan Ragan-Kelley, Wolfgang Heidrich, Gordon Wetzstein |
| ACM Transactions on Graphics 2016 (TOG) |
| ProxImaL is a domain-specific language and compiler for image optimization problems that makes it easy to experiment with different problem formulations and algorithm choices. The compiler intelligently chooses the best way to translate a problem formulation and choice of optimization algorithm into an efficient solver implementation. |
| [paper][bibtex][supplemental][project page] |
| Face2Face: Real-time Face Capture and Reenactment of RGB Videos |
|---|
| Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, Matthias Nießner |
| CVPR 2016 (Oral) |
| We present a novel approach for real-time facial reenactment of a monocular target video sequence (e.g., Youtube video). The source sequence is also a monocular video stream, captured live with a commodity webcam. Our goal is to animate the facial expressions of the target video by a source actor and re-render the manipulated output video in a photo-realistic fashion. |
| [paper][video][bibtex][supplemental][project page] |
| Volumetric and Multi-View CNNs for Object Classification on 3D Data |
|---|
| Charles Ruizhongtai Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, Leonidas Guibas |
| CVPR 2016 (Spotlight) |
| In this paper, we improve both Volumetric CNNs and Multi-view CNNs by introducing new distinct network architectures. Overall, we are able to outperform current state-of-the-art methods for both Volumetric CNNs and Multi-view CNNs. |
| [paper][code][bibtex][supplemental][project page] |
2015
| Real-time Expression Transfer for Facial Reenactment |
|---|
| Justus Thies, Michael Zollhöfer, Matthias Nießner, Levi Valgaerts, Marc Stamminger, Christian Theobalt |
| ACM Transactions on Graphics 2015 (TOG) |
| We present a method for the real-time transfer of facial expressions from an actor in a source video to an actor in a target video, thus enabling the ad-hoc control of the facial expressions of the target actor. |
| [paper][video][bibtex][project page] |
| Real-Time Pixel Luminance Optimization for Dynamic Multi-Projection Mapping |
|---|
| Christian Siegl, Matteo Colaianni, Lucas Thies, Justus Thies, Michael Zollhöfer, Shahram Izadi, Marc Stamminger, Frank Bauer |
| ACM Transactions on Graphics 2015 (TOG) |
| Using projection mapping enables us to bring virtual worlds into shared physical spaces. In this paper, we present a novel, adaptable and real-time projection mapping system, which supports multiple projectors and high quality rendering of dynamic content on surfaces of complex geometrical shape. Our system allows for smooth blending across multiple projectors using a new optimization framework that simulates the diffuse direct light transport of the physical world to continuously adapt the color output of each projector pixel. |
| [paper][video][bibtex][project page] |
| Activity-centric Scene Synthesis for Functional 3D Scene Modeling |
|---|
| Matthew Fisher, Manolis Savva, Yangyan Li, Pat Hanrahan, Matthias Nießner |
| ACM Transactions on Graphics 2015 (TOG) |
| We present a novel method to generate 3D scenes that allow the same activities as real environments captured through noisy and incomplete 3D scans. |
| [paper][video][bibtex][supplemental][project page] |
| SemanticPaint: Interactive 3D Labeling and Learning at your Fingertips |
|---|
| Julien Valentin, Vibhav Vineet, Ming-Ming Cheng, David Kim, Jamie Shotton, Pushmeet Kohli, Matthias Nießner, Antonio Criminisi, Shahram Izadi, Philip H. S. Torr |
| ACM Transactions on Graphics 2015 (TOG) |
| We present a new interactive and online approach to 3D scene understanding. Our system, SemanticPaint, allows users to simultaneously scan their environment, while interactively segmenting the scene simply by reaching out and touching any desired object or surface. |
| [paper][video][bibtex][project page] |
| Efficient Ray Tracing of Subdivision Surfaces using Tessellation Caching |
|---|
| Carsten Benthin, Sven Woop, Matthias Nießner, Kai Selgrad, Ingo Wald |
| High Performance Graphics 2015 |
| In this paper, we propose method to efficiently ray trace subdivision surfaces using a lazy-build caching scheme while exploiting the capabilities of today's many-core architectures. Our approach is part of Intel's Embree. |
| [paper][video][code][bibtex][project page] |
| Real-time Rendering Techniques with Hardware Tessellation |
|---|
| Matthias Nießner, Benjamin Keinert, Matthew Fisher, Marc Stamminger, Charles Loop, Henry Schäfer |
| Computer Graphics Forum 2015 |
| In this survey, we provide an overview of real-time rendering techniques with hardware tessellation by summarizing, discussing, and comparing state-of-the art approaches. |
| [paper][bibtex][project page] |
| Shading-based Refinement on Volumetric Signed Distance Functions |
|---|
| Michael Zollhöfer, Angela Dai, Matthias Innmann, Chenglei Wu, Marc Stamminger, Christian Theobalt, Matthias Nießner |
| ACM Transactions on Graphics 2015 (TOG) |
| We present a novel method to obtain fine-scale detail in 3D reconstructions generated with low-budget RGB-D cameras or other commodity scanning devices. |
| [paper][video][bibtex][project page] |
| Exploiting Uncertainty in Regression Forests for Accurate Camera Relocalization |
|---|
| Julien Valentin, Matthias Nießner, Jamie Shotton, Andrew Fitzgibbon, Shahram Izadi, Philip H. S. Torr |
| CVPR 2015 |
| In this paper, we train a regression forest to predict mixtures of anisotropic 3D Gaussians and show how the predicted uncertainties can be taken into account for continuous pose optimization. Experiments show that our method is able to relocalize up to 40 percent more frames than the state of the art. |
| [paper][video][bibtex][project page] |
| Incremental Dense Semantic Stereo Fusion for Large-Scale Semantic Scene Reconstruction |
|---|
| Vibhav Vineet, Ondrej Miksik, Morten Lidegaard, Matthias Nießner, Stuart Golodetz, Victor A. Prisacariu, Olaf Kähler, David W. Murray, Shahram Izadi, Patrick Pérez, Philip H. S. Torr |
| ICRA 2015 |
| In this paper, we build on a recent hash-based technique for large-scale fusion and an efficient mean-field inference algorithm for densely-connected CRFs to present what to our knowledge is the first system that can perform dense, large-scale, outdoor semantic reconstruction of a scene in (near) real time. |
| [paper][video][bibtex][project page] |
| The Semantic Paintbrush: Interactive 3D Mapping and Recognition in Large Outdoor Spaces |
|---|
| Ondrej Miksik, Vibhav Vineet, Morten Lidegaard, Ram Prasaath, Matthias Nießner, Stuart Golodetz, Stephen L. Hicks, Patrick Pérez, Shahram Izadi, Philip H. S. Torr |
| CHI 2015 |
| We present an augmented reality system for large scale 3D reconstruction and recognition in outdoor scenes. Unlike existing prior work, which tries to reconstruct scenes using active depth cameras, we use a purely passive stereo setup, allowing for outdoor use and extended sensing range. |
| [paper][bibtex][project page] |
| Database-Assisted Object Retrieval for Real-Time 3D Reconstruction |
|---|
| Yangyan Li, Angela Dai, Leonidas Guibas, Matthias Nießner |
| Eurographics 2015 |
| We present a novel reconstruction approach based on retrieving objects from a 3D shape database while scanning an environment in real-time. With this approach, we are able to replace scanned RGB-D data with complete, hand-modeled objects from shape databases. |
| [paper][video][bibtex][project page] |
| Dynamic Feature-Adaptive Subdivision |
|---|
| Henry Schäfer, Jens Raab, Mark Meyer, Marc Stamminger, Matthias Nießner |
| I3D 2015 |
| In this paper, we present dynamic feature-adaptive subdivision (DFAS), which improves upon FAS by enabling an independent subdivision depth for every irregularity. |
| [paper][video][bibtex][project page] |
2014
| SceneGrok: Inferring Action Maps in 3D Environments |
|---|
| Manolis Savva, Angel X. Chang, Pat Hanrahan, Matthew Fisher, Matthias Nießner |
| ACM Transactions on Graphics 2014 (TOG) |
| In this paper, we present a method to establish a correlation between the geometry and the functionality of 3D environments. |
| [paper][video][bibtex][project page] |
| Real-time Shading-based Refinement for Consumer Depth Cameras |
|---|
| Chenglei Wu, Michael Zollhöfer, Matthias Nießner, Marc Stamminger, Shahram Izadi, Christian Theobalt |
| ACM Transactions on Graphics 2014 (TOG) |
| We present the first real-time method for refinement of depth data using shape-from-shading in general uncontrolled scenes. |
| [paper][video][code][bibtex][project page] |
| Real-time Non-rigid Reconstruction using an RGB-D Camera |
|---|
| Michael Zollhöfer, Matthias Nießner, Shahram Izadi, Christoph Rhemann, Christopher Zach, Matthew Fisher, Chenglei Wu, Andrew Fitzgibbon, Charles Loop, Christian Theobalt, Marc Stamminger |
| ACM Transactions on Graphics 2014 (TOG) |
| We present a combined hardware and software solution for markerless reconstruction of non-rigidly deforming physical objects with arbitrary shape in real-time. |
| [paper][video][bibtex][project page] |
| Real-Time Deformation of Subdivision Surfaces from Object Collisions |
|---|
| Henry Schäfer, Benjamin Keinert, Matthias Nießner, Christoph Buchenau, Michael Guthe, Marc Stamminger |
| High Performance Graphics 2014 |
| We present a novel real-time approach for fine-scale surface deformations resulting from collisions. |
| [paper][video][bibtex][project page] |
| Local Painting and Deformation of Meshes on the GPU |
|---|
| Henry Schäfer, Benjamin Keinert, Matthias Nießner, Marc Stamminger |
| Computer Graphics Forum 2014 |
| We present a novel method to adaptively apply modifications to scene data stored in GPU memory. Such modifications |
| [paper][video][bibtex][project page] |
| RetroDepth: 3D Silhouette Sensing for High-Precision Input On and Above Physical Surfaces |
|---|
| David Kim, Shahram Izadi, Jakub Dostal, Christoph Rhemann, Cem Keskin, Christopher Zach, Jamie Shotton, Tim Large, Steven Bathiche, Matthias Nießner, Alex Butler, Sean Fanello, Vivek Pradeep |
| CHI 2014 |
| We present RetroDepth, a new vision-based system for accurately sensing the 3D silhouettes of hands, styluses, and other objects, as they interact on and above physical surfaces. |
| [video][bibtex][project page] |
| State of the Art Report on Real-time Rendering with Hardware Tessellation |
|---|
| Henry Schäfer, Matthias Nießner, Benjamin Keinert, Marc Stamminger, Charles Loop |
| Eurographics 2014 |
| In this state of the art report, we provide an overview of recent work and challenges in this topic by summarizing, discussing and comparing methods for the rendering of smooth and highly detailed surfaces in real-time. |
| [paper][bibtex][project page] |
| Combining Inertial Navigation and ICP for Real-time 3D Surface Reconstruction |
|---|
| Matthias Nießner, Angela Dai, Matthew Fisher |
| Eurographics 2014 |
| We present a novel method to improve the robustness of real-time 3D surface reconstruction by incorporating inertial sensor data when determining inter-frame alignment. With commodity inertial sensors, we can significantly reduce the number of iterative closest point (ICP) iterations required per frame. |
| [paper][video][bibtex][project page] |
2013
| Real-time 3D Reconstruction at Scale using Voxel Hashing |
|---|
| Matthias Nießner, Michael Zollhöfer, Shahram Izadi, Marc Stamminger |
| ACM Transactions on Graphics 2013 (TOG) |
| Online 3D reconstruction is gaining newfound interest due to the availability of real-time consumer depth cameras. We contribute an online system for large and fine scale volumetric reconstruction based on a memory and speed efficient data structure. |
| [paper][video][code][bibtex][project page] |
| Analytic Displacement Mapping using Hardware Tessellation |
|---|
| Matthias Nießner, Charles Loop |
| ACM Transactions on Graphics 2013 (TOG) |
| Displacement mapping is ideal for modern GPUs since it enables high-frequency geometric surface detail on models with low memory I/O. We provide a comprehensive solution to these problems by introducing a smooth analytic displacement function. |
| [paper][bibtex][project page] |
| Real-time Collision Detection for Dynamic Hardware Tessellated Objects |
|---|
| Matthias Nießner, Christian Siegl, Henry Schäfer, Charles Loop |
| Eurographics 2013 |
| We present a novel method for real-time collision detection of patch based, displacement mapped objects using hardware tessellation. Our method supports fully animated, dynamically tessellated objects and runs entirely on the GPU. |
| [paper][bibtex][project page] |
| Rendering Subdivision Surfaces using Hardware Tessellation |
|---|
| Matthias Nießner |
| PhD Thesis (Published by Dr. Hut) |
| In this thesis we present techniques that facilitate the use of high-quality movie content in real-time applications that run on commodity desktop computers. We utilize modern graphics hardware and use hardware tessellation to generate surface geometry on-the-fly based on patches. |
| [paper][bibtex][project page] |
| Real-time Simulation of Human Vision using Temporal Compositing with CUDA on the GPU |
|---|
| Matthias Nießner, Nadine Kuhnert, Kai Selgrad, Marc Stamminger, Georg Michelson |
| Proceedings 25th Workshop on Parallel Systems and Algorithms 2013 |
| We present a novel approach that simulates human vision including visual defects such as glaucoma by temporal composition of human vision in real-time on the GPU. Therefore, we determine eye focus points every time step and adapt the lens accommodation of our virtual eye model accordingly. |
| [paper][bibtex][project page] |
2012
| Feature Adaptive GPU Rendering of Catmull-Clark Subdivision Surfaces |
|---|
| Matthias Nießner, Charles Loop, Mark Meyer, Tony DeRose |
| ACM Transactions on Graphics 2012 (TOG) |
| We present a novel method for high-performance GPU based rendering of Catmull-Clark subdivision surfaces. Unlike previous methods, our algorithm computes the true limit surface up to machine precision, and is capable of rendering surfaces that conform to the full RenderMan specification for Catmull-Clark surfaces. |
| [paper][video][code][bibtex][project page] |
| Patch-based Occlusion Culling for Hardware Tessellation |
|---|
| Matthias Nießner, Charles Loop |
| Computer Graphics International 2012 |
| We present an occlusion culling algorithm that leverages the unique characteristics of patch primitives within the hardware tessellation pipeline. |
| [paper][bibtex][project page] |
| Efficient Evaluation of Semi-Smooth Creases in Catmull-Clark Subdivision Surfaces |
|---|
| Matthias Nießner, Charles Loop, Günther Greiner |
| Eurographics 2012 |
| We present a novel method to evaluate semi-smooth creases in Catmull-Clark subdivision surfaces. Our algorithm supports both integer and fractional crease tags corresponding to the RenderMan (Pixar) specification. |
| [paper][bibtex][project page] |
| Real-time Simulation and Visualization of Human Vision through Eyeglasses on the GPU |
|---|
| Matthias Nießner, Roman Sturm, Günther Greiner |
| ACM SIGGRAPH VRCAI 2012 |
| We present a novel approach that allows real-time simulation of human vision through eyeglasses. Our system supports glasses that are composed of a combination of spheric, toric and in particular of free-form surfaces. |
| [paper][bibtex][project page] |