Matterport3D: Learning from RGB-D Data in Indoor Environments
Angel X. Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Nießner, Manolis Savva, Shuran Song, Andy Zeng, Yinda Zhang
3DV 2017
In this paper, we introduce Matterport3D, a large-scale RGB-D dataset containing 10,800 panoramic views from 194,400 RGB-D images of 90 building-scale scenes.
[paper][bibtex][project page]

Multiframe Scene Flow with Piecewise Rigid Motion
Vladislav Golyanik, Kihwan Kim, Robert Maier, Matthias Nießner, Didier Stricker, Jan Kautz
3DV 2017
We introduce a novel multi-frame scene flow approach that jointly optimizes the consistency of the patch appearances and their local rigid motions from RGB-D image sequences.
[paper][bibtex][project page]

3DLite: Towards Commodity 3D Scanning for Content Creation
Jingwei Huang, Angela Dai, Leonidas Guibas, Matthias Nießner
ACM Transactions on Graphics 2017 (TOG)
We present 3DLite, a novel approach to reconstruct 3D environments using consumer RGB-D sensors, making a step towards directly utilizing captured 3D content in graphics applications, such as video games, VR, or AR. Rather than reconstructing an accurate one-to-one representation of the real world, our method computes a lightweight, low-polygonal geometric abstraction of the scanned geometry.
[paper][video][bibtex][supplemental][project page]

Autonomous Reconstruction of Unknown Indoor Scenes Guided by Time-varying Tensor Fields
Kai Xu, Lintao Zheng, Zihao Yan, Guohang Yan, Eugene Zhang, Matthias Nießner, Oliver Deussen, Daniel Cohen-Or, Hui Huang
ACM Transactions on Graphics 2017 (TOG)
We present a navigation-by-reconstruction approach to address this question where moving paths of the robot are planned to account for both global efficiency for fast exploration and local smoothness to obtain high-quality scans.
[paper][video][code][bibtex][project page]

Opt: A Domain Specific Language for Non-linear Least Squares Optimization in Graphics and Imaging
Zachary DeVito, Michael Mara, Michael Zollhöfer, Gilbert Bernstein, Jonathan Ragan-Kelley, Christian Theobalt, Pat Hanrahan, Matthew Fisher, Matthias Nießner
ACM Transactions on Graphics 2017 (TOG)
We propose a new language, Opt, in which a user simply writes energy functions over image- or graph-structured unknowns, and a compiler automatically generates state-of-the-art GPU optimization kernels.
[paper][bibtex][project page]

BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integration
Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, Christian Theobalt
ACM Transactions on Graphics 2017 (TOG)
We introduce a novel, real-time, end-to-end 3D reconstruction framework, with a robust pose optimization strategy based on sparse feature matches and dense geometric and photometric alignment. One main contribution is the ability to update the reconstructed model on-the-fly as new (global) pose optimization results become available.
[paper][video][bibtex][project page]

Intrinsic3D: High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting
Robert Maier, Kihwan Kim, Daniel Cremers, Jan Kautz, Matthias Nießner
ICCV 2017
We introduce a novel method to obtain high-quality 3D reconstructions from consumer RGB-D sensors. Our core idea is to simultaneously optimize for geometry encoded in a signed distance field (SDF), textures from automaticallyselected keyframes, and their camera poses along with material and scene lighting.
[paper][bibtex][project page]

A Lightweight Approach for On-the-Fly Reflectance Estimation
Kihwan Kim, Jinwei Gu, Stephen Tyree, Pavlo Molchanov, Matthias Nießner, Jan Kautz
ICCV 2017 (Oral)
We propose a lightweight, learning-based approach for surface reflectance estimation directly from 8-bit RGB images in real-time, which can be easily plugged into any 3D scanning-and-fusion system with a commodity RGBD sensor.
[paper][bibtex][project page]

ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner
CVPR 2017 (Spotlight)
We introduce ScanNet, an RGB-D video dataset containing 2.5M views in 1513 scenes annotated with 3D camera poses, surface reconstructions, and semantic segmentations.
[paper][video][bibtex][project page]

Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis
Angela Dai, Charles Ruizhongtai Qi, Matthias Nießner
CVPR 2017 (Spotlight)
We introduce a data-driven approach to complete partial 3D shapes through a combination of volumetric deep neural networks and 3D shape synthesis. From a partially-scanned input shape, our method first infers a low-resolution - but complete - output.
[paper][bibtex][project page]

3DMatch: Learning the Matching of Local 3D Geometry in Range Scans
Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao
CVPR 2017 (Oral)
In this paper, we introduce 3DMatch, a data-driven local feature learner that jointly learns a geometric feature representation and an associated metric function from a large collection of real-world scanning data.
[paper][video][bibtex][project page]


Learning to Navigate the Energy Landscape
Julien Valentin, Angela Dai, Matthias Nießner, Pushmeet Kohli, Philip H. S. Torr, Shahram Izadi, Cem Keskin
3DV 2016
In this paper, we present a novel, general, and efficient architecture for addressing computer vision problems that are approached from an `Analysis by Synthesis' standpoint.
[paper][video][bibtex][project page]

VolumeDeform: Real-time Volumetric Non-rigid Reconstruction
Matthias Innmann, Michael Zollhöfer, Matthias Nießner, Christian Theobalt, Marc Stamminger
ECCV 2016
We present a novel approach for the reconstruction of dynamic geometric shapes using a single hand-held consumer-grade RGB-D sensor at real-time rates. Our method does not require a pre-defined shape template to start with and builds up the scene model from scratch during the scanning process.
[paper][video][bibtex][supplemental][project page]

Efficient GPU Rendering of Subdivision Surfaces using Adaptive Quadtrees
Wade Brainerd, Tim Foley, Manuel Kraemer, Henry Moreton, Matthias Nießner
ACM Transactions on Graphics 2016 (TOG)
We present a novel method for real-time rendering of subdivision surfaces whose goal is to make subdivision faces as easy to render as triangles, points, or lines. Our approach uses the GPU tessellation hardware and processes each face of a base mesh independently and in a streaming fashion, thus allowing an entire model to be rendered in a single pass.
[paper][video][bibtex][project page]

PiGraphs: Learning Interaction Snapshots from Observations
Manolis Savva, Angel X. Chang, Pat Hanrahan, Matthew Fisher, Matthias Nießner
ACM Transactions on Graphics 2016 (TOG)
We learn a probabilistic model connecting human poses and arrangements of objects from observations of interactions collected with commodity RGB-D sensors. This model is encoded as a set of Prototypical Interaction Graphs (PiGraphs): a human-centric representation capturing physical contact and attention linkages between geometry and the human body.
[paper][video][bibtex][project page]

ProxImaL: Efficient Image Optimization using Proximal Algorithms
Felix Heide, Steven Diamond, Matthias Nießner, Jonathan Ragan-Kelley, Wolfgang Heidrich, Gordon Wetzstein
ACM Transactions on Graphics 2016 (TOG)
ProxImaL is a domain-specific language and compiler for image optimization problems that makes it easy to experiment with different problem formulations and algorithm choices. The compiler intelligently chooses the best way to translate a problem formulation and choice of optimization algorithm into an efficient solver implementation.
[paper][bibtex][supplemental][project page]

Face2Face: Real-time Face Capture and Reenactment of RGB Videos
Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, Matthias Nießner
CVPR 2016 (Oral)
We present a novel approach for real-time facial reenactment of a monocular target video sequence (e.g., Youtube video). The source sequence is also a monocular video stream, captured live with a commodity webcam. Our goal is to animate the facial expressions of the target video by a source actor and re-render the manipulated output video in a photo-realistic fashion.
[paper][video][bibtex][supplemental][project page]

Volumetric and Multi-View CNNs for Object Classification on 3D Data
Charles Ruizhongtai Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, Leonidas Guibas
CVPR 2016 (Spotlight)
In this paper, we improve both Volumetric CNNs and Multi-view CNNs by introducing new distinct network architectures. Overall, we are able to outperform current state-of-the-art methods for both Volumetric CNNs and Multi-view CNNs.
[paper][code][bibtex][supplemental][project page]


Real-time Expression Transfer for Facial Reenactment
Justus Thies, Michael Zollhöfer, Matthias Nießner, Levi Valgaerts, Marc Stamminger, Christian Theobalt
ACM Transactions on Graphics 2015 (TOG)
We present a method for the real-time transfer of facial expressions from an actor in a source video to an actor in a target video, thus enabling the ad-hoc control of the facial expressions of the target actor.
[paper][video][bibtex][project page]

Activity-centric Scene Synthesis for Functional 3D Scene Modeling
Matthew Fisher, Manolis Savva, Yangyan Li, Pat Hanrahan, Matthias Nießner
ACM Transactions on Graphics 2015 (TOG)
We present a novel method to generate 3D scenes that allow the same activities as real environments captured through noisy and incomplete 3D scans.
[paper][video][bibtex][supplemental][project page]

SemanticPaint: Interactive 3D Labeling and Learning at your Fingertips
Julien Valentin, Vibhav Vineet, Ming-Ming Cheng, David Kim, Jamie Shotton, Pushmeet Kohli, Matthias Nießner, Antonio Criminisi, Shahram Izadi, Philip H. S. Torr
ACM Transactions on Graphics 2015 (TOG)
We present a new interactive and online approach to 3D scene understanding. Our system, SemanticPaint, allows users to simultaneously scan their environment, while interactively segmenting the scene simply by reaching out and touching any desired object or surface.
[paper][video][bibtex][project page]

Efficient Ray Tracing of Subdivision Surfaces using Tessellation Caching
Carsten Benthin, Sven Woop, Matthias Nießner, Kai Selgrad, Ingo Wald
High Performance Graphics 2015
In this paper, we propose method to efficiently ray trace subdivision surfaces using a lazy-build caching scheme while exploiting the capabilities of today's many-core architectures. Our approach is part of Intel's Embree.
[paper][video][code][bibtex][project page]

Real-time Rendering Techniques with Hardware Tessellation
Matthias Nießner, Benjamin Keinert, Matthew Fisher, Marc Stamminger, Charles Loop, Henry Schäfer
Computer Graphics Forum 2015
In this survey, we provide an overview of real-time rendering techniques with hardware tessellation by summarizing, discussing, and comparing state-of-the art approaches.
[paper][bibtex][project page]

Shading-based Refinement on Volumetric Signed Distance Functions
Michael Zollhöfer, Angela Dai, Matthias Innmann, Chenglei Wu, Marc Stamminger, Christian Theobalt, Matthias Nießner
ACM Transactions on Graphics 2015 (TOG)
We present a novel method to obtain fine-scale detail in 3D reconstructions generated with low-budget RGB-D cameras or other commodity scanning devices.
[paper][video][bibtex][project page]

Exploiting Uncertainty in Regression Forests for Accurate Camera Relocalization
Julien Valentin, Matthias Nießner, Jamie Shotton, Andrew Fitzgibbon, Shahram Izadi, Philip H. S. Torr
CVPR 2015
In this paper, we train a regression forest to predict mixtures of anisotropic 3D Gaussians and show how the predicted uncertainties can be taken into account for continuous pose optimization. Experiments show that our method is able to relocalize up to 40 percent more frames than the state of the art.
[paper][video][bibtex][project page]

Incremental Dense Semantic Stereo Fusion for Large-Scale Semantic Scene Reconstruction
Vibhav Vineet, Ondrej Miksik, Morten Lidegaard, Matthias Nießner, Stuart Golodetz, Victor A. Prisacariu, Olaf Kähler, David W. Murray, Shahram Izadi, Patrick Pérez, Philip H. S. Torr
ICRA 2015
In this paper, we build on a recent hash-based technique for large-scale fusion and an efficient mean-field inference algorithm for densely-connected CRFs to present what to our knowledge is the first system that can perform dense, large-scale, outdoor semantic reconstruction of a scene in (near) real time.
[paper][video][bibtex][project page]

The Semantic Paintbrush: Interactive 3D Mapping and Recognition in Large Outdoor Spaces
Ondrej Miksik, Vibhav Vineet, Morten Lidegaard, Ram Prasaath, Matthias Nießner, Stuart Golodetz, Stephen L. Hicks, Patrick Pérez, Shahram Izadi, Philip H. S. Torr
CHI 2015
We present an augmented reality system for large scale 3D reconstruction and recognition in outdoor scenes. Unlike existing prior work, which tries to reconstruct scenes using active depth cameras, we use a purely passive stereo setup, allowing for outdoor use and extended sensing range.
[paper][bibtex][project page]

Database-Assisted Object Retrieval for Real-Time 3D Reconstruction
Yangyan Li, Angela Dai, Leonidas Guibas, Matthias Nießner
Eurographics 2015
We present a novel reconstruction approach based on retrieving objects from a 3D shape database while scanning an environment in real-time. With this approach, we are able to replace scanned RGB-D data with complete, hand-modeled objects from shape databases.
[paper][video][bibtex][project page]

Dynamic Feature-Adaptive Subdivision
Henry Schäfer, Jens Raab, Mark Meyer, Marc Stamminger, Matthias Nießner
I3D 2015
In this paper, we present dynamic feature-adaptive subdivision (DFAS), which improves upon FAS by enabling an independent subdivision depth for every irregularity.
[paper][video][bibtex][project page]


SceneGrok: Inferring Action Maps in 3D Environments
Manolis Savva, Angel X. Chang, Pat Hanrahan, Matthew Fisher, Matthias Nießner
ACM Transactions on Graphics 2014 (TOG)
In this paper, we present a method to establish a correlation between the geometry and the functionality of 3D environments.
[paper][video][bibtex][project page]

Real-time Shading-based Refinement for Consumer Depth Cameras
Chenglei Wu, Michael Zollhöfer, Matthias Nießner, Marc Stamminger, Shahram Izadi, Christian Theobalt
ACM Transactions on Graphics 2014 (TOG)
We present the first real-time method for refinement of depth data using shape-from-shading in general uncontrolled scenes.
[paper][video][code][bibtex][project page]

Real-time Non-rigid Reconstruction using an RGB-D Camera
Michael Zollhöfer, Matthias Nießner, Shahram Izadi, Christoph Rhemann, Christopher Zach, Matthew Fisher, Chenglei Wu, Andrew Fitzgibbon, Charles Loop, Christian Theobalt, Marc Stamminger
ACM Transactions on Graphics 2014 (TOG)
We present a combined hardware and software solution for markerless reconstruction of non-rigidly deforming physical objects with arbitrary shape in real-time.
[paper][video][bibtex][project page]

Real-Time Deformation of Subdivision Surfaces from Object Collisions
Henry Schäfer, Benjamin Keinert, Matthias Nießner, Christoph Buchenau, Michael Guthe, Marc Stamminger
High Performance Graphics 2014
We present a novel real-time approach for fine-scale surface deformations resulting from collisions.
[paper][video][bibtex][project page]

Local Painting and Deformation of Meshes on the GPU
Henry Schäfer, Benjamin Keinert, Matthias Nießner, Marc Stamminger
Computer Graphics Forum 2014
We present a novel method to adaptively apply modifications to scene data stored in GPU memory. Such modifications
[paper][video][bibtex][project page]

RetroDepth: 3D Silhouette Sensing for High-Precision Input On and Above Physical Surfaces
David Kim, Shahram Izadi, Jakub Dostal, Christoph Rhemann, Cem Keskin, Christopher Zach, Jamie Shotton, Tim Large, Steven Bathiche, Matthias Nießner, Alex Butler, Sean Fanello, Vivek Pradeep
CHI 2014
We present RetroDepth, a new vision-based system for accurately sensing the 3D silhouettes of hands, styluses, and other objects, as they interact on and above physical surfaces.
[video][bibtex][project page]

State of the Art Report on Real-time Rendering with Hardware Tessellation
Henry Schäfer, Matthias Nießner, Benjamin Keinert, Marc Stamminger, Charles Loop
Eurographics 2014
In this state of the art report, we provide an overview of recent work and challenges in this topic by summarizing, discussing and comparing methods for the rendering of smooth and highly detailed surfaces in real-time.
[paper][bibtex][project page]

Combining Inertial Navigation and ICP for Real-time 3D Surface Reconstruction
Matthias Nießner, Angela Dai, Matthew Fisher
Eurographics 2014
We present a novel method to improve the robustness of real-time 3D surface reconstruction by incorporating inertial sensor data when determining inter-frame alignment. With commodity inertial sensors, we can significantly reduce the number of iterative closest point (ICP) iterations required per frame.
[paper][video][bibtex][project page]


Real-time 3D Reconstruction at Scale using Voxel Hashing
Matthias Nießner, Michael Zollhöfer, Shahram Izadi, Marc Stamminger
ACM Transactions on Graphics 2013 (TOG)
Online 3D reconstruction is gaining newfound interest due to the availability of real-time consumer depth cameras. We contribute an online system for large and fine scale volumetric reconstruction based on a memory and speed efficient data structure.
[paper][video][code][bibtex][project page]

Analytic Displacement Mapping using Hardware Tessellation
Matthias Nießner, Charles Loop
ACM Transactions on Graphics 2013 (TOG)
Displacement mapping is ideal for modern GPUs since it enables high-frequency geometric surface detail on models with low memory I/O. We provide a comprehensive solution to these problems by introducing a smooth analytic displacement function.
[paper][bibtex][project page]

Real-time Collision Detection for Dynamic Hardware Tessellated Objects
Matthias Nießner, Christian Siegl, Henry Schäfer, Charles Loop
Eurographics 2013
We present a novel method for real-time collision detection of patch based, displacement mapped objects using hardware tessellation. Our method supports fully animated, dynamically tessellated objects and runs entirely on the GPU.
[paper][bibtex][project page]

Rendering Subdivision Surfaces using Hardware Tessellation
Matthias Nießner
PhD Thesis (Published by Dr. Hut)
In this thesis we present techniques that facilitate the use of high-quality movie content in real-time applications that run on commodity desktop computers. We utilize modern graphics hardware and use hardware tessellation to generate surface geometry on-the-fly based on patches.
[paper][bibtex][project page]

Real-time Simulation of Human Vision using Temporal Compositing with CUDA on the GPU
Matthias Nießner, Nadine Kuhnert, Kai Selgrad, Marc Stamminger, Georg Michelson
Proceedings 25th Workshop on Parallel Systems and Algorithms 2013
We present a novel approach that simulates human vision including visual defects such as glaucoma by temporal composition of human vision in real-time on the GPU. Therefore, we determine eye focus points every time step and adapt the lens accommodation of our virtual eye model accordingly.
[paper][bibtex][project page]


Feature Adaptive GPU Rendering of Catmull-Clark Subdivision Surfaces
Matthias Nießner, Charles Loop, Mark Meyer, Tony DeRose
ACM Transactions on Graphics 2012 (TOG)
We present a novel method for high-performance GPU based rendering of Catmull-Clark subdivision surfaces. Unlike previous methods, our algorithm computes the true limit surface up to machine precision, and is capable of rendering surfaces that conform to the full RenderMan specification for Catmull-Clark surfaces.
[paper][video][code][bibtex][project page]

Patch-based Occlusion Culling for Hardware Tessellation
Matthias Nießner, Charles Loop
Computer Graphics International 2012
We present an occlusion culling algorithm that leverages the unique characteristics of patch primitives within the hardware tessellation pipeline.
[paper][bibtex][project page]

Efficient Evaluation of Semi-Smooth Creases in Catmull-Clark Subdivision Surfaces
Matthias Nießner, Charles Loop, Günther Greiner
Eurographics 2012
We present a novel method to evaluate semi-smooth creases in Catmull-Clark subdivision surfaces. Our algorithm supports both integer and fractional crease tags corresponding to the RenderMan (Pixar) specification.
[paper][bibtex][project page]

Real-time Simulation and Visualization of Human Vision through Eyeglasses on the GPU
Matthias Nießner, Roman Sturm, Günther Greiner
We present a novel approach that allows real-time simulation of human vision through eyeglasses. Our system supports glasses that are composed of a combination of spheric, toric and in particular of free-form surfaces.
[paper][bibtex][project page]