Figure 1: Our system reconstructs a temporally coherent set of hair fibers for real-world dynamic... more Figure 1: Our system reconstructs a temporally coherent set of hair fibers for real-world dynamic hair. It accommodates a variety of hair types and styles, as well as nontrivial motion (top: input video, middle: reconstructed envelope surface, bottom: synthesized hair strands). Abstract The realistic reconstruction of hair motion is challenging because of hair's complex occlusion, lack of a well-defined surface, and non-Lambertian material. We present a system for passive capture of dynamic hair performances using a set of high-speed video cameras. Our key insight is that, while hair color is unlikely to match across multiple views, the response to oriented filters will. We combine a multi-scale version of this orientation-based matching metric with bilateral aggregation, a MRF-based stereo reconstruction technique , and algorithms for temporal tracking and de-noising. Our final output is a set of hair strands for each frame, grown according to the per-frame reconstructed rough geometry and orientation field. We demonstrate results for a number of hair styles ranging from smooth and ordered to curly and messy.
Adobe Research (a) reference photo (b) user sketches (c) example hairstyles (d) large-scale struc... more Adobe Research (a) reference photo (b) user sketches (c) example hairstyles (d) large-scale structure (e) output strands (f) another view Figure 1: Our system takes as input a reference photo (a), a few user strokes (b) and a database of example hairstyles (c) to model the 3D target hairstyle (e). The retrieved examples that best match the user strokes are highlighted with the corresponding colors in (c) which are combined consistently as shown in (d). Original image courtesy of Yung-Yuan Kao. Abstract Human hair presents highly convoluted structures and spans an extraordinarily wide range of hairstyles, which is essential for the digitization of compelling virtual avatars but also one of the most challenging to create. Cutting-edge hair modeling techniques typically rely on expensive capture devices and significant manual labor. We introduce a novel data-driven framework that can digitize complete and highly complex 3D hairstyles from a single-view photograph. We first construct a large database of manually crafted hair models from several online repositories. Given a reference photo of the target hairstyle and a few user strokes as guidance, we automatically search for multiple best matching examples from the database and combine them consistently into a single hairstyle to form the large-scale structure of the hair model. We then synthesize the final hair strands by jointly optimizing for the projected 2D similarity to the reference photo, the physical plausibility of each strand, as well as the local orientation coherency between neighboring strands. We demonstrate the effectiveness and robustness of our method on a variety of hairstyles and challenging images, and compare our system with state-of-the-art hair modeling algorithms.
input image face mesh and hair polystrips 3D avatar input image face mesh and hair polystrips 3D ... more input image face mesh and hair polystrips 3D avatar input image face mesh and hair polystrips 3D avatar input image face mesh and hair polystrips 3D avatar Fig. 1. We introduce an end-to-end framework for modeling a complete 3D avatar from a single input image for real-time rendering. We infer fully rigged textured faces models and polygonal strips for hair. Our flexible and efficient mesh-based hair representation is suitable for a wide range of hairstyles and can be readily integrated into existing real-time game engines. All of the illustrations are rendered in realtime in Unity. President Trump's picture is obtained from whitehouse.gov and Kim Jong-un's photograph was published in the Rodong Sinmun. The other celebrity pictures are used with permission from Getty Images. We present a fully automatic framework that digitizes a complete 3D head with hair from a single unconstrained image. Our system offers a practical and consumer-friendly end-to-end solution for avatar personalization in gaming and social VR applications. The reconstructed models include secondary components (eyes, teeth, tongue, and gums) and provide animation-friendly blendshapes and joint-based rigs. While the generated face is a high-quality textured mesh, we propose a versatile and efficient polygonal strips (polystrips) representation for the hair. Polystrips are suitable for an extremely wide range of hairstyles and textures and are compatible with existing game engines for real-time rendering. In addition to integrating state-of-the-art advances in facial shape modeling and appearance inference , we propose a novel single-view hair generation pipeline, based on * indicates equal contribution Permission to make digitalmodel and texture retrieval, shape refinement, and polystrip patching optimization. The performance of our hairstyle retrieval is enhanced using a deep convolutional neural network for semantic hair attribute classification. Our generated models are visually comparable to state-of-the-art game characters designed by professional artists. For real-time settings, we demonstrate the flexibility of polystrips in handling hairstyle variations, as opposed to conventional strand-based representations. We further show the effectiveness of our approach on a large number of images taken in the wild, and how compelling avatars can be easily created by anyone.
Figure 1: A live demonstration of our system. We are able to obtain high-fidelity animations of t... more Figure 1: A live demonstration of our system. We are able to obtain high-fidelity animations of the user's facial expressions in real-time using convolutional neural net regressors. Left: a user wearing our prototype system, which uses cameras attached to the HMD to track the user's eye and mouth movements. Right: a digital avatar controlled by the user. Abstract Significant challenges currently prohibit expressive interaction in virtual reality (VR). Occlusions introduced by head-mounted displays (HMDs) make existing facial tracking techniques intractable, and even state-of-the-art techniques used for real-time facial tracking in unconstrained environments fail to capture subtle details of the user's facial expressions that are essential for compelling speech animation. We introduce a novel system for HMD users to control a digital avatar in real-time while producing plausible speech animation and emotional expressions. Using a monocular camera attached to an HMD, we record multiple subjects performing various facial expressions and speaking several phonetically-balanced sentences. These images are used with artist-generated animation data corresponding to these sequences to train a convolutional neural network (CNN) to regress images of a user's mouth region to the parameters that control a digital avatar. To make training this system more tractable, we use audio-based alignment techniques to map images of multiple users making the same utterance to the corresponding animation parameters. We demonstrate that this approach is also feasible for tracking the expressions around the user's eye region with an internal infrared (IR) camera, thereby enabling full facial tracking. This system requires no user-specific calibration, uses easily obtainable consumer hardware, and produces high-quality animations of speech and emotional expressions. Finally, we demonstrate the quality of our system on a variety of subjects and evaluate its performance against state-of-the-art real-time facial tracking techniques.
Three-dimensional registration is the process of aligning scans of different incomplete views of ... more Three-dimensional registration is the process of aligning scans of different incomplete views of an object such that their corresponding regions agree in space and time. It is used to build digital complete models from partial acquisitions of real-world objects which typically contain noise, outliers, and acquisition holes due to occlusions and hardware limitations. For non-rigid models, such as humans, shapes of corresponding regions undergo complex deformations in addition to misalignments. Thus, finding explicit correspondences between two dissimilar shapes becomes considerably more challenging. In addition, finding the right deformations is also non-trivial as they depend on the correspondences which are only partially available between consecutive scans. This report presents an analysis of the 3-D registration problem, a general approach for the spatio-temporal registration of deformable scans that are acquired with a fast 3-D scanner, and extensive experiments on synthetic and real acquisition data. The objective is to un-deform the surfaces of all range maps and to fit them into a particular instance of time.
(a) An image of (b) Point cloud with (c) Failure of (d) Color-coded (e) Synthesized (f) A frame f... more (a) An image of (b) Point cloud with (c) Failure of (d) Color-coded (e) Synthesized (f) A frame from captured hairstyle 3D orientation field previous method reconstructed wisps hair strands hair simulation Figure 1: Our system takes a collection of images as input (a) and reconstructs a point cloud with a 3D orientation field (b). In contrast to previous methods (e.g. [Paris et al. 2008]) that straightforwardly grow hair strands from the scalp following the orientation field and hence cannot reconstruct complex hairstyles with convoluted curl structures (c), we reconstruct complete, coherent and plausible wisps (d) aware of the underlying hair structures. The wisps can be used to synthesize hair strands (e) that are plausible for animation or simulation (f). Abstract Existing hair capture systems fail to produce strands that reflect the structures of real-world hairstyles. We introduce a system that reconstructs coherent and plausible wisps aware of the underlying hair structures from a set of still images without any special lighting. Our system first discovers locally coherent wisp structures in the reconstructed point cloud and the 3D orientation field, and then uses a novel graph data structure to reason about both the connec-tivity and directions of the local wisp structures in a global optimization. The wisps are then completed and used to synthesize hair strands which are robust against occlusion and missing data and plausible for animation and simulation. We show reconstruction results for a variety of complex hairstyles including curly, wispy, and messy hair.
5 Artec Group 3D scanning pose change output reconstruction textured reconstruction large variety... more 5 Artec Group 3D scanning pose change output reconstruction textured reconstruction large variety of examples 3D print Figure 1: With our system, users can scan themselves with a single 3D sensor by rotating the same pose for a few different views (typically eight, ⇠45 degrees apart) to cover the full body. Our method robustly registers and merges different scans into a watertight surface with consistent texture in spite of shape changes during repositioning, and lighting differences between the scans. These surfaces are suitable for applications such as online avatars or 3D printing (the miniature shown here was printed using a ZPrinter 650.) Abstract We develop an automatic pipeline that allows ordinary users to capture complete and fully textured 3D models of themselves in minutes , using only a single Kinect sensor, in the uncontrolled lighting environment of their own home. Our method requires neither a turntable nor a second operator, and is robust to the small deformations and changes of pose that inevitably arise during scanning. After the users rotate themselves with the same pose for a few scans from different views, our system stitches together the captured scans using multi-view non-rigid registration, and produces watertight final models. To ensure consistent texturing, we recover the underlying albedo from each scanned texture and generate seamless global textures using Poisson blending. Despite the minimal requirements we place on the hardware and users, our method is suitable for full body capture of challenging scenes that cannot be handled well using previous methods, such as those involving loose clothing, complex poses, and props.
Extracting range data through active optical triangulation requires robust stripe edge detection ... more Extracting range data through active optical triangulation requires robust stripe edge detection of the emitted pattern because of undesired shape and shading variations in the scanned object. We propose several extensions to a struc-tured light system originally proposed by Zhang et al. [1]. Assuming stripes of a certain width, our edge detection criterion is achieved simply by restricting the number of consecutive edge pixels. Subpixel accuracy from one single input image can be obtained by approximating the gradient of the square local contrast with a piecewise linear function. Finally, we present a noise reduction technique for the range map through meshing followed by a parameterized face orientation culling. Experimental results have demonstrated major improvements in terms of robustness against acquisition noise, shading variations and complex shapes.
(a) no microstructure (b) static microstructure (c) dynamic microstructure (d) photograph Figure ... more (a) no microstructure (b) static microstructure (c) dynamic microstructure (d) photograph Figure 1: A rendered facial expression with (a) mesostructure only (b) static microstructure from a neutral expression (c) dynamic mi-crostructure from convolving the neutral microstructure according to local surface strain compared to a reference photograph of the similar expression. The insets show detail from the lower-left area. Abstract We present a technique for synthesizing the effects of skin mi-crostructure deformation by anisotropically convolving a high-resolution displacement map to match normal distribution changes in measured skin samples. We use a 10-micron resolution scanning technique to measure several in vivo skin samples as they are stretched and compressed in different directions, quantifying how stretching smooths the skin and compression makes it rougher. We tabulate the resulting surface normal distributions, and show that convolving a neutral skin microstructure displacement map with blurring and sharpening filters can mimic normal distribution changes and microstructure deformations. We implement the spatially-varying displacement map filtering on the GPU to interactively render the effects of dynamic microgeometry on animated faces obtained from high-resolution facial scans.
(a) input photo (b) input mesh (c) extracted structure (d) output strands (e) another view of (a)... more (a) input photo (b) input mesh (c) extracted structure (d) output strands (e) another view of (a) (f) another view of (d) Figure 1: Capturing a five-strand Dutch braid. We capture the braided hairstyle (a) using a Kinect sensor and obtain an input mesh with a local 3D orientation for each vertex (b). Based on the information provided by the example patches in a database, we extract the centerlines (c) of the braid structure to synthesize final output strands (d). (e) and (f) show the input reference photo and our output strands from another viewpoint. Abstract From fishtail to princess braids, these intricately woven structures define an important and popular class of hairstyle, frequently used for digital characters in computer graphics. In addition to the challenges created by the infinite range of styles, existing modeling and capture techniques are particularly constrained by the geometric and topological complexities. We propose a data-driven method to automatically reconstruct braided hairstyles from input data obtained from a single consumer RGB-D camera. Our approach covers the large variation of repetitive braid structures using a family of compact procedural braid models. From these models, we produce a database of braid patches and use a robust random sampling approach for data fitting. We then recover the input braid structures using a multi-label optimization algorithm and synthesize the intertwining hair strands of the braids. We demonstrate that a minimal capture equipment is sufficient to effectively capture a wide range of complex braids with distinct shapes and structures.
Figure 1: Our system takes as input a few images (a) and employs a database of simulated example ... more Figure 1: Our system takes as input a few images (a) and employs a database of simulated example strands (b) to discover structurally plausible configurations from the reconstructed cover strands (c) for final strand synthesis (d). Our method robustly fits example strands to the cover strands which are computed from unprocessed outlier affected input data (e) to generate compelling reconstruction results (f). In contrast, the state-of-the-art method of [Luo et al. 2013a] fails in the presence of strong outliers (g). Abstract We introduce a data-driven hair capture framework based on example strands generated through hair simulation. Our method can robustly reconstruct faithful 3D hair models from unprocessed input point clouds with large amounts of outliers. Current state-of-the-art techniques use geometrically-inspired heuristics to derive global hair strand structures, which can yield implausible hair strands for hairstyles involving large occlusions, multiple layers, or wisps of varying lengths. We address this problem using a voting-based fitting algorithm to discover structurally plausible configurations among the locally grown hair segments from a database of simulated examples. To generate these examples, we exhaustively sample the simulation configurations within the feasible parameter space constrained by the current input hairstyle. The number of necessary simulations can be further reduced by leveraging symmetry and constrained initial conditions. The final hairstyle can then be structurally represented by a limited number of examples. To handle constrained hairstyles such as a ponytail of which realistic simulations are more difficult, we allow the user to sketch a few strokes to generate strand examples through an intuitive interface. Our approach focuses on robustness and generality. Since our method is structurally plausible by construction, we ensure an improved control during hair digitization and avoid implausible hair synthesis for a wide range of hairstyles.
Fig. 1. Our system infers high-fidelity facial reflectance and geometry maps from a single image ... more Fig. 1. Our system infers high-fidelity facial reflectance and geometry maps from a single image (diffuse albedo, specular albedo, as well as medium-and high-frequency displacements). These maps can be used for high-fidelity rendering under novel illumination conditions. We present a deep learning-based technique to infer high-quality facial reflectance and geometry given a single unconstrained image of the subject , which may contain partial occlusions and arbitrary illumination conditions. The reconstructed high-resolution textures, which are generated in only a few seconds, include high-resolution skin surface reflectance maps, representing both the diffuse and specular albedo, and medium-and high-frequency displacement maps, thereby allowing us to render compelling digital avatars under novel lighting conditions. To extract this data, we train our deep neural networks with a high-quality skin reflectance and geometry database created with a state-of-the-art multi-view photometric stereo system using polarized gradient illumination. Given the raw facial texture map extracted from the input image, our neural networks synthesize complete * indicates equal contribution Permission to make digital reflectance and displacement maps, as well as complete missing regions caused by occlusions. The completed textures exhibit consistent quality throughout the face due to our network architecture, which propagates texture features from the visible region, resulting in high-fidelity details that are consistent with those seen in visible regions. We describe how this highly underconstrained problem is made tractable by dividing the full inference into smaller tasks, which are addressed by dedicated neural networks. We demonstrate the effectiveness of our network design with robust texture completion from images of faces that are largely occluded. With the inferred reflectance and geometry data, we demonstrate the rendering of high-fidelity 3D avatars from a variety of subjects captured under different lighting conditions. In addition, we perform evaluations demonstrating that our method can infer plausible facial reflectance and geometric details comparable to those obtained from high-end capture devices, and outperform alternative approaches that require only a single unconstrained input image.
Figure 1: Our adaptive tracking model conforms to the input expressions on-the-fly, producing a b... more Figure 1: Our adaptive tracking model conforms to the input expressions on-the-fly, producing a better fit to the user than state-of-the-art data driven techniques [Weise et al. 2011] which are confined to learned motion priors and generate plausible but not accurate tracking. Abstract We introduce a real-time and calibration-free facial performance capture framework based on a sensor with video and depth input. In this framework, we develop an adaptive PCA model using shape correctives that adjust on-the-fly to the actor's expressions through incremental PCA-based learning. Since the fitting of the adaptive model progressively improves during the performance, we do not require an extra capture or training session to build this model. As a result, the system is highly deployable and easy to use: it can faithfully track any individual, starting from just a single face scan of the subject in a neutral pose. Like many real-time methods, we use a linear subspace to cope with incomplete input data and fast motion. To boost the training of our tracking model with reliable samples, we use a well-trained 2D facial feature tracker on the input video and an efficient mesh deformation algorithm to snap the result of the previous step to high frequency details in visible depth map regions. We show that the combination of dense depth maps and texture features around eyes and lips is essential in capturing natural dialogues and nuanced actor-specific emotions. We demonstrate that using an adaptive PCA model not only improves the fitting accuracy for tracking but also increases the expressiveness of the retargeted character.
input image synthesized key expressions image-based dynamic 3D avatar real-time rendering on mobi... more input image synthesized key expressions image-based dynamic 3D avatar real-time rendering on mobile Fig. 1. Overview of our system. Using a single neutral-face input image, we are able to synthesize arbitrary expressions both in image space and UV texture space. These generated textures, which include a photoreal mouth interior and the eyes, can then be used to pilot dynamic avatars in real-time with minimal computational resources, usable even in a mobile environment. With the rising interest in personalized VR and gaming experiences comes the need to create high quality 3D avatars that are both low-cost and varie-gated. Due to this, building dynamic avatars from a single unconstrained input image is becoming a popular application. While previous techniques that attempt this require multiple input images or rely on transferring dynamic facial appearance from a source actor, we are able to do so using only one 2D input image without any form of transfer from a source image. We achieve this using a new conditional Generative Adversarial Network design that allows ï¿¿ne-scale manipulation of any facial input image into a new expression while preserving its identity. Our photoreal avatar GAN (paGAN) can also synthesize the unseen mouth interior and control the eye-gaze direction of the output, as well as produce the ï¿¿nal image from a novel viewpoint. The method is even capable of generating fully-controllable temporally stable video sequences, despite not using temporal information during training. After training, we can use our network to produce dynamic image-based avatars that are controllable on mobile devices in real time. To do this, we compute a ï¿¿xed set of output images that correspond to key blendshapes, from which we extract textures in UV space. Using a subject's expression blendshapes at run-time, we can linearly blend these key textures together to achieve the desired appearance. Furthermore, we can use the mouth interior and eye textures produced by our network to synthesize on-the-ï¿¿y avatar animations for those regions. Our work produces state-of-the-art quality image and video synthesis, and is the ï¿¿rst to our knowledge that is able to generate a dynamically textured avatar with a mouth interior, all from a single image.
HMD (CAD model) interior (CAD model) online operation RGB-D camera strain sensors foam liner faci... more HMD (CAD model) interior (CAD model) online operation RGB-D camera strain sensors foam liner facial performance capture Figure 1: To enable immersive face-to-face communication in virtual worlds, the facial expressions of a user have to be captured while wearing a virtual reality head-mounted display. Because the face is largely occluded by typical wearable displays, we have designed an HMD that combines ultra-thin strain sensors with a head-mounted RGB-D camera for real-time facial performance capture and animation. Abstract There are currently no solutions for enabling direct face-to-face interaction between virtual reality (VR) users wearing head-mounted displays (HMDs). The main challenge is that the headset obstructs a significant portion of a user's face, preventing effective facial capture with traditional techniques. To advance virtual reality as a next-generation communication platform, we develop a novel HMD that enables 3D facial performance-driven animation in real-time. Our wearable system uses ultra-thin flexible electronic materials that are mounted on the foam liner of the headset to measure surface strain signals corresponding to upper face expressions. These strain signals are combined with a head-mounted RGB-D camera to enhance the tracking in the mouth region and to account for inaccurate HMD placement. To map the input signals to a 3D face model, we perform a single-instance offline training session for each person. For reusable and accurate online operation, we propose a short calibration step to readjust the Gaussian mixture distribution of the mapping before each use. The resulting animations are visually on par with cutting-edge depth sensor-driven facial performance capture systems and hence, are suitable for social interactions in virtual worlds.
Fig. 1. FLAME example. Top: Samples of the D3DFACS dataset. Middle: Model-only registration. Bott... more Fig. 1. FLAME example. Top: Samples of the D3DFACS dataset. Middle: Model-only registration. Bottom: Expression transfer to Beeler et al. [2011] subject using model only. The field of 3D face modeling has a large gap between high-end and low-end methods. At the high end, the best facial animation is indistinguishable from real humans, but this comes at the cost of extensive manual labor. At the low end, face capture from consumer depth sensors relies on 3D face models that are not expressive enough to capture the variability in natural facial shape and expression. We seek a middle ground by learning a facial model from thousands of accurately aligned 3D scans. Our FLAME model (Faces Learned with an Articulated Model and Expressions) is designed to work with existing graphics software and be easy to fit to data. FLAME uses a linear shape space trained from 3800 scans of human heads. FLAME combines this linear shape space with an articulated jaw, neck, and eyeballs, pose-dependent corrective blendshapes, and additional global expression * Both authors contributed equally to the paper † This research was performed while TL and JR were at the MPI for Intelligent Systems. blendshapes. The pose and expression dependent articulations are learned from 4D face sequences in the D3DFACS dataset along with additional 4D sequences. We accurately register a template mesh to the scan sequences and make the D3DFACS registrations available for research purposes. In total the model is trained from over 33, 000 scans. FLAME is low-dimensional but more expressive than the FaceWarehouse model and the Basel Face Model. We compare FLAME to these models by fitting them to static 3D scans and 4D sequences using the same optimization method. FLAME is significantly more accurate and is available for research purposes (http://flame.is.tue.mpg.de).
input scans reconstruction input scans reconstruction input scans reconstruction Figure 1: Recons... more input scans reconstruction input scans reconstruction input scans reconstruction Figure 1: Reconstruction of complex deforming objects from high-resolution depth scans. Our method accurately captures the global topology and shape motion, as well as dynamic, small-scale details, such as wrinkles and folds. Abstract We present a framework and algorithms for robust geometry and motion reconstruction of complex deforming shapes. Our method makes use of a smooth template that provides a crude approximation of the scanned object and serves as a geometric and topological prior for reconstruction. Large-scale motion of the acquired object is recovered using a novel space-time adaptive, non-rigid registration method. Fine-scale details such as wrinkles and folds are synthesized with an efficient linear mesh deformation algorithm. Subsequent spatial and temporal filtering of detail coefficients allows transfer of persistent geometric detail to regions not observed by the scanner. We show how this two-scale process allows faithful recovery of small-scale shape and motion features leading to a high-quality reconstruction. We illustrate the robustness and generality of our algorithm on a variety of examples composed of different materials and exhibiting a large range of dynamic deformations.
Figure 1: Our method recovers a sequence of high-quality, temporally coherent triangle meshes fro... more Figure 1: Our method recovers a sequence of high-quality, temporally coherent triangle meshes from any sequence of closed surfaces with arbitrarily changing topology. We reliably extract correspondences from a level set and track textures backwards through a fluid simulation. Abstract We present a method for recovering a temporally coherent, deforming triangle mesh with arbitrarily changing topology from an incoherent sequence of static closed surfaces. We solve this problem using the surface geometry alone, without any prior information like surface templates or velocity fields. Our system combines a proven strategy for triangle mesh improvement, a robust multi-resolution non-rigid registration routine, and a reliable technique for changing surface mesh topology. We also introduce a novel topological constraint enforcement algorithm to ensure that the output and input always have similar topology. We apply our technique to a series of diverse input data from video reconstructions, physics simulations , and artistic morphs. The structured output of our algorithm allows us to efficiently track information like colors and displacement maps, recover velocity information, and solve PDEs on the mesh as a post process.
EPFL Figure 1: Our system captures and tracks the facial expression dynamics of the users (grey r... more EPFL Figure 1: Our system captures and tracks the facial expression dynamics of the users (grey renderings) in realtime and maps them to a digital character (colored renderings) on the opposite screen to enable engaging virtual encounters in cyberspace. Abstract This paper presents a system for performance-based character animation that enables any user to control the facial expressions of a digital avatar in realtime. The user is recorded in a natural environment using a non-intrusive, commercially available 3D sensor. The simplicity of this acquisition device comes at the cost of high noise levels in the acquired data. To effectively map low-quality 2D images and 3D depth maps to realistic facial expressions, we introduce a novel face tracking algorithm that combines geometry and texture registration with pre-recorded animation priors in a single optimization. Formulated as a maximum a posteriori estimation in a reduced parameter space, our method implicitly exploits temporal coherence to stabilize the tracking. We demonstrate that compelling 3D facial dynamics can be reconstructed in realtime without the use of face markers, intrusive lighting, or complex scanning hardware. This makes our system easy to deploy and facilitates a range of new applications, e.g. in digital gameplay or social interactions.
We present a novel shape completion technique for creating temporally coherent watertight surface... more We present a novel shape completion technique for creating temporally coherent watertight surfaces from real-time captured dynamic performances. Because of occlusions and low surface albedo, scanned mesh sequences typically exhibit large holes that persist over extended periods of time. Most conventional dynamic shape reconstruction techniques rely on template models or assume slow deformations in the input data. Our framework sidesteps these requirements and directly initializes shape completion with topology derived from the visual hull. To seal the holes with patches that are consistent with the subject's motion, we first minimize surface bending energies in each frame to ensure smooth transitions across hole boundaries. Temporally coherent dynamics of surface patches are obtained by unwarp-ing all frames within a time window using accurate inter-frame correspondences. Aggregated surface samples are then filtered with a temporal visibility kernel that maximizes the use of non-occluded surfaces. A key benefit of our shape completion strategy is that it does not rely on long-range correspondences or a template model. Consequently, our method does not suffer from error accumulation typically introduced by noise, large deformations , and drastic topological changes. We illustrate the effectiveness of our method on several high-resolution scans of human performances captured with a state-of-the-art multi-view 3D acquisition system.
Figure 1: Our system reconstructs a temporally coherent set of hair fibers for real-world dynamic... more Figure 1: Our system reconstructs a temporally coherent set of hair fibers for real-world dynamic hair. It accommodates a variety of hair types and styles, as well as nontrivial motion (top: input video, middle: reconstructed envelope surface, bottom: synthesized hair strands). Abstract The realistic reconstruction of hair motion is challenging because of hair's complex occlusion, lack of a well-defined surface, and non-Lambertian material. We present a system for passive capture of dynamic hair performances using a set of high-speed video cameras. Our key insight is that, while hair color is unlikely to match across multiple views, the response to oriented filters will. We combine a multi-scale version of this orientation-based matching metric with bilateral aggregation, a MRF-based stereo reconstruction technique , and algorithms for temporal tracking and de-noising. Our final output is a set of hair strands for each frame, grown according to the per-frame reconstructed rough geometry and orientation field. We demonstrate results for a number of hair styles ranging from smooth and ordered to curly and messy.
Adobe Research (a) reference photo (b) user sketches (c) example hairstyles (d) large-scale struc... more Adobe Research (a) reference photo (b) user sketches (c) example hairstyles (d) large-scale structure (e) output strands (f) another view Figure 1: Our system takes as input a reference photo (a), a few user strokes (b) and a database of example hairstyles (c) to model the 3D target hairstyle (e). The retrieved examples that best match the user strokes are highlighted with the corresponding colors in (c) which are combined consistently as shown in (d). Original image courtesy of Yung-Yuan Kao. Abstract Human hair presents highly convoluted structures and spans an extraordinarily wide range of hairstyles, which is essential for the digitization of compelling virtual avatars but also one of the most challenging to create. Cutting-edge hair modeling techniques typically rely on expensive capture devices and significant manual labor. We introduce a novel data-driven framework that can digitize complete and highly complex 3D hairstyles from a single-view photograph. We first construct a large database of manually crafted hair models from several online repositories. Given a reference photo of the target hairstyle and a few user strokes as guidance, we automatically search for multiple best matching examples from the database and combine them consistently into a single hairstyle to form the large-scale structure of the hair model. We then synthesize the final hair strands by jointly optimizing for the projected 2D similarity to the reference photo, the physical plausibility of each strand, as well as the local orientation coherency between neighboring strands. We demonstrate the effectiveness and robustness of our method on a variety of hairstyles and challenging images, and compare our system with state-of-the-art hair modeling algorithms.
input image face mesh and hair polystrips 3D avatar input image face mesh and hair polystrips 3D ... more input image face mesh and hair polystrips 3D avatar input image face mesh and hair polystrips 3D avatar input image face mesh and hair polystrips 3D avatar Fig. 1. We introduce an end-to-end framework for modeling a complete 3D avatar from a single input image for real-time rendering. We infer fully rigged textured faces models and polygonal strips for hair. Our flexible and efficient mesh-based hair representation is suitable for a wide range of hairstyles and can be readily integrated into existing real-time game engines. All of the illustrations are rendered in realtime in Unity. President Trump's picture is obtained from whitehouse.gov and Kim Jong-un's photograph was published in the Rodong Sinmun. The other celebrity pictures are used with permission from Getty Images. We present a fully automatic framework that digitizes a complete 3D head with hair from a single unconstrained image. Our system offers a practical and consumer-friendly end-to-end solution for avatar personalization in gaming and social VR applications. The reconstructed models include secondary components (eyes, teeth, tongue, and gums) and provide animation-friendly blendshapes and joint-based rigs. While the generated face is a high-quality textured mesh, we propose a versatile and efficient polygonal strips (polystrips) representation for the hair. Polystrips are suitable for an extremely wide range of hairstyles and textures and are compatible with existing game engines for real-time rendering. In addition to integrating state-of-the-art advances in facial shape modeling and appearance inference , we propose a novel single-view hair generation pipeline, based on * indicates equal contribution Permission to make digitalmodel and texture retrieval, shape refinement, and polystrip patching optimization. The performance of our hairstyle retrieval is enhanced using a deep convolutional neural network for semantic hair attribute classification. Our generated models are visually comparable to state-of-the-art game characters designed by professional artists. For real-time settings, we demonstrate the flexibility of polystrips in handling hairstyle variations, as opposed to conventional strand-based representations. We further show the effectiveness of our approach on a large number of images taken in the wild, and how compelling avatars can be easily created by anyone.
Figure 1: A live demonstration of our system. We are able to obtain high-fidelity animations of t... more Figure 1: A live demonstration of our system. We are able to obtain high-fidelity animations of the user's facial expressions in real-time using convolutional neural net regressors. Left: a user wearing our prototype system, which uses cameras attached to the HMD to track the user's eye and mouth movements. Right: a digital avatar controlled by the user. Abstract Significant challenges currently prohibit expressive interaction in virtual reality (VR). Occlusions introduced by head-mounted displays (HMDs) make existing facial tracking techniques intractable, and even state-of-the-art techniques used for real-time facial tracking in unconstrained environments fail to capture subtle details of the user's facial expressions that are essential for compelling speech animation. We introduce a novel system for HMD users to control a digital avatar in real-time while producing plausible speech animation and emotional expressions. Using a monocular camera attached to an HMD, we record multiple subjects performing various facial expressions and speaking several phonetically-balanced sentences. These images are used with artist-generated animation data corresponding to these sequences to train a convolutional neural network (CNN) to regress images of a user's mouth region to the parameters that control a digital avatar. To make training this system more tractable, we use audio-based alignment techniques to map images of multiple users making the same utterance to the corresponding animation parameters. We demonstrate that this approach is also feasible for tracking the expressions around the user's eye region with an internal infrared (IR) camera, thereby enabling full facial tracking. This system requires no user-specific calibration, uses easily obtainable consumer hardware, and produces high-quality animations of speech and emotional expressions. Finally, we demonstrate the quality of our system on a variety of subjects and evaluate its performance against state-of-the-art real-time facial tracking techniques.
Three-dimensional registration is the process of aligning scans of different incomplete views of ... more Three-dimensional registration is the process of aligning scans of different incomplete views of an object such that their corresponding regions agree in space and time. It is used to build digital complete models from partial acquisitions of real-world objects which typically contain noise, outliers, and acquisition holes due to occlusions and hardware limitations. For non-rigid models, such as humans, shapes of corresponding regions undergo complex deformations in addition to misalignments. Thus, finding explicit correspondences between two dissimilar shapes becomes considerably more challenging. In addition, finding the right deformations is also non-trivial as they depend on the correspondences which are only partially available between consecutive scans. This report presents an analysis of the 3-D registration problem, a general approach for the spatio-temporal registration of deformable scans that are acquired with a fast 3-D scanner, and extensive experiments on synthetic and real acquisition data. The objective is to un-deform the surfaces of all range maps and to fit them into a particular instance of time.
(a) An image of (b) Point cloud with (c) Failure of (d) Color-coded (e) Synthesized (f) A frame f... more (a) An image of (b) Point cloud with (c) Failure of (d) Color-coded (e) Synthesized (f) A frame from captured hairstyle 3D orientation field previous method reconstructed wisps hair strands hair simulation Figure 1: Our system takes a collection of images as input (a) and reconstructs a point cloud with a 3D orientation field (b). In contrast to previous methods (e.g. [Paris et al. 2008]) that straightforwardly grow hair strands from the scalp following the orientation field and hence cannot reconstruct complex hairstyles with convoluted curl structures (c), we reconstruct complete, coherent and plausible wisps (d) aware of the underlying hair structures. The wisps can be used to synthesize hair strands (e) that are plausible for animation or simulation (f). Abstract Existing hair capture systems fail to produce strands that reflect the structures of real-world hairstyles. We introduce a system that reconstructs coherent and plausible wisps aware of the underlying hair structures from a set of still images without any special lighting. Our system first discovers locally coherent wisp structures in the reconstructed point cloud and the 3D orientation field, and then uses a novel graph data structure to reason about both the connec-tivity and directions of the local wisp structures in a global optimization. The wisps are then completed and used to synthesize hair strands which are robust against occlusion and missing data and plausible for animation and simulation. We show reconstruction results for a variety of complex hairstyles including curly, wispy, and messy hair.
5 Artec Group 3D scanning pose change output reconstruction textured reconstruction large variety... more 5 Artec Group 3D scanning pose change output reconstruction textured reconstruction large variety of examples 3D print Figure 1: With our system, users can scan themselves with a single 3D sensor by rotating the same pose for a few different views (typically eight, ⇠45 degrees apart) to cover the full body. Our method robustly registers and merges different scans into a watertight surface with consistent texture in spite of shape changes during repositioning, and lighting differences between the scans. These surfaces are suitable for applications such as online avatars or 3D printing (the miniature shown here was printed using a ZPrinter 650.) Abstract We develop an automatic pipeline that allows ordinary users to capture complete and fully textured 3D models of themselves in minutes , using only a single Kinect sensor, in the uncontrolled lighting environment of their own home. Our method requires neither a turntable nor a second operator, and is robust to the small deformations and changes of pose that inevitably arise during scanning. After the users rotate themselves with the same pose for a few scans from different views, our system stitches together the captured scans using multi-view non-rigid registration, and produces watertight final models. To ensure consistent texturing, we recover the underlying albedo from each scanned texture and generate seamless global textures using Poisson blending. Despite the minimal requirements we place on the hardware and users, our method is suitable for full body capture of challenging scenes that cannot be handled well using previous methods, such as those involving loose clothing, complex poses, and props.
Extracting range data through active optical triangulation requires robust stripe edge detection ... more Extracting range data through active optical triangulation requires robust stripe edge detection of the emitted pattern because of undesired shape and shading variations in the scanned object. We propose several extensions to a struc-tured light system originally proposed by Zhang et al. [1]. Assuming stripes of a certain width, our edge detection criterion is achieved simply by restricting the number of consecutive edge pixels. Subpixel accuracy from one single input image can be obtained by approximating the gradient of the square local contrast with a piecewise linear function. Finally, we present a noise reduction technique for the range map through meshing followed by a parameterized face orientation culling. Experimental results have demonstrated major improvements in terms of robustness against acquisition noise, shading variations and complex shapes.
(a) no microstructure (b) static microstructure (c) dynamic microstructure (d) photograph Figure ... more (a) no microstructure (b) static microstructure (c) dynamic microstructure (d) photograph Figure 1: A rendered facial expression with (a) mesostructure only (b) static microstructure from a neutral expression (c) dynamic mi-crostructure from convolving the neutral microstructure according to local surface strain compared to a reference photograph of the similar expression. The insets show detail from the lower-left area. Abstract We present a technique for synthesizing the effects of skin mi-crostructure deformation by anisotropically convolving a high-resolution displacement map to match normal distribution changes in measured skin samples. We use a 10-micron resolution scanning technique to measure several in vivo skin samples as they are stretched and compressed in different directions, quantifying how stretching smooths the skin and compression makes it rougher. We tabulate the resulting surface normal distributions, and show that convolving a neutral skin microstructure displacement map with blurring and sharpening filters can mimic normal distribution changes and microstructure deformations. We implement the spatially-varying displacement map filtering on the GPU to interactively render the effects of dynamic microgeometry on animated faces obtained from high-resolution facial scans.
(a) input photo (b) input mesh (c) extracted structure (d) output strands (e) another view of (a)... more (a) input photo (b) input mesh (c) extracted structure (d) output strands (e) another view of (a) (f) another view of (d) Figure 1: Capturing a five-strand Dutch braid. We capture the braided hairstyle (a) using a Kinect sensor and obtain an input mesh with a local 3D orientation for each vertex (b). Based on the information provided by the example patches in a database, we extract the centerlines (c) of the braid structure to synthesize final output strands (d). (e) and (f) show the input reference photo and our output strands from another viewpoint. Abstract From fishtail to princess braids, these intricately woven structures define an important and popular class of hairstyle, frequently used for digital characters in computer graphics. In addition to the challenges created by the infinite range of styles, existing modeling and capture techniques are particularly constrained by the geometric and topological complexities. We propose a data-driven method to automatically reconstruct braided hairstyles from input data obtained from a single consumer RGB-D camera. Our approach covers the large variation of repetitive braid structures using a family of compact procedural braid models. From these models, we produce a database of braid patches and use a robust random sampling approach for data fitting. We then recover the input braid structures using a multi-label optimization algorithm and synthesize the intertwining hair strands of the braids. We demonstrate that a minimal capture equipment is sufficient to effectively capture a wide range of complex braids with distinct shapes and structures.
Figure 1: Our system takes as input a few images (a) and employs a database of simulated example ... more Figure 1: Our system takes as input a few images (a) and employs a database of simulated example strands (b) to discover structurally plausible configurations from the reconstructed cover strands (c) for final strand synthesis (d). Our method robustly fits example strands to the cover strands which are computed from unprocessed outlier affected input data (e) to generate compelling reconstruction results (f). In contrast, the state-of-the-art method of [Luo et al. 2013a] fails in the presence of strong outliers (g). Abstract We introduce a data-driven hair capture framework based on example strands generated through hair simulation. Our method can robustly reconstruct faithful 3D hair models from unprocessed input point clouds with large amounts of outliers. Current state-of-the-art techniques use geometrically-inspired heuristics to derive global hair strand structures, which can yield implausible hair strands for hairstyles involving large occlusions, multiple layers, or wisps of varying lengths. We address this problem using a voting-based fitting algorithm to discover structurally plausible configurations among the locally grown hair segments from a database of simulated examples. To generate these examples, we exhaustively sample the simulation configurations within the feasible parameter space constrained by the current input hairstyle. The number of necessary simulations can be further reduced by leveraging symmetry and constrained initial conditions. The final hairstyle can then be structurally represented by a limited number of examples. To handle constrained hairstyles such as a ponytail of which realistic simulations are more difficult, we allow the user to sketch a few strokes to generate strand examples through an intuitive interface. Our approach focuses on robustness and generality. Since our method is structurally plausible by construction, we ensure an improved control during hair digitization and avoid implausible hair synthesis for a wide range of hairstyles.
Fig. 1. Our system infers high-fidelity facial reflectance and geometry maps from a single image ... more Fig. 1. Our system infers high-fidelity facial reflectance and geometry maps from a single image (diffuse albedo, specular albedo, as well as medium-and high-frequency displacements). These maps can be used for high-fidelity rendering under novel illumination conditions. We present a deep learning-based technique to infer high-quality facial reflectance and geometry given a single unconstrained image of the subject , which may contain partial occlusions and arbitrary illumination conditions. The reconstructed high-resolution textures, which are generated in only a few seconds, include high-resolution skin surface reflectance maps, representing both the diffuse and specular albedo, and medium-and high-frequency displacement maps, thereby allowing us to render compelling digital avatars under novel lighting conditions. To extract this data, we train our deep neural networks with a high-quality skin reflectance and geometry database created with a state-of-the-art multi-view photometric stereo system using polarized gradient illumination. Given the raw facial texture map extracted from the input image, our neural networks synthesize complete * indicates equal contribution Permission to make digital reflectance and displacement maps, as well as complete missing regions caused by occlusions. The completed textures exhibit consistent quality throughout the face due to our network architecture, which propagates texture features from the visible region, resulting in high-fidelity details that are consistent with those seen in visible regions. We describe how this highly underconstrained problem is made tractable by dividing the full inference into smaller tasks, which are addressed by dedicated neural networks. We demonstrate the effectiveness of our network design with robust texture completion from images of faces that are largely occluded. With the inferred reflectance and geometry data, we demonstrate the rendering of high-fidelity 3D avatars from a variety of subjects captured under different lighting conditions. In addition, we perform evaluations demonstrating that our method can infer plausible facial reflectance and geometric details comparable to those obtained from high-end capture devices, and outperform alternative approaches that require only a single unconstrained input image.
Figure 1: Our adaptive tracking model conforms to the input expressions on-the-fly, producing a b... more Figure 1: Our adaptive tracking model conforms to the input expressions on-the-fly, producing a better fit to the user than state-of-the-art data driven techniques [Weise et al. 2011] which are confined to learned motion priors and generate plausible but not accurate tracking. Abstract We introduce a real-time and calibration-free facial performance capture framework based on a sensor with video and depth input. In this framework, we develop an adaptive PCA model using shape correctives that adjust on-the-fly to the actor's expressions through incremental PCA-based learning. Since the fitting of the adaptive model progressively improves during the performance, we do not require an extra capture or training session to build this model. As a result, the system is highly deployable and easy to use: it can faithfully track any individual, starting from just a single face scan of the subject in a neutral pose. Like many real-time methods, we use a linear subspace to cope with incomplete input data and fast motion. To boost the training of our tracking model with reliable samples, we use a well-trained 2D facial feature tracker on the input video and an efficient mesh deformation algorithm to snap the result of the previous step to high frequency details in visible depth map regions. We show that the combination of dense depth maps and texture features around eyes and lips is essential in capturing natural dialogues and nuanced actor-specific emotions. We demonstrate that using an adaptive PCA model not only improves the fitting accuracy for tracking but also increases the expressiveness of the retargeted character.
input image synthesized key expressions image-based dynamic 3D avatar real-time rendering on mobi... more input image synthesized key expressions image-based dynamic 3D avatar real-time rendering on mobile Fig. 1. Overview of our system. Using a single neutral-face input image, we are able to synthesize arbitrary expressions both in image space and UV texture space. These generated textures, which include a photoreal mouth interior and the eyes, can then be used to pilot dynamic avatars in real-time with minimal computational resources, usable even in a mobile environment. With the rising interest in personalized VR and gaming experiences comes the need to create high quality 3D avatars that are both low-cost and varie-gated. Due to this, building dynamic avatars from a single unconstrained input image is becoming a popular application. While previous techniques that attempt this require multiple input images or rely on transferring dynamic facial appearance from a source actor, we are able to do so using only one 2D input image without any form of transfer from a source image. We achieve this using a new conditional Generative Adversarial Network design that allows ï¿¿ne-scale manipulation of any facial input image into a new expression while preserving its identity. Our photoreal avatar GAN (paGAN) can also synthesize the unseen mouth interior and control the eye-gaze direction of the output, as well as produce the ï¿¿nal image from a novel viewpoint. The method is even capable of generating fully-controllable temporally stable video sequences, despite not using temporal information during training. After training, we can use our network to produce dynamic image-based avatars that are controllable on mobile devices in real time. To do this, we compute a ï¿¿xed set of output images that correspond to key blendshapes, from which we extract textures in UV space. Using a subject's expression blendshapes at run-time, we can linearly blend these key textures together to achieve the desired appearance. Furthermore, we can use the mouth interior and eye textures produced by our network to synthesize on-the-ï¿¿y avatar animations for those regions. Our work produces state-of-the-art quality image and video synthesis, and is the ï¿¿rst to our knowledge that is able to generate a dynamically textured avatar with a mouth interior, all from a single image.
HMD (CAD model) interior (CAD model) online operation RGB-D camera strain sensors foam liner faci... more HMD (CAD model) interior (CAD model) online operation RGB-D camera strain sensors foam liner facial performance capture Figure 1: To enable immersive face-to-face communication in virtual worlds, the facial expressions of a user have to be captured while wearing a virtual reality head-mounted display. Because the face is largely occluded by typical wearable displays, we have designed an HMD that combines ultra-thin strain sensors with a head-mounted RGB-D camera for real-time facial performance capture and animation. Abstract There are currently no solutions for enabling direct face-to-face interaction between virtual reality (VR) users wearing head-mounted displays (HMDs). The main challenge is that the headset obstructs a significant portion of a user's face, preventing effective facial capture with traditional techniques. To advance virtual reality as a next-generation communication platform, we develop a novel HMD that enables 3D facial performance-driven animation in real-time. Our wearable system uses ultra-thin flexible electronic materials that are mounted on the foam liner of the headset to measure surface strain signals corresponding to upper face expressions. These strain signals are combined with a head-mounted RGB-D camera to enhance the tracking in the mouth region and to account for inaccurate HMD placement. To map the input signals to a 3D face model, we perform a single-instance offline training session for each person. For reusable and accurate online operation, we propose a short calibration step to readjust the Gaussian mixture distribution of the mapping before each use. The resulting animations are visually on par with cutting-edge depth sensor-driven facial performance capture systems and hence, are suitable for social interactions in virtual worlds.
Fig. 1. FLAME example. Top: Samples of the D3DFACS dataset. Middle: Model-only registration. Bott... more Fig. 1. FLAME example. Top: Samples of the D3DFACS dataset. Middle: Model-only registration. Bottom: Expression transfer to Beeler et al. [2011] subject using model only. The field of 3D face modeling has a large gap between high-end and low-end methods. At the high end, the best facial animation is indistinguishable from real humans, but this comes at the cost of extensive manual labor. At the low end, face capture from consumer depth sensors relies on 3D face models that are not expressive enough to capture the variability in natural facial shape and expression. We seek a middle ground by learning a facial model from thousands of accurately aligned 3D scans. Our FLAME model (Faces Learned with an Articulated Model and Expressions) is designed to work with existing graphics software and be easy to fit to data. FLAME uses a linear shape space trained from 3800 scans of human heads. FLAME combines this linear shape space with an articulated jaw, neck, and eyeballs, pose-dependent corrective blendshapes, and additional global expression * Both authors contributed equally to the paper † This research was performed while TL and JR were at the MPI for Intelligent Systems. blendshapes. The pose and expression dependent articulations are learned from 4D face sequences in the D3DFACS dataset along with additional 4D sequences. We accurately register a template mesh to the scan sequences and make the D3DFACS registrations available for research purposes. In total the model is trained from over 33, 000 scans. FLAME is low-dimensional but more expressive than the FaceWarehouse model and the Basel Face Model. We compare FLAME to these models by fitting them to static 3D scans and 4D sequences using the same optimization method. FLAME is significantly more accurate and is available for research purposes (http://flame.is.tue.mpg.de).
input scans reconstruction input scans reconstruction input scans reconstruction Figure 1: Recons... more input scans reconstruction input scans reconstruction input scans reconstruction Figure 1: Reconstruction of complex deforming objects from high-resolution depth scans. Our method accurately captures the global topology and shape motion, as well as dynamic, small-scale details, such as wrinkles and folds. Abstract We present a framework and algorithms for robust geometry and motion reconstruction of complex deforming shapes. Our method makes use of a smooth template that provides a crude approximation of the scanned object and serves as a geometric and topological prior for reconstruction. Large-scale motion of the acquired object is recovered using a novel space-time adaptive, non-rigid registration method. Fine-scale details such as wrinkles and folds are synthesized with an efficient linear mesh deformation algorithm. Subsequent spatial and temporal filtering of detail coefficients allows transfer of persistent geometric detail to regions not observed by the scanner. We show how this two-scale process allows faithful recovery of small-scale shape and motion features leading to a high-quality reconstruction. We illustrate the robustness and generality of our algorithm on a variety of examples composed of different materials and exhibiting a large range of dynamic deformations.
Figure 1: Our method recovers a sequence of high-quality, temporally coherent triangle meshes fro... more Figure 1: Our method recovers a sequence of high-quality, temporally coherent triangle meshes from any sequence of closed surfaces with arbitrarily changing topology. We reliably extract correspondences from a level set and track textures backwards through a fluid simulation. Abstract We present a method for recovering a temporally coherent, deforming triangle mesh with arbitrarily changing topology from an incoherent sequence of static closed surfaces. We solve this problem using the surface geometry alone, without any prior information like surface templates or velocity fields. Our system combines a proven strategy for triangle mesh improvement, a robust multi-resolution non-rigid registration routine, and a reliable technique for changing surface mesh topology. We also introduce a novel topological constraint enforcement algorithm to ensure that the output and input always have similar topology. We apply our technique to a series of diverse input data from video reconstructions, physics simulations , and artistic morphs. The structured output of our algorithm allows us to efficiently track information like colors and displacement maps, recover velocity information, and solve PDEs on the mesh as a post process.
EPFL Figure 1: Our system captures and tracks the facial expression dynamics of the users (grey r... more EPFL Figure 1: Our system captures and tracks the facial expression dynamics of the users (grey renderings) in realtime and maps them to a digital character (colored renderings) on the opposite screen to enable engaging virtual encounters in cyberspace. Abstract This paper presents a system for performance-based character animation that enables any user to control the facial expressions of a digital avatar in realtime. The user is recorded in a natural environment using a non-intrusive, commercially available 3D sensor. The simplicity of this acquisition device comes at the cost of high noise levels in the acquired data. To effectively map low-quality 2D images and 3D depth maps to realistic facial expressions, we introduce a novel face tracking algorithm that combines geometry and texture registration with pre-recorded animation priors in a single optimization. Formulated as a maximum a posteriori estimation in a reduced parameter space, our method implicitly exploits temporal coherence to stabilize the tracking. We demonstrate that compelling 3D facial dynamics can be reconstructed in realtime without the use of face markers, intrusive lighting, or complex scanning hardware. This makes our system easy to deploy and facilitates a range of new applications, e.g. in digital gameplay or social interactions.
We present a novel shape completion technique for creating temporally coherent watertight surface... more We present a novel shape completion technique for creating temporally coherent watertight surfaces from real-time captured dynamic performances. Because of occlusions and low surface albedo, scanned mesh sequences typically exhibit large holes that persist over extended periods of time. Most conventional dynamic shape reconstruction techniques rely on template models or assume slow deformations in the input data. Our framework sidesteps these requirements and directly initializes shape completion with topology derived from the visual hull. To seal the holes with patches that are consistent with the subject's motion, we first minimize surface bending energies in each frame to ensure smooth transitions across hole boundaries. Temporally coherent dynamics of surface patches are obtained by unwarp-ing all frames within a time window using accurate inter-frame correspondences. Aggregated surface samples are then filtered with a temporal visibility kernel that maximizes the use of non-occluded surfaces. A key benefit of our shape completion strategy is that it does not rely on long-range correspondences or a template model. Consequently, our method does not suffer from error accumulation typically introduced by noise, large deformations , and drastic topological changes. We illustrate the effectiveness of our method on several high-resolution scans of human performances captured with a state-of-the-art multi-view 3D acquisition system.
Uploads
Papers by Hao Li