StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation

Roy Or-El1 Xuan Luo1 Mengyi Shan1 Eli Shechtman2

Jeong Joon Park3 Ira Kemelmacher-Shlizerman1
1 2 3
University of Washington Adobe Research Stanford University
arXiv:2112.11427v1 [cs.CV] 21 Dec 2021

Figure 1. Our proposed framework–StyleSDF– learns to jointly generate high resolution, 3D-consistent images (top rows) along with their
detailed view-consistent geometry represented with SDFs (depth maps in bottom rows), while being trained on single view RGB images.

Abstract 1. Introduction

We introduce a high resolution, 3D-consistent image and StyleGAN architectures [35–37] have shown an un-
shape generation technique which we call StyleSDF. Our precedented quality of RGB image generation. They are,
method is trained on single-view RGB data only, and stands however, designed to generate single RGB views rather than
on the shoulders of StyleGAN2 for image generation, while 3D content. In this paper, we introduce StyleSDF, a method
solving two main challenges in 3D-aware GANs: 1) high- for generating 3D-consistent 1024×1024 RGB images and
resolution, view-consistent generation of the RGB images, geometry, trained only on single-view RGB images.
and 2) detailed 3D shape. We achieve this by merging a Related 3D generative models [8, 49, 54, 58, 63] present
SDF-based 3D representation with a style-based 2D gener- shape and appearance synthesis via coordinate-based multi-
ator. Our 3D implicit network renders low-resolution fea- layer-perceptrons (MLP). These works, however, often re-
ture maps, from which the style-based network generates quire 3D or multi-view data for supervision, which are dif-
view-consistent, 1024×1024 images. Notably, our SDF- ficult to collect, or are limited to low-resolution rendering
based 3D modeling defines detailed 3D surfaces, leading outputs as they rely on expensive volumetric field sampling.
to consistent volume rendering. Our method shows higher Without multi-view supervision, 3D-aware GANs [8,49,58]
quality results compared to state of the art in terms of visual typically use opacity fields as geometric proxy, forgoing
and geometric quality.
Project Page: https://stylesdf.github.io/
well-defined surfaces, which results in low-quality depth pervision via fitting their 3D models to the multi-view im-
maps that are inconsistent across views. ages using differentiable rendering [46, 51, 63, 73].
At the core of our architecture lies the SDF-based 3D Two recent seminal breakthroughs are NeRF [46] and
volume renderer and the 2D StyleGAN generator. We use SIREN [62]. NeRF introduced the use of volume render-
a coordinate-based MLP to model Signed Distance Fields ing [32] for reconstructing a 3D scene as a combination of
(SDF) and radiance fields which render low resolution fea- neural radiance and density fields to synthesize novel views.
ture maps. These feature maps are then efficiently trans- SIREN replaced the popular ReLU activation function with
formed into high-resolution images using the StyleGAN sine functions with modulated frequencies, showing great
generator. Our model is trained with an adversarial loss that single scene fitting results. We refer readers to [67] for
encourages the networks to generate realistic images from more comprehensive review.
all sampled viewpoints, and an Eikonal loss that ensures Single-View Supervised 3D-Aware GANs: Rather than
proper SDF modeling. These losses automatically induce relying on 3D or multi-view supervisions, recent ap-
view-consistent, detailed 3D scenes, without 3D or multi- proaches aim at learning a 3D generative model from a
view supervision. The proposed framework effectively ad- set of unconstrained single-view images. These methods
dresses the resolution and the view-inconsistency issues of [8, 16, 25, 30, 42, 47–49, 58] typically optimize their 3D rep-
existing 3D-aware GAN approaches that base on volume resentations to render realistic 2D images from all randomly
rendering. Our system design opens the door for interesting sampled viewpoints using adversarial loss.
future research in vision and graphics that involve a latent Most inline with our work are methods that use implicit
space of high quality shape and appearance. neural radiance fields for 3D-aware image and geometry
Our approach is evaluated on the FFHQ [36] and generation (GRAF [58] and Pi-GAN [8]). However, these
AFHQ [12] datasets. We demonstrate through extensive ex- methods are limited to low-resolution outputs due to the
periments that our system outperforms the state-of-the-art high computational costs of the volume rendering. In ad-
3D-aware methods, measured by the quality of the gener- dition, the use of density fields as proxy for geometry pro-
ated images and surfaces, and their view-consistencies. vides ample amount of leeway for the networks to produce
2. Related Work realistic images while violating 3D consistency, leading to
inconsistent volume rendering w.r.t. the camera viewpoints
In this section, we review related approaches in 2D im- (the rendered RGB or depth images are not 3D-consistent).
age synthesis, 3D generative modeling, and 3D-aware im- ShadeGAN [53] introduces a shading-guided pipeline
age synthesis. which improves the surface quality, but the image output
Generative Adversarial Networks: State-of-the-art Gen- resolution (128×128) is still bounded by the computational
erative Adversarial Networks [19] (GANs) can synthesize burden of the volume rendering. GIRAFFE [49] proposed a
high-resolution RGB images that are practically indistin- dual stage rendering process, where a backbone volume ren-
guishable from real images [34–37]. Substantial work has derer generates low resolution feature maps (16×16) that
been done in order to manipulate the generated images, by are passed to a 2D CNN to generate outputs at resolutions
exploring meaningful latent space directions [1–3, 13, 24, of up to 256 × 256. Despite improved image quality, GI-
28, 59, 60, 64, 65], introducing contrastive learning [61], in- RAFFE outputs lack view consistency, as the hairstyle, fa-
verse graphics [75], examplar images [31] or multiple input cial expression, and sometimes the object’s identity, are en-
views [40]. While 2D latent space manipulation produces tangled with the camera viewpoint inputs, likely because 3D
realistic results, these methods tend to lack explicit cam- outputs at 16×16 are not descriptive enough.
era control, have no 3D understanding, require shape priors StyleNeRF [23] and CIPS-3D [76] are concurrent works
from 3DMM models [64, 65], or reconstruct the surface as that adopt the two-stage rendering process or a smart sam-
a preprocessing step [40]. pling for high-resolution image generation, yet these works
Coordinate-based 3D Models: While multiple 3D rep- still do not model well-defined, view-consistent 3D geome-
resentations have been proposed for generative modeling try.
[22, 70, 71], recent coordinate-based neural implicit mod-
els [9, 44, 54] stand out as an efficient, expressive, and dif- 3. Algorithm
ferentiable representation.
3.1. Overview
Neural implicit representations (NIR) have been widely
adopted for learning shape and appearance of objects [4,10, Our framework consists of two main components. A
14, 20, 45, 50, 52, 56, 57], local parts [17, 18], and full 3D backbone conditional SDF volume renderer, and a 2D style-
scenes [7, 11, 29, 55] from explicit 3D supervisions. More- based generator [37]. Each component also has an accom-
over, NIR approaches have been shown to be a powerful panied mapping network [36] to map the input latent vector
tool for reconstructing 3D structure from multi-view 2D su- into modulation signals for each layer. An overview of our
 Overall Architecture SDF Volume Renderer <latexit sha1_base64="gMTgjs7J9T7tfl8I4J/4iGZ8J/U=">AAAB8XicbVDLSgMxFL1TX7W+qi7dBIvgqsyIqMuiG5cV7APbUjLpnTY0kxmSjFiG/oUbF4q49W/c+Tdm2llo64HA4Zx7ybnHjwXXxnW/ncLK6tr6RnGztLW9s7tX3j9o6ihRDBssEpFq+1Sj4BIbhhuB7VghDX2BLX98k/mtR1SaR/LeTGLshXQoecAZNVZ66IbUjPwgfZr2yxW36s5AlomXkwrkqPfLX91BxJIQpWGCat3x3Nj0UqoMZwKnpW6iMaZsTIfYsVTSEHUvnSWekhOrDEgQKfukITP190ZKQ60noW8ns4R60cvE/7xOYoKrXsplnBiUbP5RkAhiIpKdTwZcITNiYgllitushI2ooszYkkq2BG/x5GXSPKt6F9Xzu/NK7TqvowhHcAyn4MEl1OAW6tAABhKe4RXeHO28OO/Ox3y04OQ7h/AHzucPADaRJQ==</latexit>

x
v x
<latexit sha1_base64="aVFEhD40Trme9YZH9TGgcE9wMKs=">AAACCXicbVDLSgMxFM3UV62vUZdugkWoIGVGirosunElFewDOqVk0kwbmmSGJCPUYbZu/BU3LhRx6x+482/MtLPQ1gOBk3Pu5d57/IhRpR3n2yosLa+srhXXSxubW9s79u5eS4WxxKSJQxbKjo8UYVSQpqaakU4kCeI+I21/fJX57XsiFQ3FnZ5EpMfRUNCAYqSN1Lehx5Ee+UHykEJPUT77Y8SSm7TinLjHfbvsVJ0p4CJxc1IGORp9+8sbhDjmRGjMkFJd14l0L0FSU8xIWvJiRSKEx2hIuoYKxInqJdNLUnhklAEMQmme0HCq/u5IEFdqwn1Tme2p5r1M/M/rxjq46CVURLEmAs8GBTGDOoRZLHBAJcGaTQxBWFKzK8QjJBHWJrySCcGdP3mRtE6r7lm1dlsr1y/zOIrgAByCCnDBOaiDa9AATYDBI3gGr+DNerJerHfrY1ZasPKeffAH1ucPVvyZcw==</latexit>

View Direction
<latexit sha1_base64="GRuxbeGZdJOlSIDiXsmmlxIc3sY=">AAAB8XicbVDLSgMxFL1TX7W+qi7dBIvgqsyIqMuiG5cV7APbUjLpnTY0kxmSTKEM/Qs3LhRx69+482/MtLPQ1gOBwzn3knOPHwuujet+O4W19Y3NreJ2aWd3b/+gfHjU1FGiGDZYJCLV9qlGwSU2DDcC27FCGvoCW/74LvNbE1SaR/LRTGPshXQoecAZNVZ66obUjPwgncz65Ypbdecgq8TLSQVy1Pvlr+4gYkmI0jBBte54bmx6KVWGM4GzUjfRGFM2pkPsWCppiLqXzhPPyJlVBiSIlH3SkLn6eyOlodbT0LeTWUK97GXif14nMcFNL+UyTgxKtvgoSAQxEcnOJwOukBkxtYQyxW1WwkZUUWZsSSVbgrd88ippXlS9q+rlw2WldpvXUYQTOIVz8OAaanAPdWgAAwnP8ApvjnZenHfnYzFacPKdY/gD5/MH/R2RIw==</latexit> <latexit sha1_base64="gMTgjs7J9T7tfl8I4J/4iGZ8J/U=">AAAB8XicbVDLSgMxFL1TX7W+qi7dBIvgqsyIqMuiG5cV7APbUjLpnTY0kxmSjFiG/oUbF4q49W/c+Tdm2llo64HA4Zx7ybnHjwXXxnW/ncLK6tr6RnGztLW9s7tX3j9o6ihRDBssEpFq+1Sj4BIbhhuB7VghDX2BLX98k/mtR1SaR/LeTGLshXQoecAZNVZ66IbUjPwgfZr2yxW36s5AlomXkwrkqPfLX91BxJIQpWGCat3x3Nj0UqoMZwKnpW6iMaZsTIfYsVTSEHUvnSWekhOrDEgQKfukITP190ZKQ60noW8ns4R60cvE/7xOYoKrXsplnBiUbP5RkAhiIpKdTwZcITNiYgllitushI2ooszYkkq2BG/x5GXSPKt6F9Xzu/NK7TqvowhHcAyn4MEl1OAW6tAABhKe4RXeHO28OO/Ox3y04OQ7h/AHzucPADaRJQ==</latexit>

z ∼ N (0, 1) Architecture
and Position

SDF Renderer
Volume Mapping !: Feature vector

Renderer Network
": RGB radiance
#: SDF value
MLP
Network
w
<latexit sha1_base64="rdxiPIdagr8cZ+NC4T6aAUakhZM=">AAAB+XicbVDLSsNAFL2pr1pfUZduBovgqiQi6rLoxmUF+4A2hMl00g6dTMLMpFJC/sSNC0Xc+ifu/BsnbRbaemDgcM693DMnSDhT2nG+rcra+sbmVnW7trO7t39gHx51VJxKQtsk5rHsBVhRzgRta6Y57SWS4ijgtBtM7gq/O6VSsVg86llCvQiPBAsZwdpIvm0PIqzHQZg95X42jXnu23Wn4cyBVolbkjqUaPn212AYkzSiQhOOleq7TqK9DEvNCKd5bZAqmmAywSPaN1TgiCovmyfP0ZlRhiiMpXlCo7n6eyPDkVKzKDCTRU617BXif14/1eGNlzGRpJoKsjgUphzpGBU1oCGTlGg+MwQTyUxWRMZYYqJNWTVTgrv85VXSuWi4V43Lh8t687asowoncArn4MI1NOEeWtAGAlN4hld4szLrxXq3PhajFavcOYY/sD5/AHszlDk=</latexit>

σ: Density Value

64x64
Image 2D feature
w
<latexit sha1_base64="rdxiPIdagr8cZ+NC4T6aAUakhZM=">AAAB+XicbVDLSsNAFL2pr1pfUZduBovgqiQi6rLoxmUF+4A2hMl00g6dTMLMpFJC/sSNC0Xc+ifu/BsnbRbaemDgcM693DMnSDhT2nG+rcra+sbmVnW7trO7t39gHx51VJxKQtsk5rHsBVhRzgRta6Y57SWS4ijgtBtM7gq/O6VSsVg86llCvQiPBAsZwdpIvm0PIqzHQZg95X42jXnu23Wn4cyBVolbkjqUaPn212AYkzSiQhOOleq7TqK9DEvNCKd5bZAqmmAywSPaN1TgiCovmyfP0ZlRhiiMpXlCo7n6eyPDkVKzKDCTRU617BXif14/1eGNlzGRpJoKsjgUphzpGBU1oCGTlGg+MwQTyUxWRMZYYqJNWTVTgrv85VXSuWi4V43Lh8t687asowoncArn4MI1NOEeWtAGAlN4hld4szLrxXq3PhajFavcOYY/sD5/AHszlDk=</latexit>

map FC Layer FC Layer v

<latexit sha1_base64="GRuxbeGZdJOlSIDiXsmmlxIc3sY=">AAAB8XicbVDLSgMxFL1TX7W+qi7dBIvgqsyIqMuiG5cV7APbUjLpnTY0kxmSTKEM/Qs3LhRx69+482/MtLPQ1gOBwzn3knOPHwuujet+O4W19Y3NreJ2aWd3b/+gfHjU1FGiGDZYJCLV9qlGwSU2DDcC27FCGvoCW/74LvNbE1SaR/LRTGPshXQoecAZNVZ66obUjPwgncz65Ypbdecgq8TLSQVy1Pvlr+4gYkmI0jBBte54bmx6KVWGM4GzUjfRGFM2pkPsWCppiLqXzhPPyJlVBiSIlH3SkLn6eyOlodbT0LeTWUK97GXif14nMcFNL+UyTgxKtvgoSAQxEcnOJwOukBkxtYQyxW1WwkZUUWZsSSVbgrd88ippXlS9q+rlw2WldpvXUYQTOIVz8OAaanAPdWgAAwnP8ApvjnZenHfnYzFacPKdY/gD5/MH/R2RIw==</latexit>

Low-Res
<latexit sha1_base64="g95WM2rz4TJpo/GyZZX0s2M1hGQ=">AAAB9HicbVDLSsNAFL2pr1pfVZduBotQNyWRoi6LblxWsA9oQ5lMJu3QySTOTIol9DvcuFDErR/jzr9x0mahrQcGDufcyz1zvJgzpW372yqsrW9sbhW3Szu7e/sH5cOjtooSSWiLRDySXQ8rypmgLc00p91YUhx6nHa88W3mdyZUKhaJBz2NqRvioWABI1gbyfWr/RDrkRekT7PzQbli1+w50CpxclKBHM1B+avvRyQJqdCEY6V6jh1rN8VSM8LprNRPFI0xGeMh7RkqcEiVm85Dz9CZUXwURNI8odFc/b2R4lCpaeiZySyiWvYy8T+vl+jg2k2ZiBNNBVkcChKOdISyBpDPJCWaTw3BRDKTFZERlpho01PJlOAsf3mVtC9qzmWtfl+vNG7yOopwAqdQBQeuoAF30IQWEHiEZ3iFN2tivVjv1sditGDlO8fwB9bnD4wnkfg=</latexit>

Generator d(x) <latexit sha1_base64="Tq+Z5G/bIypVVj/HlzCSIKs1HMM=">AAACAXicbZDLSsNAFIZP6q3WW9SN4GawCBWkJFLUZdGNywr2Am0ok+mkHTq5MDMpllA3voobF4q49S3c+TZO2gja+sPAx3/OYc753YgzqSzry8gtLa+sruXXCxubW9s75u5eQ4axILROQh6Klosl5SygdcUUp61IUOy7nDbd4XVab46okCwM7tQ4oo6P+wHzGMFKW13zwCt1fKwGrpfcT05/cDQ56ZpFq2xNhRbBzqAImWpd87PTC0ns00ARjqVs21aknAQLxQink0InljTCZIj7tK0xwD6VTjK9YIKOtdNDXij0CxSaur8nEuxLOfZd3ZmuKOdrqflfrR0r79JJWBDFigZk9pEXc6RClMaBekxQovhYAyaC6V0RGWCBidKhFXQI9vzJi9A4K9vn5cptpVi9yuLIwyEcQQlsuIAq3EAN6kDgAZ7gBV6NR+PZeDPeZ605I5vZhz8yPr4BmpGXAQ==</latexit>

f (x, v)
Discriminator Styled 2D Mapping
Generator Marching <latexit sha1_base64="zBLQWennagfZAt2j3un3tQpjQSU=">AAAB8XicjVDLSgNBEOyNrxhfUY9eBoPgKexKUI9BL4KXCOaByRJ6J7PJkNnZZWZWCMv+hRcPinj1b7z5N04eBxUFCxqKqm66u4JEcG1c98MpLC2vrK4V10sbm1vbO+XdvZaOU0VZk8YiVp0ANRNcsqbhRrBOohhGgWDtYHw59dv3TGkey1szSZgf4VDykFM0Vrq77mc9FMkI83654lXdGcjfpAILNPrl994gpmnEpKECte56bmL8DJXhVLC81Es1S5COcci6lkqMmPaz2cU5ObLKgISxsiUNmalfJzKMtJ5Ege2M0Iz0T28q/uZ1UxOe+xmXSWqYpPNFYSqIicn0fTLgilEjJpYgVdzeSugIFVJjQyr9L4TWSdU7rdZuapX6xSKOIhzAIRyDB2dQhytoQBMoSHiAJ3h2tPPovDiv89aCs5jZh29w3j4Bq6eQ7g==</latexit>

Network Kα FC Layer
Cubes <latexit sha1_base64="6cYjeJDlSs4FD97IRVVE7RgZ2io=">AAAB+3icbVDLSgMxFM3UV62vsS7dBItQN2VGirosunFZwT6gM5RMmmlDk8yQZKRlmF9x40IRt/6IO//GTDsLbT0QOJxzL/fkBDGjSjvOt1Xa2Nza3invVvb2Dw6P7ONqV0WJxKSDIxbJfoAUYVSQjqaakX4sCeIBI71gepf7vSciFY3Eo57HxOdoLGhIMdJGGtpVT9ExR3WPIz0JwnSWXQztmtNwFoDrxC1IDRRoD+0vbxThhBOhMUNKDVwn1n6KpKaYkaziJYrECE/RmAwMFYgT5aeL7Bk8N8oIhpE0T2i4UH9vpIgrNeeBmcwjqlUvF//zBokOb/yUijjRRODloTBhUEcwLwKOqCRYs7khCEtqskI8QRJhbeqqmBLc1S+vk+5lw71qNB+atdZtUUcZnIIzUAcuuAYtcA/aoAMwmIFn8ArerMx6sd6tj+VoySp2TsAfWJ8/8ZOUZA==</latexit>

σ(x)
<latexit sha1_base64="LzA4+Wbh8z+JYq5WshSxZ7rROdg=">AAACAXicbZDLSsNAFIZP6q3WW9SN4GawCBWkJFLUZdGNywr2Am0ok+mkHTq5MDMpllA3voobF4q49S3c+TZO2gja+sPAx3/OYc753YgzqSzry8gtLa+sruXXCxubW9s75u5eQ4axILROQh6Klosl5SygdcUUp61IUOy7nDbd4XVab46okCwM7tQ4oo6P+wHzGMFKW13zgJQ6PlYD10vuJ6c/OJqcdM2iVbamQotgZ1CETLWu+dnphST2aaAIx1K2bStSToKFYoTTSaETSxphMsR92tYYYJ9KJ5leMEHH2ukhLxT6BQpN3d8TCfalHPuu7kxXlPO11Pyv1o6Vd+kkLIhiRQMy+8iLOVIhSuNAPSYoUXysARPB9K6IDLDAROnQCjoEe/7kRWicle3zcuW2UqxeZXHk4RCOoAQ2XEAVbqAGdSDwAE/wAq/Go/FsvBnvs9ackc3swx8ZH9+VwJb+</latexit>

c(x, v)

Volume Aggregation
1024x1024
3D Mesh
Image

3D Mesh 64x64 2D feature

High-Res RGB Image map
Discriminator

Figure 2. StyleSDF Architecture: (Left) Overall architecture: SDF volume renderer takes in a latent code and camera parameters, queries
points and view directions in the volume, and projects the 3D surface features into the 2D view. The projected features are fed to the Styled
2D generator that creates the high resolution image. (Right) our SDF volume renderer jointly models volumetric SDF and radiance field,
providing a well defined and view consistent geometry.
architecture can be seen in Figure 2. shown in VolSDF [72], the SDF can be serve as a proxy for
To generate an image, we sample a latent vector z from the density function used for the traditional volume render-
the unit normal distribution, and camera azimuth and eleva- ing [46]. Assuming a non-hollow surface, we convert the
tion angles (φ, θ) from the dataset’s object pose distribution. SDF value into the 3D density fields σ,
For simplicity, we assume that the camera is positioned on
the unit sphere and directed towards the origin. Next, our
volume renderer outputs the signed distance value, RGB
−d(x)
 
color, and a 256 element feature vector for all the sampled 1
σ(x) = Kα (d(x)) = · Sigmoid , (1)
volume points along the camera rays. We calculate the sur- α α
face density for each sampled point from its SDF value and
apply volume rendering [46] to project the 3D surface fea-
tures into 2D feature map. The 2D generator then takes
the feature map and generates the output image from the where α is a learned parameter that controls the tightness of
desired viewpoint. The 3D surface can be visualized with the density around the surface boundary. α values that ap-
volume-rendered depths or with the mesh from marching- proach 0 represent a solid, sharp, object boundary, whereas
cubes algorithm [41]. larger α values indicate a more “fluffy” object boundary. A
large positive SDF value would drive the sigmoid function
3.2. SDF-based Volume Rendering towards 0, meaning no density outside of the surface, and
Our backbone volume renderer takes a 3D query point, x a high-magnitude negative SDF value would push the sig-
and a viewing direction v. Conditioned by the latent vector moid towards 1, which means maximal density inside the
z, it outputs an SDF value d(x, z), a view dependent color surface.
value c(x, v, z), and feature vector f (x, v, z). For clarity,
we omit z from hereon forward. We render low resolution 64 × 64 feature maps and color
The SDF value indicates the distance of the queried point images with volume rendering. For each pixel, we query
from the surface boundary, and the sign indicates whether points on a ray that originates at the camera position o, and
the point is inside or outside of a watertight surface. As points at the camera direction r(t) = o + tv. and calculate
the RGB color and feature map as follows: We observed that using view-dependent color c(x, v)
Z tf tends make the networks overfit to biases in the dataset. For
C(r) = T (t)σ(r(t))c(r(t), v)dt, instance, people in FFHQ [36] tend to smile more when
tn facing the camera. This makes the facial expression change
Z tf with the viewpoint although the geometry remains consis-
F(r) = T (t)σ(r(t))f (r(t), v)dt, (2) tent. However, when we removed view-dependent color, the
tn
 Z t  model did not converge. Therefore, to get view consistent
where T (t) = exp − σ(r(s))ds , images, we train our model with view dependent color, but
tn fix the view direction v to the frontal view during inference.
which we approximate with discrete sampling along rays. 3.3. High-Resolution Image Generation
Unlike NeRF [46] and other 3D-aware GANs such as Pi-
GAN [8] and StyleNeRF [23] we do not use stratified sam- Unlike multi-view reconstruction task [46], where the re-
pling. Instead, we split [tn , tf ] into N evenly-sized bins, construction loss for each ray can be individually computed,
t −t adversarial training needs a full image to be present. There-
draw a single offset term uniformly δ ∼ U[0, f N n ], and
fore, scaling a pure volume renderer to high-resolution
sample N evenly-spaced points,
quickly becomes untractable, as we need to sample more
tf − tn than 108 queries to render a single 1024×1024 image with
ti = · i + δ, where i ∈ {0, . . . , N − 1}. (3) 100 samples/ray. As such, we seek to fuse a volume ren-
N
derer with the StyleGAN2 network that has a proven capa-
In addition, we forgo hierarchical sampling altogether, bilities of synthesizing high-resolution images in 2D.
thereby reducing the number of samples by 50%. To combine the two architectures, we truncate the early
The incorporation of SDFs provides clear definition of layers of the StyleGAN2 generator up until the 64×64 layer
the surface, allowing us to extract the mesh via Marching and feed the generator with the 64 × 64 feature maps gen-
Cubes [41]. Moreover, the use of SDFs along with the re- erated by the backbone volume renderer. In addition, we
lated losses (Sec. 3.4.1) leads to higher quality geometry in cut StyleGAN2’s mapping network from eight layers to five
terms of expressiveness and view-consistency (as shown in layers, and feed it with the w latent code from the volume
Sec. 4.4), even with a simplified volume sampling strategy. renderer’s mapping network, instead of the original latent
The architecture of our volume renderer mostly matches vector z. The discriminator is left unchanged.
that of Pi-GAN [8]. The mapping network consists of a This design choice allows us to enjoy the best of both
3 layer MLP with LeakyReLU activation and maps an in- worlds. The volume renderer learns the underline geom-
put latent code z into w space and then generates frequecny etry, explicitly disentangles the object’s pose from it’s ap-
modulation, γi , and phase shift, βi , for each layer of the vol- pearance, and enables full control of the camera position
ume renderer. The volume rendering network contains eight during inference. The StyleGAN2 generator upsamples the
shared modulated FC layers with SIREN [62] activation: low resolution feature maps, adds high frequency details,
φi (x) = sin (γi (Wi · x + bi ) + βi ) , i ∈ {0, . . . , 7} (4) and mimics complex light transport effects such as sub-
surface scattering and inter-reflections that are difficult to
where Wi and bi are the weight matrix and bias vector of model with the low-resolution volume renderer.
the fully connected layers. The volume renderer then splits
3.4. Training
into two paths, the SDF path and the color path. The SDF
path is implemented using a single FC layer denoted φd . We employ a two-stage training procedure. First we
In the color path, the output of the last shared layer φ7 is train only the backbone SDF-based volume renderer, then
concatenated with the view direction input and passed into we freeze the volume renderer weights, and then train the
one additional FiLM siren layer φf followed by a single FC StyleGAN generator.
layer φc that generates the color output. To summarize:

σ(x) = Kα ◦ φd ◦ φ7 ◦ . . . ◦ φ0 (x), 3.4.1 Volume Renderer training

f (x, v) = φf (φ7 ◦ . . . ◦ φ0 (x), v) (5) We use the non-saturating GAN loss with R1 regulariza-
c(x, v) = φc ◦ φf . tion [43], denoted Ladv , to train our volume renderer. On
top of that, we use 3 additional regularization terms.
The output features of φf are passed to the 2D style- Pose Alignment Loss: This loss is designed to make sure
based generator, and the generated low resolution color im- that all the generated objects are globally aligned. On top of
age is fed to a discriminator for supervision. The discrimi- predicting whether the image is real or fake, the discrimina-
nator is identical to the Pi-GAN [8] discriminator. tor also tries to predict the two input camera angles (φ, θ).
Figure 3. Generated high-res RGB images (top), low-res volume rendered images (inset) and depth maps (bottom) for the same view . The
64×64 volume rendering output features are passed to the StyleGAN generator for high-resolution RGBs. Note that the object identities
and structures are preserved between the image pairs. Furthermore, as can be seen in the jaguar and cheetah examples, the StyleGAN
generator occasionally corrects badly modeled background signal from the volume renderer.

We penalize the prediction error using a smoothed L1 loss: While it is possible to have a reconstruction loss between
( the low-resolution and high-resolution output images, we
(θ̂ − θ)2 if |θ̂ − θ| ≤ 1 find that the inductive bias of the 2D convolutional archi-
Lview = . (6)
|θ̂ − θ| otherwise tecture and the sharing of style codes is strong enough to
preserve important structures and identities between the im-
This loss is applied on both view angles for the generator ages (Fig. 3).
and the discriminator, however, since we don’t have ground
truth pose data for the original dataset, this loss is only ap- 4. Experiments
plied to the fake images in the discriminator pass.
Eikonal Loss: This term ensures that the learned SDF is 4.1. Datasets & Baselines
physically valid [21]: We train and evaluate our model on the FFHQ [36] and
AFHQ [12] datasets. FFHQ contains 70,000 images of di-
Leik = Ex (k∇d(x)k2 − 1)2 . (7) verse human faces at 1024 × 1024 resolution, which are
Minimal Surface Loss: We encourage the 3D network to centered and aligned according to the procedure introduced
describe the scenes with minimal volume of zero-crossings in Karras et al. [34]. The AFHQ dataset consists of 15,630
to prevent spurious and non-visible surfaces from being images of cats, dogs and wild animals at 512 × 512 resolu-
formed within the scenes. That is, we penalize the SDF tion. Note that the AFHQ images are not aligned and con-
values that are close to zero: tain diverse animal species, posing a significant challenge
to StyleSDF.
Lsurf = Ex (exp(−100|d(x)|)) . (8) We compare our method against the state-of-the-art
3D-aware GAN baselines, GIRAFFE [49], PiGAN [8],
The overall loss function is then, GRAF [58] and HoloGAN [47], on the above datasets by
measuring the quality of the generated images, shapes, and
Lvol = Ladv + λview Lview + λeik Leik + λsurf Lsurf , (9) rendering consistency.
where λview = 15, λeik = 0.1, and λsurf = 0.05. The 4.2. Qualitative Evaluations
weight of the R1 loss is set according to the dataset.
Comparison to Baseline Methods: We compare the visual
quality of our images to the baseline methods by rendering
3.4.2 Styled Generator Training
the same identity (latent code) from 4 different viewpoints,
We train our Styled generator with the same losses and opti- results are shown in Figure 4. To compare the quality of the
mizer parameters as the original implementation, a non sat- underlying geometry, we also show the surfaces extracted
urating adversarial loss, R1 regularization, and path regu- by marching cubes from StyleSDF, Pi-GAN, and GRAF
larization. As in the volume renderer training, we set the (Note that GIRRAFE and HoloGAN pipelines do not gener-
weight of the R1 regularization according to the dataset. ate shapes). Our method generates superior images as well
 N/A N/A N/A N/A

N/A N/A N/A N/A

Figure 4. Qualitative image and geometry comparisons. We compare our sample renderings and corresponding 3D meshes against the
state-of-the-art 3D-aware GAN approaches ( [8, 47, 49, 58]). Note that HoloGAN and GIRAFFE are unable to create 3D mesh from
their representations. Both HoloGAN (a) and GRAF (b) produce renderings that are of lower quality. The 3D mesh reconstructed from
PiGAN’s learned opacity fields reveal noticeable artifacts (c). While GIRAFFE (d) produces realistic low-resolution images, the identity
of the person often changes with the viewpoints. StyleSDF (d) produces 1024×1024 realistic view consistent rgb, while also generating
high quality 3D. Best viewed digitally.

as more detailed 3D shapes. Additional generation results 4.3. Quantitative Image Evaluations
from our method can be seen in Figures 1 and 3.
We evaluate the visual quality and the diversity of the
generated images using the Frechet Inception Distance
Novel View Synthesis: Since our method learns strong 3D (FID) [26] and Kernel Inception Distance (KID) [6]. We
shape priors, it can generate images from viewpoints that compare our scores against the aforementioned baseline
are not well represented in the dataset distribution. Ex- models on the FFHQ and AFHQ datasets.
amples of out-of-distribution view synthesis are displayed All the baseline models are trained following their given
in Figure 5. pipelines to generate 256 × 256 images, with the excep-
tion of Pi-GAN, which is trained on 128 × 128 images and
Video Results: We urge readers to view our project’s web- renders 256 × 256 images at inference time. The results,
site that includes a larger set of results and videos to better summarized in Table 1, show that StyleSDF performs con-
appreciate the multi-view capabilities of StyleSDF. sistently better than all the baselines in terms of visual qual-
 Dataset: FFHQ AFHQ
PiGAN 11.04 8.66
Ours 0.40 0.63

Table 2. Depth consistency results. We measure the average modi-

fied Chamfer distance (Eq. (10)) over 1,000 random pairs of depth
maps for each dataset. Each pair contains one frontal view depth
Figure 5. Out-of-distribution view synthesis (field of view and map and one side view depth map. Our method demonstrates sig-
camera angles). Although StyleSDF was trained with a fixed field nificantly stronger consistency (see Fig. 6).
of view, increasing and decreasing FOV by 25% (columns 1-2)
still looks realistic. Similarly with 1.5 standard deviations of the
camera angles distribution used for training (columns 3-6).
is defined as the expected termination distance of 128 uni-
formly sampled points along each ray. Note that we remove
non-terminating rays whose accumulated opacity is below
Dataset: FFHQ AFHQ 0.5. We set the side viewpoint to be 1.5× the standard de-
FID KID FID KID viation of the azimuth distribution in training. See supple-
mentary for more experiment details.
HoloGAN 90.9 75.5 95.6 77.5 To measure the alignment errors between the depth
GRAF 79.2 55.0 129.5 85.1 points, we adopt a modified Chamfer distance metric. I.e.,
PiGAN 83.0 85.8 52.4 30.7 we replace the usual mean distance definition with the me-
GIRAFFE 31.2 20.1 33.5 15.1 dian of the distances to nearest points,
Ours 11.5 2.65 12.8 4.47
CD(S1 , S2 ) = med min kx − yk22 + med min kx − yk22 ,
x∈S1 y∈S2 y∈S2 x∈S1
Table 1. FID and KID evaluations. All datasets were evaluated (10)
at a resolution of 256 × 256. Our method demonstrates the best for some point sets S1 and S2 . This metric is more robust
performance. Note that we report KID × 1000 for simplicity.
to outliers that come from occlusion and background mis-
match that we are not interested in measuring. To put the
ity. It is also on par with reported scores from concurrent metric at scale, we normalize the distances by the volume
works such as StyleNerf [23] and CIPS-3D [76]. sampling bin size.
As shown in Table 2, our use of SDF representation
4.4. Volume Rendering Consistency dramatically improves depth consistency compared to the
strongest current baseline PiGAN [8]. Figure 6 shows the
Volume rendering has emerged as an essential technique
sample depth map pairs used for the evaluation and the error
to differentiably optimize a volumetric field from 2D im-
visualizations (in terms of distance to the closest point). The
ages, as its wide-coverage point sampling leads to stable
color map shows that our depth maps align well except for
gradient-flow during training. Notably, volume rendering
the occluded regions and backgrounds. In contrast, PiGAN
excels at modeling thin surfaces or transparent objects, e.g.,
depth maps show significant noise and spurious concave re-
human hairs, which are difficult to model with explicit sur-
gions (e.g., nose of the dog).
faces, e.g., 3D meshes.
Moreover, we show that our consistent volume render-
However, we notice that the volume rendering of existing ing naturally leads to high view-consistency for our RGB
3D-aware GANs [8, 58] using unregularized opacity fields renderings. As shown in Fig. 7, we visualize the reprojec-
severely lacks view-consistency due to the absence of multi- tion of side-view renderings to the frontal view, using the
view supervision. That is, depth values, computed as the depth values from volume rendering. The reprojected pix-
expected termination distance of each camera ray [15, 46], els closely match those of the original frontal view, indicat-
from different viewpoints do not consistently overlap in the ing that our high-res multi-view RGB renderings and depth
global coordinate. This means that neural implicit features maps are all consistent to each other. Refer to supplemen-
are evaluated at inconsistent 3D locations, undermining the tary for more detailed experiments.
inductive bias of the implicit 3D representation for view-
consistent renderings. As such, we measure and compare
5. Limitations & Future Work
the depth map consistency across viewpoints to gauge the
quality of volume rendering for each system. StyleSDF might exhibit minor aliasing and flickering,
We sample 1,000 identities, render their 128 × 128 depth e.g., in teeth area. We leave it for future work since we ex-
maps from the frontal view and a fixed side view, and com- pect those two to be corrected similarly to Mip-NeRF [5]
pute the alignment between the two views. The depth value and Alias-free StyleGAN [35]. See example at left two
 Ours PiGAN Ours PiGAN

Figure 7. RGB rendering view-consistency. We render two side

views (a) and project them to the frontal view (b) using the depth
maps rendered from each views, ignoring occluded pixels. Note
the high simmilarity between the reprojected images and the ren-
dered frontal view (c), as can be seen from the error map (d). The
error map shows mean absolute pixel difference for RGB channels
(0-255) for the right side-view image. The errors are mostly from
regions with high frequency textures and geometry (e.g., ear, hair),
or occlusion boundaries (right forehead).

Figure 6. Visual comparison of depth consistency. We visual-

ize the nearest neighbour distances (in sample bin units) from the
frontal depth maps to side-view depth maps. Our SDF-based tech-
nique significantly improves depth consistency compared to the Figure 8. Limitations: potential aliasing artifacts, e.g., in teeth
baseline. (left two columns). Specularities and shadows may create artifacts
(3rd column from the left, cheek and eyes area), high curvatures
are enhanced with radiance scaling filter [68]. Inconsistencies in
columns of Figure 8. Specularities or other strong light- background might decrease photorealism (right column).
ing effects currently introduce depth dents since StyleSDF
might find it hard to disambiguate with no multi-view data
require either a decreased batch size, which might hurt the
(Figure 8 third column from the left). Adjusting the losses
overall performance, or increased training time if we keep
to include those effects is left for future work. Similarly,
the batch size and accumulate gradients. Second improve-
we do not currently separate foreground from background
ment could be to create a volume sampling strategy tied
and use a single SDF for the whole image. Figure 8 (right
to SDF’s surface boundary (to reduce the number of query
column) shows how the cat’s face is rendered properly, but
points at each forward pass) and eliminate the need for a
the transition to the background is too abrupt, potentially
2D CNN that upsamples feature maps. That would tie 3D
diminishing photorealism. A potential solution could be
geometry directly to the high resolution image.
adding an additional volume renderer to model the back-
ground as suggested in NeRF++ [74].
6. Conclusions
Finally, one may consider two improvements to the al-
gorithm. First one is training the two parts as a single end- We introduced StyleSDF, a method that can render
to-end framework, instead of the current two networks. In 1024x1024 view-consistent images along with the detailed
such case the StyleGAN2 discriminator would send proper underlying geometry. The proposed architecture combines
gradients back to the volume renderer to produce optimal SDF-based volume renderer and a 2D StyleGAN network
feature maps, which might lead to even more refined geom- and is trained to generate realistic images for all sampled
etry. However, end-to-end training poses a trade-off. The viewpoints via adversarial loss, naturally inducing view-
increased GPU memory consumption of this setup would consistent 3D scenes. StyleSDF represents and learns com-
plex 3D shape and appearance without multi-view or 3D [16] Matheus Gadelha, Subhransu Maji, and Rui Wang. 3d shape
supervision, requiring only a dataset of single-view images, induction from 2d views of multiple objects. In 3DV, pages
suggesting a new route ahead for neural 3D content genera- 402–411. IEEE, 2017. 2
tion, editing, and reconstruction. [17] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna,
and Thomas Funkhouser. Local deep implicit functions for
References 3d shape. In CVPR, pages 4857–4866, 2020. 2
[18] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna,
[1] Rameen Abdal, Yipeng Qin, and Peter Wonka. Im- William T Freeman, and Thomas Funkhouser. Learning
age2stylegan: How to embed images into the stylegan latent shape templates with structured implicit functions. In CVPR,
space? In ICCV, pages 4432–4441, 2019. 2 pages 7154–7164, 2019. 2
[2] Rameen Abdal, Yipeng Qin, and Peter Wonka. Im- [19] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing
age2stylegan++: How to edit the embedded images? In Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and
CVPR, pages 8296–8305, 2020. 2 Yoshua Bengio. Generative adversarial nets. In NeurIPS,
[3] Rameen Abdal, Peihao Zhu, Niloy J Mitra, and Peter Wonka. pages 2672–2680, 2014. 2
Styleflow: Attribute-conditioned exploration of stylegan- [20] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and
generated images using conditional continuous normalizing Yaron Lipman. Implicit geometric regularization for learning
flows. ACM TOG, 40(3):1–21, 2021. 2 shapes. In ICML, pages 3569–3579, 2020. 2
[4] Matan Atzmon and Yaron Lipman. Sal: Sign agnostic learn- [21] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and
ing of shapes from raw data. In CVPR, pages 2565–2574, Yaron Lipman. Implicit geometric regularization for learning
2020. 2 shapes. arXiv preprint arXiv:2002.10099, 2020. 5
[5] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter [22] Thibault Groueix, Matthew Fisher, Vladimir G Kim,
Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. Bryan C Russell, and Mathieu Aubry. A papier-mâché ap-
Mip-nerf: A multiscale representation for anti-aliasing neu- proach to learning 3d surface generation. In Proceedings of
ral radiance fields. In ICCV, pages 5855–5864, October the IEEE conference on computer vision and pattern recog-
2021. 7, 15 nition, pages 216–224, 2018. 2
[6] Mikołaj Bińkowski, Danica J. Sutherland, Michael Arbel, [23] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian
and Arthur Gretton. Demystifying mmd gans. arXiv preprint Theobalt. Stylenerf: A style-based 3d-aware genera-
arXiv:1801.01401, 2018. 6 tor for high-resolution image synthesis. arXiv preprint
[7] Rohan Chabra, Jan E Lenssen, Eddy Ilg, Tanner Schmidt, arXiv:2110.08985, 2021. 2, 4, 7
Julian Straub, Steven Lovegrove, and Richard Newcombe. [24] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and
Deep local shapes: Learning local sdf priors for detailed 3d Sylvain Paris. Ganspace: Discovering interpretable gan con-
reconstruction. In ECCV, pages 608–625. Springer, 2020. 2 trols. In NeurIPS, 2020. 2
[8] Eric R. Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, [25] Paul Henderson, Vagia Tsiminaki, and Christoph H Lampert.
and Gordon Wetzstein. pi-gan: Periodic implicit genera- Leveraging 2d data to learn textured 3d mesh generation. In
tive adversarial networks for 3d-aware image synthesis. In CVPR, pages 7498–7507, 2020. 2
CVPR, pages 5799–5809, 2021. 1, 2, 4, 5, 6, 7, 12 [26] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner,
[9] Zhiqin Chen and Hao Zhang. Learning implicit fields for Bernhard Nessler, and Sepp Hochreiter. Gans trained by a
generative shape modeling. In CVPR, pages 5939–5948, two time-scale update rule converge to a local nash equilib-
2019. 2 rium. In NeurIPS, page 6629–6640, 2017. 6
[10] Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll. [27] K Hill and J White. Designed to deceive: Do these people
Implicit functions in feature space for 3d shape reconstruc- look real to you. New York Times, 2020. 12
tion and completion. In CVPR. IEEE, jun 2020. 2 [28] Ali Jahanian, Lucy Chai, and Phillip Isola. On the” steer-
[11] Julian Chibane, Aymen Mir, and Gerard Pons-Moll. Neural ability” of generative adversarial networks. In ICLR, 2020.
unsigned distance fields for implicit function learning. In 2
NeurIPS, December 2020. 2 [29] Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei
[12] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Huang, Matthias Nießner, Thomas Funkhouser, et al. Local
Stargan v2: Diverse image synthesis for multiple domains. implicit grid representations for 3d scenes. In CVPR, pages
In CVPR, pages 8188–8197, 2020. 2, 5 6001–6010, 2020. 2
[13] Edo Collins, Raja Bala, Bob Price, and Sabine Susstrunk. [30] Danilo Jimenez Rezende, SM Eslami, Shakir Mohamed, Pe-
Editing in style: Uncovering the local semantics of gans. In ter Battaglia, Max Jaderberg, and Nicolas Heess. Unsu-
Proceedings of the IEEE/CVF Conference on Computer Vi- pervised learning of 3d structure from images. NeurIPS,
sion and Pattern Recognition, pages 5771–5780, 2020. 2 29:4996–5004, 2016. 2
[14] Thomas Davies, Derek Nowrouzezahrai, and Alec Jacobson. [31] Omer Kafri, Or Patashnik, Yuval Alaluf, and Daniel Cohen-
Overfit neural networks as a compact shape representation. Or. Stylefusion: A generative model for disentangling spatial
arXiv preprint arXiv:2009.09808, 2020. 2 segments. arXiv preprint arXiv:2107.07437, 2021. 2
[15] Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ra- [32] James T Kajiya and Brian P Von Herzen. Ray tracing volume
manan. Depth-supervised nerf: Fewer views and faster train- densities. ACM SIGGRAPH computer graphics, 18(3):165–
ing for free. arXiv preprint arXiv:2107.02791, 2021. 7 174, 1984. 2
[33] Kimmo Karkkainen and Jungseock Joo. Fairface: Face [48] Thu Nguyen-Phuoc, Christian Richardt, Long Mai, Yong-
attribute dataset for balanced race, gender, and age for Liang Yang, and Niloy Mitra. Blockgan: Learning 3d object-
bias measurement and mitigation. In Proceedings of the aware scene representations from unlabelled images. In
IEEE/CVF Winter Conference on Applications of Computer NeurIPS, 2020. 2
Vision, pages 1548–1558, 2021. 12 [49] Michael Niemeyer and Andreas Geiger. Giraffe: Represent-
[34] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. ing scenes as compositional generative neural feature fields.
Progressive growing of GANs for improved quality, stability, In CVPR, pages 11453–11464, 2021. 1, 2, 5, 6
and variation. In ICLR, 2018. 2, 5 [50] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and
[35] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Andreas Geiger. Occupancy flow: 4d reconstruction by
Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias- learning particle dynamics. In Proceedings of the IEEE/CVF
free generative adversarial networks. arXiv preprint International Conference on Computer Vision, pages 5379–
arXiv:2106.12423, 2021. 1, 2, 7, 15 5389, 2019. 2
[36] Tero Karras, Samuli Laine, and Timo Aila. A style-based [51] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and
generator architecture for generative adversarial networks. In Andreas Geiger. Differentiable volumetric rendering: Learn-
CVPR, pages 4401–4410, 2019. 1, 2, 4, 5, 13, 15 ing implicit 3d representations without 3d supervision. In
[37] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, CVPR, pages 3504–3515, 2020. 2
Jaakko Lehtinen, and Timo Aila. Analyzing and improving [52] Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo
the image quality of stylegan. In CVPR, pages 8110–8119, Strauss, and Andreas Geiger. Texture fields: Learning tex-
2020. 1, 2, 15 ture representations in function space. In Proceedings of the
[38] Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng IEEE/CVF International Conference on Computer Vision,
Xu, Justus Thies, Matthias Niessner, Patrick Pérez, Chris- pages 4531–4540, 2019. 2
tian Richardt, Michael Zollhöfer, and Christian Theobalt. [53] Xingang Pan, Xudong Xu, Chen Change Loy, Christian
Deep video portraits. ACM Transactions on Graphics (TOG), Theobalt, and Bo Dai. A shading-guided generative im-
37(4):1–14, 2018. 12 plicit model for shape-accurate 3d-aware image synthesis. In
[39] Diederik P Kingma and Jimmy Ba. Adam: A method for NeurIPS, 2021. 2
stochastic optimization. arXiv preprint arXiv:1412.6980,
[54] Jeong Joon Park, Peter Florence, Julian Straub, Richard
2014. 15
Newcombe, and Steven Lovegrove. Deepsdf: Learning con-
[40] Thomas Leimkühler and George Drettakis. Freestylegan: tinuous signed distance functions for shape representation.
Free-view editable portrait rendering with the camera mani- In CVPR, pages 165–174, 2019. 1, 2
fold. arXiv preprint arXiv:2109.09378, 2021. 2
[55] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc
[41] William E. Lorensen and Harvey E. Cline. Marching cubes: Pollefeys, and Andreas Geiger. Convolutional occupancy
A high resolution 3d surface construction algorithm. ACM networks. In ECCV, pages 523–540. Springer, 2020. 2
TOG, 21(4):163–169, 1987. 3, 4
[56] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Mor-
[42] Sebastian Lunz, Yingzhen Li, Andrew Fitzgibbon, and
ishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned
Nate Kushman. Inverse graphics gan: Learning to gener-
implicit function for high-resolution clothed human digitiza-
ate 3d shapes from unstructured 2d data. arXiv preprint
tion. In ICCV, pages 2304–2314, 2019. 2
arXiv:2002.12674, 2020. 2
[57] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul
[43] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin.
Joo. PifuHD: Multi-level pixel-aligned implicit function for
Which training methods for gans do actually converge? In
high-resolution 3d human digitization. In CVPR, pages 84–
ICML, pages 3481–3490, 2018. 4
93, 2020. 2
[44] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Se-
bastian Nowozin, and Andreas Geiger. Occupancy networks: [58] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas
Learning 3d reconstruction in function space. In CVPR, Geiger. Graf: Generative radiance fields for 3d-aware image
pages 4460–4470, 2019. 2 synthesis. In NeurIPS, 2020. 1, 2, 5, 6, 7
[45] Mateusz Michalkiewicz, Jhony K Pontes, Dominic Jack, [59] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Inter-
Mahsa Baktashmotlagh, and Anders Eriksson. Implicit sur- preting the latent space of gans for semantic face editing. In
face representations as layers in neural networks. In ICCV, CVPR, pages 9243–9252, 2020. 2
pages 4743–4752, 2019. 2 [60] Yujun Shen and Bolei Zhou. Closed-form factorization of
[46] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, latent semantics in gans. In CVPR, pages 1532–1540, 2021.
Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: 2
Representing scenes as neural radiance fields for view syn- [61] Alon Shoshan, Nadav Bhonker, Igor Kviatkovsky, and
thesis. In ECCV, pages 405–421. Springer, 2020. 2, 3, 4, Gérard Medioni. Gan-control: Explicitly controllable gans.
7 In ICCV, pages 14083–14093, 2021. 2
[47] Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian [62] Vincent Sitzmann, Julien Martel, Alexander Bergman, David
Richardt, and Yong-Liang Yang. Hologan: Unsupervised Lindell, and Gordon Wetzstein. Implicit neural representa-
learning of 3d representations from natural images. In ICCV, tions with periodic activation functions. Advances in Neural
pages 7588–7597, 2019. 2, 5, 6 Information Processing Systems, 33, 2020. 2, 4
[63] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wet- meet differentiable rendering for inverse graphics and inter-
zstein. Scene representation networks: Continuous 3d- pretable 3d neural rendering. In ICLR, 2020. 2
structure-aware neural scene representations. NeurIPS, [76] Peng Zhou, Lingxi Xie, Bingbing Ni, and Qi Tian.
2019. 1, 2 Cips-3d: A 3d-aware generator of gans based on
[64] Ayush Tewari, Mohamed Elgharib, Florian Bernard, Hans- conditionally-independent pixel synthesis. arXiv preprint
Peter Seidel, Patrick Pérez, Michael Zollhöfer, and Christian arXiv:2110.09788, 2021. 2, 7
Theobalt. Pie: Portrait image embedding for semantic con-
trol. ACM TOG, 39(6):1–14, 2020. 2
[65] Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian
Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zoll-
hofer, and Christian Theobalt. Stylerig: Rigging stylegan for
3d control over portrait images. In CVPR, pages 6142–6151,
2020. 2
[66] Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann,
Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-
Brualla, Tomas Simon, Jason Saragih, Matthias Nießner,
et al. State of the art on neural rendering. In Computer
Graphics Forum, volume 39, pages 701–727. Wiley Online
Library, 2020. 12
[67] Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srini-
vasan, Edgar Tretschk, Yifan Wang, Christoph Lassner,
Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lom-
bardi, Tomas Simon, Christian Theobalt, Matthias Niess-
ner, Jonathan T. Barron, Gordon Wetzstein, Michael Zoll-
hoefer, and Vladislav Golyanik. Advances in neural render-
ing. arXiv preprint arXiv:2111.05849, 2021. 2
[68] Romain Vergne, Romain Pacanowski, Pascal Barla, Xavier
Granier, and Christopher M. Schlick. Radiance scaling for
versatile surface enhancement. In I3D ’10, 2010. 8
[69] Sheng-Yu Wang, Oliver Wang, Richard Zhang, Andrew
Owens, and Alexei A Efros. Cnn-generated images are
surprisingly easy to spot... for now. In Proceedings of
the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pages 8695–8704, 2020. 12
[70] Jiajun Wu, Chengkai Zhang, Tianfan Xue, William T Free-
man, and Joshua B Tenenbaum. Learning a probabilistic
latent space of object shapes via 3d generative-adversarial
modeling. In Proceedings of the 30th International Confer-
ence on Neural Information Processing Systems, pages 82–
90, 2016. 2
[71] Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge
Belongie, and Bharath Hariharan. Pointflow: 3d point cloud
generation with continuous normalizing flows. In Proceed-
ings of the IEEE/CVF International Conference on Com-
puter Vision, pages 4541–4550, 2019. 2
[72] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Vol-
ume rendering of neural implicit surfaces. arXiv preprint
arXiv:2106.12052, 2021. 3
[73] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan
Atzmon, Basri Ronen, and Yaron Lipman. Multiview neu-
ral surface reconstruction by disentangling geometry and ap-
pearance. Advances in Neural Information Processing Sys-
tems, 33, 2020. 2
[74] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen
Koltun. Nerf++: Analyzing and improving neural radiance
fields. arXiv preprint arXiv:2010.07492, 2020. 8
[75] Yuxuan Zhang, Wenzheng Chen, Huan Ling, Jun Gao, Yi-
nan Zhang, Antonio Torralba, and Sanja Fidler. Image gans
Appendix Dataset: FFHQ AFHQ
In this appendix we provide additional qualitative and PiGAN [8] 14.7 16.5
quantitative results on our approach, along with the tech- Ours (volume renderer) 2.9 2.6
nical details that supplement the main paper. In Sec. Ap-
pendix A, we discuss possible societal impacts of our tech- Table 3. Quantitative view-consistency comparison of the RGB
nology. Then, we present additional experiments on the renderings. We evaluate the color error of the RGB volume ren-
view-consistency of our RGB renderings via image repro- derings between the frontal view and the reprojection from a fixed
jection (Sec. Appendix B). In Sec. Appendix C, we de- side view. The error is measured as the median of the per-pixel
mean absolute difference (0 - 255). We average the color incon-
scribe the content of the supplementary videos and intro-
sistency over 1,000 samples for each dataset. Our underlying SDF
duce a geometry-aware noise injection procedure to re-
geometry representation promotes superior 3D consistency. (also
duce flickering. Next, we discuss implementation details see Fig. 9).
and conduct ablation studies on our approach (Sec. Ap-
pendix D Appendix E). Finally, we continue our discussion
on our method’s limitations in Sec. Appendix F. puting the median.
As shown in Tab. 3, StyleSDF presents significantly im-
A. Societal Impacts proved color consistency compared to the strongest current
baseline, PiGAN [8]. Fig. 9 shows the sample depth and
Image and 3D model-generating technologies (e.g., color rendering pairs used for the evaluation, along with the
deepfakes) could be used for spreading misinformation pixelwise error maps. The error maps demonstrate that our
about existing or non-existing people [27,38]. Our proposed volume RGB renderings have high view consistency, as the
technology allows generating multi-view renderings of a large reprojection errors are mostly in the occluded regions.
person, and might be used for creating more realistic fake On the other hand, PiGAN’s reprojections do not align well
videos. These problems could potentially be addressed by with the frontal view, showing big errors also near the eyes,
developing algorithms to detect neural network-generated mouth, in presence of specular highlights, etc.
content [69]. We refer readers to [66] for strategies of mit-
igating negative social impacts of neural rendering. More- B.2. High-Resolution RGB Consistency
over, image generative models are optimized to follow the
In Fig. 10, we present the reprojection experiment results
training distribution, and thus could inherit the ethnic, gen-
using our high-resolution RGB outputs. As in the volume
der, or other biases present in the training dataset. A pos-
rendering consistency experiment, we reproject the high-res
sible solution is creating a more balanced dataset, e.g., as
RGB pixels from non-frontal views (with varying azimuth
in [33].
and elevation) to the frontal views. The results demonstrate
the strong 3D-consistency of our high-resolution images, as
B. View Consistency of RGB Renderings the reprojected non-frontal images are similar to the frontal
B.1. Volume Rendering Consistency renderings. However, as mentioned in the limitation sec-
tion of the main paper, the current implementation of Style-
The consistent volume rendering from our SDF-based GAN networks come with significant aliasing of the high-
technique naturally leads to high view consistency of our frequency components, resulting in noticeable pixel errors
RGB renderings. To show the superior 3D-consistency of on the regions with high-frequency details, e.g., hair, ears,
our SDF-based volume rendering, we measure the reprojec- eyes, etc.
tion error when a side view pixels are warped to the frontal To identify the errors in the high-frequency details, we
view. We randomly sample 1,000 identities and render the visualize the mean reprojection images. I.e. we project
depth and RGB images at 256×256 and set the side view to non-frontal views and average the pixel values across views.
be 1.5× the standard deviation of the azimuth distribution in As can be seen in Fig. 11, the mean reprojection images
training (which is 0.45 radians for FFHQ and 0.225 radians closely replicate the identities and important structures of
for AFHQ). We reproject the side-view RGB renderings to the frontal view, demonstrating strong view-consistencies.
the frontal view using the side-view depth, and we do not ig- The error map confirms that most of the errors are concen-
nore occluded pixels. We measure color inconsistency with trated on the high-frequency noise of the StyleGAN gener-
the median of pixel-wise L1 error in RGB (0 - 255), aver- ator.
aged over the 1,000 samples. The use of median effectively
removes the large errors coming from occlusions. Note that C. Video Results
since PiGAN is trained with center-cropped FFHQ images
(resized to 320 × 320 and center-cropped to 256 × 256), we Since our 3D-consistent high-resolution image genera-
apply the same transformation on our results before com- tion can be better appreciated with videos, we have at-
 FFHQ (azimuth=0.45 radians) AFHQ (azimuth=0.225 radians)
Side depth Side rendering Reprojection Frontal rendering Error Side depth Side rendering Reprojection Frontal rendering Error
Ours (volume renderer)
PiGAN

Figure 9. Qualitative view consistency comparison of RGB renderings. We project the rendering from a side view using its corresponding
depth map to the frontal view. We compare the reprojection to the frontal-view rendering and compute the error map showing mean
absolute difference in RGB channels (0 - 255). Our SDF-based technique generate superior depth quality and significantly improves the
view-consistency of the RGB renderings. Most of our errors concentrate on the occlusion boundaries whereas PiGAN’s errors spread
across the whole subject (e.g., eyes, mouth, specular highlights, fur patterns).

tached 24 sequences in the supplementary material, featur- frames. The flickering artifacts are especially prominent for
ing view-generation results on the two datasets using two the AFHQ dataset due to high-frequency textures from the
different camera trajectories. For each identity, we provide fur patterns.
two videos, one for RGB and another for depth rendering. Therefore, we aim at reducing this flickering by adding
The videos are presented in the project’s website. the Gaussian noise in a 3D-consistent manner, i.e., we want
to attach the noise on the 3D surface. We achieve this by
C.1. Geometry-Aware StyleGAN Noise
extracting a mesh (at 128 resolution grid) for each sequence
Even though the images shown in the main paper on from our SDF representation and attach a unit Gaussian
multi-view RGB generation look highly realistic, we note noise to each vertex, and render the mesh using vertex col-
that for generating a video sequence, the random noise of oring. Since higher resolution intermediate features require
StyleGAN2 [36], when naı̈vely applied to 2D images, could up to 1024×1024 noise map, we subdivide triangle faces of
result in severe flickering of high-frequency details between the extracted mesh once every layer, starting from 128×128
 Frontal φ = −0.45 φ = −0.3 φ = −0.15 φ = 0.15 φ = 0.3 φ = 0.45 θ = −0.225 θ = −0.15 θ = −0.075 θ = 0.075 θ = 0.15 θ = 0.225

Figure 10. View-consistency visualization of high-resolution renderings. We use the side-view depth maps (first rows) to warp the side-
view RGB renderings (second rows) to the frontal view (first column). The reprojected pixels that pass the occlusion testing are shown
in the third row. We compare the reprojections with the frontal-view renderings and show the per-pixel error maps (fourth rows). Our
reprojections well align with the frontal view with errors mostly in the occlusion boundaries and high-frequency details.

feature layers. erator for 300k iterations.

The video results show that the geometry-aware noise in- AFHQ: The AFHQ dataset contains training and validation
jection reduce the flickering problem on the AFHQ dataset, sets for 3 classes, cats, dogs and wild animals. We merged
but noticeable flickering still exist. Furthermore, we ob- all the training data into a single training set. We apply
serve that the geometry-aware noise slightly sacrifices in- R1 regularization loss of 50. Both azimuth and elevation
dividual frame’s image quality, presenting less pronounced angles are sampled from a Gaussian distribution with zero
high-frequency details, likely due to the change of the Gaus- mean and standard deviation of 0.15 and a camera field of
sian noise distribution during the rendering process. view of 12◦ . The near and far fields as well as the number
of samples per ray are identical to the FFHQ setup. Our
D. Implementation Details volume renderer as well as the 2D-Styled generator were
D.1. Dataset Details trained for 200k iterations.

FFHQ: We trained FFHQ with R1 regularization loss of 10. D.2. Training Details
The camera field of view was fixed to 12◦ and its azimuth
and elevation angles are sampled from Normal distributions Sphere Initialization: During our experiments we have no-
with zero mean and standard deviations of 0.3 and 0.15 re- ticed that our SDF volume renderer can get stuck at a local
spectively. We set the near and far fields to [0.88, 1.12] and minimum, which generates concave surfaces. To avoid this
sample 24 points per ray during training .We trained our optimization failure, we first initialize the MLP to generate
volume renderer for 200k iterations and the 2D-Styled gen- an SDF of a sphere centered at the origin with a fixed radius.
 FFHQ AFHQ

Frontal rendering
Mean reproduction
Difference

Figure 11. Color consistency visualization with mean faces. We reproject the RGB renderings from the side views to the frontal view (as
in Fig. 10). We show the mean reprojections that pass the occlusion testing and their differences to the frontal-view renderings. The mean
reprojections are well aligned with the frontal rendering. The majority of the errors are in the high-frequency details, generated from the
random noise maps in the StyleGAN component. This demonstrates the strong view consistency of our high-resolution renderings.
We analytically compute the signed distance of the sampled biguate between shape and radiance. Penalizing values that
points from the sphere and fit the MLP to match these dis- are close to zero essentially minimizes the surface area and
tances. We run this procedure for 10k iterations before the makes the network prefer smooth SDFs.
main training. The importance of sphere initialization is In Figure 13, we show the importance of the sphere ini-
discussed in Appendix E. tialization in breaking the concave/convex ambiguity. With-
Training setup: Our system is trained in a two-stage strat- out properly initializing the weights, the network gets stuck
egy. First, we train the backbone SDF volume renderer on at a local minimum that generates concave surfaces. Al-
64×64 images with a batch size of 24 using the ADAM [39] though concave surfaces are physically incorrect, they can
optimizer with learning rates of 2 · 10−5 and 2 · 10−4 for the perfectly explain multi-view images, as they are essentially
generator and discriminator respectively and β1 = 0, β2 = the ”mirror” surface. Concave surfaces cause the images to
0.9. We accumulate gradients in order to fit to the GPU be rendered in the opposite azimuth angle, an augmentation
memory constraints. For instance, a setup of 2 NVIDIA that the discriminator cannot detect as fake. Therefore, the
A6000 GPUs (a batch of 12 images per GPU) requires the generator cannot recover from the this local minima.
accumulation of two forward passes (6 images per forward
pass) and takes roughly 3.5 days to train. We use an expo-
nential moving average model during inference. F. Limitations (continued)
In the second phase, we freeze the volume renderer
As mentioned in the main paper, our high-resolution
weights and train the 2D styled generator with identical
generation network is based on the implementation of Style-
setup to StyleGAN2 [37]. This includes ADAM optimizer
GAN2 [36], and thus might experience the same aliasing
with 0.002 learning rate and β1 = 0, β2 = 0.99, equalized
and flickering at regions with high-frequency details (e.g.,
learning rate, lazy R1 and path regularization, batch size of
hair), which are recently addressed in Alias-free GAN [35]
32, and exponential moving average. We trained the styled
or Mip-NeRF [5]. Moreover, we observe that the recon-
generator on 8 NVIDIA TeslaV100 GPUs for 7 days.
structed geometry for human eyes contain artifacts, charac-
E. Ablation studies terized by concave, instead of convex, eye balls. We believe
that these artifacts often lead to slight gaze changes along
We perform two ablation studies to show the neces- with the camera views. As stated in the main paper, our cur-
sity of the minimal surface loss (see main paper) and the rent implementation of volume rendering during inference
sphere initialization. As can be seen in Figure 12, on top uses fixed frontal view directions for RGB queries c(x, v),
of preventing spurious and non-visible surfaces from be- and thus cannot express moving specular highlights along
ing formed, the minimal surface loss also helps to disam- with the camera.
Without Minimal
Surface Loss

With Minimal
Surface Loss

Color Image Frontal view Side view Color Image Frontal view Side view

Figure 12. Minimal surface loss ablation study. We visualize the volume rendered RGB and depth images from volume renderers trained
with and without the minimal surface loss. The Depth map meshes are visualized from the front and side views. Note how a model trained
with the minimal surface loss generates smoother surfaces and is less prone to shape-radiance ambiguities, e.g., specular highlights are
baked into the geometry.

Without sphere
initialization

With sphere
initialization

Color Image Frontal view Side view Color Image Frontal view Side view

Figure 13. Sphere initialization ablation study. We visualize volume-rendered RGB and depth images from volume renderers trained with
and without sphere initialization. The Depth map meshes are visualized from the front and side views. Note how a model trained without
model sphere initialization generates concave surfaces.

G. Additional Results
We show uncurated set of images generated by our net-
works (Fig. 14).
Figure 14. Uncurated high-resolution RGB images that are randomly generated by StyleSDF.