IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION.

ACCEPTED JANUARY 2023 1

NeurAR: Neural Uncertainty for Autonomous 3D

Reconstruction with Implicit Neural Representations
Yunlong Ran1 , Jing Zeng1 , Shibo He2 , Jiming Chen2 , Lincheng Li3 , Yingfeng Chen3 , Gimhee Lee4 , Qi Ye2

Abstract—Implicit neural representations have shown com- and at each viewpoint, the information of a scene is captured
pelling results in offline 3D reconstruction and also recently by an RGB image (with an optional depth image).
demonstrated the potential for online SLAM systems. How- Implicit neural representations for 3D objects have shown
ever, applying them to autonomous 3D reconstruction, where
arXiv:2207.10985v2 [cs.CV] 8 Feb 2023

a robot is required to explore a scene and plan a view path their potential to be precise in geometry encoding, efficient
for the reconstruction, has not been studied. In this paper, in memory consumption (adaptive to scene size and com-
we explore for the first time the possibility of using implicit plexity), predictive in filling unseen regions and flexible in
neural representations for autonomous 3D scene reconstruction the amount of training data. The reconstruction with offline
by addressing two key challenges: 1) seeking a criterion to images [1] or online images with cameras held by human [2]
measure the quality of the candidate viewpoints for the view
planning based on the new representations, and 2) learning the has achieved compelling results recently with implicit neural
criterion from data that can generalize to different scenes instead representations. However, leveraging these advancements to
of a hand-crafting one. To solve the challenges, firstly, a proxy achieve high-quality autonomous 3D reconstruction has not
of Peak Signal-to-Noise Ratio (PSNR) is proposed to quantify a been studied.
viewpoint quality; secondly, the proxy is optimized jointly with Previous 3D representations for autonomous 3D reconstruc-
the parameters of an implicit neural network for the scene.
With the proposed view quality criterion from neural networks
tion include point cloud, volume, and surface. To plan the view
(termed as Neural Uncertainty), we can then apply implicit without global information of a scene, previous work resorts to
representations to autonomous 3D reconstruction. Our method a greedy strategy: given the current position of a robot and the
demonstrates significant improvements on various metrics for reconstruction status, they quantify the quality of the candidate
the rendered image quality and the geometry quality of the viewpoints via information gain to plan the next best view
reconstructed 3D models when compared with variants using
TSDF or reconstruction without view planning. Project webpage
(NBV). In these works, the information gain relies on hand-
https://kingteeloki-ran.github.io/NeurAR/ crafted criteria, each designed ad-hoc for a particular combina-
tion of a 3D representation and a reconstruction algorithm. For
Index Terms—Computer Vision for Automation; Motion and
Path Planning; Planning under Uncertainty example, Mendez et al. [3] define view cost by triangulated
uncertainty given by the algorithm inferring depth from stereo
RGB images, Isler et al. [4] quantify the information gain
I. I NTRODUCTION for a view using entropy in voxels seen in this viewpoint, Wu
UTONOMOUS 3D reconstruction has a wide range of
A applications, e.g. augmented/virtual reality, autonomous
driving, filming, gaming, medicine, architecture. The problem
et al. [5] identify the quality of the view by a Poisson field
from point clouds and Song et al. [6] leverage mesh holes and
boundaries to guide the view planning.
requires a robot to make decisions about moving towards To use implicit neural representations for autonomous 3D
which viewpoint in each step to get the best reconstruction reconstruction, a key capability is to quantify the quality of
quality of an unknown scene with the lowest cost, i.e. view the candidate viewpoints. For implicit neural representations,
planning. In this work, we assume a robot can localize itself how to define the viewpoint quality? Is it possible for the
neural network to learn a measurement of the quality from
Manuscript received: August 7, 2022; Revised: November 23, 2022; Ac-
cepted: December 21, 2022. This paper was recommended for publication by data instead of defining it heuristically? In this paper, we make
Editor Cesar Cadena upon evaluation of the Associate Editor and Reviewers’ efforts to answer both questions.
comments. This work was supported in part by NSFC under Grants 62233013, The quality of reconstructed 3D models can be measured by
62088101, 62103372, and the Fundamental Research Funds for the Central
Universities, China. (Corresponding Author: Qi Ye)
the quality of images rendered from different viewpoints and
1 Yunlong Ran and Jing Zeng are with the College of Control Science the measurement is adopted in many offline 3D reconstruction
and Engineering, Zhejiang University, Hangzhou, China. {yunlong_ran, works. PSNR is one popular measurement and it is defined
zengjing}@zju.edu.cn
2 Shibo He, Jiming Chen, and Qi Ye are with the College of Control Science according to the difference between the images rendered from
and Engineering, the State Key Laboratory of Industrial Control Technol- the reconstructed model and the ground truth model.
ogy, Zhejiang University, and the Key Laboratory of Collaborative Sensing The ground truth images, however, are unavailable for
and Autonomous Unmanned Systems of Zhejiang Province. {s18he, cjm unvisited viewpoints to calculate PSNR during autonomous
qi.ye}@zju.edu.cn
3 Lincheng Li and Yingfeng Chen are with Fuxi AI reconstruction. Is it possible to learn a proxy for PSNR?
Lab, NetEase Inc., Hangzhou, China. {lilincheng, In [7], [8], the authors point out that if the target variable
chenyingfeng1}@corp.netease.com to regress is under a Gaussian noise model and the target
4 Gimhee Lee is with the Department of Computer Science, National
University of Singapore, Singapore. dcslgh@nus.edu.sg distribution conditioned on the input is optimized by maximum
Digital Object Identifier (DOI): see top of this page. likelihood, the optimum value of the noise variance is given
 View Planner
RGB Images Depth Images

2 3D Reconstruction Poses
IEEE ROBOTICS AND Renderer
AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JANUARY 2023

5 U:0.0367 15 U:0.0100

15

6 U:0.0677 16 U:0.0139

6 16

View Path Uncertainty Prediction GT Uncertainty Prediction GT

Fig. 1. View paths planned by NeurAR and the uncertainty maps used to guide the planner. Given current pose 5 and 15, paths are planned toward viewpoints
having higher uncertainties, i.e. pose 6 and 16. Uncertainties of the viewpoints are shown in red text. Notice that darker regions in uncertainty maps relate to
worse quality regions of rendered images.
by the residual variance of the target values and the regressed completeness and smoothness of the constructed Poisson iso-
ones. Inspired by this, we assume the color to regress for a surface and the confidence map is used to guide the calculation
spatial point in a scene as a random variable modeled by a of NBV. Schmid et al. [9] propose information gain to evaluate
Gaussian distribution. The Gaussian distribution models the the quality of observed surfaces and unknown voxels near the
uncertainty of the reconstruction and the variance quantifies observed surfaces, then plan a path by RRT.
the uncertainty. When the regression network converges, the
variance of the distribution is given by the squared error of In contrast, some methods explore function learning to
the predicted color and the ground truth color; the integral of solve the NBV problem [11]–[13]. Supervised learning-based
the uncertainty of points in the frustum of a viewpoint can be methods [13] learn information gain of a viewpoint given a
taken as a proxy of PSNR to measure the quality of candidate partial occupancy map and [12] learns an informed distribu-
viewpoints. tion of high-utility viewpoints based on a partial occupancy
With the key questions solved, we are able to build an map. For the reinforcement learning-based methods [11], no
autonomous 3D reconstruction system (NeurAR) using an hand-crafted heuristics are required and an agent explores the
implicit neural network. In summary, the contributions of the viewpoints with high overall coverage. These learning-based
paper are: methods require a large-scale dataset for training and may be
hard to generalize to different scenes.
• We propose the first autonomous 3D reconstruction sys-
tem using an implicit neural representation. Different from the hand-crafted heuristics, the learned NBV
• We propose a novel view quality criterion that learns policy, and the existing learning-based information gain, our
online from continuously added input images per target proposed neural uncertainty is learned per target scene during
scene instead of hand-engineering or learning from a large the reconstruction, requiring no manual definition, no large
corpus of 3D scenes. training set, and being able to work in any new scene.
• Our proposed method significantly improves on various
metrics from alternatives using voxel-based representa- Online Dense Reconstruction For online 3D reconstruc-
tions or using man-designed paths for the reconstruction. tion from RGB images, most methods use Multi-View-Stereo
(MVS) [14] to reconstruct dense models by first getting a
II. R ELATED W ORK sparse set of initial matches, iteratively expanding matches
to nearby locations and performing surface reconstruction.
View Planning Most view planning methods focus on the With the release of commodity RGB-D sensors, the fusion of
NBV problem which uses feedback from current partial recon- point clouds reprojected from depth images gains popularity.
struction to determine NBV. According to the representations KinectFusion [15] achieves real-time 3D reconstruction with
for the 3D models, these methods can be divided into voxel- a moving depth camera by integrating points from depth
based method [3], [4] and surface-based method [5], [9]. images with Truncated Signed Distance Functions (TSDFs).
The voxel-based methods are most commonly used due OctoMap [16] builds a probabilistic occupancy volume based
to their simplicity in representing space. Vasquez-Gomez et on octree. Recently, implicit representations have shown com-
al. [10] analyze a set of boundary voxels and determine pelling results in 3D reconstruction, either for the radiance
NBV for dense 3D modeling of small-scale objects. Stefan field approximation from RGB images [17] or the shape ap-
Isler et al. [4] provide several metrics to quantify the volume proximation from point clouds [18]. The novel representations
information contained in the voxels. Mendez et al. [3] define are also studied for online dense reconstruction. iMAP [2]
the information gain by the triangulation uncertainty of stereo adopts MLPs as the scene representation and reconstructs the
images. Despite the simplicity, these methods suffer from scene from RGBD images. In addition to using MLPs to
memory consumption with growing scene complexity and represent a scene implicitly, NeRFusion [19] further combines
higher spatial resolutions. a feature volume to fuse information from different views as
A complete volumetric map does not necessarily guarantee a latent scene representation. Similarly, we use the implicit
a perfect 3D surface. Therefore, researchers propose to analyze neural function to represent 3D models but we focus on how
the shape and quality of the reconstructed surface for NBV [5], to leverage this representation for autonomous view planning.
[9]. Wu et al. [5] estimate a confidence map representing the
RAN et al.: NEURAL UNCERTAINTY FOR AUTONOMOUS 3D RECONSTRUCTION WITH IMPLICIT NEURAL REPRESENTATIONS 3

Uncertainties Poses
acceleration in Section III-C. The view planner module (Sec-
View Planner
tion III-D) samples viewpoints from empty space, measures
3D Reconstruction
the contributions of these sampled viewpoints by composing
Renderer

RGB Images Depth Images

Poses
the neural uncertainty, chooses NBV and plans a view path to
NBV. The simulation module, built upon Unity Engine, renders
Fig. 2. The pipeline of our method RGBD images from new viewpoints and provides both the
images and the camera poses to the 3D reconstruction module.
III. M ETHOD
A. Problem Description and System Overview B. 3D Reconstruction with Neural Uncertainty
The problem considered is to generate a trajectory for a In the section, we first formulate Neural Uncertainty when
robot that yields high-quality 3D models of a bounded priori a target scene is approximated by an implicit neural network.
unknown scene and fulfills robot constraints. The trajectory is Then we show how to optimize the uncertainty together
defined as a sequence of viewpoints V = (v1 , ..., vn ) where with the network parameters. With the neural uncertainty
vi ∈ R3 × SO(2) (usually roll is not considered). Let S be formulation, a strong linear relationship is established between
the set of all the sequences. The problem can therefore be the uncertainty and the image quality metric, i.e. PSNR and
expressed as therefore a proxy for PSNR is proposed.
1) Problem formulation: NeRF [1] represents a continuous
\centering \begin {aligned} V^* = \mathop {\arg \max }\limits _{V \in S} I(V) \text { s.t. } L(V) \leq L_{max}, \end {aligned} \eqlabel {opttraj} (1) scene with an implicit neural function by taking the location of
a point x on a ray of direction d as inputs and its color value
where I(V ) equals ΣI(vi ), an objective function measuring c, density ρ as outputs, i.e. Fθ (x, d) = (c, ρ). The function Fθ
the contributions from different viewpoints to the reconstruc- is represented by an MLP.
tion quality and L(V ) is the length of the trajectory. To learn the neural uncertainty for view planning, we treat
One difficulty of the problem is defining the objective the color as a random variable under a Gaussian distribu-
function itself as there is no ground truth data to measure the tion instead of taking it as a deterministic one. The color
quality of the reconstruction. Hence, one major line of work distribution can be represented as p(C) = N (µ, σ 2 ), where
in this field is seeking a surrogate quantity that can be used the RGB channels of C share the same σ. To estimate the
for the view sequence optimization, which is also the focus of distribution, we map the inputs (x, d) to its parameters (µ, σ)
this paper. of the distribution for the color variable C; in other words,
Finding the best sequence for (1) is prohibitively expensive the outputs of the MLP is µ, σ rather than c, so we have
and most work resolves this problem by a greedy strategy. At Fθ (x, d) = (µ, σ, ρ). Having defined the color distribution
each step of the reconstruction process, a number of candidate for a point x on a ray d, we now deduce the rendered color
viewpoints are sampled according to some constraints, the distribution for a camera ray. Following NeRF, we define the
contributions of these viewpoints to the reconstruction quality rendered color as
are evaluated based on the current reconstructed scene and the \rayclr = \sum _{i=1}^{N} \omega _i \clr _i, \eqlabel {calccolor} (2)
viewpoint with the most contribution is chosen as NBV. The
process is repeated until a maximum length is reached. To Qi−1
move a robot to NBV, various path planning methods can be where ωi is oi j=1 (1 − oj ), oi = (1 − exp(−ρi δi )) and
applied to navigate the robot. δi = di+1 − di which represents the inter-sample distance.
p(Ci ) = N (µi , σi2 ) is the color distribution for a point on the
In this work, we follow the greedy strategy and choose an
ray. Assuming {Ci } for points on the ray are independent from
existing RRT* based view planner. For each viewpoint, an
each other, the distribution for Cr is a Gaussian distribution.
RGB image (with an optional depth image) is captured. To
The probability of the rendered color value being cr is p(Cr =
achieve an autonomous 3D reconstruction system depending
cr ) = N (cr |µr , σr ), where
PN the 2mean and variance are µr =
only on the input images, joint estimation of camera poses P N 2
and the target scene is required. To focus on view planning ω
i=1 i iµ and σ r = i=1 ωi σi .
Similarly, assuming {Cr } for different rays are independent,
with implicit scene representations, we assume the ground
the random variable CI for the mean rendered color of
truth camera poses are given and leave the localization of the
rays sampled from an image pool is also under a Gaussian
camera as future work.
distribution, whose mean and variance are
Accordingly, our proposed autonomous 3D reconstruction
system can be divided into three modules (shown in the top of
Fig. 1): a 3D reconstruction module, a view planner module, \mean _I = \frac 1 {\numrayset }\sum _{r=1}^{\numrayset }\mean _r = \frac 1 {\numrayset } \sum _{r=1}^{\numrayset } \sum _{i=1}^{N} \omega _{ri} \mean _{ri}, \eqlabel {meanimg} (3)
and a simulation module. At each step, the 3D reconstruc-
tion module reconstructs a scene represented by an implicit
neural network with new images from the simulation module \uncert _I^2=\frac 1 {\numrayset } \sum _{r=1}^{\numrayset } \uncert _r^2=\frac 1 {\numrayset } \sum _{r=1}^{\numrayset }\sum _{i=1}^{N}\omega _{ri} \uncert _{ri}^2, \eqlabel {uncertimg} (4)
and provides a quantity (neural uncertainty) measuring the
reconstructed quality of each position (Section III-B). To adapt where r represents a camera ray tracing through a pixel from
the implicit neural representation to online reconstruction, an image pool and R the number of rays sampled for the image
we further propose several strategies for online training and pool.
4 IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JANUARY 2023

2) Optimization: We now consider how to optimize the

network parameters θ and determine µ, σ, ρ for each point on a
ray (we ignore the subscript ri for brevity). Consider a pool of
images, the likelihood for the mean color value cI of a set of
rays shooting from the pixels sampled from the image pool are
obtained as p(CI = cI ) = N (cI |µI , σI ). Taking the negative
logarithm of the likelihood and ignoring the constant, we have
(a) Online and offline training (b) The uncertainty and ratio loss curves
L_{color}=\log {\uncert _I} + \frac {L_{mean}}{2\uncert _I^2} \leq \log {\uncert _I} + \frac {L_I}{2\uncert _I^2}, \eqlabel {logloss} (5)

L_{mean} = \lVert \raysetclrval - \mean _I \rVert _2^2 \leq L_{I}=\frac 1 {\numrayset } \sum _{r=1}^{\numrayset } \lVert \rayclrval - \mean _r \rVert _2^2. \eqlabel {mseloss} (6)

For Lmean of (6), the constraint for each pixel is too weak
and the network is not able to converge a meaningful result
(PSNR about 10). As we have supervision for the color of each
(c) PCC -0.96 (d) PCC -0.92
pixel and Lmean is smaller than or equal to LI , we choose to
minimize LI instead. The loss function above is differentiable Fig. 3. Loss curves and linear relationship between log σI2 and PSNR. PCC
w.r.t θ and µ, σ, ρ are the outputs of MLPs (Fθ ); we can use value of -1 signifies strong negative correlation. (a) Training loss curves for
online training and offline training. (b) The loss curves for the uncertainty part
gradient descent to determine the network parameters θ and
log σI and the ratio part L I
in (5). (c) Linearity when the scene is optimized
µ, σ, ρ. To make σ positive, we let the network estimate σ 2 σI2
using offline images. (d) Linearity when the scene is optimized using online
and the output of MLPs s is activated by es to get σ 2 . images.
Consider the minimization with respect to θ. Given σI , we
can see the minimization of the maximum likelihood under a the uncertainty part log σI and the ratio part L I
σI2
in (5) for
conditional Gaussian distribution for each point is equivalent online are shown in Fig. 3(b). Notice that the loss curve for
to minimizing a mean-of-squares error function given by LI in the ratio part stays almost constant during training, which also
(6). Apply Fθ on a point on a ray and µ, ρ, σ can be obtained. verifies the effectiveness using uncertainty as a proxy of PSNR.
3) Neural Uncertainty and PSNR: For the variance σI2 , or For the verification and the usage of the linear relationship
Neural Uncertainty, the optimum value can be achieved by for autonomous reconstruction, we adopt NeRF [1] as our
setting the derivative of Lcolor with respect to σI to zero, implicit representation for a scene. From the formulation and
giving the derivation of the relation between neural uncertainty and
\sigma _I^2 = L_{I}. \eqlabel {relation} (7) PSNR, our neural uncertainty is agnostic to the underlying
function Fθ , which can be MLPs like NeRF or networks based
The equation above indicates that the optimal solution for σI2 is on trainable feature vectors with MLP Decoder [19]
the squared errors between the predicted image and the ground
truth image. C. Online Training and Acceleration
M AX 2
On the other hand, PSNR is defined as 10 log10 M SEI , Though online reconstruction with Neural Uncertainty su-
where M AXI is the maximum possible pixel value of the pervised by images can achieve similar accuracy with NeRF,
image. When a pixel is represented using 8 bits, it is 255 and the convergence is too slow for view planning. We accelerate
MSE is the mean squared error between two images, the same training by introducing particle filter, depth supervision and a
as LI . Then we establish a linear relationship between the keyframe strategy.
logarithm of Neural Uncertainty and PSNR, i.e. Particle filter keeps particles (rays) active in high loss region,
PSNR = A\log {\sigma }_I^2, (8) which helps the network optimize details faster. At each step,
when a new image is added to an image pool for the training,
where A is a constant coefficient. a set of particles are randomly sampled from the image. After
To verify the linear relationship, we scatter data pairs of an iteration of training, a quarter of particles are resampled
(P SN R, log σI2 ) for images in the testing set evaluated at according to the weight of particles, which is defined according
different iterations when optimizing Fθ for a cabin scene (the to the loss of a ray, and the other particles are uniformly
scene is shown in Section IV). Two different training strategies sampled from the image.
are conducted: online training with images captured along a Particle filter is applied after coarse learning of the whole
planned trajectory added sequentially and offline batch training scene is done. At the early stage of training, the model knows
using all the images precaptured from the trajectory. As can be little information about the scene and tends to have a higher
seen Fig. 3(c-d), a strong correlation exists between P SN R loss for rays shooting at objects than that for empty space. This
and log σ 2 , whose Pearson Correlation Coefficient (PCC) is results in particles always staying at the surfaces of objects
-0.96 for online and -0.92 for offline. The two variables and therefore the network learns the whole space slower.
are almost perfectly negatively linearly related. During the Considering this, at the beginning iterations of the training,
training, σI and LI are jointly optimized. The loss curves of we use random sampling.
RAN et al.: NEURAL UNCERTAINTY FOR AUTONOMOUS 3D RECONSTRUCTION WITH IMPLICIT NEURAL REPRESENTATIONS 5

Depth supervision can greatly speed up training. Depth im- other scenes. RGBD images are rendered by Unity Engine and
ages for NeRF are rendered similar to color images in [1]. We we assume their corresponding camera poses are known. To
define depth loss and our final loss as simulate depth noise, all rendered depth images are added with
noise scaling approximately quadratically with depth z [20].
L_{depth} = \frac 1 {\numrayset }\sum _{r=1}^{\numrayset } \lVert \hat {z}_r- z_r \rVert _2^2 \text { , }\text { } L = L_{color} + \lambda _d L_{depth}, \eqlabel {finalloss} (9) The depth noise model is ϵ = N (µ(z), σ(z)) where µ(z) =
0.0001125z 2 +0.0048875 , σ(z) = 0.002925z 2 +0.003325 and
the constant parameters are acquired by fitting the model to
where ẑr and zr represent the rendered depth from recon- the noise reported for Intel Realsense L515 2 . The depth noise
structed 3D model Fθ and the ground truth depth for a pixel. model for Alexander of large size, µ(z) = 0.00001235z 2 +
As depth captured from real sensors typically has noise, we 0.00004651 , σ(z) = 0.00001228z 2 +0.00001571, the constant
find using depth supervision may make the model not able parameters acquired by fitting the model to the noise reported
to converge well due to the conflict between noisy depth and for Lidar VLP163 .
the depth inferred from multiview RGB images. A balance Implementation details The networks and the hyperparam-
needs to be made between the two cases. We strengthen depth eters of all the experiments for different scenes below are
supervision at the early stage of training to accelerate training set to the same value. Most hyper parameters use the default
and decrease it after getting a coarse 3D structure. The weight setting in NeRF, including parameters of Adam optimizer (with
of depth loss is decreased from 1 to 1/10 after Nd iterations hyperparameters β1 = 0.9, β2 = 0.999, ϵ = 10−7 ), 64/128
to emphasize structure from multiview images. sampling points on a ray for coarse/fine sampling, a batch
Keyframe pool We follow iMAP [2] to maintain a keyframe size of 1024 etc.
pool containing 4 images for continual training. The pool is
initialized with the first four views and during training, the 𝜸 𝑿
60
Neural Uncertainty
branch
ρ
image with minimum image loss in the pool is replaced with +
a new image or an image from a seen images set. 128 Uncertainty

𝜸 𝒙
256 256 256 256 256 256 256 256 256 256
60

128 RGB

D. View Planning with Neural Uncertainty +

𝜸 𝒅
24
As explained in Section III-A, we follow the greedy strategy.
For the view path planning, we exploit the scenic path planner Fig. 4. The network architecture of the 3D reconstruction module of NeurAR.
based on RRTs* [3] to evaluate the efficacy of the proposed Similar to NeRF. The only difference is an additional branch of a 128 fully
Neural Uncertainty in guiding the view planning. In the scenic connected layer for the estimation of uncertainty
path planner, the RRTs* algorithm samples from a prior
distribution for the view quality cost-space approximated by GT 2K iterations 14K iterations 100K iterations 150K iterations
Sequential Monte-Carlo (SMC) instead from SE(3), which
baises the growth of the tree towards areas with good NBV
cost while also aims at a shortest Euclidean path. Our view
planner replaces their view cost with
C_{view} =\sigma _I^2, \eqlabel {costv} (10)
where σI2 is defined in (4) and here the image pool contains Fig. 5. Uncertainty maps from different viewpoints and training iterations.
Top: Uncertainty maps for seen viewpoints; Bottom: uncertainty maps for
only one image. an unseen viewpoint. The overall uncertainty decreases with training while
For the view planning, we introduce a prior of objects in the uncertainty for the unseen viewpoint stays high even when the network
the center of the space for the candidate viewpoint sampling: converges.
the camera moves in a band with the nearest distance to the NeurAR reconstructs scenes and plans the view path simul-
object Dnear meters and the farthest distance Df ar meters; taneously, running parallel on two RTX3090 GPUs: the 3D
the camera looks at directions pointing roughly to the center reconstruction module, i.e. optimization of NeRF on a GPU;
of the object; candidate viewpoints are sampled in a sphere the view planner module and the renderer module on the other
with the center at the current camera position and the radius one. After a step of the view planner (also 700 iterations for
Rsample meters. the NeRF optimization), the optimization receives one or two
images collected on the view path from cameras. We set the
IV. R ESULTS maximum views for planning to be 28 views, i.e. 11k training
iterations and about 15 steps for the optimization during the
Data The experiments are conducted on five 3D models, view planning. In each step, 1 or 2 views are selected along
drums from NeRF synthetic dataset, Alexander1 , cabin, the planned path. The network architecture of our NeurAR is
tank and monsters we collect online. The scenes used in the shown in Fig. 4.
paper are mostly of the size 5m×5m×5m, and the larger scene Algorithmic Variants To evaluate the efficacy of the proposed
Alexander is about 80m×80m×80m. Dnear , Df ar , Rsample method, we designed baselines and variants of our method.
are 80m, 90m, 80m for Alexander and 3m, 4m, 3m for
2 https://www.intelrealsense.com/lidar-camera-l515/
1 https://3dwarehouse.sketchup.com/ 3 https://usermanual.wiki/Pdf/VLP16Manual.1719942037/view/
6 IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JANUARY 2023

Ground Truth Online RRT (our) Online FT Offline FT Online RS TSDF FT TSDF RS TSDF RRT

cabin

drums

alexander

Fig. 6. Comparison of the reconstruction models with different methods. Refer to the supplementary video for higher resolution, more comparison and more
viewpoints.
tank
TABLE I
E VALUATIONS ON THE RECONSTRUCTED 3D MODELS USING DIFFERENT METHODS

cabin drums alexander tank monsters

Methodmonsters PSNR↑ SSIM↑ LPIPS↓ PSNR↑ SSIM↑ LPIPS↓ PSNR↑ SSIM↑ LPIPS↓ PSNR↑ SSIM↑ LPIPS↓ PSNR↑ SSIM↑ LPIPS↓
TSDF FT 21.17 0.768 0.140 18.67 0.746 0.150 19.30 0.662 0.190 20.39 0.782 0.151 20.42 0.786 0.126
TSDF RS 20.87 0.75 0.151 18.68 0.738 0.160 19.15 0.649 0.195 18.64 0.749 0.171 20.19 0.777 0.131
TSDF RRT [9] 20.34 0.739 0.171 17.95 0.716 0.181 18.86 0.644 0.199 21.36 0.772 0.172 20.14 0.773 0.146
Offline FT 24.31 0.842 0.108 21.17 0.831 0.117 8.40 0.431 0.606 21.71 0.819 0.142 25.01 0.880 0.076
Online FT 23.42 0.833 0.114 21.49 0.831 0.117 8.71 0.419 0.597 23.03 0.839 0.124 24.42 0.878 0.077
Online RS 26.74 0.864 0.082 23.66 0.859 0.0838 13.08 0.533 0.373 21.97 0.789 0.162 24.68 0.871 0.063
Online even cover. 26.56 0.868 0.106 24.82 0.913 0.049 24.30 0.767 0.182 24.81 0.875 0.108 25.90 0.909 0.065
Ours Offline 28.86 0.917 0.048 26.45 0.916 0.051 23.98 0.758 0.188 27.54 0.909 0.064 27.57 0.927 0.039
Ours 28.35 0.902 0.062 25.73 0.905 0.058 24.07 0.757 0.199 25.83 0.874 0.097 26.57 0.908 0.054
Method Acc↓ Comp↓ C.R.↑ Acc↓ Comp↓ C.R.↑ Acc↓ Comp↓ C.R.↑ Acc↓ Comp↓ C.R.↑ Acc.↓ Comp↓ C.R.↑
TSDF FT 2.47 2.68 0.39 2.51 1.64 0.21 87.16 164.30 0.44 2.46 3.14 0.40 2.44 1.21 0.43
TSDF RS 2.83 2.81 0.42 3.09 1.56 0.22 107.85 159.78 0.44 3.04 3.53 0.40 2.93 1.19 0.48
TSDF RRT [9] 2.05 2.53 0.44 2.90 1.33 0.25 100.85 169.56 0.43 2.01 1.62 0.48 2.97 1.14 0.48
Offline FT 1.09 1.02 0.70 1.77 1.21 0.66 1582.74 193.77 0.04 1.38 1.28 0.63 1.10 1.10 0.71
Online FT 1.19 1.06 0.67 1.79 1.22 0.65 1506.59 202.52 0.05 1.37 1.30 0.62 1.19 1.09 0.71
Online RS 1.88 1.20 0.57 1.53 1.10 0.70 652.66 211.00 0.27 2.62 2.25 0.37 1.89 1.11 0.66
Online even cover. 1.21 1.01 0.74 1.15 1.08 0.77 54.24 35.69 0.48 1.50 1.29 0.64 1.17 1.06 0.76
Ours Offline 0.96 0.93 0.76 1.31 1.07 0.74 31.66 21.90 0.73 1.51 1.19 0.68 1.27 1.01 0.76
Ours 1.07 0.95 0.74 1.29 1.04 0.75 48.48 34.16 0.60 1.37 1.20 0.65 1.43 1.02 0.74

For the implicit scene representation, we adopt TSDF as the sampled NBVs. 7) Online even cover.: NeurAR with views
baseline as it is oen of the most used representation for SLAM evenly distributed in a dome to cover the whole space. 8)
and autonomous reconstruction [4], [9], [21]. For view path Ours offline: NeurAR trained offline, i.e. trained from scratch
planning based on the proposed Neural Uncertainty, we con- with images precaptured from all the planned views from our
struct two variants: one using a pre-defined circular trajectory proposed method.
with which existing work [21], [22] usually compares and the Metrics The quality of the reconstructed models can be
other one randomly sampling a viewpoint instead of using the measured in two aspects: the quality of the rendered im-
view cost to choose NBV at each step. ages (measure both the geometry and the texture) and the
The variants are 1) TSDF FT: RGBD images and corre- quality of the geometry of the constructed surface. For the
sponding poses are collected from a Fixed circular Trajectory former, we evenly distribute about 200 testing views 80m for
and are fed into the system sequentially. The voxel resolution Alexander, 3m and 3.4m from the center for other scenes,
of TSDF is 1cm. 2) TSDF RS: replace the fixed trajectory in render images for the reconstructed models and evaluate
TSDF FT with Randomly Sampled NBVs. 3)TSDF RRT: PSNR, SSIM and LPIPS for these images. For the latter, we
As for autonomous scene reconstruction most work is not adopt metrics from iMAP [2]: Accuracy (cm), Completion
open source, we re-implement the view cost defined in [9] (cm), CompletionRatio (the percentage of points in the
which adopts TSDF representation. The online trajectory is reconstructed mesh with Completion under a threshold, 30
planned with RRT and the view cost is defined according to cm for Alexander and 1cm for others). For the geometry
the quality of a reconstructed voxel, measured by the number metrics, about 300k points are sampled from the surfaces.
of rays traversing through it. Images and corresponding poses
for the fusion are collected from the planned views and are A. Neural Uncertainty
fed into the system sequentially. 4) Offline FT: The variant Fig. 3 has quantitatively verified the correlation between
is NeurAR with views from the fixed trajectory and trained Neural Uncertainty and the image quality. Here, we further
offline. 5) Online FT: The variant is NeurAR with views from demonstrate the correlation with examples. In this experi-
the fixed trajectory. 6) Online RS: NeurAR with randomly ment, the reconstruction network is optimized by the loss
RAN et al.: NEURAL UNCERTAINTY FOR AUTONOMOUS 3D RECONSTRUCTION WITH IMPLICIT NEURAL REPRESENTATIONS 7

defined in (5) with 73 images collected from cameras placed for many scenes of simple shape and texture is an optimum
roughly at one hemisphere. Fig. 5 shows the images rendered solution. Metrics for Online even cover. and Ours in Table I
from different viewpoints for the reconstructed 3D models demonstrate that our planned views can achieve similar or even
and their corresponding uncertainty map during training. The better reconstruction results and our method has no prior or
uncertainty map is rendered from an uncertainty field where only a minor knowledge about the scene.
1
the value of each point in the field is logσ . Viewpoints of Ours offline in most scenes gives better results than NeurAR
Row 1 are in the hemisphere seen in the training and the trained online as using all views enforces multiview constraints
viewpoint of the last row is in the other hemisphere. At the at the start of the training.
beginning, for all viewpoints, the object region and the empty In addition to the lower mean PSNR shown in the Table I,
space both have high uncertainty. For the seen viewpoints, our method achieves much smaller variance regarding the
with training continuing, the uncertainty of the whole space PSNR of the rendered images from different viewpoints. For
decreases and the quality of the rendered images improve. example, the PSNR variance of the rendered images for
When the network converges, the uncertainty only remains reconstructed cabins using Online FT, Online RS and our
high in the local areas having complicated geometries. For method is 32.55dB, 15.44dB to 4.78dB, indicating our method
the unseen viewpoint, though the uncertainty for the empty more even. Further, for the average path length in the smaller
space decreases dramatically with training, the uncertainty is scenes, our NeurAR traverses 43.27m while Online RS about
still very high on the whole surface of the object even when 70.24m; in the larger scene, our NeurAR traverses 907.60m
the network converges. while Online RS about 1329.75m.
Fig. 1 shows the Neural Uncertainty can successfully guide For models using the reconstruction with TSDF (TSDF
the planner to planning view paths for cameras to look at FT, TSDF RS, TSDF RRT), we render images via volume
regions that are not well reconstructed. For example, given rendering. From the images in the last three columns in Fig. 6,
current pose 15 and Fθ , a path is planned toward viewpoint the surfaces exhibit many holes; in addition, the surfaces of the
16 having a higher uncertainty map. The rendered image from reconstructed models are rugged due to the noise in the depth
viewpoint 16 from Fθ exhibits poor quality (zoom in the image images. Though NeurAR uses depth too, it depends on the
and the uncertainty map for details under the roof). depth images to accelerate convergence only at the early stage
of training and decreases its effect after coarse structures have
been learned. The finer geometry is acquired by multiview
B. View planning with Neural Uncertainty
image supervision. NeurAR fills holes in TSDF RRT and
To show the efficacy of our proposed method, we compare provide finer details. It outperforms TSDF RRT in the image
metrics in Table I and the rendered images of reconstructed 3D quality, geometry quality and path length (43.27m vs 57.39m).
models using different methods in Fig. 6. Table I demonstrates Though post-processing can be applied to extra finer meshes
except for the even coverage, our method outperforms all for the reconstructions using TSDF and get smoother images
variants on all metrics significantly. Our proposed NeurAR than our rendering images from TSDF directly, denoising and
achieves better reconstruction results with shorter view paths filling the holes are non-trivial tasks, particularly in scenes
compared with other variants and existing work. having complex structures.
Compared with methods using the implicit representation
without path planning (Offline FT, Online FT, Online RS),
our method demonstrates significant improvements in the C. Ablation study
image quality and geometry accuracy. Methods without the Training Iterations between Steps The number of iterations
planned views 1) are even unable to converge to a reasonable for NeRF optimization iter allowed for planning a view step
result in the large scene (NeRF will fail due to overfitting affects final results. We choose different iterations to run our
without carefully planned input viewpoint coverage), 2) have NeurAR system in the scene cabin and compare PSNR of the
many holes in the objects as these regions are unseen in the rendered images of the reconstructed models. PSNR for the
images and 3) inferior image details on the surface of objects models optimized using 300, 700, 1400 iterations is 26.16,
as these regions have a little overlap of different views, making 28.58, 26.91. This is because too few iterations may lead to
inferring the 3D geometry hard (check red box in Fig. 6 uncertainty not being optimized well while too many steps
for visual comparison). In addition, these variants tend to may lead to overfitting to images added.
have ghost effects in the empty space. This is largely because Depth noise We construct variants of NeurAR using different
the viewpoints are designed to make the camera look at the noise magnitude from no noise, the noise magnitude of L515,
objects and the empty space is not considered. Notice that to the double (2 × µ(z), 2 × σ(z) ) and the triple (3 × µ(z),
placing viewpoints covering the whole space without the aid 3×σ(z) ) noise magnitude of L515. Table II shows the metrics
of visualization tools as feedback is non-trivial for humans. measuring the cabin models reconstructed by these variants:
Assuming we have the 3D shape of a scene with the target NeurAR can be accelerated with very noisy depth at a cost of
object in the center (which is our goal), to cover the whole only a minor drop in the reconstruction quality. The robustness
space, we distribute viewpoints evenly at a dome centered at verifies the effect of dampening the depth supervision during
the scene center with a radius of 80 meters for Alexander and the training in Section III-C .
3 meters for others. All viewpoints look at the center. An even Number of views To further study the influence, we set the
coverage path can provide a good reference as even coverage maximum allowed views from 18 to 58 for the cabin scene and
8 IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JANUARY 2023

TABLE II [3] O. Mendez, S. Hadfield, N. Pugeault, and R. Bowden, “Taking the

I NFLUENCE OF DIFFERENT NOISE MAGNITUDE scenic route to 3d: Optimising reconstruction from moving cameras,” in
Proceedings of the IEEE International Conference on Computer Vision
Method PSNR↑ SSIM↑ LPIPS↓ Acc↓ Comp↓ C.R.↑ (ICCV), 2017, pp. 4677–4685.
[4] S. Isler, R. Sabzevari, J. Delmerico, and D. Scaramuzza, “An information
Ours (no noise) 28.71 0.926 0.043 0.92 0.90 0.75 gain formulation for active volumetric 3d reconstruction,” in 2016 IEEE
Ours (with noise) 28.35 0.902 0.062 1.07 0.95 0.74
Ours (noise x2) 28.19 0.903 0.062 0.99 0.97 0.74
International Conference on Robotics and Automation (ICRA), 2016, pp.
Ours (noise x3) 27.65 0.900 0.063 1.00 0.94 0.77 3477–3484.
[5] S. Wu, W. Sun, P. Long, H. Huang, D. Cohen-Or, M. Gong, O. Deussen,
and B. Chen, “Quality-driven poisson-guided autoscanning,” ACM Trans-
actions on Graphics (TOG), vol. 33, no. 6, pp. 1–12, 2014.
provide the PSNR for the reconstructed models under different [6] S. Song and S. Jo, “Surface-based exploration for autonomous 3d
modeling,” in 2018 IEEE International Conference on Robotics and
view settings. Automation (ICRA), 2018, pp. 4319–4326.
TABLE III [7] C. M. Bishop, Pattern Recognition and Machine Learning. Springer-
E VALUATIONS ON THE RECONSTRUCTED 3D MODELS USING DIFFERENT Verlag New York, 2006.
NUMBER OF VIEWS [8] Q. Ye and T.-K. Kim, “Occlusion-aware hand pose estimation using
hierarchical mixture density network,” in Proceedings of the European
Views PSNR↑ SSIM↑ LPIPS↓ Acc↓ Comp↓ C.R.↑ Conference on Computer Vision (ECCV), 2018, pp. 801–817.
[9] L. Schmid, M. Pantic, R. Khanna, L. Ott, R. Siegwart, and J. Nieto, “An
18 22.47 0.802 0.155 2.18 1.31 0.59 efficient sampling-based method for online informative path planning in
28 28.35 0.902 0.062 1.07 0.95 0.74 unknown environments,” IEEE Robotics and Automation Letters (RAL),
38 30.03 0.919 0.053 0.90 0.86 0.77
vol. 5, no. 2, pp. 1500–1507, 2020.
58 29.39 0.926 0.050 0.82 0.90 0.79
[10] J. I. Vasquez-Gomez, L. E. Sucar, and R. Murrieta-Cid, “View/state
planning for three-dimensional object reconstruction under uncertainty,”
Autonomous Robots, vol. 41, no. 1, pp. 89–109, 2017.
[11] M. Devrim Kaba, M. Gokhan Uzunbas, and S. Nam Lim, “A reinforce-
V. C ONCLUSION ment learning approach to the view planning problem,” in Proceedings
of the IEEE Conference on Computer Vision and Pattern Recognition
In this paper, we have presented an autonomous 3D recon- (CVPR), 2017, pp. 6933–6941.
[12] L. Schmid, C. Ni, Y. Zhong, R. Siegwart, and O. Andersson, “Fast
struction method based on an implicit neural representation, and compute-efficient sampling-based local exploration planning via
with the view path planned according to a proxy of PSNR. The distribution learning,” IEEE Robotics and Automation Letters (RAL),
proxy is acquired by learning uncertainty for the estimation of vol. 7, no. 3, pp. 7810–7817, 2022.
[13] B. Hepp, D. Dey, S. N. Sinha, A. Kapoor, N. Joshi, and O. Hilliges,
the color of a spatial point on a ray during the reconstruction. “Learn-to-score: Efficient 3d scene exploration by predicting view util-
A strong correlation is discovered between the uncertainty and ity,” in Proceedings of the European Conference on Computer Vision
PSNR, which is verified by extensive experiments. Compared (ECCV), 2018, pp. 437–452.
with variants with a pre-defined trajectory and variants using [14] J. L. Schönberger, E. Zheng, J.-M. Frahm, and M. Pollefeys, “Pixelwise
view selection for unstructured multi-view stereo,” in Proceedings of the
TSDF, our method demonstrates significant improvements. Our European Conference on Computer Vision (ECCV), 2016, pp. 501–518.
work shows a promising potential of implicit neural represen- [15] S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe, P. Kohli,
tations for high-quality autonomous 3D reconstruction. J. Shotton, S. Hodges, D. Freeman, A. Davison et al., “Kinectfusion:
real-time 3d reconstruction and interaction using a moving depth cam-
One limitation of NeurAR is the optimization of implicit era,” in Proceedings of the 24th annual ACM symposium on User
neural representation is slow and the computation consumption interface software and technology (UIST), 2011, pp. 559–568.
is high. The optimization of the model between view steps [16] A. Hornung, K. M. Wurm, M. Bennewitz, C. Stachniss, and W. Burgard,
“Octomap: An efficient probabilistic 3d mapping framework based on
takes about 50-120 seconds depending on the number of octrees,” Autonomous robots, vol. 34, no. 3, pp. 189–206, 2013.
iterations, which is much slower than existing unknown area [17] “Nerf project page,” https://www.matthewtancik.com/nerf, 2020.
explorations. However, our method is agnostic to the under- [18] A. Gropp, L. Yariv, N. Haim, M. Atzmon, and Y. Lipman, “Implicit
geometric regularization for learning shapes,” in Proceedings of the
lying NeRF realization and therefore, we can easily swap the International Conference on Machine Learning (ICML), 2020, pp. 3789–
module for more advanced NeRF variants emerging lately [23]. 3799.
Future work includes achieving acceleration of the training and [19] X. Zhang, S. Bi, K. Sunkavalli, H. Su, and Z. Xu, “Nerfusion: Fusing
radiance fields for large-scale scene reconstruction,” in Proceedings of
rendering for NeurAR. the IEEE/CVF Conference on Computer Vision and Pattern Recognition
The other limitation is that we assume known camera poses, (CVPR), 2022, pp. 5449–5458.
which require extra location systems to make NeurAR work. [20] C. V. Nguyen, S. Izadi, and D. Lovell, “Modeling kinect sensor noise for
improved 3d reconstruction and tracking,” in 2012 Second International
Future work includes joint optimization of the camera poses, Conference on 3D Imaging, Modeling, Processing, Visualization &
reconstruction, and view planning. Also, extending NeurAR to Transmission, 2012, pp. 524–530.
multiple agents for large scene reconstructions with distributed [21] S. Song, D. Kim, and S. Choi, “View path planning via online multiview
stereo for 3-d modeling of large-scale structures,” IEEE Transactions on
learning [24] and adverse environments with the mmWave Robotics, 2021.
radar [25] are interesting directions. [22] D. Peralta, J. Casimiro, A. M. Nilles, J. A. Aguilar, R. Atienza, and
R. Cajote, “Next-best view policy for 3d reconstruction,” in Proceedings
R EFERENCES of the European Conference on Computer Vision (ECCV), 2020, pp. 558–
573.
[1] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, [23] T. Müller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics
and R. Ng, “Nerf: Representing scenes as neural radiance fields for view primitives with a multiresolution hash encoding,” ACM Transactions on
synthesis,” in Proceedings of the European Conference on Computer Graphics (TOG), vol. 41, no. 4, pp. 1–15, 2022.
Vision (ECCV), 2020, pp. 405–421. [24] Y. Huang, Y. Sun, Z. Zhu, C. Yan, and J. Xu, “Tackling data heterogene-
[2] E. Sucar, S. Liu, J. Ortiz, and A. J. Davison, “imap: Implicit mapping and ity: A new unified framework for decentralized sgd with sample-induced
positioning in real-time,” in Proceedings of the IEEE/CVF International topology,” in Proceedings of the International Conference on Machine
Conference on Computer Vision (ICCV), 2021, pp. 6229–6238. Learning (ICML), 2022, pp. 9310–9345.
 9

[25] A. Chen, X. Wang, S. Zhu, Y. Li, J. Chen, and Q. Ye, “mmbody

benchmark: 3d body reconstruction dataset and analysis for millimeter
wave radar,” in Proceedings of the 30th ACM International Conference
on Multimedia, 2022, pp. 3501–3510.