(Arxiv 2021) NeRF--: Neural Radiance Fields Without Known Camera Parameters

NeRF--: Neural Radiance Fields Without Known Camera Parameters

Project Page | Arxiv | Colab Notebook | Data

Zirui Wang¹, Shangzhe Wu², Weidi Xie², Min Chen³, Victor Adrian Prisacariu¹.

¹Active Vision Lab + ²Visual Geometry Group + ³e-Research Centre, University of Oxford.

Overview

We provide 3 training targets in this repository, under the tasks directory:

  1. task/nerfmm/train.py: This is our main training script for the NeRF-LLFF dataset, which estimates camera poses, focal lenghts and a NeRF jointly and monitors the absolute trajectory error (ATE) between our estimation of camera parameters and COLMAP estimation during training. This target can also start training from a COLMAP initialisation and refine the COLMAP camera parameters.
  2. task/refine_nerfmm/train.py: This is the training script that refines a pretrained nerfmm system.
  3. task/any_folder/train.py: This is a training script that takes a folder that contains forward-facing images and trains with our nerfmm system without making any comparison with COLMAP. It is similar to what we offer in our CoLab notebook and we treat this any_folder target as a playgraound, where users can try novel view synthesis by just providing an image folder and do not care how the camera parameter estimation compares with COLMAP.

For each target, we provide relevant utilities to evaluate our system. Specifically,

  • for the nerfmm target, we provide three utility files:
    • eval.py to evaluate image rendering quality on validation splits with PSNR, SSIM and LPIPS, i.e, results in Table 1.
    • spiral.py to render novel views using a spiral camera trajectory, i.e. results in Figure 1.
    • vis_learned_poses.py to visualise our camera parameter estimation with COLMAP estimation in 3D. It also computes ATE between them, i.e. E1 in Table 2.
  • for the refine_nerfmm target, all utilities in nerfmm target above are compatible with refine_nerfmm target, since it just refines a pretrained nerfmm system.
  • for the any_folder target, it has its own spiral.py and vis_learned_poses.py utilities, as it does not compare with COLMAP. It does not have a eval.py file as this target is treated as a playground and does not split images to train/validation sets. It only provides novel view synthesis results via the spiral.py file.

Table of Content

Environment

We provide a requirement.yml file to set up a conda environment:

git clone https://github.com/ActiveVisionLab/nerfmm.git
cd nerfmm
conda env create -f environment.yml

Generally, our code should be able to run with any pytorch >= 1.1 .

(Optional) Install open3d for visualisation. You might need a physical monitor to install this lib.

pip install open3d

Get Data

We use the NeRF-LLFF dataset with two small structural changes:

  1. We remove their image_4 and image_8 folder and downsample images to any desirable resolution during data loading dataloader/with_colmap.py, by calling PyTorch's interpolate function.
  2. We explicitly generate two txt files for train/val image ids. i.e. take every 8th image as the validation set, as in the official NeRF train/val split. The only difference is that we store them as txt files while NeRF split them during data loading. The file produces these two txt files is utils/split_dataset.py.

In addition to the NeRF-LLFF dataset, we provide two demo scenes to demonstrate how to use the any_folder target.

We pack the re-structured LLFF data and our data to a tar ball (~1.8G), to get it, run:

wget https://www.robots.ox.ac.uk/~ryan/nerfmm2021/nerfmm_release_data.tar.gz

Untar the data:

tar -xzvf path/to/the/tar.gz

Training

We show how to:

  1. train a nerfmm from scratch, i.e. initialise camera poses with identity matrices and focal lengths with image resolution:
    python tasks/nerf/train.py \
    --base_dir='path/to/nerfmm_release/data' \
    --scene_name='LLFF/fern'
  2. train a nerfmm from COLMAP initialisation:
    python tasks/nerf/train.py \
    --base_dir='path/to/nerfmm_release/data' \
    --scene_name='LLFF/fern' \
    --start_refine_pose_epoch=1000 \
    --start_refine_focal_epoch=1000
    This command initialises a nerfmm target with COLMAP parameters, trains with them for 1000 epochs, and starts refining those parameters after 1000 epochs.
  3. train a nerfmm from a pretrained nerfmm:
    python tasks/refine_nerfmm/train.py \
    --base_dir='path/to/nerfmm_release/data' \
    --scene_name='LLFF/fern' --start_refine_epoch=1000 \
    --ckpt_dir='path/to/a/dir/contains/nerfmm/ckpts'
    This command initialises a refine_nerfmm target with a set of pretrained nerfmm parameters, trains with them for 1000 epochs, and starts refining those parameters after 1000 epochs.
  4. train an any_folder from scratch given an image folder:
    python tasks/any_folder/train.py \
    --base_dir='path/to/nerfmm_release/data' \
    --scene_name='any_folder_demo/desk'
    This command trains an any_folder target using a provided demo scene desk.

(Optional) set a symlink to the downloaded data:

mkdir data_dir  # do it in this nerfmm repo
cd data_dir
ln -s /path/to/downloaded/data ./nerfmm_release_data
cd ..

this can simplify the above training commands, for example:

python tasks/nerfmm/train.py

Evaluation

Compute image quality metrics

Call eval.py in nerfmm target:

python tasks/nerfmm/eval.py \
--base_dir='path/to/nerfmm_release/data' \
--scene_name='LLFF/fern' \
--ckpt_dir='path/to/a/dir/contains/nerfmm/ckpts'

This file can be used to evaluate a checkpoint trained with refine_nerfmm target. For some scenes, you might need to tweak with --opt_eval_lr option to get the best results. Common values for opt_eval_lr are 0.01 / 0.005 / 0.001 / 0.0005 / 0.0001. The default value is 0.001. Overall, it finds validation poses that can produce highest PSNR on validation set while freezing NeRF and focal lengths. We do this because the learned camera pose space is different from the COLMAP estimated camera pose space.

Render novel views

Call spiral.py in each target. The spiral.py in nerfmm is compatible with refine_nerfmm target:

python spiral.py \
--base_dir='path/to/nerfmm_release/data' \
--scene_name='LLFF/fern' \
--ckpt_dir='path/to/a/dir/contains/nerfmm/ckpts'

Visualise estimated poses in 3D

Call vis_learned_poses.py in each target. The vis_learned_poses.py in nerfmm is compatible with refine_nerfmm target:

python spiral.py \
--base_dir='path/to/nerfmm_release/data' \
--scene_name='LLFF/fern' \
--ckpt_dir='path/to/a/dir/contains/nerfmm/ckpts'

Acknowledgement

Shangzhe Wu is supported by Facebook Research. Weidi Xie is supported by Visual AI (EP/T028572/1).

The authors would like to thank Tim Yuqing Tang for insightful discussions and proofreading.

During our NeRF implementation, we referenced several open sourced NeRF implementations, and we thank their contributions. Specifically, we referenced functions from nerf and nerf-pytorch, and borrowed/modified code from nerfplusplus and nerf_pl. We especially appreciate the detailed code comments and git issue answers in nerf_pl.

Citation

@article{wang2021nerfmm,
  title={Ne{RF}$--$: Neural Radiance Fields Without Known Camera Parameters},
  author={Zirui Wang and Shangzhe Wu and Weidi Xie and Min Chen and Victor Adrian Prisacariu},
  journal={arXiv preprint arXiv:2102.07064},
  year={2021}
}
Comments
  • Test with few images

    Test with few images

    Hi,

    Thank you for sharing this work.

    I tested the code with 6 images of one scene. I found if colmap can converge on these images, original NeRF can produce better performance than nerfmm. In my experiment, nerfmm's output is very blurry. It seems nerfmm get stuck in a local minima because I got same result when tseting with 10000 epochs and 20000 epochs.

    Did you try some experiments using less than 10 images? I hope nerfmm can process few images better.

    By the way, your paper shows one limitation of nerfmm is that it cannot produce a good result with images which have few overlaps. I think it is a big problem of nerf-related models. I expect new method coming out to solve this problem.

  • BLEFF data download too slow

    BLEFF data download too slow

    Hi, thanks for the BLEFF data. However, it is downloading too slowly on my side, only 80kB/s and that would take 3 days to download... Is it possible to upload it to e.g. google drive for a faster download?

  • eval.py test on 'latest_nerf.pth'?

    eval.py test on 'latest_nerf.pth'?

    Dear authors: I have a question about the evaluation implementation. I found on "eval.py" that you only test the results on "latest_nerf.pth". This indicates that you are using the latest NERF model, which is trained with 10,000 epochs (as set on args) for comparing the performance of each baseline method (like the original NERF model). Am I right? Thank you for your time!

  • Gap between retraining results and demonstrated results.

    Gap between retraining results and demonstrated results.

    Dear authors: Thanks for your excellent work. I try to reproduce the results as shown in the paper (e.g., PSNR 22.48 in Tab.1). I use the scene LLFF/fern and the following command python tasks/nerfmm/train.py \ --base_dir='/mnt/cephfs/dataset/NVS/' \ --scene_name='LLFF/fern' However, the best PSNR only reaches 21.171 as recorded in the log.txt. 9TCMD~@XQ9 T{NLE22J5K)6

    Are there details I neglected when reproducing the results?
    
  • Dataloader is slow

    Dataloader is slow

    Hello,

    First of all, thank you for your work!

    When I train the any_folder task, I don't know why the speed of the data loader is too slow.

    Thank you in advance.

    Best regards

  • about gpu usage

    about gpu usage

    https://github.com/ActiveVisionLab/nerfmm/blob/27faab66a927ea14259125e1140231f0c8f6d14c/tasks/any_folder/train.py#L206 In this line, you backward gradient per image, so I guess it will consume more GPU resources when compute all gradients. I want to know how much GPU memory do you use in your practice. Thanks in advance!

  • Question about the pos_enc.py

    Question about the pos_enc.py

    Hi, I highly appreciate your work and I am learning your code recently. But I noticed that the position encoder didn't multiple the pi, I wandered whether it was a mistake. BTW I also found some other NeRF codes didn't multiple pi.

  • Multiply the distance by the norm of direction

    Multiply the distance by the norm of direction

    Hi there, thanks for sharing the code!

    When computing the distances of samples during rendering, some open sourced codes such as nerf_pl and mip-nerf multiplies the distances with the norms of direction. Seems like there's no such operation in this repo. (volume_op.py#L191) Could you give any hint on the difference?

  • get_ndc_rays function returns inf ray_o and ray_d.

    get_ndc_rays function returns inf ray_o and ray_d.

    Hello,

    While testing out with a custom unbounded dataset (with colmap init) the get_ndc_rays function returns ray_o and ray_d with inf values.

    More specifically, in the below snippet (from the get_ndc_rays) the rays_o is being reinitialised in such a way that the last column (rays_o[...,2]) is zeros. And in the last two equations of the snippet a divide by zero is leading to inf.

     t = -(near + rays_o[..., 2]) / rays_d[..., 2]
    rays_o = rays_o + t[..., None] * rays_d   ##### last column of rays_o are all zeros.
    
    # Store some intermediate homogeneous results
    ox_oz = rays_o[..., 0] / rays_o[..., 2]
    oy_oz = rays_o[..., 1] / rays_o[..., 2]
    

    I am unable to figure out why something like this could be happening.

    And also it is worth mentioning that this happened with only one of the three dataset that I tested with, the other two are working fine.

    It would be great if I would get any suggestions on why this might be happening. And please do let me know if more information is needed.

    Thanks in Advance,

  • Cannot reproduce horns scene on experiment E1

    Cannot reproduce horns scene on experiment E1

    Hi, firstly thank you for your amazing work and beautiful codes.

    I have been trying to reproduce the horns scene of E1 table 2 (reported PSNR=23.13). The provided setting seemed to not work in this scene. For my case, the train PSNR reached around 21 near the end of training and the average test PSNR is around 18, translation error is around 0.14 and rotation error is around 65.0. Yet the same setting works well for the other 7 scenes and basically match the results in the paper.

    I tried to lower the nerf_lr to 5e-4 and the results are equally poor. So I guess there is something wrong with the hyperparameter for this scene? I would like to know whether you have observed such poor training and testing on the horns scene before. It would be great if you could provide some advice.

    Thanks in advance.

  • Mesh ouput

    Mesh ouput

    First ! thanks for the very useful code, brilliant work ! I would like to get a mesh output (3D) in .ply format ! do you have any idea how to do that?

  • `nerfmm/spiral.py` bug if trained with colmap initialized pose

    `nerfmm/spiral.py` bug if trained with colmap initialized pose

    Hi,

    Thanks for you great work!

    One bug I have encountered here is if train with:

    --start_refine_pose_epoch 1000
    --start_refine_focal_epoch 1000
    

    There will be a field named init_c2ws in the models state dict of LearnPose.

    However if use spiral for inference, the default action is to initialize a LearnPose with self.init_c2ws as None, which leads to Unexpected key(s) in state_dict: "init_c2w". when load it in spiral.py.

    Maybe this can be fixed by add a arg that controlls whether the R, t in LearnPose is learnt from scratch or the offset from COLMAP ones in nerfmm/spiral.py

Blender add-on: Add to Cameras menu: View → Camera, View → Add Camera, Camera → View, Previous Camera, Next Camera
Blender add-on: Add to Cameras menu: View → Camera, View → Add Camera, Camera → View, Previous Camera, Next Camera

Blender add-on: Camera additions In 3D view, it adds these actions to the View|Cameras menu: View → Camera : set the current camera to the 3D view Vie

Feb 8, 2022
Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields.
Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields.

This repository contains the code release for Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. This implementation is written in JAX, and is a fork of Google's JaxNeRF implementation. Contact Jon Barron if you encounter any issues.

Aug 1, 2022
This repository contains a PyTorch implementation of "AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis".
This repository contains a PyTorch implementation of

AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis | Project Page | Paper | PyTorch implementation for the paper "AD-NeRF: Audio

Aug 1, 2022
Code release for DS-NeRF (Depth-supervised Neural Radiance Fields)
Code release for DS-NeRF (Depth-supervised Neural Radiance Fields)

Depth-supervised NeRF: Fewer Views and Faster Training for Free Project | Paper | YouTube Pytorch implementation of our method for learning neural rad

Aug 8, 2022
A PyTorch implementation of NeRF (Neural Radiance Fields) that reproduces the results.
A PyTorch implementation of NeRF (Neural Radiance Fields) that reproduces the results.

NeRF-pytorch NeRF (Neural Radiance Fields) is a method that achieves state-of-the-art results for synthesizing novel views of complex scenes. Here are

Jul 30, 2022
D-NeRF: Neural Radiance Fields for Dynamic Scenes
 D-NeRF: Neural Radiance Fields for Dynamic Scenes

D-NeRF: Neural Radiance Fields for Dynamic Scenes [Project] [Paper] D-NeRF is a method for synthesizing novel views, at an arbitrary point in time, of

Jul 29, 2022
Code release for NeRF (Neural Radiance Fields)
Code release for NeRF (Neural Radiance Fields)

NeRF: Neural Radiance Fields Project Page | Video | Paper | Data Tensorflow implementation of optimizing a neural representation for a single scene an

Aug 9, 2022
Pytorch implementation for A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose
Pytorch implementation for A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose

A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose Paper | Website | Data A-NeRF: Articulated Neural Radiance F

Aug 3, 2022
A minimal TPU compatible Jax implementation of NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

NeRF Minimal Jax implementation of NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Result of Tiny-NeRF RGB Depth

Jul 24, 2022
Build upon neural radiance fields to create a scene-specific implicit 3D semantic representation, Semantic-NeRF
Build upon neural radiance fields to create a scene-specific implicit 3D semantic representation, Semantic-NeRF

Semantic-NeRF: Semantic Neural Radiance Fields Project Page | Video | Paper | Data In-Place Scene Labelling and Understanding with Implicit Scene Repr

Aug 4, 2022
Point-NeRF: Point-based Neural Radiance Fields
Point-NeRF: Point-based Neural Radiance Fields

Point-NeRF: Point-based Neural Radiance Fields Project Sites | Paper | Primary c

Aug 2, 2022
CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields
CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields

CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields Paper | Supplementary | Video | Poster If you find our code or paper useful, please

Jul 25, 2022
Plenoxels: Radiance Fields without Neural Networks

Plenoxels: Radiance Fields without Neural Networks Alex Yu*, Sara Fridovich-Keil*, Matthew Tancik, Qinhong Chen, Benjamin Recht, Angjoo Kanazawa UC Be

Aug 1, 2022
BARF: Bundle-Adjusting Neural Radiance Fields 🤮 (ICCV 2021 oral)

BARF ?? : Bundle-Adjusting Neural Radiance Fields Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey IEEE International Conference on Comp

Jul 28, 2022
[ICCV 2021 Oral] NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo
[ICCV 2021 Oral] NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo

NerfingMVS Project Page | Paper | Video | Data NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo Yi Wei, Shaohui

Aug 5, 2022
Instant-nerf-pytorch - NeRF trained SUPER FAST in pytorch

instant-nerf-pytorch This is WORK IN PROGRESS, please feel free to contribute vi

Jul 25, 2022
This is the code for Deformable Neural Radiance Fields, a.k.a. Nerfies.

Deformable Neural Radiance Fields This is the code for Deformable Neural Radiance Fields, a.k.a. Nerfies. Project Page Paper Video This codebase conta

Aug 5, 2022
Open source repository for the code accompanying the paper 'Non-Rigid Neural Radiance Fields Reconstruction and Novel View Synthesis of a Deforming Scene from Monocular Video'.
Open source repository for the code accompanying the paper 'Non-Rigid Neural Radiance Fields Reconstruction and Novel View Synthesis of a Deforming Scene from Monocular Video'.

Non-Rigid Neural Radiance Fields This is the official repository for the project "Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synt

Jul 29, 2022
Code for KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs
Code for KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs

KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs Check out the paper on arXiv: https://arxiv.org/abs/2103.13744 This repo cont

Jul 28, 2022