BARF: Bundle-Adjusting Neural Radiance Fields 🤮 (ICCV 2021 oral)

BARF 🤮 : Bundle-Adjusting Neural Radiance Fields

Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey
IEEE International Conference on Computer Vision (ICCV), 2021 (oral presentation)

Project page: https://chenhsuanlin.bitbucket.io/bundle-adjusting-NeRF
arXiv preprint: https://arxiv.org/abs/2104.06405

We provide PyTorch code for the NeRF experiments on both synthetic (Blender) and real-world (LLFF) datasets.


Prerequisites

This code is developed with Python3 (python3). PyTorch 1.9+ is required.
It is recommended use Anaconda to set up the environment. Install the dependencies and activate the environment barf-env with

conda env create --file requirements.yaml python=3
conda activate barf-env

Initialize the external submodule dependencies with

git submodule update --init --recursive

Dataset

  • Synthetic data (Blender) and real-world data (LLFF)

    Both the Blender synthetic data and LLFF real-world data can be found in the NeRF Google Drive. For convenience, you can download them with the following script: (under this repo)
    # Blender
    gdown --id 18JxhpWD-4ZmuFKLzKlAw-w5PpzZxXOcG # download nerf_synthetic.zip
    unzip nerf_synthetic.zip
    rm -f nerf_synthetic.zip
    mv nerf_synthetic data/blender
    # LLFF
    gdown --id 16VnMcF1KJYxN9QId6TClMsZRahHNMW5g # download nerf_llff_data.zip
    unzip nerf_llff_data.zip
    rm -f nerf_llff_data.zip
    mv nerf_llff_data data/llff
    The data directory should contain the subdirectories blender and llff. If you already have the datasets downloaded, you can alternatively soft-link them within the data directory.
  • iPhone (TODO)


Running the code

  • BARF models

    To train and evaluate BARF:

    # <GROUP> and <NAME> can be set to your likes, while <SCENE> is specific to datasets
    
    # Blender (<SCENE>={chair,drums,ficus,hotdog,lego,materials,mic,ship})
    python3 train.py --group=<GROUP> --model=barf --yaml=barf_blender --name=<NAME> --data.scene=<SCENE> --barf_c2f=[0.1,0.5]
    python3 evaluate.py --group=<GROUP> --model=barf --yaml=barf_blender --name=<NAME> --data.scene=<SCENE> --data.val_sub= --resume
    
    # LLFF (<SCENE>={fern,flower,fortress,horns,leaves,orchids,room,trex})
    python3 train.py --group=<GROUP> --model=barf --yaml=barf_llff --name=<NAME> --data.scene=<SCENE> --barf_c2f=[0.1,0.5]
    python3 evaluate.py --group=<GROUP> --model=barf --yaml=barf_llff --name=<NAME> --data.scene=<SCENE> --resume

    All the results will be stored in the directory output/<GROUP>/<NAME>. You may want to organize your experiments by grouping different runs in the same group.

    To train baseline models:

    • Full positional encoding: omit the --barf_c2f argument.
    • No positional encoding: add --arch.posenc!.

    If you want to evaluate a checkpoint at a specific iteration number, use --resume=<ITER_NUMBER> instead of just --resume.

  • Training the original NeRF

    If you want to train the reference NeRF models (assuming known camera poses):

    # Blender
    python3 train.py --group=<GROUP> --model=nerf --yaml=nerf_blender --name=<NAME> --data.scene=<SCENE>
    python3 evaluate.py --group=<GROUP> --model=nerf --yaml=nerf_blender --name=<NAME> --data.scene=<SCENE> --data.val_sub= --resume
    
    # LLFF
    python3 train.py --group=<GROUP> --model=nerf --yaml=nerf_llff --name=<NAME> --data.scene=<SCENE>
    python3 evaluate.py --group=<GROUP> --model=nerf --yaml=nerf_llff --name=<NAME> --data.scene=<SCENE> --resume

    If you wish to replicate the results from the original NeRF paper, use --yaml=nerf_blender_repr or --yaml=nerf_llff_repr instead for Blender or LLFF respectively. There are some differences, e.g. NDC will be used for the LLFF forward-facing dataset. (The reference NeRF models considered in the paper do not use NDC to parametrize the 3D points.)

  • Visualizing the results

    We have included code to visualize the training over TensorBoard and Visdom. The TensorBoard events include the following:

    • SCALARS: the rendering losses and PSNR over the course of optimization. For BARF, the rotational/translational errors with respect to the given poses are also computed.
    • IMAGES: visualization of the RGB images and the RGB/depth rendering.

    We also provide visualization of 3D camera poses in Visdom. Run visdom -port 9000 to start the Visdom server.
    The Visdom host server is default to localhost; this can be overridden with --visdom.server (see options/base.yaml for details). If you want to disable Visdom visualization, add --visdom!.


Codebase structure

The main engine and network architecture in model/barf.py inherit those from model/nerf.py. This codebase is structured so that it is easy to understand the actual parts BARF is extending from NeRF. It is also simple to build your exciting applications upon either BARF or NeRF -- just inherit them again! This is the same for dataset files (e.g. data/blender.py).

To understand the config and command lines, take the below command as an example:

python3 train.py --group=<GROUP> --model=barf --yaml=barf_blender --name=<NAME> --data.scene=<SCENE> --barf_c2f=[0.1,0.5]

This will run model/barf.py as the main engine with options/barf_blender.yaml as the main config file. Note that barf hierarchically inherits nerf (which inherits base), making the codebase customizable.
The complete configuration will be printed upon execution. To override specific options, add --<key>=value or --<key1>.<key2>=value (and so on) to the command line. The configuration will be loaded as the variable opt throughout the codebase.

Some tips on using and understanding the codebase:

  • The computation graph for forward/backprop is stored in var throughout the codebase.
  • The losses are stored in loss. To add a new loss function, just implement it in compute_loss() and add its weight to opt.loss_weight.<name>. It will automatically be added to the overall loss and logged to Tensorboard.
  • If you are using a multi-GPU machine, you can add --gpu=<gpu_number> to specify which GPU to use. Multi-GPU training/evaluation is currently not supported.
  • To resume from a previous checkpoint, add --resume=<ITER_NUMBER>, or just --resume to resume from the latest checkpoint.
  • (to be continued....)

If you find our code useful for your research, please cite

@inproceedings{lin2021barf,
  title={BARF: Bundle-Adjusting Neural Radiance Fields},
  author={Lin, Chen-Hsuan and Ma, Wei-Chiu and Torralba, Antonio and Lucey, Simon},
  booktitle={IEEE International Conference on Computer Vision ({ICCV})},
  year={2021}
}

Please contact me ([email protected]) if you have any questions!

Owner
Chen-Hsuan Lin
Research scientist @NVIDIA, PhD in Robotics @ CMU
Chen-Hsuan Lin
Comments
  • How to create a 3D-model from the output of iphone experiment?

    How to create a 3D-model from the output of iphone experiment?

    Hi Chen-Hsuan,

    I successfully created an experiment in the iphone.py style. That is, without any information of camera poses. Is it possible to create a 3D model from the output?

    Thanks in advance,

  • Cannot reproduce results on llff:fern

    Cannot reproduce results on llff:fern

    Thanks for your sharing of codes.

    I followed all your tips in the ReadME and tried to reproduce the results of your paper. However, the rotation error of my test is much higher than yours in the paper.

    | llff:fern | rotation | translation( x100 ) | | :-----| ----: | :----: | | paper | 0.191 | 0.192 | | reproduce | 0.689 | 0.193 |

    Can you give some advice? Besides, the depth map and the rendered RGB seem good. image

    Thanks!

  • How was focal length calculated for the iPhone 12 in the example?

    How was focal length calculated for the iPhone 12 in the example?

    In the example in data/iphone.py the focal length is computed as self.focal = self.raw_W * 4.2 / (12.8 / 2.55). I am wondering what the specific constants mean?

  • Documentation on the LLFF data scene?

    Documentation on the LLFF data scene?

    Hi Chen-Hsuan,

    Do you have any documentation regarding how to set up another scene experiment? I did not found anything on the readme nor in the paper.

    I'm refering to this folders and files: Data_LLFF

    Thanks in advance,

  • Out of memory problem

    Out of memory problem

    Hi! Thank you for sharing your fantastic work! When I was trying to train the network with a 3090 in barf mode with position encoding with blender datasets, after validating, it always show that RuntimeError: CUDA out of memory. Tried to allocate 5.25 GiB (GPU 0; 23.70 GiB total capacity; 15.75 GiB already allocated; 4.71 GiB free; 15.77 GiB reserved in total by PyTorch) Could you please tell me which parameter I should change to avoid this problem? Thank you very much!

  • Camera Pose Optimization with Geometry Prior

    Camera Pose Optimization with Geometry Prior

    Hey,

    I am trying to optimize the camera positions given a non textured mesh. Right now NeRF implementation does not consider this prior and samples random rays. I was wondering how do I provide my mesh as initial input to BARF? Since I already have the mesh it should be instant to fix the camera poses!

    I've tried to use PyTorch3D as differentiable renderer with BARF's additional parameters for camera pose optimisation, but it doesn't work. The cameras drift away and loss becomes NaN.

  • Result seems not right

    Result seems not right

    Hello, thanks for your great work! I ran the barf code on the chair of the blender dataset, but the validation result doesn't seem right on tensorboard, is this normal? Train: MZO$U7N$IG($FQ(L(}BOT

    Test: loss

  • Question about camera pose transformation for LLFF

    Question about camera pose transformation for LLFF

    Hi, Chen-Hsuan Lin. Thank you for sharing the great work!

    I have been reading the code and I did not understand very well about camera pose transformation when calling __getitem__ method for LLFF dataset: https://github.com/chenhsuanlin/bundle-adjusting-NeRF/blob/main/data/llff.py#L104

    In my understanding, camera pose in returned values from parse_cameras_and_bounds is camera-to-world matrix and its coordinate system is [right, up, backwards]. https://github.com/chenhsuanlin/bundle-adjusting-NeRF/blob/main/data/llff.py#L42

    Then, the camera pose is transformed by parse_raw_camera when calling __getitem__, but I could not follow what the transformation did: https://github.com/chenhsuanlin/bundle-adjusting-NeRF/blob/main/data/llff.py#L104 Could you please let me know?

  • Multi-GPU training (DataParallel)

    Multi-GPU training (DataParallel)

    Hi @chenhsuanlin

    Thank you for sharing this nice work. I'm just curious if you happen to have multi-gpu training code by hand? I was trying to train BARF with multi GPU, but got stuck in a weird OOM issue: the GPU memory explode into over 50G, while your original code base takes less than 10G on blender/lego

    Here's the edit I made: https://github.com/Ir1d/bundle-adjusting-NeRF/commit/904228c3a243e939d96e5595f7073779f95b997a The command to run: CUDA_VISIBLE_DEVICES=0,1 python train.py --group=blender --model=barf --yaml=barf_blender --name=lego_baseline --data.scene=lego --gpu=1 --visdom! --batch_size=2

    Do you know what might be the leading to the OOM here? Thank you!

  • Training BARF with another real data

    Training BARF with another real data

    Hello Chen-Hsuan,

    Thank you again for releasing the code. I have tried to train BARF with Freiburg Cars dataset. (please see the attached mp4 file) As you can see, it is a real dataset with a spherical sequence. (hybrid between LLFF and NeRF-Synthetic)

    I ran COLMAP to obtain camera pose, and have tried to train the model but it was not successful. I have tried

    1. Starting from zero 6-dimensional se3 vectors (as in LLFF), but the model could not find the proper pose.
    2. Adding noise (0.15) from the GT pose, but the error on Rotation and Translation keeps increasing. As far as I know, the model finds the correct pose first and optimizes the NeRF parameters - but they do not work. Please see the images. Validation error keeps increasing as the pose estimation was not successful.

    Separately, NeRF could be trained with this dataset successfully. Please advise me on how I should do with this.

    Cheers, Wonbong

    https://user-images.githubusercontent.com/32883157/165571787-dc6c6ec7-ba8a-4a9d-8bd9-01a7da850ae5.mp4

    Screenshot from 2022-04-27 17-50-05 Screenshot from 2022-04-27 17-55-06

  • Training are very sensitive to network initialization

    Training are very sensitive to network initialization

    Hello, Thank you for the great work! I really like the neat project structure so I am experimenting with NeRF on this codebase. However, I find the training is very sensitive to network initialization.

    First, if I trained on the hotdog scene for 2000 iterations using default configs in the options/nerf_blender_repr.yaml, I would get a normal render from self.nerf, but an empty render from self.nerf_fine.

    The command is

    python train.py --group=nerf-debug --model=nerf --yaml=nerf_blender_repr --name=debug-hotdog-nerf-9 --data.scene=hotdog
    

    The result shown in tensorboard is image

    After some investigation, I found that the reason lay in the different initialization weights of the two networks. Then I tried changing the random seed from 0 to 233, this time both networks rendered an empty scene.

    The command is

    python train.py --group=nerf-debug --model=nerf --yaml=nerf_blender_repr --name=debug-hotdog-nerf-10 --data.scene=hotdog --seed=233
    

    The result is image

    Finally, I tried copying the self.nerf's weights to self.nerf_fine, using the following code

              if opt.nerf.fine_sampling:
                  self.nerf_fine = NeRF(opt)
                  self.nerf_fine.load_state_dict(self.nerf.state_dict()) // here
    

    and set the random seed back to 0. This time the result was fine.

    The command is

    python train.py --group=nerf-debug --model=nerf --yaml=nerf_blender_repr --name=debug-hotdog-nerf-11 --data.scene=hotdog
    

    The result is image

    Here, I want to post the results to discuss the general stability of NeRF training. I wonder if other NeRF repositories all have this kind of sensitivity to network initialization, or are we missing some initialization trick in this repo?

    I believe this can relate to a more fundamental nature of NeRF. Do you have any ideas about this phenomenon? Thank you!

  • What does the numbers in get_camera() mean?

    What does the numbers in get_camera() mean?

    Thanks for the wonderful project. I wonder what does these numbers mean, and should I change them when testing my own dataset? https://github.com/chenhsuanlin/bundle-adjusting-NeRF/blob/803291bd0ee91c7c13fb5cc42195383c5ade7d15/data/iphone.py#L65

  • How to optimize intrinsics as well?

    How to optimize intrinsics as well?

    Hi @chenhsuanlin great work!

    I'm trying BARF on my custom data, and it shows promising results. However, I'm wondering whether the performance will be better if we optimize the intrinsic as well. Do you feel it's doable? If so, could you please guide me on where I should be modifying? Thanks!

  • How to train with my own images?

    How to train with my own images?

    thanks for the code!that‘s really a nice work! But there seems to be a little description of how to use my own images for training, could you tell me more about it?

  • Weird traning behavior of barf

    Weird traning behavior of barf

    Hi, @chenhsuanlin

    Thanks for your great work. I'm trying barf on other scenes. However, the training behavior seems weird. As you can see from the image below, the rotation error is decreasing, but the translation error keeps increasing. image

    When I took a look at the synthesized validation image, it seems the result was biased by several pixels from the original image, and also the scale is not consistent with the original image. image

    For my experimental setting, I used COLMAP to compute the ground truth camera poses and intrinsics. The initial camera poses for barf are not identities instead of perturbing by a small pose with noise to be 0.15. I wonder if there are any parameters we need to fine tune?

    Part of the scene looks like this: P1000686

    And the reconstructed scene: image

  • AttributeError: 'Model' object has no attribute 'train_data'

    AttributeError: 'Model' object has no attribute 'train_data'

    When I run “extract_mesh.py”. It seems that there is no data loaded. What's the matter? image At the same time, the external parameters of the program I run are as follows “python extract_mesh.py --group=G1 --model=barf --yaml=barf_blender --name=First --data.scene=chair --data.val_sub= --resume”

[ICCV'21] UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction
[ICCV'21] UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction

UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction Project Page | Paper | Supplementary | Video This reposit

Sep 16, 2022
(Arxiv 2021) NeRF--: Neural Radiance Fields Without Known Camera Parameters

NeRF--: Neural Radiance Fields Without Known Camera Parameters Project Page | Arxiv | Colab Notebook | Data Zirui Wang¹, Shangzhe Wu², Weidi Xie², Min

Sep 21, 2022
This is the code for Deformable Neural Radiance Fields, a.k.a. Nerfies.

Deformable Neural Radiance Fields This is the code for Deformable Neural Radiance Fields, a.k.a. Nerfies. Project Page Paper Video This codebase conta

Sep 22, 2022
Open source repository for the code accompanying the paper 'Non-Rigid Neural Radiance Fields Reconstruction and Novel View Synthesis of a Deforming Scene from Monocular Video'.
Open source repository for the code accompanying the paper 'Non-Rigid Neural Radiance Fields Reconstruction and Novel View Synthesis of a Deforming Scene from Monocular Video'.

Non-Rigid Neural Radiance Fields This is the official repository for the project "Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synt

Sep 22, 2022
Unofficial & improved implementation of NeRF--: Neural Radiance Fields Without Known Camera Parameters
Unofficial & improved implementation of NeRF--: Neural Radiance Fields Without Known Camera Parameters

[Unofficial code-base] NeRF--: Neural Radiance Fields Without Known Camera Parameters [ Project | Paper | Official code base ] ⬅️ Thanks the original

Sep 20, 2022
Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields.
Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields.

This repository contains the code release for Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. This implementation is written in JAX, and is a fork of Google's JaxNeRF implementation. Contact Jon Barron if you encounter any issues.

Sep 26, 2022
Code for KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs
Code for KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs

KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs Check out the paper on arXiv: https://arxiv.org/abs/2103.13744 This repo cont

Sep 16, 2022
This repository contains a PyTorch implementation of "AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis".
This repository contains a PyTorch implementation of

AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis | Project Page | Paper | PyTorch implementation for the paper "AD-NeRF: Audio

Sep 23, 2022
Code release for DS-NeRF (Depth-supervised Neural Radiance Fields)
Code release for DS-NeRF (Depth-supervised Neural Radiance Fields)

Depth-supervised NeRF: Fewer Views and Faster Training for Free Project | Paper | YouTube Pytorch implementation of our method for learning neural rad

Sep 25, 2022
PyTorch implementation for MINE: Continuous-Depth MPI with Neural Radiance Fields
PyTorch implementation for  MINE: Continuous-Depth MPI with Neural Radiance Fields

MINE: Continuous-Depth MPI with Neural Radiance Fields Project Page | Video PyTorch implementation for our ICCV 2021 paper. MINE: Towards Continuous D

Sep 16, 2022
This repository contains the source code for the paper "DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks",
This repository contains the source code for the paper

DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks Project Page | Video | Presentation | Paper | Data L

Sep 23, 2022
[ICCV21] Self-Calibrating Neural Radiance Fields
[ICCV21] Self-Calibrating Neural Radiance Fields

Self-Calibrating Neural Radiance Fields, ICCV, 2021 Project Page | Paper | Video Author Information Yoonwoo Jeong [Google Scholar] Seokjun Ahn [Google

Sep 22, 2022
This is the code for "HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields".

HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields This is the code for "HyperNeRF: A Higher-Dimensional

Sep 19, 2022
A PyTorch implementation of NeRF (Neural Radiance Fields) that reproduces the results.
A PyTorch implementation of NeRF (Neural Radiance Fields) that reproduces the results.

NeRF-pytorch NeRF (Neural Radiance Fields) is a method that achieves state-of-the-art results for synthesizing novel views of complex scenes. Here are

Sep 22, 2022
D-NeRF: Neural Radiance Fields for Dynamic Scenes
 D-NeRF: Neural Radiance Fields for Dynamic Scenes

D-NeRF: Neural Radiance Fields for Dynamic Scenes [Project] [Paper] D-NeRF is a method for synthesizing novel views, at an arbitrary point in time, of

Sep 7, 2022
pixelNeRF: Neural Radiance Fields from One or Few Images
 pixelNeRF: Neural Radiance Fields from One or Few Images

pixelNeRF: Neural Radiance Fields from One or Few Images Alex Yu, Vickie Ye, Matthew Tancik, Angjoo Kanazawa UC Berkeley arXiv: http://arxiv.org/abs/2

Sep 18, 2022
Code release for NeRF (Neural Radiance Fields)
Code release for NeRF (Neural Radiance Fields)

NeRF: Neural Radiance Fields Project Page | Video | Paper | Data Tensorflow implementation of optimizing a neural representation for a single scene an

Sep 21, 2022
A PyTorch re-implementation of Neural Radiance Fields
A PyTorch re-implementation of Neural Radiance Fields

nerf-pytorch A PyTorch re-implementation Project | Video | Paper NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis Ben Mildenhall

Sep 26, 2022
This is a JAX implementation of Neural Radiance Fields for learning purposes.

learn-nerf This is a JAX implementation of Neural Radiance Fields for learning purposes. I've been curious about NeRF and its follow-up work for a whi

Aug 22, 2022