Unofficial & improved implementation of NeRF--: Neural Radiance Fields Without Known Camera Parameters

[Unofficial code-base] NeRF--: Neural Radiance Fields Without Known Camera Parameters

[ Project | Paper | Official code base ] ⬅️ Thanks the original authors for the great work!

  • ⚠️ This is an unofficial pytorch re-implementation of the paper NeRF--: Neural Radiance Fields Without Known Camera Parameters.
  • I have reproduced the results on the LLFF-fern dataset, LLFF-flower dataset, personal photos, and some YouTube video clips chosen by myself.
  • This repo contains implementation of both the original paper and my personal modifications & refinements.

Example results

input raw images of the same scene (order doesn't matter, could be in arbitrary order)
output
(after joint optimization)
camera intrinsics (focal_x and focal_y)
camera extrinsics (inverse of poses, rotations and translations) of each image
a 3D implicit representation framework [NeRF] that models both appearance and geometry of the scene

Source 1: random YouTube video clips, time from 00:10:36 to 00:10:42

ReLU-based NeRF--
(no refinement, stuck in local minima)
SIREN-based NeRF--
(no refinement)
input
32 raw photos, sampled at 5 fps
32 x [540 x 960 x 3]
castle_input
learned scene model size 1.7 MiB / 158.7k params
8+ MLPs with width of 128
learned camera poses castle_1041_relu_pose castle_1041_pose_siren
predicted rgb
(appearance)
(with novel view synthesis)
castle_1041_relu castle_1041_siren
predicted depth
(geometry)
(with novel view synthesis)
castle_1041_relu_depth castle_1041_siren

Source 2: random YouTube video clips, time from 00:46:17 to 00:46:28

ReLU-based NeRF--
(with refinement, still stuck in local minima)
SIREN-based NeRF--
(with refinement)
input
27 raw photos, sampled at 2.5 fps
27 x [540 x 960 x 3]
castle_4614_input
learned scene model size 1.7 MiB / 158.7k params
8+ MLPs with width of 128
learned camera poses castle_4614_pose_siren castle_4614_pose_siren
predicted rgb
(appearance)
(with novel view synthesis)
castle_1041_siren castle_1041_siren
predicted depth
(geometry)
(with novel view synthesis)
castle_1041_siren castle_1041_siren

Source 3: photos by @crazyang

ReLU-based NeRF--
(no refinement)
SIREN-based NeRF--
(no refinement)
input
22 raw photos
22 x [756 x 1008 x3]
piano_input
learned scene model size 1.7 MiB / 158.7k params
8+ MLPs with width of 128
learned camera poses piano_relu_pose piano_siren_pose
predicted rgb
(appearance)
(with novel view synthesis)
piano_relu_rgb piano_siren_rgb
predicted depth
(geometry)
(with novel view synthesis)
piano_relu_depth piano_siren_depth

Notice that the reflectance of the piano's side is misunderstood as transmittance, which is reasonable and acceptable since no prior of the shape of the piano is provided.

What is NeRF and what is NeRF--

NeRF

NeRF is a neural (differentiable) rendering framework with great potentials. Please view [NeRF Project Page] for more details.

It represents scenes as a continuous function (typically modeled by several layers of MLP with non-linear activations); the same ideas within DeepSDF, SRN, DVR, and so on.

It is suggested to refer to [awesome-NeRF] and [awesome-neural-rendering] to catch up with recent 'exploding' development in these areas.

NeRF--

NeRF-- modifies the original NeRF from requiring known camera parameters to supporting unknown and learnable camera parameters.

  • NeRF-- does the following work in the training process:

    • Joint optimization of
      • camera intrinsics
      • camera extrinsics
      • a NeRF model (appearance and geometry)
    • Using pure raw real-world images and using just photometric loss (image reconstruction loss).
  • SfM+MVS

    • In other words, NeRF-- tackles exactly the same problem with what a basic SfM+MVS system like COLMAP does, but learns the camera parameters, geometry and appearance of the scene simultaneously in a more natural and holistic way, requiring no hand-crafted feature extraction procedures like SIFT or points, lines, surfaces etc.
  • How?

    • Since NeRF is a neural rendering framework (which means the whole framework is differentiable), one can directly compute the gradients of the photometric loss with respect to the camera parameters.
  • 🚀 Wide future of NeRF-based framework --- vision by inverse computer graphics

    • Expect more to come! Imagine direct computing of gradients of photometric loss w.r.t. illumination? object poses? object motion? object deformation? objects & background decomposition ? object relationships?...

My modifications & refinements / optional features

This repo first implements NeRF-- with nothing changed from the original paper. But it also support the following optional modifications, and will keep updating.

All the options are configured using yaml configuration files in the configs folder. See details about how to use these configs in the configuration section.

SIREN-based NeRF as backbone

Replace the ReLU activations of NeRF with sinusoidal(sin) activation. Codes borrowed and modified from [lucidrains' implementation of pi-GAN]. Please refer to SIREN and pi-GAN for more theoretical details.

To config:

model:
  framework: SirenNeRF # options: [NeRF, SirenNeRF]

📌 SIREN-based NeRF compared with ReLU-based NeRF

  • SirenNeRF could lead to smoother learned scenes (especially smoother shapes). (below, left for ReLU-based, right for SIREN-based)
ReLU-based NeRF-- (no refinement) SIREN-based NeRF-- (no refinement)
image-20210418015802348 depth_siren
  • SirenNeRF could lead to better results (smaller losses at convergence, better SSIM/PSNR metrics) .

siren_vs_relu_loss

The above two conclusions are also evidenced by the DeepSDF results shown in the SIREN project.

  • SirenNeRF is a little bad for scenes with lots of sharp and messy edges (for its continuously differentiable & smoothing nature).

e.g. LLFF-flower scene

ReLU-based NeRF-- (with refinement) SIREN-based NeRF-- (with refinement)
relu_rgb rgb_siren
relu_depth depth_siren

Note: since the raw output of SirenNeRF is relatively slower to grow, I multiply the raw output (sigma) of SirenNeRF with a factor of 10 30. To config, use model:siren_sigma_mul

[WIP] Perceptual model

For fewer shots with large viewport changes, I add an option to use a perceptual model (CLIP) and an additional perceptual loss along with the reconstruction loss, as in DietNeRF.

To config:

data:
  N_rays: -1 # options: -1 for whole image and no sampling, a integer > 0 for number of ray samples
training:
  w_perceptual: 0.01 # options: 0. for no perceptual model & loss, >0 to enable

Note: as the CLIP model requires at least 224x224 resolution and a whole image (not sampled rays) as input

  • data:N_rays must set to -1 for generating whole images when training
  • data:downscale must set to a proper value, and a GPU with larger memory size is required
    • or proper up-sampling is required

More choices of rotation representations / intrinsics parameterization

  • rotation representation

Refer to this paper for theoretical suggestions for different choices of SO(3) representations.

To config:

model:
  so3_representation: 'axis-angle' # options: [quaternion, axis-angle, rotation6D]
  • intrinsics parameterization

To config:

model:
  intrinsics_representation: 'square' # options: [square, ratio, exp]

Usage

hardware

  • 💻 OS: tested on Ubuntu 16 & 18
  • GPU (all assume 1024 ray samples(by default) + 756 x 1000 resolution + 128 network width)
    • scene model parameter size
      • 🌟 1.7 MiB for float32.
      • For NeRF scene model, it's just 8+ layers of MLPs with ReLU/sin activation, with width of 128.
    • 🕐 training time on 2080Ti
      • <10 mins or 0-200 epochs: learning poses mainly, and rough appearances
      • ~4 hours, from ~300 to 10000 epochs: the poses has little further changes; the NeRF model learns fine details (geometry & appearance) of the scene
    • GPU memory:
      • (training) ~3300 MiB GPU memory usage
      • (testing / rendering) lower GPU memory usage, but potentially more GPU usage since testing is on full resolution, while training is on a small batch of sampled pixel for each iteration.

software

  • Python >= 3.5

  • To install requirements, run:

    • Simply just run: (suggested used in anaconda environments):

      ## install torch & cuda & torchvision using your favorite tools, conda/pip
      # pip install torch torchvision
      
      ## install other requirements
      pip install numpy pyyaml addict imageio imageio-ffmpeg scikit-image tqdm tensorboardX "pytorch3d>=0.3.0" opencv-python
    • Or

      conda env create -f environment.yml
  • Before running any python scripts for the first time, cd to the project root directory and add the root project directory to the PYTHONPATH by running:

    cd /path/to/improved-nerfmm
    source set_env.sh

configuration

There are three choices for giving configuration values:

  • [DO NOT change] configs/base.yaml contains all the default values of the whole config.
  • [Your playground] Specific config yamls in configs folder are for specific tasks. You only need to put related config keys here. It is given to the python scripts using python xxx.py --config /path/to/xxx.yaml.
  • You can also give additional runtime command-line arguments with python xxx.py --xxx:yyy val, to change the config dict: 'xxx':{'yyy': val}

The configuration overwriting priority order:

  • command line args >>overwrites>> --config /path/to/xxx.yaml >>overwrites>> configs/base.yaml

data

dataset source link / script file path
LLFF Download LLFF example data using the scripts (run in project root directory):
bash dataio/download_example_data.sh
(automatic)
Youtube video clips https://www.youtube.com/watch?v=hWagaTjEa3Y ./data/castle_1041
./data/castle_4614
piano photos by @crazyang Google-drive ./data/piano

pre-trained models

You can get pre-trained models in either of the following two ways:

  • Clone the repo using git-lfs, and you will get the pre-trained models and configs in pretrained folder.
  • From pretrained folder in the google-drive

Training

Before running any python scripts for the first time, cd to the project root directory and add the root project directory to the PYTHONPATH by running:

cd /path/to/improved-nerfmm
source set_env.sh

Train on example data (without refinement)

Download LLFF example data using the scripts (run in project root directory)

bash dataio/download_example_data.sh

Start training:

python train.py --config configs/fern.yaml
  • To specify used GPUs: (e.g. 2 and 3) (in most cases, one GPU is quite enough.)

    python train.py --config configs/fern.yaml --device_ids 2,3
    • Currently, this repo use torch.DataParallel for multi-GPU training.
  • View the training logs and stats output in the experiment folder: ./logs/fern

  • Run tensorboard to monitor the training process:

    tensorboard --logdir logs/fern/events
  • To resume previously interrupted training:

    python train.py --load_dir logs/fern
    • Note: Full config is automatically backed up in the experiment directory when start training. Thus when loading from a directory, the scripts will only read from your_exp_dir/config.yaml, and configs/base.yaml will not be used.

🚀 Train on your own data

  • 📌 Note on suitable input

    • ①static scene ②forward-facing view ③with small view-port changes.
      • Smaller viewport change / forward facing views
        • So that a certain face of a certain object should appear in all views
        • Otherwise the training would fail in the early stages (failed to learn reasonable camera poses, and hence no chance for the NeRF).
        • This is mostly because it processes all input images at once.
      • No moving / deforming objects. (e.g. a car driving across the street, a bird flying across the sky, people waving hands)
      • No significant illumination/exposure changes. (e.g. camera moving from pointing towards the sun to back to the sun)
      • No focal length changes. Currently assume all input share the same camera intrinsics.
      • Just temporarily! (All imaginable limitations have imaginable solutions. Stay tuned!)
  • 📌 Note on training

    • When training with no refinement, the training process is roughly split into two phases:
      • [0 to about 100-300 epochs] The NeRF model learns some rough blurry pixel blocks, and these rough blocks helps with optimizing the camera extrinsics.
      • [300 epochs+ to end] The camera extrinsics are almost fixed, with very small further changes; the NeRF model learns the fine details of the scene.
    • You should monitor the early 100~300 epochs of the training process. If no meaningful camera poses (especially the camera translation on xy-plane) are learned during this early stages, there almost won't be any miracle further.
    • I have not tested on >50 images, but you can give it a try.
  • First, prepare your photos and put them into one separate folder, say /path/to/your_photos/xxx.png.

  • Second:

    • Write a new config file for your data: (you can put any config key mentioned in configs/base.yaml)

      expname: your_expname
      data:
        data_dir: /path/to/your_photos
        #=========
        N_rays: 1024 # numer of sampled rays in training.
        downscale: 4.
        #=========
    • And run

      python train.py --config /path/to/your_config.yaml
    • Or you can use some existing config file and run:

      python train.py --config /path/to/xxx.yaml --data:data_dir /path/to/your_photos --expname your_expname
  • The logging and stats would be in logs/your_expname folder.

  • Monitor the training process with:

    tensorboard --logdir logs/your_expname/events

Train on video clips

  • First, clip your video.mp4 with ffmepg.

    ffmpeg -ss 00:10:00 -i video.mp4 -to 00:00:05 -c copy video_clip.mp4
    • Note:

      • time format: hh:mm:ss.xxx
      • -ss means starting timestamp
      • -to means duration length, not end timestamp.
  • Second, convert video_clip.mp4 into images:

    mkdir output_dir
    ffmpeg -i video_clip.mp4 -filter:v fps=fps=3/1 output_dir/img-%04d.png
    • Note:

      • 3/1 means 3 frames per second. 3 is the nominator, 1 is the denominator.
  • Then train on your images with instructions in 🚀 ​Train on your own data

Automatic training with a pre-train stage and refine stage

Run

python train.py --config ./configs/fern_prefine.yaml

Or

python train.py --config ./configs/fern.yaml --training:num_epoch_pre 1000 --expname fern_prefine

You can also try on your own photos using similar configurations.

Refining a pre-trained NeRF--

This is the step suggested by original NeRF-- paper: drop all pre-trained parameters except for camera parameters, and refine.

For example, refine a pre-trained LLFF-fern scene, with original config stored in ./configs/fern.yaml, a pre-trained checkpoint in ./logs/fern/ckpts/final_xxxx.pt, and with a new experiment name fern_refine:

python train.py --config ./configs/fern.yaml --expname fern_refine --training:ckpt_file ./logs/fern/ckpts/final_xxxx.pt  --training:ckpt_only_use_keys cam_params

Note:

  • --training:ckpt_only_use_keys cam_params is used to drop all the keys in the pre-trained state_dict except cam_params when loading the checkpoints.
    • Some warnings like Could not find xxx in checkpoint will be prompted, which is OK and is the exact desired behavior.
  • a new expname is specified and hence a new experiment directory would be used, since we do not want to concatenate and mix the new logging stats with the old ones.

Testing

Free view port rendering

  • To render with camera2world matrices interpolated from the learned pose
python vis/free_viewport_rendering.py --load_dir /path/to/pretrained/exp_dir --render_type interpolate
  • To render with spiral camera paths as in the original NeRF repo
python vis/free_viewport_rendering.py --load_dir /path/to/pretrained/exp_dir --render_type spiral

Visualize learned camera pose

python vis/plot_camera_pose.py --load_dir /path/to/pretrained/exp_dir

Notice that the learned camera phi & t is actually for camera2world matrices, the inverse of camera extrinsics

You will get a matplotlib window like this:

image-20210425181218826

Road-map & updates

Basic NeRF model

  • 2021-04-17 Basic implementation of the original paper, including training & logging
    • Add quaternion, axis-angle, rotation6D as the rotation representation
    • Add exp, square, ratio for different parameterizations of camera focal_x and focal_y
    • Siren-ized NeRF
    • refinement process as in the original NeRF-- paper
  • Change DataParallel to DistributedDataParallel

Efficiency & training

  • 2021-04-19 Add pre-train for 1000 epochs then refine for 10000 epochs, similar with the official code base.

  • tolist - recent works in speeding up NeRF

More experiments

  • vSLAM tasks and datasets
  • traditional SfM datasets
  • 2021-04-15 raw videos handling

Better SfM strategy

  • tolist

More applicable for more scenes

  • NeRF++ & NeRF-- for handling unconstrained scenes

  • NeRF-W

  • some dynamic NeRF framework for dynamic scenes

  • Finish perceptual loss, for fewer shots

Related/used code bases

Citations

  • NeRF--
@article{wang2021nerf,
  title={Ne{RF}$--$: Neural Radiance Fields Without Known Camera Parameters},
  author={Wang, Zirui and Wu, Shangzhe and Xie, Weidi and Chen, Min and Prisacariu, Victor Adrian},
  journal={arXiv preprint arXiv:2102.07064},
  year={2021}
}
  • SIREN
@inproceedings{sitzmann2020siren,
  author={Sitzmann, Vincent and Martel, Julien NP and Bergman, Alexander W and Lindell, David B and Wetzstein, Gordon},
  title={Implicit neural representations with periodic activation functions},
  booktitle={Proc. NeurIPS},
  year={2020}
}
  • Perceptual model / semantic consistency from DietNeRF
@article{jain2021dietnerf,
  title={Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis},
  author={Ajay Jain and Matthew Tancik and Pieter Abbeel},
  journal={arXiv},
  year={2021}
}
Owner
Jianfei Guo
Thrive, don't just exist.
Jianfei Guo
Similar Resources

Point-NeRF: Point-based Neural Radiance Fields

Point-NeRF: Point-based Neural Radiance Fields

Point-NeRF: Point-based Neural Radiance Fields Project Sites | Paper | Primary c

Aug 2, 2022

CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields

CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields

CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields Paper | Supplementary | Video | Poster If you find our code or paper useful, please

Jul 25, 2022

Plenoxels: Radiance Fields without Neural Networks

Plenoxels: Radiance Fields without Neural Networks Alex Yu*, Sara Fridovich-Keil*, Matthew Tancik, Qinhong Chen, Benjamin Recht, Angjoo Kanazawa UC Be

Aug 1, 2022

Instant-nerf-pytorch - NeRF trained SUPER FAST in pytorch

instant-nerf-pytorch This is WORK IN PROGRESS, please feel free to contribute vi

Jul 25, 2022

PyTorch implementation for MINE: Continuous-Depth MPI with Neural Radiance Fields

PyTorch implementation for  MINE: Continuous-Depth MPI with Neural Radiance Fields

MINE: Continuous-Depth MPI with Neural Radiance Fields Project Page | Video PyTorch implementation for our ICCV 2021 paper. MINE: Towards Continuous D

Jul 31, 2022

A PyTorch re-implementation of Neural Radiance Fields

A PyTorch re-implementation of Neural Radiance Fields

nerf-pytorch A PyTorch re-implementation Project | Video | Paper NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis Ben Mildenhall

Jul 31, 2022

This is a JAX implementation of Neural Radiance Fields for learning purposes.

learn-nerf This is a JAX implementation of Neural Radiance Fields for learning purposes. I've been curious about NeRF and its follow-up work for a whi

Aug 6, 2022

Implementation of "Generalizable Neural Performer: Learning Robust Radiance Fields for Human Novel View Synthesis"

Implementation of

Generalizable Neural Performer: Learning Robust Radiance Fields for Human Novel View Synthesis Abstract: This work targets at using a general deep lea

Jul 29, 2022

This is the code for Deformable Neural Radiance Fields, a.k.a. Nerfies.

Deformable Neural Radiance Fields This is the code for Deformable Neural Radiance Fields, a.k.a. Nerfies. Project Page Paper Video This codebase conta

Aug 5, 2022
Comments
  • Skip connect for SirenNeRF

    Skip connect for SirenNeRF

    Hi @ventusff , thank you for your work!

    I was wondering why do you remove the skip connection in SirenNeRF? Does it harm the performance? Or does skip connection make it difficult to decide the initialization method for SirenLayer?

    Thanks.

  • Speeding up Nerf Training with Nvidia's latest paper

    Speeding up Nerf Training with Nvidia's latest paper

    Nvidia introduced a new method for training Nerf that takes only 2 seconds to get results … a project like this will be amazing if it used it.

    https://nvlabs.github.io/instant-ngp/

  • Motivation for using SIREN-based NeRF

    Motivation for using SIREN-based NeRF

    Hi @ventusff, thanks for the interesting experiments and nice visualizations!

    I wonder why you choose to use SIREN activation in NeRF-- in the first place? People seems to have trouble making SIREN work on vanilla NeRF. Do you have any clue why it works in this setting?

  • no module named clip

    no module named clip

    Trying to experiment with perceptual model, upon training it says:

    Traceback (most recent call last):
      File "train.py", line 490, in <module>
        main_function(config)
      File "train.py", line 224, in main_function
        perceptual_net = CLIP_for_Perceptual()
      File "/content/improved-nerfmm/models/perceptual_model.py", line 65, in __init__
        import clip
    ModuleNotFoundError: No module named 'clip'
    
Blender add-on: Add to Cameras menu: View → Camera, View → Add Camera, Camera → View, Previous Camera, Next Camera
Blender add-on: Add to Cameras menu: View → Camera, View → Add Camera, Camera → View, Previous Camera, Next Camera

Blender add-on: Camera additions In 3D view, it adds these actions to the View|Cameras menu: View → Camera : set the current camera to the 3D view Vie

Feb 8, 2022
This repository contains a PyTorch implementation of "AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis".
This repository contains a PyTorch implementation of

AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis | Project Page | Paper | PyTorch implementation for the paper "AD-NeRF: Audio

Aug 1, 2022
A PyTorch implementation of NeRF (Neural Radiance Fields) that reproduces the results.
A PyTorch implementation of NeRF (Neural Radiance Fields) that reproduces the results.

NeRF-pytorch NeRF (Neural Radiance Fields) is a method that achieves state-of-the-art results for synthesizing novel views of complex scenes. Here are

Jul 30, 2022
Pytorch implementation for A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose
Pytorch implementation for A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose

A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose Paper | Website | Data A-NeRF: Articulated Neural Radiance F

Aug 3, 2022
A minimal TPU compatible Jax implementation of NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

NeRF Minimal Jax implementation of NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Result of Tiny-NeRF RGB Depth

Jul 24, 2022
Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields.
Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields.

This repository contains the code release for Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. This implementation is written in JAX, and is a fork of Google's JaxNeRF implementation. Contact Jon Barron if you encounter any issues.

Aug 1, 2022
Code release for DS-NeRF (Depth-supervised Neural Radiance Fields)
Code release for DS-NeRF (Depth-supervised Neural Radiance Fields)

Depth-supervised NeRF: Fewer Views and Faster Training for Free Project | Paper | YouTube Pytorch implementation of our method for learning neural rad

Aug 8, 2022
D-NeRF: Neural Radiance Fields for Dynamic Scenes
 D-NeRF: Neural Radiance Fields for Dynamic Scenes

D-NeRF: Neural Radiance Fields for Dynamic Scenes [Project] [Paper] D-NeRF is a method for synthesizing novel views, at an arbitrary point in time, of

Jul 29, 2022
Code release for NeRF (Neural Radiance Fields)
Code release for NeRF (Neural Radiance Fields)

NeRF: Neural Radiance Fields Project Page | Video | Paper | Data Tensorflow implementation of optimizing a neural representation for a single scene an

Aug 9, 2022
Build upon neural radiance fields to create a scene-specific implicit 3D semantic representation, Semantic-NeRF
Build upon neural radiance fields to create a scene-specific implicit 3D semantic representation, Semantic-NeRF

Semantic-NeRF: Semantic Neural Radiance Fields Project Page | Video | Paper | Data In-Place Scene Labelling and Understanding with Implicit Scene Repr

Aug 4, 2022