Code release for DS-NeRF (Depth-supervised Neural Radiance Fields)

Depth-supervised NeRF: Fewer Views and Faster Training for Free

Project | Paper | YouTube

Pytorch implementation of our method for learning neural radiance fields that takes advantage of depth supervised by 3D point clouds. It can be used to train NeRF models given only very few input views.

Depth-supervised NeRF: Fewer Views and Faster Training for Free

arXiv 2107.02791, 2021

Kangle Deng1, Andrew Liu2, Jun-Yan Zhu1, Deva Ramanan1,3,

1CMU, 2Google, 3Argo AI


We propose DS-NeRF (Depth-supervised Neural Radiance Fields), a model for learning neural radiance fields that takes advantage of depth supervised by 3D point clouds.

NeRF trained with 2 views:

DS-NeRF trained with 2 views:


Quick Start

Dependencies

Install requirements:

pip install -r requirements.txt

You will also need COLMAP installed to compute poses if you want to run on your data.

Data

Download data for the example scene: fern_2v

bash download_example_data.sh

To play with other scenes presented in the paper, download the data here.

Pre-trained Models

You can download the pre-trained models here. Place the downloaded directory in ./logs in order to test it later. See the following directory structure for an example:

├── logs 
│   ├── fern_2v    # downloaded logs
│   ├── flower_2v  # downloaded logs

How to Run?

Generate camera poses and sparse depth information using COLMAP (optional)

This step is necessary only when you want to run on your data.

First, place your scene directory somewhere. See the following directory structure for an example:

├── data
│   ├── fern_2v
│   ├── ├── images
│   ├── ├── ├── image001.png
│   ├── ├── ├── image002.png

To generate the poses and sparse point cloud:

python imgs2poses.py <your_scenedir>

Testing

Once you have the experiment directory (downloaded or trained on your own) in ./logs,

  • to render a video:
python run_nerf.py --config configs/fern_dsnerf.txt --render_only

Training

To train a DS-NeRF on the example fern dataset:

python run_nerf.py --config configs/fern_dsnerf.txt

You can create your own experiment configuration to try other datasets.


Citation

If you find this repository useful for your research, please cite the following work.

@article{kangle2021dsnerf,
  title={Depth-supervised NeRF: Fewer Views and Faster Training for Free},
  author={Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan},
  journal={arXiv preprint arXiv:2107.02791},
  year={2021}
}

Credits

This code borrows heavily from nerf-pytorch.

Comments
  • NDC space and depth loss

    NDC space and depth loss

    I am using NDC space and I have depth values collected from a sensor, should I normalize these values between 0 and 1 or should I convert them into NDC space? How should we calculate the depth loss while using NDC space as the depth_col value comes between 0 and 1 in NDC space but the target_depth is bigger than 1.

    Thanks.

  • Question about the regularization added to the loss

    Question about the regularization added to the loss

  • Potential view inconsistency in depth loss

    Potential view inconsistency in depth loss

    Hi,

    I noticed that you compute depth in input views in https://github.com/dunbar12138/DSNeRF/blob/main/load_llff.py#L371 but the depth given by NeRF model is in cannonical view https://github.com/dunbar12138/DSNeRF/blob/main/run_nerf_helpers.py#L377

    Shouldn't both the depths be brought to same view before computing depth loss? Please let me know if I missed something

  • Configs does not use weighted loss

    Configs does not use weighted loss

    Hi,

    I noticed that the configs you've provided has not enabled weighted loss (reprojection error), but in the paper you've used weighted loss. Can you please confirm if it's a typo in configs and I have to enable weighted loss or did you deliberately disable weighted loss?

  • Standard rendering demo mentioned in the tutorial outputs corrupted images and video.

    Standard rendering demo mentioned in the tutorial outputs corrupted images and video.

    No code changes, just followed the readme

    I downloaded the pretrained model to the ./logs folder, and initiated:

    run_nerf.py --config configs/fern_dsnerf.txt --render_only
    

    I keep seeing a compression warning every iteration as follows:

    max: 5.9706993
    Lossy conversion from float32 to uint8. Range [0.10505260527133942, 5.970699310302734]. Convert image to uint8 prior to saving to suppress this warning.
     99%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏   | 119/120 [14:05<00:07,  7.12s/it]119 7.150676250457764
    max: 5.921188
    Lossy conversion from float32 to uint8. Range [0.11592616140842438, 5.921187877655029]. Convert image to uint8 prior to saving to suppress this warning.
    100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [14:12<00:00,  7.10s/it]
    Done rendering ./logs/fern_dsnerf_2v/renderonly_path_000000
    

    And the output mentioned above says the following: 000

  • Scene with known camera poses

    Scene with known camera poses

    Hello, thank you for this project :)

    I got a problem while training on a scene.

    When I extracted the camera poses from COLMAP this is the output

    https://user-images.githubusercontent.com/19538303/142732551-c89f3ed4-06c9-4567-a4d4-d2d5e3272bd9.mp4

    the result is not so crispy as the scene is a little bit hard to learn but

    But when I use the camera poses from the dataset I got a result like this

    https://user-images.githubusercontent.com/19538303/142732596-d1f68062-72f8-46f1-9608-36568be8fee7.mp4

    The difference between the two is really great. I changed the gt camera poses to the coordinate system of right, upwards, backwards.

    What am I doing wrong?

    Thanks.

  • Depth loss only applied to fine network?

    Depth loss only applied to fine network?

    Hi, thanks for your awsome work and code!

    I noticed that in your code the depth loss is applied to the fine network only, and the coarse network only has RGB loss. Is there any reasons behind this choice?

  • Unreasobable result with the pre-trained model.

    Unreasobable result with the pre-trained model.

    Hi, thanks for your great work and code!

    I'm trying to reproduce the demo results with the pre-trained model, but I got unreasonable frame outputs as below: image

    The checkpoint fern_2v/020000.tar was loaded. The only modification to the code is replacing from torchsearchsorted import searchsorted withfrom torch import searchsorted and the corresponding usage of it. Do you have any idea what may have gone wrong here?

    Thanks in advance!

  • Unable to get sparse point cloud with colmap with 2 images with known pose

    Unable to get sparse point cloud with colmap with 2 images with known pose

    Hi,

    I'm trying to train DS-NeRF on two images whose intrinsics and extrinsics are known. I followed this link you suggested in Issue-20, but colmap point_triangulator is failing because it triangulated 0 points.

    To debug, I used the fern_2view data and ran imgs2poses.py to obtain camera params and point cloud. Then I followed the known-camera-poses procedure (I removed every second line in images.txt), but it fails even for that. In other words, for the exact same camera params for which first approach worked, the second approach didn't work. So, it can't be a problem with the data, but instead it should be something that I'm doing incorrectly.

    I'm new to NeRF and Colmap. Can you please help me out.

    More details: I created a new folder fern_trial and in images folder, added the two images and renamed them as image001.png and image002.png. Also created an empty sparse folder.

    This works:

    fern_trial$ colmap feature_extractor --database_path database.db --image_path images --ImageReader.single_camera 1
    fern_trial$ colmap exhaustive_matcher --database_path database.db
    fern_trial$ colmap mapper --database_path database.db --image_path images --output_path sparse --Mapper.num_threads 16 --Mapper.init_min_tri_angle 4 --Mapper.multiple_models 0 --Mapper.extract_colors 0
    

    This does not work:

    fern_trial2$ colmap feature_extractor --database_path database.db --image_path images --ImageReader.single_camera 1
    

    I copied camera.txt and images.txt (deleted every second line) from fern_trial to this folder.

    fern_trial2$ colmap exhaustive_matcher --database_path database.db
    fern_trial2$ colmap point_triangulator --database_path database.db --image_path images --input_path sparse/0 --output_path sparse2 --Mapper.num_threads 16 --Mapper.init_min_tri_angle 4 --Mapper.multiple_models 0 --Mapper.extract_colors 0
    
  • License

    License

    First off, excellent work on this project!

    Can you guys add the MIT License to your codebase? It's the licensed used by the nerf-pytorch project that this was based off of. Thanks!

  • Performance aspect of w/o batching for few-shot learning

    Performance aspect of w/o batching for few-shot learning

    I think the released code is based on batching(use_batching=True) for depth supervision setting. Is there a performance drop when not using batching (especially for few-shot learning & sparse depth supervision)?

    Thank you

  • Training on DTU

    Training on DTU

    Hi, I am a little confused about training and evaluating on DTU. You have mentioned in the paper that

    We run COLMAP with the ground truth calibrated camera poses to get keypoints. Images are down-sampled to a resolution of 400 × 300 for training and evaluation.

    1- Did you make any changes to the ground truth camera poses after you down-sampled images to 400 x 300? 2- imgs2poses.py does not create colmap_depth.npy. 3- Do you have a separate config file for DTU? 4- Where can I get cameras.npz from?

  • Directly use Redwood depth for supervision

    Directly use Redwood depth for supervision

    Hi, thanks for your excellent work! I found that one of your set of experiments was trained directly using RedWood Depth as the supervision source instead of COLMAP, I would like to read the code for this part of the work, please where can I find it, thanks!

    image

  • Shooting angle problem and camera internal parameter adjustment

    Shooting angle problem and camera internal parameter adjustment

    Hi, thanks for your excellent work, while reproducing your paper, I have the following three questions:

    1. Which file should I modify the camera's internal parameters in the program?
    2. How do you choose the camera angle when you create the dataset?
    3. During training, the following 4 videos (see picture) are generated, do they have to be generated? Because I always run out of memory when generating and rendering video during training, when I block this part of the code, the reproduction effect is not good. image Looking forward to your reply
  • RuntimeError: CUDA out of memory.

    RuntimeError: CUDA out of memory.

    RuntimeError: CUDA out of memory. Tried to allocate 360.00 MiB (GPU 0; 10.76 GiB total capacity; 6.21 GiB already allocated; 200.56 MiB free; 6.76 GiB reserved in total by PyTorch)

    image

    I use my data but produce some problems, I don't know how to solve this problem,please help me? I use 2080Ti

(Arxiv 2021) NeRF--: Neural Radiance Fields Without Known Camera Parameters

NeRF--: Neural Radiance Fields Without Known Camera Parameters Project Page | Arxiv | Colab Notebook | Data Zirui Wang¹, Shangzhe Wu², Weidi Xie², Min

Aug 6, 2022
Unofficial & improved implementation of NeRF--: Neural Radiance Fields Without Known Camera Parameters
Unofficial & improved implementation of NeRF--: Neural Radiance Fields Without Known Camera Parameters

[Unofficial code-base] NeRF--: Neural Radiance Fields Without Known Camera Parameters [ Project | Paper | Official code base ] ⬅️ Thanks the original

Jul 31, 2022
Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields.
Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields.

This repository contains the code release for Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. This implementation is written in JAX, and is a fork of Google's JaxNeRF implementation. Contact Jon Barron if you encounter any issues.

Aug 1, 2022
This repository contains a PyTorch implementation of "AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis".
This repository contains a PyTorch implementation of

AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis | Project Page | Paper | PyTorch implementation for the paper "AD-NeRF: Audio

Aug 1, 2022
A PyTorch implementation of NeRF (Neural Radiance Fields) that reproduces the results.
A PyTorch implementation of NeRF (Neural Radiance Fields) that reproduces the results.

NeRF-pytorch NeRF (Neural Radiance Fields) is a method that achieves state-of-the-art results for synthesizing novel views of complex scenes. Here are

Jul 30, 2022
D-NeRF: Neural Radiance Fields for Dynamic Scenes
 D-NeRF: Neural Radiance Fields for Dynamic Scenes

D-NeRF: Neural Radiance Fields for Dynamic Scenes [Project] [Paper] D-NeRF is a method for synthesizing novel views, at an arbitrary point in time, of

Jul 29, 2022
Pytorch implementation for A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose
Pytorch implementation for A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose

A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose Paper | Website | Data A-NeRF: Articulated Neural Radiance F

Aug 3, 2022
A minimal TPU compatible Jax implementation of NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

NeRF Minimal Jax implementation of NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Result of Tiny-NeRF RGB Depth

Jul 24, 2022
Build upon neural radiance fields to create a scene-specific implicit 3D semantic representation, Semantic-NeRF
Build upon neural radiance fields to create a scene-specific implicit 3D semantic representation, Semantic-NeRF

Semantic-NeRF: Semantic Neural Radiance Fields Project Page | Video | Paper | Data In-Place Scene Labelling and Understanding with Implicit Scene Repr

Aug 4, 2022
Point-NeRF: Point-based Neural Radiance Fields
Point-NeRF: Point-based Neural Radiance Fields

Point-NeRF: Point-based Neural Radiance Fields Project Sites | Paper | Primary c

Aug 2, 2022
This repository contains the source code for the paper "DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks",
This repository contains the source code for the paper

DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks Project Page | Video | Presentation | Paper | Data L

Jul 30, 2022
PyTorch implementation for MINE: Continuous-Depth MPI with Neural Radiance Fields
PyTorch implementation for  MINE: Continuous-Depth MPI with Neural Radiance Fields

MINE: Continuous-Depth MPI with Neural Radiance Fields Project Page | Video PyTorch implementation for our ICCV 2021 paper. MINE: Towards Continuous D

Jul 31, 2022
Official code release for "GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis"
Official code release for

GRAF This repository contains official code for the paper GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. You can find detailed usage i

Aug 5, 2022
Instant-nerf-pytorch - NeRF trained SUPER FAST in pytorch

instant-nerf-pytorch This is WORK IN PROGRESS, please feel free to contribute vi

Jul 25, 2022
This is the code for Deformable Neural Radiance Fields, a.k.a. Nerfies.

Deformable Neural Radiance Fields This is the code for Deformable Neural Radiance Fields, a.k.a. Nerfies. Project Page Paper Video This codebase conta

Aug 5, 2022
Open source repository for the code accompanying the paper 'Non-Rigid Neural Radiance Fields Reconstruction and Novel View Synthesis of a Deforming Scene from Monocular Video'.
Open source repository for the code accompanying the paper 'Non-Rigid Neural Radiance Fields Reconstruction and Novel View Synthesis of a Deforming Scene from Monocular Video'.

Non-Rigid Neural Radiance Fields This is the official repository for the project "Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synt

Jul 29, 2022
Code for KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs
Code for KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs

KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs Check out the paper on arXiv: https://arxiv.org/abs/2103.13744 This repo cont

Jul 28, 2022
This is the code for "HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields".

HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields This is the code for "HyperNeRF: A Higher-Dimensional

Aug 3, 2022
BARF: Bundle-Adjusting Neural Radiance Fields 🤮 (ICCV 2021 oral)

BARF ?? : Bundle-Adjusting Neural Radiance Fields Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey IEEE International Conference on Comp

Jul 28, 2022