Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields.

mip-NeRF

This repository contains the code release for Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. This implementation is written in JAX, and is a fork of Google's JaxNeRF implementation. Contact Jon Barron if you encounter any issues.

rays

Abstract

The rendering procedure used by neural radiance fields (NeRF) samples a scene with a single ray per pixel and may therefore produce renderings that are excessively blurred or aliased when training or testing images observe scene content at different resolutions. The straightforward solution of supersampling by rendering with multiple rays per pixel is impractical for NeRF, because rendering each ray requires querying a multilayer perceptron hundreds of times. Our solution, which we call "mip-NeRF" (à la "mipmap"), extends NeRF to represent the scene at a continuously-valued scale. By efficiently rendering anti-aliased conical frustums instead of rays, mip-NeRF reduces objectionable aliasing artifacts and significantly improves NeRF's ability to represent fine details, while also being 7% faster than NeRF and half the size. Compared to NeRF, mip-NeRF reduces average error rates by 17% on the dataset presented with NeRF and by 60% on a challenging multiscale variant of that dataset that we present. mip-NeRF is also able to match the accuracy of a brute-force supersampled NeRF on our multiscale dataset while being 22x faster.

Installation

We recommend using Anaconda to set up the environment. Run the following commands:

# Clone the repo
git clone https://github.com/google/mipnerf.git
# Create a conda environment, note you can use python 3.6-3.8 as
# one of the dependencies (TensorFlow) hasn't supported python 3.9 yet.
conda create --name mipnerf python=3.6.13; conda activate mipnerf
# Prepare pip
conda install pip; pip install --upgrade pip
# Install requirements
pip install -r requirements.txt

[Optional] Install GPU and TPU support for Jax

# Remember to change cuda101 to your CUDA version, e.g. cuda110 for CUDA 11.0.
pip install --upgrade jax jaxlib==0.1.65+cuda101 -f https://storage.googleapis.com/jax-releases/jax_releases.html

Data

Then, you'll need to download the datasets from the NeRF official Google Drive. Please download and unzip nerf_synthetic.zip and nerf_llff_data.zip.

TODO(bmild): Add instructions for synthesizing the multiscale Blender dataset.

Running

Example scripts for training mip-NeRF on individual scenes from the three datasets used in the paper can be found in scripts/. You'll need to change the paths to point to wherever the datasets are located. Gin configuration files for our model and some ablations can be found in configs/. An example script for evaluating on the test set of each scene can be found in scripts/, after which you can use scripts/summarize.ipynb to produce error metrics across all scenes in the same format as was used in tables in the paper.

Citation

If you use this software package, please cite our paper:

@misc{barron2021mipnerf,
      title={Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields},
      author={Jonathan T. Barron and Ben Mildenhall and Matthew Tancik and Peter Hedman and Ricardo Martin-Brualla and Pratul P. Srinivasan},
      year={2021},
      eprint={2103.13415},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Acknowledgements

Thanks to Boyang Deng for JaxNeRF.

Owner
Google
Google ❤️ Open Source
Google
Comments
  • Question about mipnerf360

    Question about mipnerf360

    Hello! Sorry if this is the wrong place to post this question. In mipnerf, during inference, LLFF scenes have their near and far distances set as 0 and 1 with the use of NDC. However in mipnerf360, during training, without the usage of NDC, I assume near and far distances are used directly from the COLMAP calculations. If so, how do you set these values during inference? Has it got something to do with the contract(.) operator? Or perhaps I am approaching this wrongly?

  • IPE explained

    IPE explained

    Hello. First of all, thank you for sharing this great work.

    I would like to kindly ask you to elaborate more on how did you derive the formulas for IPE.

    Why do you concatenate it this way? image

    How did you get the formula for the y_var? image

    I cannot quiet get it from the article. Thank you :)

  • Why positional encoding for viewing direction?

    Why positional encoding for viewing direction?

    Positional encoding (PE) or IPE is required to allow NeRF/MipNeRF learn high frequencies. I understand that volume density and color as a function of 3d point can have discontinuities, especially at object boundaries. However, for a given 3d position, the color as a function of viewing direction is a smooth function, right? I can't imagine a scenario where it may have high frequencies. So, why do we need PE for viewing direction?

    @jonbarron, I've pondered over this question for quite some time now, but didn't understand. Any intuition will be helpful.

  • Radii computation with and without NDC

    Radii computation with and without NDC

    The radii computation code is different for non-ndc and ndc spaces. In particular, without ndc, radii computation uses only dx derived from directions, but when ndc is enabled, (dx+dy)/2 is used, which is derived from ndc_origins. Can you please shed some light on why it is done so?

  • keras version match tensorflow

    keras version match tensorflow

    Hi, when I am trying to reproduce the result, I installed on cuda10.1 py3.6 tensorflow2.3.1, keras2.7.0

    Then I met the issue when I import tensorflow, and from keras import optimizers,

    "another metric with the same name already exists".

    I would like to know what's the version you use during your experiments?

    Thanks in advance!

  • Question about CPU OOM.

    Question about CPU OOM.

    I use the following command to train on multiscale datasets, but get "killed" output.

    bash ./scripts/train_multiblender.sh
    

    I have generated multiscale datesets, and changed the correct path in ./scripts/train_multiblender.sh. It works well on ./scripts/train_blender.sh with original datasets.

    My computer has 16G MEM and 4G SWAP and I'd like to know the minimum requirements.

    Thanks.

  • Are the signs inverted on rgb padding?

    Are the signs inverted on rgb padding?

    After the sigmoid activation I noticed that you are doing this

    rgb = rgb * (1 + 2 * self.rgb_padding) - self.rgb_padding
    

    where rgb_padding = 0.001 by default.

    Is this intentional or did you mean to move the range to (rgb_padding, 1 - rgb_padding)?

  • Reconstructing using colmap poses

    Reconstructing using colmap poses

    If we estimate the poses of the lego dataset using colmap, will it work? I tried doing the same, and tried training it using train_llff.sh but it showed inconsistent results.

  • Use of sensor size in focal length calculation

    Use of sensor size in focal length calculation

    Hi,

    Thanks for making the mip-NeRF codebase public. For the focal length calculation, why isn't the actual sensor size used (example 36mm for the Blender dataset) here? I believe the formula for calculating focal length from FOV uses the true sensor size. https://github.com/google/mipnerf/blob/84c969e0a623edd183b75693aed72a7e7c22902d/internal/datasets.py#L349

  • Need some suggestions for reimplementation in pytorch

    Need some suggestions for reimplementation in pytorch

    Hi, thanks for this remarkable work.

    I'm trying to reimplement mipnerf in pytorch on LLFF. Currently, I've finished the pipeline identical to this repo. But the training converged to a lower psnr on the same validation images, e.g. 22.5 in my implementation vs 24.9 in this repo for 20w iterations . Is this phenomenon possibly caused by the natural of the optimizers in pytorch? Or maybe I missed some points in mipnerf implementation? Would you please provide some suggestions?

    Thank you!

  • Extracting isosurfaces of Mip-NeRF

    Extracting isosurfaces of Mip-NeRF

    Hello!

    Thanks for sharing this awesome work! :)

    I'm curious, have you thought about the correct way of extracting the isosurfaces from the trained implicit function?

    For vanilla NeRF model it is possible to extract the level surface at two scales using separately trained coarse and fine networks. Here, as far as I understand, it is possible to extract the level surface at an arbitrary scale, and for that I could just query the network with a positional encoding obtained for a desired point x and some manually selected variance, which determines the scale.

    Does this approach makes sense to you, or are there some reasons why it could fail?

  • Batch_size Can't reduce GPU Memory

    Batch_size Can't reduce GPU Memory

    Hi, I am using RTX3080 for training and will crash every 5000 iterations when executing this code vis_suite = vis.visualize_suite(pred_distance, pred_acc) And here is the error message

    Traceback (most recent call last):
      File "/home/feihu/.conda/envs/metanerf/lib/python3.9/runpy.py", line 197, in _run_module_as_main
        return _run_code(code, main_globals, None,
      File "/home/feihu/.conda/envs/metanerf/lib/python3.9/runpy.py", line 87, in _run_code
        exec(code, run_globals)
      File "/data/feihu/mipnerf-main/train.py", line 321, in <module>
        app.run(main)
      File "/home/feihu/.conda/envs/metanerf/lib/python3.9/site-packages/absl/app.py", line 312, in run
        _run_main(main, args)
      File "/home/feihu/.conda/envs/metanerf/lib/python3.9/site-packages/absl/app.py", line 258, in _run_main
        sys.exit(main(argv))
      File "/data/feihu/mipnerf-main/train.py", line 295, in main
        vis_suite = vis.visualize_suite(pred_distance, pred_acc)
      File "/data/feihu/mipnerf-main/internal/vis.py", line 140, in visualize_suite
        'depth_normals': visualize_normals(depth, acc)
      File "/data/feihu/mipnerf-main/internal/vis.py", line 125, in visualize_normals
        normals = depth_to_normals(scaled_depth)
      File "/data/feihu/mipnerf-main/internal/vis.py", line 38, in depth_to_normals
        dy = convolve2d(depth, f_blur[None, :] * f_edge[:, None])
      File "/data/feihu/mipnerf-main/internal/vis.py", line 30, in convolve2d
        return jsp.signal.convolve2d(
      File "/home/feihu/.conda/envs/metanerf/lib/python3.9/site-packages/jax/_src/scipy/signal.py", line 85, in convolve2d
        return _convolve_nd(in1, in2, mode, precision=precision)
      File "/home/feihu/.conda/envs/metanerf/lib/python3.9/site-packages/jax/_src/scipy/signal.py", line 65, in _convolve_nd
        result = lax.conv_general_dilated(in1[None, None], in2[None, None], strides,
      File "/home/feihu/.conda/envs/metanerf/lib/python3.9/site-packages/jax/_src/lax/convolution.py", line 147, in conv_general_dilated
        return conv_general_dilated_p.bind(
      File "/home/feihu/.conda/envs/metanerf/lib/python3.9/site-packages/jax/core.py", line 323, in bind
        return self.bind_with_trace(find_top_trace(args), args, params)
      File "/home/feihu/.conda/envs/metanerf/lib/python3.9/site-packages/jax/core.py", line 326, in bind_with_trace
        out = trace.process_primitive(self, map(trace.full_raise, args), params)
      File "/home/feihu/.conda/envs/metanerf/lib/python3.9/site-packages/jax/core.py", line 675, in process_primitive
        return primitive.impl(*tracers, **params)
      File "/home/feihu/.conda/envs/metanerf/lib/python3.9/site-packages/jax/_src/dispatch.py", line 98, in apply_primitive
        compiled_fun = xla_primitive_callable(prim, *unsafe_map(arg_spec, args),
      File "/home/feihu/.conda/envs/metanerf/lib/python3.9/site-packages/jax/_src/util.py", line 219, in wrapper
        return cached(config._trace_context(), *args, **kwargs)
      File "/home/feihu/.conda/envs/metanerf/lib/python3.9/site-packages/jax/_src/util.py", line 212, in cached
        return f(*args, **kwargs)
      File "/home/feihu/.conda/envs/metanerf/lib/python3.9/site-packages/jax/_src/dispatch.py", line 148, in xla_primitive_callable
        compiled = _xla_callable_uncached(lu.wrap_init(prim_fun), device, None,
      File "/home/feihu/.conda/envs/metanerf/lib/python3.9/site-packages/jax/_src/dispatch.py", line 230, in _xla_callable_uncached
        return lower_xla_callable(fun, device, backend, name, donated_invars, False,
      File "/home/feihu/.conda/envs/metanerf/lib/python3.9/site-packages/jax/_src/dispatch.py", line 704, in compile
        self._executable = XlaCompiledComputation.from_xla_computation(
      File "/home/feihu/.conda/envs/metanerf/lib/python3.9/site-packages/jax/_src/dispatch.py", line 806, in from_xla_computation
        compiled = compile_or_get_cached(backend, xla_computation, options)
      File "/home/feihu/.conda/envs/metanerf/lib/python3.9/site-packages/jax/_src/dispatch.py", line 768, in compile_or_get_cached
        return backend_compile(backend, computation, compile_options)
      File "/home/feihu/.conda/envs/metanerf/lib/python3.9/site-packages/jax/_src/profiler.py", line 206, in wrapper
        return func(*args, **kwargs)
      File "/home/feihu/.conda/envs/metanerf/lib/python3.9/site-packages/jax/_src/dispatch.py", line 713, in backend_compile
        return backend.compile(built_c, compile_options=options)
    jaxlib.xla_extension.XlaRuntimeError: UNKNOWN: Failed to determine best cudnn convolution algorithm for:
    %cudnn-conv = (f32[1,1,800,800]{3,2,1,0}, u8[0]{0}) custom-call(f32[1,1,800,800]{3,2,1,0} %Arg_0.1, f32[1,1,3,3]{3,2,1,0} %Arg_1.2), window={size=3x3 pad=1_1x1_1}, dim_labels=bf01_oi01->bf01, custom_call_target="__cudnn$convForward", metadata={op_name="jit(conv_general_dilated)/jit(main)/conv_general_dilated[window_strides=(1, 1) padding=((1, 1), (1, 1)) lhs_dilation=(1, 1) rhs_dilation=(1, 1) dimension_numbers=ConvDimensionNumbers(lhs_spec=(0, 1, 2, 3), rhs_spec=(0, 1, 2, 3), out_spec=(0, 1, 2, 3)) feature_group_count=1 batch_group_count=1 lhs_shape=(1, 1, 800, 800) rhs_shape=(1, 1, 3, 3) precision=(<Precision.HIGHEST: 2>, <Precision.HIGHEST: 2>) preferred_element_type=None]" source_file="/data/feihu/mipnerf-main/internal/vis.py" source_line=30}, backend_config="{\"conv_result_scale\":1,\"activation_mode\":\"0\",\"side_input_scale\":0}"
    
    Original error: UNIMPLEMENTED: DNN library is not found.
    
    To ignore this failure and try to use a fallback algorithm (which may have suboptimal performance), use XLA_FLAGS=--xla_gpu_strict_conv_algorithm_picker=false.  Please also file a bug for the root cause of failing autotuning.
    

    I have found that jax will show this message when OOM, so i changed my batch_size from 1024 to 512, but it still takes 10GB when training, how can I reduce the usage of GPU Memory?

  • Could you provide the Pre-Trained model?

    Could you provide the Pre-Trained model?

    I've been using this for a while, and it works well, but still isn't nearly as good as the demo, could you provide your pre-trained model for download?

  • why only multiscale blender dataset? What about multiscale LLFF dataset?

    why only multiscale blender dataset? What about multiscale LLFF dataset?

    I do not understand why you only have provided the code for multiscale blender dataset. What about multiscale LLFF dataset? Is it because multiscale representation does not work for LLFF dataset? Looking forward to your reply.

  • small typo in instructions

    small typo in instructions

    python scripts/convert_blender_data.py --blenderdir /nerf_synthetic --outdir /multiscale

    should be:

    python scripts/convert_blender_data.py --blenderdir ./nerf_synthetic --outdir ./multiscale

(Arxiv 2021) NeRF--: Neural Radiance Fields Without Known Camera Parameters

NeRF--: Neural Radiance Fields Without Known Camera Parameters Project Page | Arxiv | Colab Notebook | Data Zirui Wang¹, Shangzhe Wu², Weidi Xie², Min

Aug 6, 2022
Unofficial & improved implementation of NeRF--: Neural Radiance Fields Without Known Camera Parameters
Unofficial & improved implementation of NeRF--: Neural Radiance Fields Without Known Camera Parameters

[Unofficial code-base] NeRF--: Neural Radiance Fields Without Known Camera Parameters [ Project | Paper | Official code base ] ⬅️ Thanks the original

Jul 31, 2022
This repository contains a PyTorch implementation of "AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis".
This repository contains a PyTorch implementation of

AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis | Project Page | Paper | PyTorch implementation for the paper "AD-NeRF: Audio

Aug 1, 2022
Code release for DS-NeRF (Depth-supervised Neural Radiance Fields)
Code release for DS-NeRF (Depth-supervised Neural Radiance Fields)

Depth-supervised NeRF: Fewer Views and Faster Training for Free Project | Paper | YouTube Pytorch implementation of our method for learning neural rad

Aug 8, 2022
A PyTorch implementation of NeRF (Neural Radiance Fields) that reproduces the results.
A PyTorch implementation of NeRF (Neural Radiance Fields) that reproduces the results.

NeRF-pytorch NeRF (Neural Radiance Fields) is a method that achieves state-of-the-art results for synthesizing novel views of complex scenes. Here are

Jul 30, 2022
D-NeRF: Neural Radiance Fields for Dynamic Scenes
 D-NeRF: Neural Radiance Fields for Dynamic Scenes

D-NeRF: Neural Radiance Fields for Dynamic Scenes [Project] [Paper] D-NeRF is a method for synthesizing novel views, at an arbitrary point in time, of

Jul 29, 2022
Code release for NeRF (Neural Radiance Fields)
Code release for NeRF (Neural Radiance Fields)

NeRF: Neural Radiance Fields Project Page | Video | Paper | Data Tensorflow implementation of optimizing a neural representation for a single scene an

Aug 9, 2022
Pytorch implementation for A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose
Pytorch implementation for A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose

A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose Paper | Website | Data A-NeRF: Articulated Neural Radiance F

Aug 3, 2022
A minimal TPU compatible Jax implementation of NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

NeRF Minimal Jax implementation of NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Result of Tiny-NeRF RGB Depth

Jul 24, 2022
Point-NeRF: Point-based Neural Radiance Fields
Point-NeRF: Point-based Neural Radiance Fields

Point-NeRF: Point-based Neural Radiance Fields Project Sites | Paper | Primary c

Aug 2, 2022
Unofficial pytorch-lightning implement of Mip-NeRF
Unofficial pytorch-lightning implement of Mip-NeRF

mipnerf_pl Unofficial pytorch-lightning implement of Mip-NeRF, Here are some results generated by this repository (pre-trained models are provided bel

Aug 6, 2022
This is the code for "HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields".

HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields This is the code for "HyperNeRF: A Higher-Dimensional

Aug 3, 2022
With this package, you can generate mixed-integer linear programming (MIP) models of trained artificial neural networks (ANNs) using the rectified linear unit (ReLU) activation function

With this package, you can generate mixed-integer linear programming (MIP) models of trained artificial neural networks (ANNs) using the rectified linear unit (ReLU) activation function. At the moment, only TensorFlow sequential models are supported. Interfaces to either the Pyomo or Gurobi modeling environments are offered.

Aug 2, 2022
Instant-nerf-pytorch - NeRF trained SUPER FAST in pytorch

instant-nerf-pytorch This is WORK IN PROGRESS, please feel free to contribute vi

Jul 25, 2022
Implementation for "Exploiting Aliasing for Manga Restoration" (CVPR 2021)
Implementation for

[CVPR Paper](To appear) | [Project Website](To appear) | BibTex Introduction As a popular entertainment art form, manga enriches the line drawings det

Aug 9, 2022
Eff video representation - Efficient video representation through neural fields

Neural Residual Flow Fields for Efficient Video Representations 1. Download MPI

Jul 20, 2022
This is the code for Deformable Neural Radiance Fields, a.k.a. Nerfies.

Deformable Neural Radiance Fields This is the code for Deformable Neural Radiance Fields, a.k.a. Nerfies. Project Page Paper Video This codebase conta

Aug 5, 2022
Open source repository for the code accompanying the paper 'Non-Rigid Neural Radiance Fields Reconstruction and Novel View Synthesis of a Deforming Scene from Monocular Video'.
Open source repository for the code accompanying the paper 'Non-Rigid Neural Radiance Fields Reconstruction and Novel View Synthesis of a Deforming Scene from Monocular Video'.

Non-Rigid Neural Radiance Fields This is the official repository for the project "Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synt

Jul 29, 2022
Code for KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs
Code for KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs

KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs Check out the paper on arXiv: https://arxiv.org/abs/2103.13744 This repo cont

Jul 28, 2022