This is the code for "HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields".

HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields

This is the code for "HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields".

This codebase implements HyperNeRF using JAX, building on JaxNeRF.

Demo

We provide an easy-to-get-started demo using Google Colab!

These Colabs will allow you to train a basic version of our method using Cloud TPUs (or GPUs) on Google Colab.

Note that due to limited compute resources available, these are not the fully featured models and will train quite slowly and the quality will likely not be that great. If you would like to train a fully featured model, please refer to the instructions below on how to train on your own machine.

Description Link
Process a video into a dataset Open In Colab
Train HyperNeRF Open In Colab
Render HyperNeRF Videos Open In Colab

Setup

The code can be run under any environment with Python 3.8 and above. (It may run with lower versions, but we have not tested it).

We recommend using Miniconda and setting up an environment:

conda create --name hypernerf python=3.8

Next, install the required packages:

pip install -r requirements.txt

Install the appropriate JAX distribution for your environment by following the instructions here. For example:

# For CUDA version 11.1
pip install --upgrade "jax[cuda111]" -f https://storage.googleapis.com/jax-releases/jax_releases.html

Training

After preparing a dataset, you can train a Nerfie by running:

export DATASET_PATH=/path/to/dataset
export EXPERIMENT_PATH=/path/to/save/experiment/to
python train.py \
    --base_folder $EXPERIMENT_PATH \
    --gin_bindings="data_dir='$DATASET_PATH'" \
    --gin_configs configs/test_local.gin

To plot telemetry to Tensorboard and render checkpoints on the fly, also launch an evaluation job by running:

python eval.py \
    --base_folder $EXPERIMENT_PATH \
    --gin_bindings="data_dir='$DATASET_PATH'" \
    --gin_configs configs/test_local.gin

The two jobs should use a mutually exclusive set of GPUs. This division allows the training job to run without having to stop for evaluation.

Configuration

  • We use Gin for configuration.
  • We provide a couple preset configurations.
  • Please refer to config.py for documentation on what each configuration does.
  • Preset configs:
    • hypernerf_vrig_ds_2d.gin: The deformable surface configuration for the validation rig (novel-view synthesis) experiments.
    • hypernerf_vrig_ap_2d.gin: The axis-aligned plane configuration for the validation rig (novel-view synthesis) experiments.
    • hypernerf_interp_ds_2d.gin: The deformable surface configuration for the interpolation experiments.
    • hypernerf_interp_ap_2d.gin: The axis-aligned plane configuration for the interpolation experiments.

Dataset

The dataset uses the same format as Nerfies.

Citing

If you find our work useful, please consider citing:

@article{park2021hypernerf
  author    = {Park, Keunhong and Sinha, Utkarsh and Hedman, Peter and Barron, Jonathan T. and Bouaziz, Sofien and Goldman, Dan B and Martin-Brualla, Ricardo and Seitz, Steven M.},
  title     = {HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields},
  journal   = {arXiv preprint arXiv:2106.13228},
  year      = {2021},
}
Comments
  • Some tips on collecting selfie video for hypernerf?

    Some tips on collecting selfie video for hypernerf?

    Hi, Keunhong!

    I was wondering if you could provide some rules-of-thumb on how to collect good selfie video so that hypernerf can converge to its best global optima. (My first trial wasn't so much successful despite calibrating my camera extrinsics using background only w/ COLMAP)

    Any tips from others successfully trained their own selfie videos are welcomed to comment!

  • code to generate figures in notebooks file

    code to generate figures in notebooks file

    Hi Keunhong,

    This is really nice work and I am wondering if you have tested the code to generate figures in the notebooks file (like hypernerf_ap_ds_figure.ipynb). I run this code and seems cannot work. Could you give some hints on how to use these code in the notebooks file? Thank you so much!

  • ValueError: Non-hashable static arguments are not supported

    ValueError: Non-hashable static arguments are not supported

    When I run the training script, I encounter the problems as follow. By the way, it's ok when I run training script from nerfies with similar config.

    train.py:302] Starting training
    Traceback (most recent call last):
      File "/root/anaconda3/lib/python3.9/site-packages/jax/api_util.py", line 146, in argnums_partial_except
        hash(static_arg)
    TypeError: unhashable type: 'NerfModel'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/bfs/sz/Research/hypernerf/train.py", line 370, in <module>
        app.run(main)
      File "/root/anaconda3/lib/python3.9/site-packages/absl/app.py", line 312, in run
        _run_main(main, args)
      File "/root/anaconda3/lib/python3.9/site-packages/absl/app.py", line 258, in _run_main
        sys.exit(main(argv))
      File "/bfs/sz/Research/hypernerf/train.py", line 330, in main
        state, stats, keys, model_out = ptrain_step(
      File "/root/anaconda3/lib/python3.9/site-packages/jax/_src/traceback_util.py", line 183, in reraise_with_filtered_traceback
        return fun(*args, **kwargs)
      File "/root/anaconda3/lib/python3.9/site-packages/jax/_src/api.py", line 1669, in f_pmapped
        out = pxla.xla_pmap(
      File "/root/anaconda3/lib/python3.9/site-packages/jax/core.py", line 1620, in bind
        return call_bind(self, fun, *args, **params)
      File "/root/anaconda3/lib/python3.9/site-packages/jax/core.py", line 1551, in call_bind
        outs = primitive.process(top_trace, fun, tracers, params)
      File "/root/anaconda3/lib/python3.9/site-packages/jax/core.py", line 1623, in process
        return trace.process_map(self, fun, tracers, params)
      File "/root/anaconda3/lib/python3.9/site-packages/jax/core.py", line 606, in process_call
        return primitive.impl(f, *tracers, **params)
      File "/root/anaconda3/lib/python3.9/site-packages/jax/interpreters/pxla.py", line 624, in xla_pmap_impl
        compiled_fun, fingerprint = parallel_callable(fun, backend, axis_name, axis_size,
      File "/root/anaconda3/lib/python3.9/site-packages/jax/linear_util.py", line 262, in memoized_fun
        ans = call(fun, *args)
      File "/root/anaconda3/lib/python3.9/site-packages/jax/interpreters/pxla.py", line 712, in parallel_callable
        jaxpr, out_sharded_avals, consts = pe.trace_to_jaxpr_final(
      File "/root/anaconda3/lib/python3.9/site-packages/jax/interpreters/partial_eval.py", line 1284, in trace_to_jaxpr_final
        jaxpr, out_avals, consts = trace_to_subjaxpr_dynamic(fun, main, in_avals)
      File "/root/anaconda3/lib/python3.9/site-packages/jax/interpreters/partial_eval.py", line 1262, in trace_to_subjaxpr_dynamic
        ans = fun.call_wrapped(*in_tracers)
      File "/root/anaconda3/lib/python3.9/site-packages/jax/linear_util.py", line 166, in call_wrapped
        ans = self.f(*args, **dict(self.params, **kwargs))
      File "/root/anaconda3/lib/python3.9/site-packages/jax/_src/traceback_util.py", line 183, in reraise_with_filtered_traceback
        return fun(*args, **kwargs)
      File "/root/anaconda3/lib/python3.9/site-packages/jax/_src/api.py", line 413, in cache_miss
        f, args = argnums_partial_except(f, static_argnums, args, allow_invalid=True)
      File "/root/anaconda3/lib/python3.9/site-packages/jax/api_util.py", line 148, in argnums_partial_except
        raise ValueError(
    jax._src.traceback_util.UnfilteredStackTrace: ValueError: Non-hashable static arguments are not supported, as this can lead to unexpected cache-misses. Static argument (index 0) of type <class 'hypernerf.models.NerfModel'> for function train_step is non-hashable.
    
    The stack trace below excludes JAX-internal frames.
    The preceding is the original exception that occurred, unmodified.
    
    --------------------
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/bfs/sz/Research/hypernerf/train.py", line 370, in <module>
        app.run(main)
      File "/root/anaconda3/lib/python3.9/site-packages/absl/app.py", line 312, in run
        _run_main(main, args)
      File "/root/anaconda3/lib/python3.9/site-packages/absl/app.py", line 258, in _run_main
        sys.exit(main(argv))
      File "/bfs/sz/Research/hypernerf/train.py", line 330, in main
        state, stats, keys, model_out = ptrain_step(
      File "/root/anaconda3/lib/python3.9/site-packages/jax/api_util.py", line 148, in argnums_partial_except
        raise ValueError(
    ValueError: Non-hashable static arguments are not supported, as this can lead to unexpected cache-misses. Static argument (index 0) of type <class 'hypernerf.models.NerfModel'> for function train_step is non-hashable.
    

    the jax version is as below:

    jax                                0.2.17
    jaxlib                             0.1.66+cuda110
    
  • Bump tensorflow from 2.6.0 to 2.6.1

    Bump tensorflow from 2.6.0 to 2.6.1

    ⚠️ Dependabot is rebasing this PR ⚠️

    Rebasing might not happen immediately, so don't worry if this takes some time.

    Note: if you make any changes to this PR yourself, they will take precedence over the rebase.


    Bumps tensorflow from 2.6.0 to 2.6.1.

    Release notes

    Sourced from tensorflow's releases.

    TensorFlow 2.6.1

    Release 2.6.1

    This release introduces several vulnerability fixes:

    • Fixes a code injection issue in saved_model_cli (CVE-2021-41228)
    • Fixes a vulnerability due to use of uninitialized value in Tensorflow (CVE-2021-41225)
    • Fixes a heap OOB in FusedBatchNorm kernels (CVE-2021-41223)
    • Fixes an arbitrary memory read in ImmutableConst (CVE-2021-41227)
    • Fixes a heap OOB in SparseBinCount (CVE-2021-41226)
    • Fixes a heap OOB in SparseFillEmptyRows (CVE-2021-41224)
    • Fixes a segfault due to negative splits in SplitV (CVE-2021-41222)
    • Fixes segfaults and vulnerabilities caused by accesses to invalid memory during shape inference in Cudnn* ops (CVE-2021-41221)
    • Fixes a null pointer exception when Exit node is not preceded by Enter op (CVE-2021-41217)
    • Fixes an integer division by 0 in tf.raw_ops.AllToAll (CVE-2021-41218)
    • Fixes a use after free and a memory leak in CollectiveReduceV2 (CVE-2021-41220)
    • Fixes an undefined behavior via nullptr reference binding in sparse matrix multiplication (CVE-2021-41219)
    • Fixes a heap buffer overflow in Transpose (CVE-2021-41216)
    • Prevents deadlocks arising from mutually recursive tf.function objects (CVE-2021-41213)
    • Fixes a null pointer exception in DeserializeSparse (CVE-2021-41215)
    • Fixes an undefined behavior arising from reference binding to nullptr in tf.ragged.cross (CVE-2021-41214)
    • Fixes a heap OOB read in tf.ragged.cross (CVE-2021-41212)
    • Fixes a heap OOB in shape inference for QuantizeV2 (CVE-2021-41211)
    • Fixes a heap OOB read in all tf.raw_ops.QuantizeAndDequantizeV* ops (CVE-2021-41205)
    • Fixes an FPE in ParallelConcat (CVE-2021-41207)
    • Fixes FPE issues in convolutions with zero size filters (CVE-2021-41209)
    • Fixes a heap OOB read in tf.raw_ops.SparseCountSparseOutput (CVE-2021-41210)
    • Fixes vulnerabilities caused by incomplete validation in boosted trees code (CVE-2021-41208)
    • Fixes vulnerabilities caused by incomplete validation of shapes in multiple TF ops (CVE-2021-41206)
    • Fixes a segfault produced while copying constant resource tensor (CVE-2021-41204)
    • Fixes a vulnerability caused by unitialized access in EinsumHelper::ParseEquation (CVE-2021-41201)
    • Fixes several vulnerabilities and segfaults caused by missing validation during checkpoint loading (CVE-2021-41203)
    • Fixes an overflow producing a crash in tf.range (CVE-2021-41202)
    • Fixes an overflow producing a crash in tf.image.resize when size is large (CVE-2021-41199)
    • Fixes an overflow producing a crash in tf.tile when tiling tensor is large (CVE-2021-41198)
    • Fixes a vulnerability produced due to incomplete validation in tf.summary.create_file_writer (CVE-2021-41200)
    • Fixes multiple crashes due to overflow and CHECK-fail in ops with large tensor shapes (CVE-2021-41197)
    • Fixes a crash in max_pool3d when size argument is 0 or negative (CVE-2021-41196)
    • Fixes a crash in tf.math.segment_* operations (CVE-2021-41195)
    • Updates curl to 7.78.0 to handle CVE-2021-22922, CVE-2021-22923, CVE-2021-22924, CVE-2021-22925, and CVE-2021-22926.
    Changelog

    Sourced from tensorflow's changelog.

    Release 2.6.1

    This release introduces several vulnerability fixes:

    • Fixes a code injection issue in saved_model_cli (CVE-2021-41228)
    • Fixes a vulnerability due to use of uninitialized value in Tensorflow (CVE-2021-41225)
    • Fixes a heap OOB in FusedBatchNorm kernels (CVE-2021-41223)
    • Fixes an arbitrary memory read in ImmutableConst (CVE-2021-41227)
    • Fixes a heap OOB in SparseBinCount (CVE-2021-41226)
    • Fixes a heap OOB in SparseFillEmptyRows (CVE-2021-41224)
    • Fixes a segfault due to negative splits in SplitV (CVE-2021-41222)
    • Fixes segfaults and vulnerabilities caused by accesses to invalid memory during shape inference in Cudnn* ops (CVE-2021-41221)
    • Fixes a null pointer exception when Exit node is not preceded by Enter op (CVE-2021-41217)
    • Fixes an integer division by 0 in tf.raw_ops.AllToAll (CVE-2021-41218)
    • Fixes a use after free and a memory leak in CollectiveReduceV2 (CVE-2021-41220)
    • Fixes an undefined behavior via nullptr reference binding in sparse matrix multiplication (CVE-2021-41219)
    • Fixes a heap buffer overflow in Transpose (CVE-2021-41216)
    • Prevents deadlocks arising from mutually recursive tf.function objects (CVE-2021-41213)
    • Fixes a null pointer exception in DeserializeSparse (CVE-2021-41215)
    • Fixes an undefined behavior arising from reference binding to nullptr in tf.ragged.cross (CVE-2021-41214)
    • Fixes a heap OOB read in tf.ragged.cross (CVE-2021-41212)
    • Fixes a heap OOB in shape inference for QuantizeV2 (CVE-2021-41211)
    • Fixes a heap OOB read in all tf.raw_ops.QuantizeAndDequantizeV* ops (CVE-2021-41205)
    • Fixes an FPE in ParallelConcat (CVE-2021-41207)
    • Fixes FPE issues in convolutions with zero size filters (CVE-2021-41209)
    • Fixes a heap OOB read in tf.raw_ops.SparseCountSparseOutput (CVE-2021-41210)

    ... (truncated)

    Commits
    • 3aa40c3 Merge pull request #52889 from tensorflow/fix-build-1-on-r2.6
    • 1a97260 Upper bound tensorflow_estimator to match release
    • b03a4f1 Merge pull request #52874 from tensorflow-jenkins/relnotes-2.6.1-11115
    • 1067732 Update RELEASE.md
    • 7882f9c Merge pull request #52875 from tensorflow-jenkins/version-numbers-2.6.1-6784
    • bf0618d Update version numbers to 2.6.1
    • edf2a35 Insert release notes place-fill
    • ea77035 Merge pull request #52865 from tensorflow/fix-build-2-on-r2.6
    • c42e447 Merge pull request #52864 from tensorflow/fix-build-1-on-r2.6
    • f0258a5 Fix build on tensorflow/core/kernels/boosted_trees/stats_ops.cc
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

  • Questions about camera parameters and scene parameter in dataset

    Questions about camera parameters and scene parameter in dataset

    Hello, I really appreciate to this awesome work!!

    To experiment on your dataset with pytorch framework, First, I''m trying to reimplemen nerfies data loader(HyperNeRF data loader) in pytorch.

    However, I have some questions about camera parameters and scene parameters in dataset.

    1. What does the scale, scene_to_metric and bbox in scene.json file?? What is differences between above scale parameter and 'scale' of images like 4x? ( scale of image vs scale of scene )

    [ scene_scale is applied ] image

    1. For llff data format, we generate rays using this code.

    [llff data format - get ray function] image

    I think that is the simple version of yours(ignore the distortion and assume that principal points is almost same to the center of image)

    [ yours ] image

    But you undistorted to generate dirs using principal points, tangential distortion, radial_distortion, and pixel_aspect_ratio. Is there huge difference?

    1. Are HyperNerf and Nerfies is implemented in NDC(Normalized Device Coordinate)? Usually many Nerf-based models have been implemented in NDC.

    It will be really helpful for me if you answer this question!! Thanks for your awesome work again!

    Best regards Dogyoon Lee

  • Questions about camera coordinate and poses

    Questions about camera coordinate and poses

    I really appreciate to your awesome work!!

    I have some questions about the camera pose matrix in camera folder's json files.

    I'm reimplementing your paper on pytorch based nerf code.

    However, I have confused about camera coordinate in nerfies dataset.

    1. In load_llff.py in nerf pytorch code, they load the camera poses from given poses_bounds.npy file and correct their correct rotation matrix like follow codes.(poses matrix format is (3, 5, frame_num)

    image

    However, I could not find any correction part on your code when you generate the poses of dataset after loading them from given camera information json files.

    I guess you used different coordinate system compared to original nerf(using llff dataset)

    Then if I want to use your dataset as llff format and ray function, should I correct poses matrix like above code?

    If you don't mind, could you explain the camera coordinate axis format of COLMAP results, NeRF(llff), and Nerfies(HyperNerf)??

    1. for dynamic scene dataset(called nvidia-dataset), there is no undistortion part for camera pose like follow part of your code.
    image

    I wonder the necessity of above part because nerfies dataset images seems to not be distorted. and the dynamic scene dataset is pre-undistorted image?

    1. Did you implement your code on NDC coordinate?? I wonder should I use the ndc_rays

    I really hope your answer and appreciate to your great work again!!

    Thank you.

  • Pipeline of the data preprocess

    Pipeline of the data preprocess

    Hi, thanks for your great work! And I'm trying to run hyperNeRF on my own data captured with a mobile phone. But after running COLMAP on those frames, I don't know how to get all those parameters in your data class. Could you please offer a brief guide or something to refer to? I worry that I can't get the best performance of your work without a correct data preprocess. Thanks!

  • Inconsistent shapes between value and initializer for parameter

    Inconsistent shapes between value and initializer for parameter "kernel" in "/warp_field/trunk/hidden_0"

    in running eval.py with "hypernerf_vrig_ap_2d.gin" configuration I got this error:

    raise errors.ScopeParamShapeError(name, self.path_text, flax.errors.ScopeParamShapeError: Inconsistent shapes between value and initializer for parameter "kernel" in "/warp_field/trunk/hidden_0": (32, 128), (47, 128). (https://flax.readthedocs.io/en/latest/flax.errors.html#flax.errors.ScopeParamShapeError)

    anyone has any idea?

  • Bump tensorflow from 2.6.1 to 2.6.3

    Bump tensorflow from 2.6.1 to 2.6.3

    Bumps tensorflow from 2.6.1 to 2.6.3.

    Release notes

    Sourced from tensorflow's releases.

    TensorFlow 2.6.3

    Release 2.6.3

    This releases introduces several vulnerability fixes:

    • Fixes a floating point division by 0 when executing convolution operators (CVE-2022-21725)
    • Fixes a heap OOB read in shape inference for ReverseSequence (CVE-2022-21728)
    • Fixes a heap OOB access in Dequantize (CVE-2022-21726)
    • Fixes an integer overflow in shape inference for Dequantize (CVE-2022-21727)
    • Fixes a heap OOB access in FractionalAvgPoolGrad (CVE-2022-21730)
    • Fixes an overflow and divide by zero in UnravelIndex (CVE-2022-21729)
    • Fixes a type confusion in shape inference for ConcatV2 (CVE-2022-21731)
    • Fixes an OOM in ThreadPoolHandle (CVE-2022-21732)
    • Fixes an OOM due to integer overflow in StringNGrams (CVE-2022-21733)
    • Fixes more issues caused by incomplete validation in boosted trees code (CVE-2021-41208)
    • Fixes an integer overflows in most sparse component-wise ops (CVE-2022-23567)
    • Fixes an integer overflows in AddManySparseToTensorsMap (CVE-2022-23568)
    • Fixes a number of CHECK-failures in MapStage (CVE-2022-21734)
    • Fixes a division by zero in FractionalMaxPool (CVE-2022-21735)
    • Fixes a number of CHECK-fails when building invalid/overflowing tensor shapes (CVE-2022-23569)
    • Fixes an undefined behavior in SparseTensorSliceDataset (CVE-2022-21736)
    • Fixes an assertion failure based denial of service via faulty bin count operations (CVE-2022-21737)
    • Fixes a reference binding to null pointer in QuantizedMaxPool (CVE-2022-21739)
    • Fixes an integer overflow leading to crash in SparseCountSparseOutput (CVE-2022-21738)
    • Fixes a heap overflow in SparseCountSparseOutput (CVE-2022-21740)
    • Fixes an FPE in BiasAndClamp in TFLite (CVE-2022-23557)
    • Fixes an FPE in depthwise convolutions in TFLite (CVE-2022-21741)
    • Fixes an integer overflow in TFLite array creation (CVE-2022-23558)
    • Fixes an integer overflow in TFLite (CVE-2022-23559)
    • Fixes a dangerous OOB write in TFLite (CVE-2022-23561)
    • Fixes a vulnerability leading to read and write outside of bounds in TFLite (CVE-2022-23560)
    • Fixes a set of vulnerabilities caused by using insecure temporary files (CVE-2022-23563)
    • Fixes an integer overflow in Range resulting in undefined behavior and OOM (CVE-2022-23562)
    • Fixes a vulnerability where missing validation causes tf.sparse.split to crash when axis is a tuple (CVE-2021-41206)
    • Fixes a CHECK-fail when decoding resource handles from proto (CVE-2022-23564)
    • Fixes a CHECK-fail with repeated AttrDef (CVE-2022-23565)
    • Fixes a heap OOB write in Grappler (CVE-2022-23566)
    • Fixes a CHECK-fail when decoding invalid tensors from proto (CVE-2022-23571)
    • Fixes a null-dereference when specializing tensor type (CVE-2022-23570)
    • Fixes a crash when type cannot be specialized (CVE-2022-23572)
    • Fixes a heap OOB read/write in SpecializeType (CVE-2022-23574)
    • Fixes an unitialized variable access in AssignOp (CVE-2022-23573)
    • Fixes an integer overflow in OpLevelCostEstimator::CalculateTensorSize (CVE-2022-23575)
    • Fixes an integer overflow in OpLevelCostEstimator::CalculateOutputSize (CVE-2022-23576)
    • Fixes a null dereference in GetInitOp (CVE-2022-23577)
    • Fixes a memory leak when a graph node is invalid (CVE-2022-23578)
    • Fixes an abort caused by allocating a vector that is too large (CVE-2022-23580)
    • Fixes multiple CHECK-failures during Grappler's IsSimplifiableReshape (CVE-2022-23581)
    • Fixes multiple CHECK-failures during Grappler's SafeToRemoveIdentity (CVE-2022-23579)
    • Fixes multiple CHECK-failures in TensorByteSize (CVE-2022-23582)

    ... (truncated)

    Changelog

    Sourced from tensorflow's changelog.

    Release 2.6.3

    This releases introduces several vulnerability fixes:

    • Fixes a floating point division by 0 when executing convolution operators (CVE-2022-21725)
    • Fixes a heap OOB read in shape inference for ReverseSequence (CVE-2022-21728)
    • Fixes a heap OOB access in Dequantize (CVE-2022-21726)
    • Fixes an integer overflow in shape inference for Dequantize (CVE-2022-21727)
    • Fixes a heap OOB access in FractionalAvgPoolGrad (CVE-2022-21730)
    • Fixes an overflow and divide by zero in UnravelIndex (CVE-2022-21729)
    • Fixes a type confusion in shape inference for ConcatV2 (CVE-2022-21731)
    • Fixes an OOM in ThreadPoolHandle (CVE-2022-21732)
    • Fixes an OOM due to integer overflow in StringNGrams (CVE-2022-21733)
    • Fixes more issues caused by incomplete validation in boosted trees code (CVE-2021-41208)
    • Fixes an integer overflows in most sparse component-wise ops (CVE-2022-23567)
    • Fixes an integer overflows in AddManySparseToTensorsMap (CVE-2022-23568)
    • Fixes a number of CHECK-failures in MapStage (CVE-2022-21734)
    • Fixes a division by zero in FractionalMaxPool (CVE-2022-21735)
    • Fixes a number of CHECK-fails when building invalid/overflowing tensor shapes (CVE-2022-23569)
    • Fixes an undefined behavior in SparseTensorSliceDataset (CVE-2022-21736)
    • Fixes an assertion failure based denial of service via faulty bin count operations (CVE-2022-21737)
    • Fixes a reference binding to null pointer in QuantizedMaxPool (CVE-2022-21739)
    • Fixes an integer overflow leading to crash in SparseCountSparseOutput (CVE-2022-21738)
    • Fixes a heap overflow in SparseCountSparseOutput (CVE-2022-21740)
    • Fixes an FPE in BiasAndClamp in TFLite (CVE-2022-23557)
    • Fixes an FPE in depthwise convolutions in TFLite (CVE-2022-21741)

    ... (truncated)

    Commits
    • 92a6bb0 Merge pull request #54204 from tensorflow/disable-flaky-tests-on-r2.6
    • 3dc94ed Merge pull request #54203 from tensorflow/fix-sanity-on-r2.6
    • 52f4b51 Disable flaky test
    • ad069c0 Reorder tags to fix buildifier linting
    • 1db0b90 Merge pull request #54200 from tensorflow/cherrypick-510ae18200d0a4fad797c0bf...
    • dcbbb77 Set Env Variable to override Setuptools new behavior
    • fd16fa9 Merge pull request #54174 from tensorflow-jenkins/relnotes-2.6.3-32160
    • bc97b92 Update RELEASE.md
    • 1e6335b Merge pull request #54184 from tensorflow/cherrypick-d437dec4d549fc30f9b85c75...
    • 083c3c6 Update third_party/icu/workspace.bzl
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

  • Bump numpy from 1.19.5 to 1.21.0

    Bump numpy from 1.19.5 to 1.21.0

    Bumps numpy from 1.19.5 to 1.21.0.

    Release notes

    Sourced from numpy's releases.

    v1.21.0

    NumPy 1.21.0 Release Notes

    The NumPy 1.21.0 release highlights are

    • continued SIMD work covering more functions and platforms,
    • initial work on the new dtype infrastructure and casting,
    • universal2 wheels for Python 3.8 and Python 3.9 on Mac,
    • improved documentation,
    • improved annotations,
    • new PCG64DXSM bitgenerator for random numbers.

    In addition there are the usual large number of bug fixes and other improvements.

    The Python versions supported for this release are 3.7-3.9. Official support for Python 3.10 will be added when it is released.

    :warning: Warning: there are unresolved problems compiling NumPy 1.21.0 with gcc-11.1 .

    • Optimization level -O3 results in many wrong warnings when running the tests.
    • On some hardware NumPy will hang in an infinite loop.

    New functions

    Add PCG64DXSM BitGenerator

    Uses of the PCG64 BitGenerator in a massively-parallel context have been shown to have statistical weaknesses that were not apparent at the first release in numpy 1.17. Most users will never observe this weakness and are safe to continue to use PCG64. We have introduced a new PCG64DXSM BitGenerator that will eventually become the new default BitGenerator implementation used by default_rng in future releases. PCG64DXSM solves the statistical weakness while preserving the performance and the features of PCG64.

    See upgrading-pcg64 for more details.

    (gh-18906)

    Expired deprecations

    • The shape argument numpy.unravel_index cannot be passed as dims keyword argument anymore. (Was deprecated in NumPy 1.16.)

    ... (truncated)

    Commits
    • b235f9e Merge pull request #19283 from charris/prepare-1.21.0-release
    • 34aebc2 MAINT: Update 1.21.0-notes.rst
    • 493b64b MAINT: Update 1.21.0-changelog.rst
    • 07d7e72 MAINT: Remove accidentally created directory.
    • 032fca5 Merge pull request #19280 from charris/backport-19277
    • 7d25b81 BUG: Fix refcount leak in ResultType
    • fa5754e BUG: Add missing DECREF in new path
    • 61127bb Merge pull request #19268 from charris/backport-19264
    • 143d45f Merge pull request #19269 from charris/backport-19228
    • d80e473 BUG: Removed typing for == and != in dtypes
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

  • Bump pillow from 8.3.2 to 9.0.0

    Bump pillow from 8.3.2 to 9.0.0

    Bumps pillow from 8.3.2 to 9.0.0.

    Release notes

    Sourced from pillow's releases.

    9.0.0

    https://pillow.readthedocs.io/en/stable/releasenotes/9.0.0.html

    Changes

    ... (truncated)

    Changelog

    Sourced from pillow's changelog.

    9.0.0 (2022-01-02)

    • Restrict builtins for ImageMath.eval(). CVE-2022-22817 #5923 [radarhere]

    • Ensure JpegImagePlugin stops at the end of a truncated file #5921 [radarhere]

    • Fixed ImagePath.Path array handling. CVE-2022-22815, CVE-2022-22816 #5920 [radarhere]

    • Remove consecutive duplicate tiles that only differ by their offset #5919 [radarhere]

    • Improved I;16 operations on big endian #5901 [radarhere]

    • Limit quantized palette to number of colors #5879 [radarhere]

    • Fixed palette index for zeroed color in FASTOCTREE quantize #5869 [radarhere]

    • When saving RGBA to GIF, make use of first transparent palette entry #5859 [radarhere]

    • Pass SAMPLEFORMAT to libtiff #5848 [radarhere]

    • Added rounding when converting P and PA #5824 [radarhere]

    • Improved putdata() documentation and data handling #5910 [radarhere]

    • Exclude carriage return in PDF regex to help prevent ReDoS #5912 [hugovk]

    • Fixed freeing pointer in ImageDraw.Outline.transform #5909 [radarhere]

    • Added ImageShow support for xdg-open #5897 [m-shinder, radarhere]

    • Support 16-bit grayscale ImageQt conversion #5856 [cmbruns, radarhere]

    • Convert subsequent GIF frames to RGB or RGBA #5857 [radarhere]

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

  • Problem with Geman-McClure loss implementation

    Problem with Geman-McClure loss implementation

    I have been testing the Geman-McClure loss implemented in the repo, more specifically the model_utils.general_loss_with_squared_residual() with alpha = -2.

    However, the output, as shown below, is very different from my expectation. (alpha = -2, scale = 0.1) image Shouldn't it be like the quadratic shape when |x| < scale

  • TypeError: broadcast_to requires ndarray or scalar arguments, got <class 'list'> at position 0.

    TypeError: broadcast_to requires ndarray or scalar arguments, got at position 0.

    Hi authors I am getting this error while running train.py script. I have not made any changes to the code. If I may ask, what was your build setup and configuration ? Also, is this code specific only for a jax version or does it have a forward compatibility with new version of jax. ?

  • There is no rendered video when run locally.

    There is no rendered video when run locally.

    Hello everyone, Can someone help me to run this code locally on my computer? I run train.py and eval.py for my own dataset and my render folder is empty. actually there is no code for render video in eval.py!

  • dataset

    dataset

    Hello, I would like to know how to process my own data set. The medical data set in the video of my data set moves slowly, the data I processed cannot meet the data requirements in the paper, and the training results are also poor. May I ask what I need to pay attention to most when processing other video data

  • Hyper-Space Template

    Hyper-Space Template

    Hello! I want to produce a hyper-space template with my own dataset, but I have no idea how I can do it. Could you please give me some instructions on it?

TensorFlow code for the neural network presented in the paper: "Structural Language Models of Code" (ICML'2020)
TensorFlow code for the neural network presented in the paper:

SLM: Structural Language Models of Code This is an official implementation of the model described in: "Structural Language Models of Code" [PDF] To ap

Sep 21, 2022
Inference code for "StylePeople: A Generative Model of Fullbody Human Avatars" paper. This code is for the part of the paper describing video-based avatars.

NeuralTextures This is repository with inference code for paper "StylePeople: A Generative Model of Fullbody Human Avatars" (CVPR21). This code is for

Aug 3, 2022
A code generator from ONNX to PyTorch code

onnx-pytorch Generating pytorch code from ONNX. Currently support onnx==1.9.0 and torch==1.8.1. Installation From PyPI pip install onnx-pytorch From

Aug 29, 2022
This is the code for our KILT leaderboard submission to the T-REx and zsRE tasks. It includes code for training a DPR model then continuing training with RAG.

KGI (Knowledge Graph Induction) for slot filling This is the code for our KILT leaderboard submission to the T-REx and zsRE tasks. It includes code fo

Sep 13, 2022
Convert Python 3 code to CUDA code.

Py2CUDA Convert python code to CUDA. Usage To convert a python file say named py_file.py to CUDA, run python generate_cuda.py --file py_file.py --arch

Jul 14, 2021
Empirical Study of Transformers for Source Code & A Simple Approach for Handling Out-of-Vocabulary Identifiers in Deep Learning for Source Code

Transformers for variable misuse, function naming and code completion tasks The official PyTorch implementation of: Empirical Study of Transformers fo

Sep 21, 2022
Reference implementation of code generation projects from Facebook AI Research. General toolkit to apply machine learning to code, from dataset creation to model training and evaluation. Comes with pretrained models.

This repository is a toolkit to do machine learning for programming languages. It implements tokenization, dataset preprocessing, model training and m

Sep 20, 2022
Code for the prototype tool in our paper "CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning".

CoProtector Code for the prototype tool in our paper "CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning".

Oct 26, 2021
Low-code/No-code approach for deep learning inference on devices
Low-code/No-code approach for deep learning inference on devices

EzEdgeAI A concept project that uses a low-code/no-code approach to implement deep learning inference on devices. It provides a componentized framewor

Apr 5, 2022
Code for all the Advent of Code'21 challenges mostly written in python

Advent of Code 21 Code for all the Advent of Code'21 challenges mostly written in python. They are not necessarily the best or fastest solutions but j

May 26, 2022
Code to use Augmented Shapiro Wilks Stopping, as well as code for the paper "Statistically Signifigant Stopping of Neural Network Training"

This codebase is being actively maintained, please create and issue if you have issues using it Basics All data files are included under losses and ea

Nov 9, 2021
Opinionated code formatter, just like Python's black code formatter but for Beancount

beancount-black Opinionated code formatter, just like Python's black code formatter but for Beancount Try it out online here Features MIT licensed - b

Sep 8, 2022
a delightful machine learning tool that allows you to train, test and use models without writing code
a delightful machine learning tool that allows you to train, test and use models without writing code

igel A delightful machine learning tool that allows you to train/fit, test and use models without writing code Note I'm also working on a GUI desktop

Sep 21, 2022
Pytorch Lightning code guideline for conferences

Deep learning project seed Use this seed to start new deep learning / ML projects. Built in setup.py Built in requirements Examples with MNIST Badges

Sep 21, 2022
Automatically Build Multiple ML Models with a Single Line of Code. Created by Ram Seshadri. Collaborators Welcome. Permission Granted upon Request.
Automatically Build Multiple ML Models with a Single Line of Code. Created by Ram Seshadri. Collaborators Welcome. Permission Granted upon Request.

Auto-ViML Automatically Build Variant Interpretable ML models fast! Auto_ViML is pronounced "auto vimal" (autovimal logo created by Sanket Ghanmare) N

Sep 19, 2022
Code samples for my book "Neural Networks and Deep Learning"

Code samples for "Neural Networks and Deep Learning" This repository contains code samples for my book on "Neural Networks and Deep Learning". The cod

Sep 23, 2022
Code for: https://berkeleyautomation.github.io/bags/

DeformableRavens Code for the paper Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks. Here is the

Sep 14, 2022
Code for our method RePRI for Few-Shot Segmentation. Paper at http://arxiv.org/abs/2012.06166
Code for our method RePRI for Few-Shot Segmentation. Paper at http://arxiv.org/abs/2012.06166

Region Proportion Regularized Inference (RePRI) for Few-Shot Segmentation In this repo, we provide the code for our paper : "Few-Shot Segmentation Wit

Sep 11, 2022
Applications using the GTN library and code to reproduce experiments in "Differentiable Weighted Finite-State Transducers"

gtn_applications An applications library using GTN. Current examples include: Offline handwriting recognition Automatic speech recognition Installing

Aug 31, 2022