Rendering color and depth images for ShapeNet models.

Color & Depth Renderer for ShapeNet


This library includes the tools for rendering multi-view color and depth images of ShapeNet models. Physically based rendering (PBR) is featured based on blender2.79.


Outputs

  1. Color image (20 views)

color_1.png color_2.PNG

  1. Depth image (20 views)

depth_1.png depth_2.PNG

  1. Point cloud and normals (Back-projected from color & depth images)

point_cloud_1.png point_cloud_2.png

  1. Watertight meshes (fused from depth maps)

mesh_1.png mesh_2.png


Install

  1. We recommend to install this repository with conda.
    conda env create -f environment.yml
    conda activate renderer
    
  2. Install Pyfusion by
    cd ./external/pyfusion
    mkdir build
    cd ./build
    cmake ..
    make
    
    Afterwards, compile the Cython code in ./external/pyfusion by
    cd ./external/pyfusion
    python setup.py build_ext --inplace
    
  3. Download & Extract blender2.79b, and specify the path of your blender executable file at ./setting.py by
    g_blender_excutable_path = '../../blender-2.79b-linux-glibc219-x86_64/blender'
    

Usage

  1. Normalize ShapeNet models to a unit cube by

    python normalize_shape.py
    

    The ShapeNetCore.v2 dataset is put in ./datasets/ShapeNetCore.v2. Here we only present some samples in this repository.

  2. Generate multiple camera viewpoints for rendering by

    python create_viewpoints.py
    

    The camera extrinsic parameters will be saved at ./view_points.txt, or you can customize it in this script.

  3. Run renderer to render color and depth images by

    python run_render.py
    

    The rendered images are saved in ./datasets/ShapeNetRenderings. The camera intrinsic and extrinsic parameters are saved in ./datasets/camera_settings. You can change the rendering configurations at ./settings.py, e.g. image sizes and resolution.

  4. The back-projected point cloud and corresponding normals can be visualized by

    python visualization/draw_pc_from_depth.py
    
  5. Watertight meshes can be obtained by

    python depth_fusion.py
    

    The reconstructed meshes are saved in ./datasets/ShapeNetCore.v2_watertight


Citation

This library is used for data preprocessing in our work SK-PCN. If you find it helpful, please consider citing

@inproceedings{NEURIPS2020_ba036d22,
 author = {Nie, Yinyu and Lin, Yiqun and Han, Xiaoguang and Guo, Shihui and Chang, Jian and Cui, Shuguang and Zhang, Jian.J},
 booktitle = {Advances in Neural Information Processing Systems},
 editor = {H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin},
 pages = {16119--16130},
 publisher = {Curran Associates, Inc.},
 title = {Skeleton-bridged Point Completion: From Global Inference to Local Adjustment},
 url = {https://proceedings.neurips.cc/paper/2020/file/ba036d228858d76fb89189853a5503bd-Paper.pdf},
 volume = {33},
 year = {2020}
}


License

This repository is relased under the MIT License.

Owner
Yinyu Nie
Currently a Post-doc researcher in the Visual Computing Group, Technical University of Munich.
Yinyu Nie
Similar Resources

(CVPR 2022 - oral) Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry

(CVPR 2022 - oral) Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry

Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry Official implementation of the paper Multi-View Depth Est

Aug 6, 2022

An image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testingAn image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testing

An image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testingAn image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testing

SVM Données Une base d’images contient 490 images pour l’apprentissage (400 voitures et 90 bateaux), et encore 21 images pour fait des tests. Prétrait

Nov 30, 2021

MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images

MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images This repository contains the implementation of our paper MetaAvatar: Learni

Jul 29, 2022

Official Repo for ICCV2021 Paper: Learning to Regress Bodies from Images using Differentiable Semantic Rendering

[ICCV2021] Learning to Regress Bodies from Images using Differentiable Semantic Rendering Getting Started DSR has been implemented and tested on Ubunt

Jul 29, 2022

ViewFormer: NeRF-free Neural Rendering from Few Images Using Transformers

ViewFormer: NeRF-free Neural Rendering from Few Images Using Transformers Official implementation of ViewFormer. ViewFormer is a NeRF-free neural rend

Jul 29, 2022

PN-Net a neural field-based framework for depth estimation from single-view RGB images.

PN-Net a neural field-based framework for depth estimation from single-view RGB images.

PN-Net We present a neural field-based framework for depth estimation from single-view RGB images. Rather than representing a 2D depth map as a single

Oct 2, 2021

Honours project, on creating a depth estimation map from two stereo images of featureless regions

image-processing This module generates depth maps for shape-blocked-out images Install If working with anaconda, then from the root directory: conda e

Nov 4, 2021

Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging

 Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging

Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging This repository contains an implementation

Aug 2, 2022
Comments
  • Build correspondence between pixels and points

    Build correspondence between pixels and points

    Hi, thanks for your helpful work. I would like to ask if there is a convenient way of outputing correspondence relationships between pixels on the rendered image and points on the corresponding 3D model. I notice that "visualization/draw_pc_from_depth.py" can generate partial point clouds from specific viewpoints. Is there any function to obtain such "pixel-point" mapping? Look forward to your reply as soon as possible!

  • Data Preprocessing

    Data Preprocessing

    Thanks a lot for the awesome work from the authors. But I am a bit confused with data preprocessing. I noticed that in the dataset attached each object has two folders which are respectively image and model. But in the original dataset, each object only has one file which is called XXX.obj. How could I change the XXX.obj to the form of the current dataset?

  •  No module named 'cyfusion'

    No module named 'cyfusion'

    `(renderer) [email protected]:~/lkq/depth_renderer# python depth_fusion.py Traceback (most recent call last): File "depth_fusion.py", line 9, in from external import pyfusion

    File "/root/lkq/depth_renderer/external/pyfusion/init.py", line 8, in from cyfusion import * ModuleNotFoundError: No module named 'cyfusion'` many thanks for your great work. I follow the install step, but can not run the code. can you give me some help?

  • Points not lying on the surface

    Points not lying on the surface

    Hi,

    thanks for open-sourcing this project. I have used it to render some of the shapenet models but have observed that the 3D point do not lie exactly on the surface once back-projected to 3D. The cloud2mesh distance lie in the range of 0.002.

    Have you maybe observed the same? Do you know if this is an inherent precision of Blender depth rendering or if there is maybe some precision loss occurring at some point?

    Zan

Adversarial Color Enhancement: Generating Unrestricted Adversarial Images by Optimizing a Color Filter

ACE Please find the preliminary version published at BMVC 2020 in the folder BMVC_version, and its extended journal version in Journal_version. Datase

Jun 15, 2022
Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model in ONNX
Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model in ONNX

ONNX msg_chn_wacv20 depth completion Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20 model in

Feb 5, 2022
Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model in Tensorflow Lite.
Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model in Tensorflow Lite.

TFLite-msg_chn_wacv20-depth-completion Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model

Oct 4, 2021
Monocular Depth Estimation - Weighted-average prediction from multiple pre-trained depth estimation models
Monocular Depth Estimation - Weighted-average prediction from multiple pre-trained depth estimation models

merged_depth runs (1) AdaBins, (2) DiverseDepth, (3) MiDaS, (4) SGDepth, and (5) Monodepth2, and calculates a weighted-average per-pixel absolute dept

Jun 16, 2022
This repository contains the source code for the paper "DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks",
This repository contains the source code for the paper

DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks Project Page | Video | Presentation | Paper | Data L

Jul 30, 2022
The implemention of Video Depth Estimation by Fusing Flow-to-Depth Proposals

Flow-to-depth (FDNet) video-depth-estimation This is the implementation of paper Video Depth Estimation by Fusing Flow-to-Depth Proposals Jiaxin Xie,

Jun 14, 2022
Monocular Depth Estimation Using Laplacian Pyramid-Based Depth Residuals
Monocular Depth Estimation Using Laplacian Pyramid-Based Depth Residuals

LapDepth-release This repository is a Pytorch implementation of the paper "Monocular Depth Estimation Using Laplacian Pyramid-Based Depth Residuals" M

Jul 28, 2022
Beyond Image to Depth: Improving Depth Prediction using Echoes (CVPR 2021)
 Beyond Image to Depth: Improving Depth Prediction using Echoes (CVPR 2021)

Beyond Image to Depth: Improving Depth Prediction using Echoes (CVPR 2021) Kranti Kumar Parida, Siddharth Srivastava, Gaurav Sharma. We address the pr

Jun 27, 2022
Light-weight network, depth estimation, knowledge distillation, real-time depth estimation, auxiliary data.
Light-weight network, depth estimation, knowledge distillation, real-time depth estimation, auxiliary data.

light-weight-depth-estimation Boosting Light-Weight Depth Estimation Via Knowledge Distillation, https://arxiv.org/abs/2105.06143 Junjie Hu, Chenyou F

May 24, 2022
Data-depth-inference - Data depth inference with python
Data-depth-inference - Data depth inference with python

Welcome! This readme will guide you through the use of the code in this reposito

Feb 8, 2022