This is an old revision of the document!
Photometric Depth Super-Resolution Dataset
Photometric Depth Super-Resolution
Bjoern Haefner1\ \ \ \ Songyou Peng2\ \ \ \ Alok Verma1\ \ \ \ Yvain Quéau3\ \ \ \ Daniel Cremers1
1Technical University of Munich\ \ \ \ 2University of Illinois
at Urbana-Champaign\ \ \ \ 3University of Caen
Submitted to IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) Special Issue on RGB-D Vision: Methods and Applications
This study explores the use of photometric techniques (shape-from-shading and uncalibrated photometric stereo) for upsampling the low-resolution depth map from an RGB-D sensor to the higher resolution of the companion RGB image. A single-shot variational approach is first put forward, which is effective as long as the target's reflectance is piecewise-constant. It is then shown that this dependency upon a specific reflectance model can be relaxed by focusing on a specific class of objects (e.g., faces), and delegate reflectance estimation to a deep neural network. A multi-shot strategy based on randomly varying lighting conditions is eventually discussed. It requires no training or prior on the reflectance, yet this comes at the price of a dedicated acquisition setup. Both quantitative and qualitative evaluations illustrate the effectiveness of the proposed methods on synthetic and real-world scenarios.
Code
The code that generated the data shown here is available on github:
https://github.com/BjoernHaefner/DepthSRfromShading
https://github.com/pengsongyou/SRmeetsPS.
Dataset
The following dataset contains RGB-D sequences and reconstructed 3D models of multiple different scenes. We captured the RGB-D data under different scaling factors using an Asus Xtion Pro and an Intel RealSense D415 RGB-D sensor. Please refer to the respective publication when using this data.
Format
For each scene of the Photometric Depth Super-Resolution dataset, we provide the respective RGB-D sequence as well as the refined 3D models. Each RGB-D sequence contains:
- Color frames
- Depth frames
- Masks
- Intrinsic parameters (default factory calibration).
Rucksack
Download (mat-files):
- rucksack_sfs.zip | rucksack_deep.zip | rucksack_ups.zip
Download (png-file of deep net albedo estimate):
- rucksack_albedo_deep.png
Download (obj-files):
- rucksack_input_mesh.zip | rucksack_sfs_mesh.zip | rucksack_deep_mesh.zip | rucksack_ups_mesh.zip
Android
Download (mat-files):
- android_sf2_sfs.zip
- android_sf4_sfs.zip
Download (obj-files):
- android_sf2_input_mesh.zip | android_sf2_sfs_mesh.zip
- android_sf4_input_mesh.zip | android_sf4_sfs_mesh.zip
Basecap
Download (mat-files):
- basecap_sfs.zip
Download (obj-files):
- basecap_input_mesh.zip | basecap_sfs_mesh.zip
Minion
Download (mat-files):
- minion_sfs.zip
Download (obj-files):
- minion_input_mesh.zip | minion_sfs_mesh.zip
Blanket
Download (mat-files):
- blanket_sf2_sfs.zip
- blanket_sf4_sfs.zip
Download (obj-files):
- blanket_sf2_input_mesh.zip | blanket_sf2_sfs_mesh.zip
- blanket_sf4_input_mesh.zip | blanket_sf4_sfs_mesh.zip
Clothes
Download (mat-files):
- clothes_sf2_sfs.zip
- clothes_sf4_sfs.zip
Download (obj-files):
- clothes_sf2_input_mesh.zip | clothes_sf2_sfs_mesh.zip
- clothes_sf4_input_mesh.zip | clothes_sf4_sfs_mesh.zip
Monkey
Download (mat-files):
- monkey_sf2_sfs.zip
- monkey_sf4_sfs.zip
Download (obj-files):
- monkey_sf2_input_mesh.zip | monkey_sf2_sfs_mesh.zip
- monkey_sf4_input_mesh.zip | monkey_sf4_sfs_mesh.zip
Wool
Download (mat-files):
- wool_sf2_sfs.zip
- wool_sf4_sfs.zip
Download (obj-files):
- wool_sf2_input_mesh.zip | wool_sf2_sfs_mesh.zip
- wool_sf4_input_mesh.zip | wool_sf4_sfs_mesh.zip
Face 1
Download (mat-files):
- face1_sfs.zip | face1_deep.zip | face1_ups.zip
Download (png-file of deep net albedo estimate):
- face1_albedo_deep.png
Download (obj-files):
- face1_input_mesh.zip | face1_sfs_mesh.zip | face1_deep_mesh.zip | face1_ups_mesh.zip
Face 2
Download (mat-files):
- face2_deep.zip | face2_ups.zip
Download (png-file of deep net albedo estimate):
- face2_albedo_deep.png
Download (obj-files):
- face2_input_mesh.zip | face2_deep_mesh.zip | face2_ups_mesh.zip
Face 3
Download (mat-files):
- face3_deep.zip
Download (png-file of deep net albedo estimate):
- face3_albedo_deep.png
Download (obj-files):
- face3_input_mesh.zip | face3_deep_mesh.zip
Face 4
Download (mat-files):
- face4_deep.zip
Download (png-file of deep net albedo estimate):
- face4_albedo_deep.png
Download (obj-files):
- face4_input_mesh.zip | face4_deep_mesh.zip
Face 5
Download (mat-files):
- face5_deep.zip
Download (png-file of deep net albedo estimate):
- face5_albedo_deep.png
Download (obj-files):
- face5_input_mesh.zip | face5_deep_mesh.zip
Face 6
Download (mat-files):
- face6_deep.zip
Download (png-file of deep net albedo estimate):
- face6_albedo_deep.png
Download (obj-files):
- face6_input_mesh.zip | face6_deep_mesh.zip
Tabletcase
Download (mat-files):
- tabletcase_sfs.zip | tabletcase_deep.zip | tabletcase_ups.zip
Download (png-file of deep net albedo estimate):
- tabletcase_albedo_deep.png
Download (obj-files):
- tabletcase_input_mesh.zip | tabletcase_sfs_mesh.zip | tabletcase_deep_mesh.zip | tabletcase_ups_mesh.zip
Shirt
Download (mat-files):
- shirt_sf2_ups.zip
- shirt_sf4_ups.zip
Download (obj-files):
- shirt_sf2_input_mesh.zip | shirt_sf2_ups_mesh.zip
- shirt_sf4_input_mesh.zip | shirt_sf4_ups_mesh.zip
Backpack
Download (mat-files):
- backpack_sf4_ups.zip
Download (obj-files):
- backpack_sf4_input_mesh.zip | backpack_sf4_ups_mesh.zip
Ovenmitt
Download (mat-files):
- ovenmitt_sf2_ups.zip
- ovenmitt_sf4_ups.zip
Download (obj-files):
- ovenmitt_sf2_input_mesh.zip | ovenmitt_sf2_ups_mesh.zip
- ovenmitt_sf4_input_mesh.zip | ovenmitt_sf4_ups_mesh.zip
Hat
Download (mat-files):
- hat_sf2_ups.zip
Download (obj-files):
- hat_sf2_input_mesh.zip | hat_sf2_ups_mesh.zip
Vase
Download (mat-files):
- vase_sf4_ups.zip
Download (obj-files):
- vase_sf4_input_mesh.zip | vase_sf4_ups_mesh.zip
License
Unless stated otherwise, all data in the Photometric Depth Super-Resolution Dataset is licensed under a Creative Commons 4.0 Attribution License (CC BY 4.0).
Related Publications
Export as PDF, XML, TEX or BIB
Journal Articles
2020
[] Photometric Depth Super-Resolution , In IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), volume 42, 2020. ([supp] [project page])
Conference and Workshop Papers
2018
[] Fight ill-posedness with ill-posedness: Single-shot variational depth super-resolution from shading , In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018. ([supp] [poster] [slides] [code] [cvf] [video])
Spotlight Presentation
2017
[] Depth Super-Resolution Meets Uncalibrated Photometric Stereo , In IEEE International Conference on Computer Vision Workshops (ICCVW), 2017. ([code][slides] [cvf])
Oral Presentation at ICCV Workshop on Color and Photometry in Computer Vision