Working in a specific subject area within the framework of machine learning (in our case, this is 3D), you need to understand what are the main datasets on the basis of which the models are trained and tested, as well as what libraries and programs exist for comfortable work, taking into account the specifics of the data.
3D ML 3D .
3D ML :
IT- “VR/AR & AI” — PHYGITALISM.
Datasets
, , . , . 3D ML .1. github ( geometricdeeplearning.com Tutorials).
.1 3D ML.
, ( , ), . -, : , ( ShapeNet). -, , , , .
.2 3D Geometric Deep Learning SGP 2018.
, , , , , .
, GDL.
1. ShapeNet (2015) [1]
3 (.obj ), 4 . . : , , . (SHREC, ICCV) 3D ML.
:
- ShapeNetCore: 51300 55 .
- ShapeNetSem: 12000 270 .
2D-to-3D , 3D . , , , .
2. ModelNet (2015) [2]
127915 3D 662 ( .off).
:
- ModelNet10: 4899 10 .
- ModelNet40: 12311 40 , .
, “ModelNet Benchmark”. ShapeNet , , .
3. Pix3D (2018) [3]
395 3D .obj .mat (.mat Matlab , scipy.io). 9 , . , , 3D .
4. ABC dataset (2019) [4]
CAD . , , , , .
, , GDL ADASE . 3D ML ADASE , .
.3 3D [4].
5. VOCASET: Speech-4D Head Scan Dataset (2019) [5]
VOCASET — 4D face dataset 29- , 60 . 12 480 3-4 , , .
6. ScanNet (2017) [6]
RGB-D , 2,5 1500 , 3D-, .
, (indoor scene), , .
7. Semantic3D (2017) [7]
, 3D , (outdoor scene) 4 , .
KITTY, Semantic3D , .
8. Campus3D (2020) [8]
. 3D ( 900 ).
.4 [8].
Frameworks
.5 , Kaolin [9].
, , , , , (.. train/test pipelines). , . .
, , , 3D . Blender, PCL (Point cloud library) MeshLab. , Python, ( Tensorflow Pytorch) .
Unity 3D Unity ML agents , ( RL ). ML agents RL : , , , .
, 3D ML (. Kaolin .5):
- 3D ;
- ( );
- — , (, , ) . — , , . ( : 1, 2, 3; : redner, softras, pytorch3d expl; medium paper);
- 3D ;
- (model zoo) SOTA 3D ;
- 3D ;
- .
, [9] NVIDIA Kaolin, [10] PyTorch3D, TensorFlow Graphics [11].
, , : , polyscope, , , mesh_to_sdf.
1. PyTorch Geometric (Fey & Lenssen: Department of Computer Graphics TU Dortmund University | 2019) [12]
— PyTorch, , . , . .
, PyTorch Geometric 3D ML, . — . . PyTorch3D.
: Linux; Windows; Mac (CPU only).
2. TensorFlow Graphics (Google Brain | 2019) [11]
PyTorch Geometric, TensorFlow Graphics .
, TensorBoard, 3D . ( TensorFlow PyTorch), TensorBoard 3D.
, TensorFlow Graphics . .
: Linux; Windows; Mac.
3. NVidia Kaolin (NVidia | 2019) [9]
3D ML, ( ). — , .
, , .
: Linux; Windows (unstable).
.7 Kaolin [9].
4. PyTorch 3D (Facebook research | 2019) [10]
Linux , , (. PyTorch Geometric + trimesh + polyscope) 3D ML.
CPU GPU. , 3D ML: chamfer loss, , , , 3D PyTorch .
: Linux; Mac; Windows (unstable).
5. Points 3D (Chaton & Chaulet: Principia Labs | 2020)
. SOTA , .
CPU GPU. , .
: Linux; Mac; Windows.
.8 Jupyter Lab , Points 3D .
1. Kornia (2019) [50]
3D ML , , geometrical deep learning Kornia PyTorch.
.9 [13].
, , , Kornia , 3D ML , RGB-D .
2. Polyscope
. trimesh ( ). , . .
.10 Polyscope.
3. trimesh
:
- ;
- (ICP .);
- , .;
- .
.
.
4. mesh_to_sdf
.11 SDF mesh_to_sdf .
SDF . non-watertight meshes: , , , .. non-manifold .
DeepSDF [14].
5. Open3D [15]
trimesh. Python C++. .
— deep learning machine learning. 3D ML , , , .
, , . PyTroch3D + , trimesh, PCL Open3D ( C++ ), Python API Blender, .
"SGP 2020 Graduate School: Black Box Geometric Computing with Python", Python, , 3D .
[1] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015 [project page]
[2] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang and J. Xiao 3D ShapeNets: A Deep Representation for Volumetric Shapes Proceedings of 28th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2015) [project page]
[3] X. Sun, J. Wu, X. Zhang, Z. Zhang, C. Zhang, T. Xue, J. B. Tenenbaum, and W. T. Freeman. Pix3d: Dataset and methods for single-image 3d shape modeling. In CVPR, 2018 [project page]
[4] Koch, S., Matveev, A., Jiang, Z., Williams, F., Artemov, A., Burnaev, E., Alexa, M., Zorin, D. and Panozzo, D., 2019. ABC: A Big CAD Model Dataset For Geometric Deep Learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 9601–9611). [project page]
[5] Cudeiro, D., Bolkart, T., Laidlaw, C., Ranjan, A. and Black, M.J., 2019. Capture, learning, and synthesis of 3d speaking styles. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 10101-10111). [project page]
[6] Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T. and Nießner, M., 2017. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 5828-5839). [project page]
[7] Hackel, T., Savinov, N., Ladicky, L., Wegner, J.D., Schindler, K. and Pollefeys, M., 2017. Semantic3d. net: A new large-scale point cloud classification benchmark. arXiv preprint arXiv:1704.03847. [project page]
[8] Li, X., Li, C., Tong, Z., Lim, A., Yuan, J., Wu, Y., Tang, J. and Huang, R., 2020. Campus3D: A Photogrammetry Point Cloud Benchmark for Hierarchical Understanding of Outdoor Scene. arXiv preprint arXiv:2008.04968. [project page]
[9] Jatavallabhula, Krishna Murthy, Edward Smith, Jean-Francois Lafleche, Clement Fuji Tsang, Artem Rozantsev, Wenzheng Chen, Tommy Xiang, Rev Lebaredian and Sanja Fidler. “Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research.” ArXivabs/1911.05063 (2019): n. pag. [project page]
[10] Ravi, N., Reizenstein, J., Novotny, D., Gordon, T., Lo, W.Y., Johnson, J. and Gkioxari, G., 2020. Accelerating 3D Deep Learning with PyTorch3D. arXiv preprint arXiv:2007.08501. [github page]
[11] Valentin, J., Keskin, C., Pidlypenskyi, P., Makadia, A., Sud, A. and Bouaziz, S., 2019. Tensorflow graphics: Computer graphics meets deep learning. [project page]
[12] Fey, M. and Lenssen, J.E., 2019. Fast graph representation learning with PyTorch Geometric. arXiv preprint arXiv:1903.02428. [project page]
[13] Riba, E., Mishkin, D., Ponsa, D., Rublee, E. and Bradski, G., 2019. Kornia: an open source differentiable computer vision library for pytorch. arXiv preprint arXiv:1910.02190. [project page]
[14] Park, J.J., Florence, P., Straub, J., Newcombe, R. and Lovegrove, S., 2019. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 165-174). [github page]
[15] Zhou, Q.Y., Park, J. and Koltun, V., 2018. Open3D: A modern library for 3D data processing. arXiv preprint arXiv:1801.09847. [project page]