Tanks and Temples

Tanks and Temples is a benchmark for image-based 3D reconstruction. The benchmark sequences were acquired outside the lab, in realistic conditions. Ground-truth data was captured using an industrial laser scanner. The benchmark includes both outdoor scenes and indoor environments. The dataset is split into three subsets: training, intermediate, and advanced.

Web: https://www.tanksandtemples.org/
Paper: Tanks and Temples: Benchmarking Large-Scale Scene Reconstruction
Authors: Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, Vladlen Koltun
Method PSNR SSIM LPIPS Time GPU Mem.
Instant NGP 21.62 0.712 0.340 4m 27s 4.13 GB
auditorium 20.67 0.761 0.429 4m 31s 3.92 GB
ballroom 21.62 0.652 0.352 4m 5s 4.02 GB
courtroom 19.44 0.640 0.448 4m 35s 3.93 GB
museum 15.19 0.471 0.606 5m 53s 3.93 GB
palace 19.09 0.668 0.440 4m 46s 4.64 GB
temple 17.84 0.689 0.424 4m 40s 3.94 GB
family 22.59 0.761 0.235 3m 42s 3.42 GB
francis 24.38 0.824 0.265 4m 10s 3.94 GB
horse 21.82 0.784 0.225 3m 47s 3.42 GB
lighthouse 21.65 0.765 0.281 4m 35s 4.02 GB
m60 25.82 0.832 0.202 4m 34s 4.04 GB
panther 26.81 0.844 0.208 4m 29s 4.04 GB
playground 23.33 0.696 0.344 4m 14s 3.97 GB
train 20.01 0.658 0.334 4m 39s 3.94 GB
barn 25.90 0.772 0.271 4m 41s 4.31 GB
caterpillar 21.72 0.633 0.360 4m 30s 4.22 GB
church 19.92 0.650 0.419 4m 21s 4.64 GB
courthouse 20.80 0.681 0.414 4m 55s 6.72 GB
ignatius 19.40 0.613 0.343 3m 58s 3.81 GB
meetingroom 23.24 0.783 0.326 3m 59s 4.18 GB
truck 22.85 0.770 0.216 4m 20s 3.77 GB
NerfStudio 22.04 0.743 0.270 19m 27s 3.74 GB
auditorium 20.77 0.771 0.330 19m 46s 3.88 GB
ballroom 22.68 0.705 0.261 19m 44s 3.88 GB
courtroom 20.24 0.673 0.336 19m 17s 3.88 GB
museum 17.84 0.648 0.311 18m 49s 3.88 GB
palace 17.68 0.640 0.452 20m 9s 3.64 GB
temple 17.06 0.678 0.392 19m 37s 3.88 GB
family 24.32 0.822 0.158 18m 54s 3.63 GB
francis 24.60 0.851 0.190 18m 48s 3.63 GB
horse 24.31 0.847 0.139 18m 53s 3.88 GB
lighthouse 20.85 0.768 0.245 19m 33s 3.89 GB
m60 26.54 0.843 0.179 19m 47s 3.64 GB
panther 27.57 0.858 0.174 20m 10s 3.89 GB
playground 24.69 0.755 0.249 19m 33s 3.64 GB
train 20.43 0.693 0.261 19m 42s 3.88 GB
barn 26.40 0.794 0.215 19m 16s 3.63 GB
caterpillar 21.71 0.666 0.302 19m 52s 3.63 GB
church 20.06 0.671 0.338 19m 29s 3.63 GB
courthouse 18.11 0.632 0.465 19m 37s 3.63 GB
ignatius 20.44 0.689 0.251 19m 5s 3.63 GB
meetingroom 23.21 0.793 0.261 19m 3s 3.63 GB
truck 23.37 0.797 0.167 19m 28s 3.63 GB
Gaussian Opacity Fields 22.39 0.825 0.172 N/A N/A
auditorium 23.20 0.871 0.194 - -
ballroom 22.84 0.818 0.107 - -
courtroom 21.15 0.781 0.168 - -
museum 19.92 0.761 0.152 - -
palace 16.46 0.683 0.443 - -
temple 20.29 0.794 0.234 - -
family 22.31 0.875 0.084 41m 32s 35.16 GB
francis 24.76 0.901 0.158 35m 52s 12.30 GB
horse 23.73 0.881 0.092 38m 29s 24.41 GB
lighthouse 21.80 0.833 0.181 - -
m60 28.04 0.906 0.104 40m 9s 26.18 GB
panther 28.47 0.910 0.102 40m 13s 30.99 GB
playground 23.89 0.869 0.142 - -
train 19.69 0.796 0.164 - -
barn 25.72 0.866 0.140 - -
caterpillar 21.78 0.791 0.187 - -
church 19.65 0.775 0.208 48m 2s 36.45 GB
courthouse 19.60 0.726 0.346 35m 12s 12.82 GB
ignatius 20.34 0.769 0.162 43m 50s 34.97 GB
meetingroom 24.31 0.862 0.140 40m 29s 23.30 GB
truck 22.33 0.860 0.099 - -
Gaussian Splatting 23.83 0.831 0.165 13m 48s 6.95 GB
auditorium 24.13 0.871 0.193 10m 54s 4.98 GB
ballroom 24.07 0.824 0.101 19m 55s 9.08 GB
courtroom 23.12 0.790 0.165 17m 12s 9.02 GB
museum 20.92 0.764 0.160 20m 55s 11.38 GB
palace 19.63 0.736 0.350 11m 21s 5.85 GB
temple 20.85 0.806 0.222 10m 56s 5.13 GB
family 24.43 0.865 0.095 14m 60s 6.31 GB
francis 27.22 0.897 0.169 10m 20s 4.28 GB
horse 23.82 0.875 0.103 11m 35s 4.50 GB
lighthouse 22.11 0.843 0.156 12m 3s 6.09 GB
m60 27.84 0.901 0.113 13m 18s 6.80 GB
panther 28.32 0.908 0.107 13m 19s 7.01 GB
playground 25.37 0.848 0.170 15m 33s 7.44 GB
train 21.67 0.791 0.171 11m 56s 5.39 GB
barn 27.51 0.852 0.160 11m 9s 6.22 GB
caterpillar 23.38 0.791 0.190 11m 32s 5.89 GB
church 22.79 0.811 0.177 17m 15s 8.36 GB
courthouse 22.22 0.779 0.266 11m 8s 10.34 GB
ignatius 21.53 0.776 0.153 16m 36s 8.82 GB
meetingroom 25.19 0.866 0.141 12m 39s 5.50 GB
truck 24.25 0.853 0.108 15m 18s 7.47 GB
Mip-Splatting 23.93 0.833 0.166 15m 56s 7.27 GB
auditorium 24.41 0.872 0.196 11m 47s 4.82 GB
ballroom 24.15 0.826 0.098 25m 4s 9.76 GB
courtroom 23.00 0.791 0.165 20m 46s 9.27 GB
museum 20.88 0.768 0.158 24m 55s 12.57 GB
palace 19.63 0.731 0.354 12m 53s 6.48 GB
temple 20.55 0.805 0.226 12m 3s 5.93 GB
family 24.55 0.872 0.095 15m 44s 6.99 GB
francis 27.61 0.899 0.172 11m 47s 4.40 GB
horse 23.94 0.879 0.104 12m 47s 4.58 GB
lighthouse 22.25 0.844 0.159 13m 47s 7.01 GB
m60 27.98 0.904 0.112 15m 13s 7.13 GB
panther 28.27 0.908 0.109 15m 43s 7.38 GB
playground 25.87 0.861 0.155 18m 26s 8.30 GB
train 21.82 0.795 0.172 13m 26s 5.65 GB
barn 27.75 0.855 0.161 12m 42s 6.12 GB
caterpillar 23.42 0.790 0.197 13m 16s 5.69 GB
church 22.76 0.812 0.176 19m 53s 8.37 GB
courthouse 22.15 0.779 0.265 13m 57s 10.28 GB
ignatius 21.73 0.780 0.159 18m 44s 8.46 GB
meetingroom 25.46 0.870 0.137 14m 13s 5.73 GB
truck 24.36 0.857 0.108 17m 32s 7.68 GB
Zip-NeRF 24.63 0.840 0.131 5h 44m 9s 26.61 GB
auditorium 24.52 0.877 0.153 5h 24m 16s 26.61 GB
ballroom 25.45 0.835 0.113 5h 25m 1s 26.61 GB
courtroom 22.17 0.790 0.153 5h 17m 54s 26.61 GB
museum 19.34 0.746 0.159 5h 24m 17s 26.61 GB
palace 19.11 0.718 0.317 6h 42m 11s 26.61 GB
temple 20.58 0.805 0.183 5h 37m 11s 26.61 GB
family 27.10 0.889 0.067 5h 32m 15s 26.61 GB
francis 29.10 0.915 0.106 5h 34m 11s 26.61 GB
horse 26.82 0.897 0.069 5h 55m 60s 26.61 GB
lighthouse 23.07 0.849 0.131 5h 52m 55s 26.63 GB
m60 29.01 0.912 0.076 6h 3m 19s 26.63 GB
panther 28.76 0.909 0.081 5h 36m 45s 26.63 GB
playground 27.13 0.880 0.095 5h 29m 4s 26.61 GB
train 22.19 0.814 0.119 6h 14m 13s 26.61 GB
barn 29.26 0.884 0.083 5h 28m 43s 26.61 GB
caterpillar 23.94 0.802 0.152 5h 53m 16s 26.61 GB
church 23.14 0.807 0.153 6h 4m 32s 26.61 GB
courthouse 22.88 0.780 0.218 6h 4m 6s 26.61 GB
ignatius 22.61 0.789 0.127 5h 48m 32s 26.61 GB
meetingroom 25.93 0.875 0.121 5h 21m 9s 26.61 GB
truck 25.09 0.864 0.081 5h 37m 11s 26.61 GB

PSNR

Peak Signal to Noise Ratio. The higher the better.

Method auditorium ballroom courtroom museum palace temple family francis horse lighthouse m60 panther playground train barn caterpillar church courthouse ignatius meetingroom truck
Instant NGP 20.67 21.62 19.44 15.19 19.09 17.84 22.59 24.38 21.82 21.65 25.82 26.81 23.33 20.01 25.90 21.72 19.92 20.80 19.40 23.24 22.85
NerfStudio 20.77 22.68 20.24 17.84 17.68 17.06 24.32 24.60 24.31 20.85 26.54 27.57 24.69 20.43 26.40 21.71 20.06 18.11 20.44 23.21 23.37
Gaussian Opacity Fields 23.20 22.84 21.15 19.92 16.46 20.29 22.31 24.76 23.73 21.80 28.04 28.47 23.89 19.69 25.72 21.78 19.65 19.60 20.34 24.31 22.33
Gaussian Splatting 24.13 24.07 23.12 20.92 19.63 20.85 24.43 27.22 23.82 22.11 27.84 28.32 25.37 21.67 27.51 23.38 22.79 22.22 21.53 25.19 24.25
Mip-Splatting 24.41 24.15 23.00 20.88 19.63 20.55 24.55 27.61 23.94 22.25 27.98 28.27 25.87 21.82 27.75 23.42 22.76 22.15 21.73 25.46 24.36
Zip-NeRF 24.52 25.45 22.17 19.34 19.11 20.58 27.10 29.10 26.82 23.07 29.01 28.76 27.13 22.19 29.26 23.94 23.14 22.88 22.61 25.93 25.09

SSIM

Structural Similarity Index. The higher the better. The implementation matches JAX's SSIM and torchmetrics's SSIM (with default parameters).

Method auditorium ballroom courtroom museum palace temple family francis horse lighthouse m60 panther playground train barn caterpillar church courthouse ignatius meetingroom truck
Instant NGP 0.761 0.652 0.640 0.471 0.668 0.689 0.761 0.824 0.784 0.765 0.832 0.844 0.696 0.658 0.772 0.633 0.650 0.681 0.613 0.783 0.770
NerfStudio 0.771 0.705 0.673 0.648 0.640 0.678 0.822 0.851 0.847 0.768 0.843 0.858 0.755 0.693 0.794 0.666 0.671 0.632 0.689 0.793 0.797
Gaussian Opacity Fields 0.871 0.818 0.781 0.761 0.683 0.794 0.875 0.901 0.881 0.833 0.906 0.910 0.869 0.796 0.866 0.791 0.775 0.726 0.769 0.862 0.860
Gaussian Splatting 0.871 0.824 0.790 0.764 0.736 0.806 0.865 0.897 0.875 0.843 0.901 0.908 0.848 0.791 0.852 0.791 0.811 0.779 0.776 0.866 0.853
Mip-Splatting 0.872 0.826 0.791 0.768 0.731 0.805 0.872 0.899 0.879 0.844 0.904 0.908 0.861 0.795 0.855 0.790 0.812 0.779 0.780 0.870 0.857
Zip-NeRF 0.877 0.835 0.790 0.746 0.718 0.805 0.889 0.915 0.897 0.849 0.912 0.909 0.880 0.814 0.884 0.802 0.807 0.780 0.789 0.875 0.864

LPIPS

Learned Perceptual Image Patch Similarity. The lower the better. The implementation uses AlexNet backbone and matches lpips pip package with checkpoint version 0.1

Method auditorium ballroom courtroom museum palace temple family francis horse lighthouse m60 panther playground train barn caterpillar church courthouse ignatius meetingroom truck
Instant NGP 0.429 0.352 0.448 0.606 0.440 0.424 0.235 0.265 0.225 0.281 0.202 0.208 0.344 0.334 0.271 0.360 0.419 0.414 0.343 0.326 0.216
NerfStudio 0.330 0.261 0.336 0.311 0.452 0.392 0.158 0.190 0.139 0.245 0.179 0.174 0.249 0.261 0.215 0.302 0.338 0.465 0.251 0.261 0.167
Gaussian Opacity Fields 0.194 0.107 0.168 0.152 0.443 0.234 0.084 0.158 0.092 0.181 0.104 0.102 0.142 0.164 0.140 0.187 0.208 0.346 0.162 0.140 0.099
Gaussian Splatting 0.193 0.101 0.165 0.160 0.350 0.222 0.095 0.169 0.103 0.156 0.113 0.107 0.170 0.171 0.160 0.190 0.177 0.266 0.153 0.141 0.108
Mip-Splatting 0.196 0.098 0.165 0.158 0.354 0.226 0.095 0.172 0.104 0.159 0.112 0.109 0.155 0.172 0.161 0.197 0.176 0.265 0.159 0.137 0.108
Zip-NeRF 0.153 0.113 0.153 0.159 0.317 0.183 0.067 0.106 0.069 0.131 0.076 0.081 0.095 0.119 0.083 0.152 0.153 0.218 0.127 0.121 0.081