Depth video super-resolution using a high-speed time-of-flight sensor

Germán Mora-Martín; Stirling Scholes; Alice Ruget; Robert Henderson; Jonathan Leach; Istvan Gyongy

doi:10.1117/12.2647260

15 March 2023 Depth video super-resolution using a high-speed time-of-flight sensor

Germán Mora-Martín, Stirling Scholes, Alice Ruget, Robert Henderson, Jonathan Leach, Istvan Gyongy

Proceedings Volume 12438, AI and Optical Data Sciences IV; 124380J (2023) https://doi.org/10.1117/12.2647260
Event: SPIE OPTO, 2023, San Francisco, California, United States

Abstract

Three-dimensional (3D) imaging captures depth information from a given scene and is used in a wide range of fields like industrial environments, smartphones and autonomous driving, among others. This paper summarises the results of a depth video super-resolution scheme that is tailored for single-photon avalanche diode (SPAD) image sensors, which produces 3D maps at frame rates > 100 FPS (32×64 pixels). Consecutive frames are used to super-resolve and denoise depth maps via 3D convolutional neural networks with an upscaling factor of 4. Due to the lack of noise-free, high-resolution depth maps captured with high-speed cameras, the neural network is trained with synthetic data using Unreal Engine, which is later processed to resemble the data outputted by a SPAD sensor. The model is then tested with different video sequences captured with a high-speed SPAD dToF, which processes frames at >30 frames per second. The super-resolved data shows a significant reduction in noise and presents enhanced edge details in objects. We believe these results are relevant to improve the accuracy of object detection in autonomous driving cars for collision avoidance or AR/VR systems.

Conference Presentation

Citation Download Citation

Germán Mora-Martín, Stirling Scholes, Alice Ruget, Robert Henderson, Jonathan Leach, and Istvan Gyongy "Depth video super-resolution using a high-speed time-of-flight sensor", Proc. SPIE 12438, AI and Optical Data Sciences IV, 124380J (15 March 2023); https://doi.org/10.1117/12.2647260

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available