Skip to content

CalibRCNN: Calibrating Camera and LiDAR by recurrent convolutional neural network and geometric constraints

Research output: Chapter in Book/Report/Conference proceedingConference contribution

In this paper, we present Calibration Recurrent Convolutional Neural Network (CalibRCNN) to infer a 6 degrees of freedom (DOF) rigid body transformation between 3D LiDAR and 2D camera. Different from the existing methods, our 3D-2D CalibRCNN not only uses the LSTM network to extract the temporal features between 3D point clouds and RGB images of consecutive frames, but also uses the geometric loss and photometric loss obtained by the interframe constraint to refine the calibration accuracy of the predicted transformation parameters. The CalibRCNN aims at inferring the correspondence between projected depth image and RGB image to learn the underlying geometry of 2D-3D calibration. Thus, the proposed calibration model achieves a good generalization ability to adapt to unknown initial calibration error ranges, and other 3D LiDAR and 2D camera pairs with different intrinsic parameters from the training dataset. Extensive experiments have demonstrated that our CalibRCNN can achieve state-of-the-art accuracy by comparison with other CNN based methods.

Original languageEnglish
Title of host publication2020 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2020
Number of pages6
ISBN (Electronic)9781728162126
ISBN (Print)9781728162133
Publication statusPublished - 10 Feb 2021
Event2020 IEEE/RSJ International Conference on Intelligent Robots and Systems - Las Vegas, United States
Duration: 24 Oct 202024 Jan 2021

Publication series

NameIEEE IROS Proceedings Series
ISSN (Print)2153-0858
ISSN (Electronic)2153-0866


Conference2020 IEEE/RSJ International Conference on Intelligent Robots and Systems
Abbreviated titleIROS 2020
CountryUnited States
CityLas Vegas


  • CalibRCNN post-print

    Rights statement: © 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

    Accepted author manuscript (Post-print), 1.56 MB, PDF document

Related information

Relations Get citation (various referencing formats)

ID: 26873916