CalibRCNN: Calibrating Camera and LiDAR by recurrent convolutional neural network and geometric constraints

Jieying Shi; Ziheng Zhu; Jianhua Zhang; Ruyu Liu; Zhenhua Wang; Shengyong Chen; Honghai Liu

doi:10.1109/IROS45743.2020.9341147

CalibRCNN: Calibrating Camera and LiDAR by recurrent convolutional neural network and geometric constraints

Jieying Shi
, Ziheng Zhu
, Jianhua Zhang
, Ruyu Liu
, Zhenhua Wang
, Shengyong Chen
, Honghai Liu

University of Portsmouth

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

2045 Downloads (Pure)

Abstract

In this paper, we present Calibration Recurrent Convolutional Neural Network (CalibRCNN) to infer a 6 degrees of freedom (DOF) rigid body transformation between 3D LiDAR and 2D camera. Different from the existing methods, our 3D-2D CalibRCNN not only uses the LSTM network to extract the temporal features between 3D point clouds and RGB images of consecutive frames, but also uses the geometric loss and photometric loss obtained by the interframe constraint to refine the calibration accuracy of the predicted transformation parameters. The CalibRCNN aims at inferring the correspondence between projected depth image and RGB image to learn the underlying geometry of 2D-3D calibration. Thus, the proposed calibration model achieves a good generalization ability to adapt to unknown initial calibration error ranges, and other 3D LiDAR and 2D camera pairs with different intrinsic parameters from the training dataset. Extensive experiments have demonstrated that our CalibRCNN can achieve state-of-the-art accuracy by comparison with other CNN based methods.

Original language	English
Title of host publication	2020 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2020
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	10197-10202
Number of pages	6
ISBN (Electronic)	9781728162126
ISBN (Print)	9781728162133
DOIs	https://doi.org/10.1109/IROS45743.2020.9341147
Publication status	Published - 10 Feb 2021
Event	2020 IEEE/RSJ International Conference on Intelligent Robots and Systems - Las Vegas, United States Duration: 24 Oct 2020 → 24 Jan 2021

Publication series

Name	IEEE IROS Proceedings Series
Publisher	IEEE
ISSN (Print)	2153-0858
ISSN (Electronic)	2153-0866

Conference

Conference	2020 IEEE/RSJ International Conference on Intelligent Robots and Systems
Abbreviated title	IROS 2020
Country/Territory	United States
City	Las Vegas
Period	24/10/20 → 24/01/21

Access to Document

10.1109/IROS45743.2020.9341147

CalibRCNN post-print
© 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Accepted author manuscript (Post-print), 1.56 MB

Cite this

Shi, J., Zhu, Z., Zhang, J., Liu, R., Wang, Z., Chen, S., & Liu, H. (2021). CalibRCNN: Calibrating Camera and LiDAR by recurrent convolutional neural network and geometric constraints. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2020 (pp. 10197-10202). (IEEE IROS Proceedings Series). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IROS45743.2020.9341147

@inproceedings{a901bae38f6e49d389e21c503f95db11,

title = "CalibRCNN: Calibrating Camera and LiDAR by recurrent convolutional neural network and geometric constraints",

abstract = "In this paper, we present Calibration Recurrent Convolutional Neural Network (CalibRCNN) to infer a 6 degrees of freedom (DOF) rigid body transformation between 3D LiDAR and 2D camera. Different from the existing methods, our 3D-2D CalibRCNN not only uses the LSTM network to extract the temporal features between 3D point clouds and RGB images of consecutive frames, but also uses the geometric loss and photometric loss obtained by the interframe constraint to refine the calibration accuracy of the predicted transformation parameters. The CalibRCNN aims at inferring the correspondence between projected depth image and RGB image to learn the underlying geometry of 2D-3D calibration. Thus, the proposed calibration model achieves a good generalization ability to adapt to unknown initial calibration error ranges, and other 3D LiDAR and 2D camera pairs with different intrinsic parameters from the training dataset. Extensive experiments have demonstrated that our CalibRCNN can achieve state-of-the-art accuracy by comparison with other CNN based methods.",

author = "Jieying Shi and Ziheng Zhu and Jianhua Zhang and Ruyu Liu and Zhenhua Wang and Shengyong Chen and Honghai Liu",

note = "Funding Information: This work was partially supported by National Key R\&D Program of China (2018YFB1305200). This publication was partially funded by the National Natural Science Foundation of China (61876167 and 61802348) and the Natural Science Foundation of Zhejiang Province (LY20F030017). Publisher Copyright: {\textcopyright} 2020 IEEE. Copyright: Copyright 2021 Elsevier B.V., All rights reserved.; 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2020 ; Conference date: 24-10-2020 Through 24-01-2021",

year = "2021",

month = feb,

day = "10",

doi = "10.1109/IROS45743.2020.9341147",

language = "English",

isbn = "9781728162133",

series = "IEEE IROS Proceedings Series",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "10197--10202",

booktitle = "2020 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2020",

address = "United States",

}

Shi, J, Zhu, Z, Zhang, J, Liu, R, Wang, Z, Chen, S & Liu, H 2021, CalibRCNN: Calibrating Camera and LiDAR by recurrent convolutional neural network and geometric constraints. in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2020. IEEE IROS Proceedings Series, Institute of Electrical and Electronics Engineers Inc., pp. 10197-10202, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems, Las Vegas, Nevada, United States, 24/10/20. https://doi.org/10.1109/IROS45743.2020.9341147

CalibRCNN: Calibrating Camera and LiDAR by recurrent convolutional neural network and geometric constraints. / Shi, Jieying; Zhu, Ziheng; Zhang, Jianhua et al.
2020 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2020. Institute of Electrical and Electronics Engineers Inc., 2021. p. 10197-10202 (IEEE IROS Proceedings Series).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - CalibRCNN

T2 - 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems

AU - Shi, Jieying

AU - Zhu, Ziheng

AU - Zhang, Jianhua

AU - Liu, Ruyu

AU - Wang, Zhenhua

AU - Chen, Shengyong

AU - Liu, Honghai

N1 - Funding Information: This work was partially supported by National Key R&D Program of China (2018YFB1305200). This publication was partially funded by the National Natural Science Foundation of China (61876167 and 61802348) and the Natural Science Foundation of Zhejiang Province (LY20F030017). Publisher Copyright: © 2020 IEEE. Copyright: Copyright 2021 Elsevier B.V., All rights reserved.

PY - 2021/2/10

Y1 - 2021/2/10

N2 - In this paper, we present Calibration Recurrent Convolutional Neural Network (CalibRCNN) to infer a 6 degrees of freedom (DOF) rigid body transformation between 3D LiDAR and 2D camera. Different from the existing methods, our 3D-2D CalibRCNN not only uses the LSTM network to extract the temporal features between 3D point clouds and RGB images of consecutive frames, but also uses the geometric loss and photometric loss obtained by the interframe constraint to refine the calibration accuracy of the predicted transformation parameters. The CalibRCNN aims at inferring the correspondence between projected depth image and RGB image to learn the underlying geometry of 2D-3D calibration. Thus, the proposed calibration model achieves a good generalization ability to adapt to unknown initial calibration error ranges, and other 3D LiDAR and 2D camera pairs with different intrinsic parameters from the training dataset. Extensive experiments have demonstrated that our CalibRCNN can achieve state-of-the-art accuracy by comparison with other CNN based methods.

AB - In this paper, we present Calibration Recurrent Convolutional Neural Network (CalibRCNN) to infer a 6 degrees of freedom (DOF) rigid body transformation between 3D LiDAR and 2D camera. Different from the existing methods, our 3D-2D CalibRCNN not only uses the LSTM network to extract the temporal features between 3D point clouds and RGB images of consecutive frames, but also uses the geometric loss and photometric loss obtained by the interframe constraint to refine the calibration accuracy of the predicted transformation parameters. The CalibRCNN aims at inferring the correspondence between projected depth image and RGB image to learn the underlying geometry of 2D-3D calibration. Thus, the proposed calibration model achieves a good generalization ability to adapt to unknown initial calibration error ranges, and other 3D LiDAR and 2D camera pairs with different intrinsic parameters from the training dataset. Extensive experiments have demonstrated that our CalibRCNN can achieve state-of-the-art accuracy by comparison with other CNN based methods.

UR - https://www.scopus.com/pages/publications/85102413165

U2 - 10.1109/IROS45743.2020.9341147

DO - 10.1109/IROS45743.2020.9341147

M3 - Conference contribution

AN - SCOPUS:85102413165

SN - 9781728162133

T3 - IEEE IROS Proceedings Series

SP - 10197

EP - 10202

BT - 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2020

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 24 October 2020 through 24 January 2021

ER -

Shi J, Zhu Z, Zhang J, Liu R, Wang Z, Chen S et al. CalibRCNN: Calibrating Camera and LiDAR by recurrent convolutional neural network and geometric constraints. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2020. Institute of Electrical and Electronics Engineers Inc. 2021. p. 10197-10202. (IEEE IROS Proceedings Series). doi: 10.1109/IROS45743.2020.9341147

CalibRCNN: Calibrating Camera and LiDAR by recurrent convolutional neural network and geometric constraints

Abstract

Publication series

Conference

Access to Document

Fingerprint

Cite this