基于多模态中间表示的端到端自动驾驶模型*

doi:10.16731/j.cnki.1671-3133.2024.03.010

现代制造工程 ›› 2024, Vol. 522 ›› Issue (3): 70-78.doi: 10.16731/j.cnki.1671-3133.2024.03.010

基于多模态中间表示的端到端自动驾驶模型^*

孔慧芳, 刘润武, 胡杰

合肥工业大学电气与自动化工程学院,合肥 230009

收稿日期:2023-05-09 出版日期:2024-03-18 发布日期:2024-05-31
通讯作者: 刘润武,硕士研究生,主要研究方向为自动驾驶。E-mail:liurunwu@yeah.net
作者简介:孔慧芳,博士,教授,博士生导师,主要研究方向为汽车电子和物联网等领域的控制理论和应用研究等。E-mail:konghuifang@hfut.edu.cn
基金资助:
* 安徽省重点研发计划项目 (202104a05020035)

End-to-end autonomous driving model based on multi-modal intermediate representations

KONG Huifang, LIU Runwu, HU Jie

School of Electrical Engineering and Automation,Hefei University of Technology,Hefei 230009,China

Received:2023-05-09 Online:2024-03-18 Published:2024-05-31

摘要/Abstract

摘要： 对驾驶环境的准确理解是实现自动驾驶的先决条件之一。为提高自动驾驶车辆的场景理解能力,提出了一种基于语义分割、水平视差和角度编码的多模态中间表示的端到端自动驾驶模型。该端到端自动驾驶模型利用深度学习技术构建感知-规划网络。感知网络以RGB和深度图为输入生成多模态中间表示,实现道路环境及周围障碍物的空间分布描述;规划网络使用多模态中间表示进行道路环境特征提取和航路点预测。基于CARLA仿真平台进行模型的训练和性能测试,结果表明:该端到端自动驾驶模型能够实现对城市道路环境的场景理解,有效地减少了碰撞;相较于单模态中间表示的基线模型,其驾驶性能指标提升了31.47 %。

关键词: 自动驾驶, 场景理解, 模仿学习, 轨迹规划

Abstract: An accurate understanding of the driving environment is one of the prerequisites for autonomous driving.In order to improve the scene understanding ability of autonomous driving vehicles,an end-to-end autonomous driving model based on semantic segmentation,horizontal disparity,and angle coding multi-modal intermediate representations was proposed.The end-to-end autonomous driving model used deep learning technology to build perception-planning network.The perception network generated multi-modal intermediate representations with RGB and depth images as inputs to realize the spatial distribution description of road environment and surrounding obstacles.The planning network used multi-modal intermediate representations to extract road environment features and predict waypoints. Model training and performance testing were conducte based on the CARLA simulation platform.The results showed that the end-to-end autonomous driving model can realize the scene understanding of urban road environment and effectively reduce collisions.Compared with the baseline model based on the single modal intermediate representation,its driving performance index is 31.47 % better than the baseline model.

Key words: autonomous driving, scene understanding, imitation learning, trajectory planning

中图分类号:

TP181

孔慧芳, 刘润武, 胡杰. 基于多模态中间表示的端到端自动驾驶模型^*[J]. 现代制造工程, 2024, 522(3): 70-78.

KONG Huifang, LIU Runwu, HU Jie. End-to-end autonomous driving model based on multi-modal intermediate representations[J]. Modern Manufacturing Engineering, 2024, 522(3): 70-78.

参考文献

[1] POMERLEAU D A.ALVINN:an autonomous land vehicle in a neural network[C]//Proceedings of the 1st International Conference on Neural Information Processing Systems.Cambridge,MA,USA:[s.n.],1988:305-313.
[2] BOJARSKI M,DELTESTA D,DWORAKOWSKI D,et al.End to End Learning for Self-Driving Cars[EB/OL].(2016-04-25) [2022-12-20].http://arxiv.org/abs/1604.07316.
[3] DOSOVITSKIY A,ROS G,CODEVILLA F,et al.CARLA:An open urban driving simulator[C]//Conference on robot learning.New York,USA:PMLR,2017:1-16.
[4] CODEVILLA F,MÜLLER M,LÓPEZ A,et al.End-to-end driving via conditional imitation learning[C]//2018 IEEE international conference on robotics and automation (ICRA).Brisbane,QLD,Australia:IEEE,2018:4693-4700.
[5] LI Z,MOTOYOSHI T,SASAKI K,et al.Rethinking self-driving:Multi-task knowledge for better generalization and accident explanation ability[EB/OL].(2018-09-28) [2022-12-20].https://arxiv.org/abs/1809.11100.
[6] HU J,KONG H,ZHANG Q,et al.Enhancing scene understanding based on deep learning for end-to-end autonomous driving[J].Engineering Applications of Artificial Intellig-ence,2022,116:105474.
[7] CODEVILLA F,SANTANA E,LOPEZ A,et al.Exploring the Limitations of Behavior Cloning for Autonomous Driving[C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV).Seoul,Korea (South):IEEE,2019:9328-9337.
[8] HUANG Z,LV C,XING Y,et al.Multi-modal sensor fusion-based deep neural network for end-to-end autonomous driving with scene understanding[J].IEEE Sensors Journal,2020,21(10):11781-11790.
[9] WEI B,REN M,ZENG W,et al.Perceive,attend,and drive:Learning spatial attention for safe self-driving[C]//2021 IEEE International Conference on Robotics and Automation (ICRA).Xi′an,China:IEEE,2021:4875-4881.
[10] ZHAO A,HE T,LIANG Y,et al.Sam:Squeeze-and-mimic networks for conditional visual driving policy learning[C]//Conference on Robot Learning.New York,USA:PMLR,2021:156-175.
[11] HUANG J,XIE S,SUN J,et al.Learning a decision module by imitating driver’s control behaviors[C]//Conference on Robot Learning.[S.l.]:PMLR,2021:1-10.
[12] BEHL A,CHITTA K,PRAKASH A,et al.Label efficient visual abstractions for autonomous driving[C]//2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).Las Vegas,USA:IEEE,2020:2338-2345.
[13] MUELLER M,DOSOVITSKIY A,GHANEM B,et al.Driving Policy Transfer via Modularity and Abstraction[C]//Conference on Robot Learning.New York,USA:PMLR,2018:1-15.
[14] SAX A,ZHANG J O,EMI B,et al.Learning to Navigate Using Mid-Level Visual Priors[C]//Conference on Robot Learning.New York,USA:PMLR,2020:791-812.
[15] SHELHAMER E,LONG J,DARRELL T.Fully Convolutional Networks for Semantic Segmentation[J].IEEE Transactions on Pattern Analysis & Machine Intelligence,2017,39(4):640-651.
[16] KONG H,HU J,FAN L,et al.Encoder-decoder with double spatial pyramid for semantic segmentation[J].Journal of Electronic Imaging,2019,28(6):063007(1)-063007(10).
[17] HU J,KONG H,FAN L,et al.Enhancing feature fusion with spatial aggregation and channel fusion for semantic segmentation[J].IET Computer Vision,2021,15(6):418-427.
[18] DENG J,DONG W,SOCHER R,et al.ImageNet:A large-scale hierarchical image database[C]//2009 IEEE Conference on Computer Vision and Pattern Recognition.Miami,USA:IEEE Computer Society,2009:248-255.
[19] HE K,ZHANG X,REN S,et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE conference on computer vision and pattern recognition.Las Vegas,USA:[s.n.],2016:770-778.
[20] GUPTA S,GIRSHICK R,ARBELÁEZ P,et al.Learning rich features from RGB-D images for object detection and segmentation[C]//Computer Vision-ECCV 2014:13th European Conference,Zurich,Switzerland,Proceedings,Part VII 13.[S.l.]: Springer International Publishing,2014:345-360.
[21] FAN R,WANG H,CAI P,et al.Sne-roadseg:Incorporating surface normal information into semantic segmentation for accurate freespace detection[C]//European Conference on Computer Vision.Cham:Springer International Publishing,2020:340-356.
[22] 李鑫.城市结构化道路下智能汽车的轨迹规划与控制研究[D].长春:吉林大学,2022.
[23] CHEN Y,WANG J,LI J,et al.LiDAR-Video Driving Dataset:Learning Driving Policies Effectively[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Salt Lake City,USA:IEEE Computer Society,2018:5870-5878.
[24] Carla autonomous driving leaderboard (accessed november 2021) [EB/OL]. (2021-12-08)(2022-12-20). https://leaderboardcarla.org/leaderboard/,2021.
[25] PRAKASH A,CHITTA K,GEIGER A.Multi-Modal Fusion Transformer for End-to-End Autonomous Driving[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Nashville,USA:IEEE Computer Society,2021:7073-7083.
[26] CHEN D,ZHOU B,KOLTUN V,et al.Learning by cheating[C]//Conference on Robot Learning.New York,USA:PMLR,2020:66-75.
[27] HINTON G,VINYALS O,DEAN J. Distilling the Knowledge in a Neural Network[EB/OL].(2015-03-09) [2022-12-20].https://arxiv.org/abs/1503.02531.

基于多模态中间表示的端到端自动驾驶模型^*

End-to-end autonomous driving model based on multi-modal intermediate representations

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 13

编辑推荐

Metrics

[1]	马国庆, 刘启畅, 刘珺玮, 曲合拉·金恩斯. 基于点云的航发叶片分割及打磨轨迹规划方法^*[J]. 现代制造工程, 2024, 523(4): 102-108.
[2]	张学聪, 晁永生, 李纯艳, 周江林. 基于机器学习的工业机器人多目标轨迹规划^*[J]. 现代制造工程, 2024, 521(2): 31-37.
[3]	陈广玲, 张天赐, 付涛, 王林涛, 宋学官. 基于NSGA-Ⅱ的智能化电铲多目标最优挖掘轨迹规划[J]. 现代制造工程, 2024, 521(2): 142-149.
[4]	金宇杰, 龚堰珏, 赵罘. 基于模拟退火量子遗传算法的焊接机器人轨迹规划^*[J]. 现代制造工程, 2024, 520(1): 33-38.
[5]	何成刚;朱润智;黄挺博;汪晓鑫;黄尊地;刘吉华. 工业机器人轨迹规划与优化研究进展[J]. 现代制造工程, 2023, 514(7): 150-159.
[6]	刘阳，徐巧玉，陈翰文，张正. 凿岩机械臂轨迹规划及电液跟踪控制研究[J]. 现代制造工程, 2023, 519(12): 61-67.
[7]	韩军，庞楠楠，沈伟东，董冰洋，邵帅. 工业机器人大规格叶轮零件加工技术研究[J]. 现代制造工程, 2023, 519(12): 84-90.
[8]	黄龙,徐武彬,杨宇恒,李冰. 挖掘机多目标关节空间最优轨迹规划[J]. 现代制造工程, 2023, 508(1): 83-89.
[9]	李东阳，谢海龙，王清辉，廖昭洋，. 基于力反馈的机器人砂带磨削虚拟示教轨迹规划方法[J]. 现代制造工程, 2022, 500(5): 30-35.
[10]	王秋爽,钟羽中,郭斌,佃松宜. 基于改进OMP算法的喷涂机器人轨迹规划研究[J]. 现代制造工程, 2021, 495(12): 28-34.
[11]	曹培,杨涛,王天琪,张保家. 面向飞机蒙皮的机器人喷涂轨迹规划[J]. 现代制造工程, 2020, 481(10): 72-77.
[12]	樊旭, 杨冬, 张改萍, 李贺立, 李铁军. 一种新型板材安装机器人的空间直线轨迹规划[J]. 现代制造工程, 2017, 441(6): 62-68.
[13]	陈少克, 樊璇, 李汉阳. 基于CAD/CAM技术的五轴焊接机器人轨迹规划[J]. 现代制造工程, 2017, 446(11): 50-53.