基于改进TD3的四足机器人非结构化地形运动控制*

doi:10.16731/j.cnki.1671-3133.2025.01.004

现代制造工程 ›› 2025, Vol. 532 ›› Issue (1): 33-41.doi: 10.16731/j.cnki.1671-3133.2025.01.004

基于改进TD3的四足机器人非结构化地形运动控制^*

谢子健^1,2, 秦建军^1,2, 曹钰^1,2

1 北京建筑大学机电与车辆工程学院,北京 100044;
2 北京市建筑安全监测工程技术研究中心, 北京 100044

收稿日期:2024-02-06 出版日期:2025-01-18 发布日期:2025-02-10
通讯作者: 秦建军,博士,教授,主要研究方向为机器人学、智能设计方法、工程教育。E-mail:13639777766@163.com;qinjianjun@bucea.edu.cn
作者简介:谢子健,硕士研究生,主要研究方向为机器人控制、强化学习。
基金资助:
* 北京市属高校基本科研业务费专项资金项目(X20060);北京市建筑安全监测工程技术研究中心研究基金项目(BJC2020K012);北京建筑大学研究生创新项目(PG2024139);载运工具先进制造与测试技术教育部重点实验室(北京交通大学)开放课题基金项目

Unstructured terrain motion control for quadruped robot based on improved TD3

XIE Zijian^1,2, QIN Jianjun^1,2, CAO Yu^1,2

1 School of Mechanical-Electronic and Vehicle Engineering,Beijing University of Civil Engineering and Architecture,Beijing 100044,China;
2 Beijing Engineering Research Center of Monitoring for Construction Safety,Beijing 100044,China

Received:2024-02-06 Online:2025-01-18 Published:2025-02-10

摘要/Abstract

摘要： 四足机器人在非结构化地形的运动控制高度依赖于复杂的动力学模型和控制器设计,利用深度强化学习方法设计四足机器人控制器已成为趋势。针对在深度强化学习训练过程中收敛较慢、容易陷入局部最优解及计算资源消耗较大等问题,提出一种融合记忆组件的双延迟深度确定性策略梯度(Memory-integrated Twin Delayed Deep Deterministic policy gradient,M-TD3)算法。首先,对四足机器人以及非结构化地形建模;其次,分析M-TD3算法收敛状态与学习效率;最后,为验证控制器性能,针对多种地形进行运动控制仿真对比并制作样机进行测试。仿真结果表明,相较于传统TD3算法,M-TD3算法收敛更快,效率更高,运动控制性能有显著改善,样机测试结果证明基于改进TD3算法所设计的控制器能够让四足机器人在非结构化地形进行有效的运动越障。

关键词: 四足机器人, 非结构化地形, 深度强化学习, TD3算法

Abstract: The motion control of quadruped robots in unstructured terrains heavily relies on complex dynamics models and controller designs. Utilizing Deep Reinforcement Learning (DRL) methods to design controllers for quadruped robots has become a trend. To address issues such as slow convergence, susceptibility to local optima, and high computational resource consumption during the training process of DRL, an improved algorithm called Memory-integrated Twin Delayed Deep Deterministic policy gradient (M-TD3) was proposed. Firstly, models for both the quadruped robot and unstructured terrains were established. Then, the convergence state and learning efficiency of M-TD3 algorithm were analyzed. Finally, motion control simulation comparisons for various terrains were carried out, and production of prototype was tested to verify controller performance. Simulation results show that compared with traditional TD3 algorithm, M-TD3 algorithm has faster convergence, higher efficiency, and significantly improves motion control performance. Prototype test results show that the controller designed based on the improved TD3 algorithm can make quadruped robot effectively move over obstacles in unstructured terrain.

Key words: quadruped robot, unstructured terrains, Deep Reinforcement Learning(DRL), TD3 algorithm

中图分类号:

TP242.6

谢子健, 秦建军, 曹钰. 基于改进TD3的四足机器人非结构化地形运动控制^*[J]. 现代制造工程, 2025, 532(1): 33-41.

XIE Zijian, QIN Jianjun, CAO Yu. Unstructured terrain motion control for quadruped robot based on improved TD3[J]. Modern Manufacturing Engineering, 2025, 532(1): 33-41.

参考文献

[1] SEHOON H .Quadrupedal robots trot into the wild[J].Science Robotics,2020,5(47):eabe5218.
[2] 谢惠祥. 四足机器人对角小跑步态虚拟模型直觉控制方法研究[D].长沙:国防科技大学,2015.
[3] CHAI H, LI Y, SONG R, et al. A survey of the development of quadruped robots: joint configuration, dynamic locomotion control method and mobile manipulation approach[J].Biomimetic Intelligence and Robotics,2022,2(1):100029.
[4] CARLO J D, WENSING P M, KATZ B, et al. Dynamic locomotion in the MIT cheetah 3 through convex model-predictive control[C]//2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Madrid:IEEE, 2018.DOI:10.1109/IROS.2018.8594448.
[5] ZHAO J, MA S, NIU S, et al. Fractional-order virtual model control for trotting motion of quadruped robot[C]//2020 Chinese Control and Decision Conference (CCDC).Hefei:[s.n.],2020.DOI:10.1109/CCDC49329.2020.9164655.
[6] ZHAO W, QUERALTA J P, WESTERLUND T. Sim-to-real transfer in deep reinforcement learning for robotics: a survey[C]//2020 IEEE Symposium Series on Computational Intelligence (SSCI). Canberra:[s.n.],2020.DOI:10.1109/SSCI47803.2020.9308468.
[7] 多南讯,吕强,林辉灿,等.迈进高维连续空间:深度强化学习在机器人领域中的应用[J].机器人,2019,41(2):276-288.
[8] BELLEGARDA G , NGUYEN Q .Robust quadruped jumping via deep reinforcement learning[EB/OL]. (2020-11-13)[2023-11-06]. http://researchgate.net/pubtication/345971287. DOI:10.48550/arXiv.2011.07089.
[9] JI Q, FU S, TAN K, et al. Synthesizing the optimal gait of a quadruped robot with soft actuators using deep reinforcement learning[J]. Robotics and Computer-Integrated Manufacturing,2022,78:102382.
[10] ZHU K, ZHANG T. Deep reinforcement learning based mobile robot navigation: a review[J]. Tsinghua Science and Technology, 2021,26(5):674-691.
[11] RUDIN N, HOELLER D,REIST P,et al. Learning to walk in minutes using massively parallel deep reinforcement learning[C]//5th Annual Conference on Robot Learning.[S.l.]:[s.n.],2022:91-100.
[12] LI Y, HAO X, SHE Y, et al. Constrained motion planning of free-float dual-arm space manipulator via deep reinforcement learning[J]. Aerospace Science and Technology,2021,109:106446.
[13] LEE J, HWANGBO J, WELLHAUSEN L, et al. Learning quadrupedal locomotion over challenging terrain[J]. Science Robotics,2020,5(47):5986.
[14] MOCK J W, MUKNAHALLIPATNA S S. A comparison of ppo, TD3 and sac reinforcement algorithms for quadruped walking gait generation[J]. Journal of Intelligent Learning Systems and Applications, 2023,15(1):36-56.
[15] 陈恺丰,田博睿,李和清,等.基于DDPG算法的双轮腿机器人运动控制研究[J].系统工程与电子技术,2023,45(4):1144-1151.

基于改进TD3的四足机器人非结构化地形运动控制^*

Unstructured terrain motion control for quadruped robot based on improved TD3

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 6

编辑推荐

Metrics

[1]	杨逢海, 杨晓英, 裴志杰, 武亚琪, 张志伟. 基于深度强化学习的风电拉挤板生产智能排程^*[J]. 现代制造工程, 2025, 532(1): 23-32.
[2]	李伟豪;茅健;郑武;张婷. 四足机器人腿部结构拓扑优化设计及力学性能分析[J]. 现代制造工程, 2023, 516(9): 45-53.
[3]	陶鑫钰;王艳;纪志成. 基于A3C的特征重构工艺路线规划方法[J]. 现代制造工程, 2023, 517(10): 15-26.
[4]	王银浩，颉潭成，徐彦伟，桂林，. 四足机器人侧摆型trot步态的运动学分析及仿真研究[J]. 现代制造工程, 2022, 500(5): 24-29.
[5]	钟敬伟，石宇强. 基于DQN的智能工厂作业车间调度[J]. 现代制造工程, 2021, 492(9): 17-23.
[6]	高彤, 袁立鹏, 宫赤坤, 勾文浩. 基于刚柔耦合的四足机器人动力性能研究[J]. 现代制造工程, 2017, 441(6): 52-55.