基于改进SAC算法的机械臂运动规划

doi:10.16180/j.cnki.issn1007-7820.2024.11.007

摘要/Abstract

摘要：

针对深度强化学习算法在高维状态空间和高精度需求下的机械臂运动规划任务中存在探索效率低、收敛速度慢以及不收敛等问题,文中以SAC(Soft Actor-Critic)算法为基础,引入异步优势机制,提出了一种融合异步优势的AA-SAC(Asynchronous Advantage Soft Actor-Critic)算法。该算法使用Q_target网络代替了原V网络,有效降低了Q网络的方差,n个独立的进程可并行训练,提升了训练效率。将AA-SAC算法的经验回放池划分成两个部分,将高质量的经验数据单独存放、单独采样,以提高有效经验数据的利用率。仿真结果表明,AA-SAC算法在收敛速度、成功率和稳定性上表现最优。相较于SAC算法,AA-SAC算法的收敛时间提前了3 000回合。收敛后AA-SAC算法的成功率达到了96%,比SAC算法提升了6%,比DDPG(Deep Deterministic Policy Gradient)算法提升了26%。

关键词: 深度强化学习, 异步优势, SAC算法, 经验回放池, 机械臂, 运动规划, 微创手术, CoppeliaSim

Abstract:

In view of the problems such as low exploration efficiency, slow convergence speed or even non-convergence of deep reinforcement learning algorithm in the motion planning task of robot arm under the requirement of high dimensional state space and high precision, this study introduces asynchronous advantage mechanism based on SAC(Soft Actor-Critic) algorithm, and proposes an AA-SAC(Asynchronous Advantage Soft Actor-Critic) algorithm integrating asynchronous advantage. This algorithm replaces the original V network with a Q_target network,which effectively reduces the variance of the Q network. The n independent processes can be trained in parallel, which improves the training efficiency. The study also divides the experience playback pool of the AA-SAC algorithm into two parts, store and sample high-quality empirical data separately to improve the utilization of effective empirical data. The simulation results show that AA-SAC algorithm has the best performance in convergence speed, success rate and stability. Compared with the SAC algorithm, the convergence time of AA-SAC algorithm is 3 000 rounds earlier. After convergence, the success rate of AA-SAC algorithm reaches 96%, which is 6% higher than SAC algorithm and 26% higher than DDPG(Deep Deterministic Policy Gradient) algorithm.

Key words: deep reinforcement learning, asynchronous advantage, SAC algorithm, experience playback pool, mechanical arm, motion planning, minimally invasive surgery, CoppeliaSim

中图分类号:

TP241

唐超, 张帆. 基于改进SAC算法的机械臂运动规划[J]. 电子科技, 2024, 37(11): 47-54.

TANG Chao, ZHANG Fan. Motion Planning of Manipulator Based on Improved SoftActor-Critic Algorithm[J]. Electronic Science and Technology, 2024, 37(11): 47-54.

图/表 12

图1

图2

表1

图3

表2

图4

图5

图6

图7

表3

图8

图9

参考文献 21

[1]	Mitchell T M. Machine learning[M]. New York: McGraw-hill, 1997:4-16.
[2]	Winston P H. Artificial intelligence[M]. London: Addison-Wesley Longman Publishing Company, 1984: 12-18.
[3]	Murphy K P. Machine learning:A probabilistic perspective[M]. Cambridge: MIT Press, 2012:62-63.
[4]	Neunert M, Abdolmaleki A, Wulfmeier M, et al. Continuous-discrete reinforcement learning for hybrid control in robotics[C]. Osaka:Conference on Robot Learning, 2020:735-751.
[5]	Xie Z, Berseth G, Clary P, et al. Feedback control for cassie with deep reinforcement learning[C]. Madrid:International Conference on Intelligent Robots and Systems, 2018:1241-1246.
[6]	张佳鹏, 李琳, 朱叶. 基于强化学习的无人驾驶车辆行为决策方法研究进展[J]. 电子科技, 2021, 34(5):66-71.
	Zhang Jiapeng, Li Lin, Zhu Ye. A review of research on decision-making method of autonomous vehicle based on reinforcement learning[J]. Electronic Science and Technology, 2021, 34(5):66-71.
[7]	Vinyals O, Babuschkin I, Czarnecki W M, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning[J]. Nature, 2019, 575(7782):350-354.
[8]	Haarnoja T, Zhou A, Abbeel P, et al. Soft actor-critic:Off-policy maximum entropy deep reinforcement learning with a stochastic actor[C]. Stockholm:International Conference on Machine Learning, 2018:1861-1870.
[9]	王骏超. 基于SAC算法的机械臂控制策略的研究[D]. 南昌: 华东交通大学, 2020:33-42.
	Wang Junchao. Research on manipulator control strategy based on SAC algorithm[D]. Nanchang: East China Jiaotong University, 2020:33-42.
[10]	Wong C C, Chien S Y, Feng H M, et al. Motion planning for dual-arm robot based on soft actor-critic[J]. IEEE Access, 2021, 9(2):26871-26885.
[11]	De Jesus J C, Kich V A, Kolling A H, et al. Soft actor-critic for navigation of mobile robots[J]. Journal of Intelligent & Robotic Systems, 2021, 102(2):31-36.
[12]	Banerjee C, Chen Z, Noman N. Improved soft actor-critic:Mixing prioritized off-policy samples with on-policy experiences[J]. IEEE Transactions on Neural Networks and Learning Systems, 2022, 65(9):1-9.
[13]	Chen Y, Ying F, Li X, et al. Deep reinforcement learning in maximum entropy framework with automatic adjustment of mixed temperature parameters for path planning[C]. Taizhou:The Seventh International Conference on Robotics,Control and Automation, 2023:78-82.
[14]	肖硕, 黄珍珍, 张国鹏, 等. 基于SAC的多智能体深度强化学习算法[J]. 电子学报, 2021, 49(9):1675-1681. doi: 10.12263/DZXB.20200243
	Xiao Shuo, Huang Zhenzhen, Zhang Guopeng, et al. Deep reinforcement learning algorithm of multi-agent based on SAC[J]. Acta Electronica Sinica, 2021, 49(9):1675-1681. doi: 10.12263/DZXB.20200243
[15]	范静宇. 基于熵的深度强化学习优化算法[D]. 苏州: 苏州大学, 2021:7-15.
	Fan Jingyu. Optimized algorithm of deep reinforcement learning based on entropy[D]. Suzhou: Soochow University, 2021:7-15.
[16]	Konda V, Tsitsiklis J. Actor-critic algorithms[J]. Advances in Neural Information Processing Systems, 1999, 12(5):1008-1014.
[17]	Mnih V, Badia A P, Mirza M, et al. Asynchronous methods for deep reinforcement learning[C]. New York: International Conference on Machine Learning, 2016:1928-1937.
[18]	Prianto E, Kim M S, Park J H, et al. Path planning for multi-arm manipulators using deep reinforcement learning:Soft actor-critic with hindsight experience replay[J]. Sensors, 2020, 20(20):5911-5933.
[19]	Kim S K, Shin W H, Ko S Y, et al. Design of a compact 5-DOF surgical robot of a spherical mechanism: CURES[C]. Xi'an:IEEE/ASME International Conference on Advanced Intelligent Mechatronics, 2008:990-995.
[20]	马保平. 微创手术机器人机构设计与人机交互研究[D]. 上海: 上海工程技术大学, 2020:11-21.
	Ma Baoping. Mechanism design and human-robot interaction of endoscopic surgery robot[D]. Shanghai: Shanghai University of Engineering Science, 2020:11-21.
[21]	Farley A, Wang J, Marshall J A. How to pick a mobile robot simulator:A quantitative comparison of CoppeliaSim,Gazebo,MORSE and Webots with a focus on accuracy of motion[J]. Simulation Modelling Practice and Theory, 2022, 120(7):1026-1058.

神经网络	输入	输出	名称
第1层	15	300	全连接层
第1层	300	300	ReLU层
第2层	300	300	全连接层
第2层	300	300	ReLU层
第3层	300	3	策略均值
第3层	300	3	策略标准差
第4层	6	3	高斯分布
第5层	3	3	tanh层

神经网络	输入	输出	名称
第1层	18	300	全连接层
第1层	300	300	ReLU层
第2层	300	300	全连接层
第2层	300	300	ReLU层
第3层	300	1	全连接层

超参数	数值
学习率	0.001
熵的重要性权重α	0.200
软更新系数τ	0.005
折扣率γ	0.990
经验回放池总容量	10⁶
特殊经验回放池容量	10⁴
采样batch	256
训练回合数episode	5 000
每回合的最大步数	100