studies just apply simple distributed RL into a multi-vehicle environment, in which it
will lead to inefficient and uncoordinated driving decisions [6].
This paper aims to solve the coordination lane change problem by using multiagent RL (MARL) technology. Since the main strategies of autonomous driving are car
following and lane changing for overtaking [18], we pay more attention to the study of
lane changing and car-follow decision on the road. We use the coordination graph
(CG) to achieve the cooperative decision making for multi-vehicle. However, due to
the motion characteristic of the vehicles continuously change topology, the direct
application of CG to autonomous driving is a thorny issue. In order to solve this
problem, we propose a MARL approach based on the combination of dynamic graph
and RL, which use driving risk evaluation and graph theory to model the interaction
between vehicles, then use the consistency of latent conflicts relations to achieve
effective distributed coordination learning. The experiment verifies that the proposed
approach allows the vehicles to achieve higher driving safety under cooperative
decision making. Compared with other approaches that have no coordination or rely on
some mobile models and expert driving rules, the proposed approach is more
advantageous.
2 Related Work
RL has been widely used in the field of autonomous driving. Some researchers have
used RL to learn driving strategies. For example, Wang et al. [20] use deep reinforcement learning with rule-based constraints to learn driving strategies. Other work
uses Deep RL to make driving decisions directly from real-world scenarios, such as
virtual to real reinforcement learning for autonomous driving [14]. All of these studies
only focus on the driving strategies of a single vehicle, without considering the
interactions and coordination between multiple vehicles.
A large number of scholars have studied lane changing before, such as Moriarty
and Langley [11] used RL to solve the lane selection problem of multiple vehicles.
Although these methods have good results in their respective applications, they either
focus on the relatively simple traffic environment or do not simulate the interaction
between multiple vehicles. Some studies [12] have proposed methods to simulate
vehicle interaction and formation using graph theory, but these methods all focus on
formation and communication. Recently, automobile safety accidents happen frequently, so our work is more dedicated to driving safety issues. We realize distributed
learning in multi-vehicle through dynamic CG and RL, which can provide more safe
and accurate lane-changing decision for autonomous driving.
3 Related Theories and Approaches
Traditional coordination graph can only model closed, static problems, which does not
change during the solution process. In order to realize coordinated learning in dynamic
environments, we propose an approach to construct dynamic coordination graph and a
MARL approach to solve it.
600
H. Zuo et al.
will lead to inefficient and uncoordinated driving decisions [6].
This paper aims to solve the coordination lane change problem by using multiagent RL (MARL) technology. Since the main strategies of autonomous driving are car
following and lane changing for overtaking [18], we pay more attention to the study of
lane changing and car-follow decision on the road. We use the coordination graph
(CG) to achieve the cooperative decision making for multi-vehicle. However, due to
the motion characteristic of the vehicles continuously change topology, the direct
application of CG to autonomous driving is a thorny issue. In order to solve this
problem, we propose a MARL approach based on the combination of dynamic graph
and RL, which use driving risk evaluation and graph theory to model the interaction
between vehicles, then use the consistency of latent conflicts relations to achieve
effective distributed coordination learning. The experiment verifies that the proposed
approach allows the vehicles to achieve higher driving safety under cooperative
decision making. Compared with other approaches that have no coordination or rely on
some mobile models and expert driving rules, the proposed approach is more
advantageous.
2 Related Work
RL has been widely used in the field of autonomous driving. Some researchers have
used RL to learn driving strategies. For example, Wang et al. [20] use deep reinforcement learning with rule-based constraints to learn driving strategies. Other work
uses Deep RL to make driving decisions directly from real-world scenarios, such as
virtual to real reinforcement learning for autonomous driving [14]. All of these studies
only focus on the driving strategies of a single vehicle, without considering the
interactions and coordination between multiple vehicles.
A large number of scholars have studied lane changing before, such as Moriarty
and Langley [11] used RL to solve the lane selection problem of multiple vehicles.
Although these methods have good results in their respective applications, they either
focus on the relatively simple traffic environment or do not simulate the interaction
between multiple vehicles. Some studies [12] have proposed methods to simulate
vehicle interaction and formation using graph theory, but these methods all focus on
formation and communication. Recently, automobile safety accidents happen frequently, so our work is more dedicated to driving safety issues. We realize distributed
learning in multi-vehicle through dynamic CG and RL, which can provide more safe
and accurate lane-changing decision for autonomous driving.
3 Related Theories and Approaches
Traditional coordination graph can only model closed, static problems, which does not
change during the solution process. In order to realize coordinated learning in dynamic
environments, we propose an approach to construct dynamic coordination graph and a
MARL approach to solve it.
600
H. Zuo et al.