The vehicles with the greatest driving risk are updated by the following rules:
Dx ij ¼ a r þ cmaxQ ij ðs
0
; a
0
; xÞ À Q ij ðs; a; xÞ
Â
à  r x ij Q ij ðs ij ; a ij ; x ij Þ
ð4Þ
Through pretraining, the local cooperative control model can make coordinated
decisions on the basic pair of vehicles in various scenarios and thus can accurately
calculate the coordination payoff functions between pairs of vehicles with latent
conflicting relations.
3.4 The Coordinated Learning Mechanisms
Autonomous driving can be achieved by using MARL technology. Coordination graph
(CG) [8] is an effective way to solve the MARL problems. In order to reduce the
computational complexity, the global payoff function can be decomposed into
Q
p
ðs; aÞ ¼
P
i2V q i ðs i ; a i Þ þ
P
ði;jÞ2E
Q ij ðs ij ; a i ; a j Þ, where V defines the set of vertices
that is equivalent to the set of vehicles, E defines the set of edges, q i ðs i ; a i Þ defines the
individual payoff function of vehicle i, and Q ij ðs ij ; a i ; a j Þ defines the payoff function of
neighboring vehicles i; j
ð Þ.
Although several coordinated RL methods have been proposed previously [3], it is
not feasible to directly apply these methods to the multi-vehicle coordination problem
because the dependencies of moving vehicles are continuously changing. Therefore,
inspired by action-based methods, we propose to use the above DCG model to guide
the driving strategy of the autonomous vehicle and realize distributed coordinated
learning between multiple vehicles. Previous work used manual definition of expert
rules to guide the action selection of agents. However, it is difficult to define accurate
expert rules to guide coordinated decision making in actual collaborative environment.
So, we guide the autonomous vehicle to make decisions through the global optimal
decision of the DCG. Our autonomous vehicles are controlled using an independent
neural network while using the local cooperative control model after pretraining to
calculate the local Q ij ðs ij ; a i ; a j Þ value for each edge.
In each time step, the DCG is constructed for vehicles with latent conflicting
relations. Each vehicle uses the global payoff function arg max ja Q global when the agent
action is selected. The decision of the vehicles is made by maximizing the global payoff
function but using its own local payoff function to update the vehicles’ q i . It can be
explained as guiding the vehicles to explore toward the global optimal state-action
pairs. The neural network of each vehicle can be updated in the following ways:
Dw i ¼ a½r i þ c max
ja
q
0
i ðs
0
; a
0
ja ; wÞ À q i ðs; a ja ; wÞr w i q i ðs i ; a i ; w i Þ
ð 5Þ
4 Experimental Evaluation
In order to verify the effectiveness of the above MDP formulation, local cooperative
control model and dynamic coordination graph, we compare the learning result of RL
method, the famous mobility model MOBIL [7] and the expert rule-based approach
Coordinated Learning for Lane Changing Based on Coordination …
603
Dx ij ¼ a r þ cmaxQ ij ðs
0
; a
0
; xÞ À Q ij ðs; a; xÞ
Â
à  r x ij Q ij ðs ij ; a ij ; x ij Þ
ð4Þ
Through pretraining, the local cooperative control model can make coordinated
decisions on the basic pair of vehicles in various scenarios and thus can accurately
calculate the coordination payoff functions between pairs of vehicles with latent
conflicting relations.
3.4 The Coordinated Learning Mechanisms
Autonomous driving can be achieved by using MARL technology. Coordination graph
(CG) [8] is an effective way to solve the MARL problems. In order to reduce the
computational complexity, the global payoff function can be decomposed into
Q
p
ðs; aÞ ¼
P
i2V q i ðs i ; a i Þ þ
P
ði;jÞ2E
Q ij ðs ij ; a i ; a j Þ, where V defines the set of vertices
that is equivalent to the set of vehicles, E defines the set of edges, q i ðs i ; a i Þ defines the
individual payoff function of vehicle i, and Q ij ðs ij ; a i ; a j Þ defines the payoff function of
neighboring vehicles i; j
ð Þ.
Although several coordinated RL methods have been proposed previously [3], it is
not feasible to directly apply these methods to the multi-vehicle coordination problem
because the dependencies of moving vehicles are continuously changing. Therefore,
inspired by action-based methods, we propose to use the above DCG model to guide
the driving strategy of the autonomous vehicle and realize distributed coordinated
learning between multiple vehicles. Previous work used manual definition of expert
rules to guide the action selection of agents. However, it is difficult to define accurate
expert rules to guide coordinated decision making in actual collaborative environment.
So, we guide the autonomous vehicle to make decisions through the global optimal
decision of the DCG. Our autonomous vehicles are controlled using an independent
neural network while using the local cooperative control model after pretraining to
calculate the local Q ij ðs ij ; a i ; a j Þ value for each edge.
In each time step, the DCG is constructed for vehicles with latent conflicting
relations. Each vehicle uses the global payoff function arg max ja Q global when the agent
action is selected. The decision of the vehicles is made by maximizing the global payoff
function but using its own local payoff function to update the vehicles’ q i . It can be
explained as guiding the vehicles to explore toward the global optimal state-action
pairs. The neural network of each vehicle can be updated in the following ways:
Dw i ¼ a½r i þ c max
ja
q
0
i ðs
0
; a
0
ja ; wÞ À q i ðs; a ja ; wÞr w i q i ðs i ; a i ; w i Þ
ð 5Þ
4 Experimental Evaluation
In order to verify the effectiveness of the above MDP formulation, local cooperative
control model and dynamic coordination graph, we compare the learning result of RL
method, the famous mobility model MOBIL [7] and the expert rule-based approach
Coordinated Learning for Lane Changing Based on Coordination …
603