3.1 MDP Formalization for Autonomous Driving
The decision-making process of each vehicle can be modeled as the Markov Decision
Process (MDP) by four tuples of ðS; A; P; RÞ. The state S is mainly composed of factors
that affect the decision of the vehicles, which mainly includes the following parts: the
lane l in which the vehicle is currently located; the remaining reaction time t i of the
vehicle and the neighboring vehicles. The state of each vehicle consists of a five tuple
of l; t i
ð Þ i ¼ 1; 2; 3; 4
ð
Þthat can be computed from the data illustrated in Fig. 1.
Finally, the most important safety in autonomous driving is used to evaluate the
chosen actions. We define the reward function as follows:
r s ¼
À40
if collision
minðt 1 ; t 2 Þ if l ¼ 0 and d 1 [ 3 and d 2 [ 3
minðt 3 ; t 4 Þ if l ¼ 1 and d 3 [ 3 and d 4 [ 3
À5
else
8
> > <
> > :
ð1Þ
In the experiment, we will use function approximated RL based on neural network
to solve the above-mentioned MDP problem and compare it with the famous mobile
model Mobil [7] and expert method [10] to verify the validity of the MDP formulation.
3.2 Dynamic Coordination Graph for Multi-vehicle
In the dynamic environment, the dependencies between agents change rapidly as the
state changes. In order to apply the CG to the continuous domain and realize the
coordination among the agents, we propose an approach of constructing the dynamic
graph model based on evaluating the driving risk between each pairs of the vehicles.
A general highway scene is used to illustrate how to construct the coordination
relation between the two vehicles. The vehicle 1 is the lag vehicle, the vehicle 2 is the
lead vehicle, the speeds of vehicle 1 and vehicle 2 are v 1 and v 2 , respectively, and the
following distance between the two vehicles is d. The formula [19] for evaluating the
driving risk between two vehicles is as follows:
3
V
1
V
0
V
2
V
4
V
b v
v
f
1
d
1
d
4
d
2
d
Fig. 1. State of vehicle CAR0
Coordinated Learning for Lane Changing Based on Coordination …
601
The decision-making process of each vehicle can be modeled as the Markov Decision
Process (MDP) by four tuples of ðS; A; P; RÞ. The state S is mainly composed of factors
that affect the decision of the vehicles, which mainly includes the following parts: the
lane l in which the vehicle is currently located; the remaining reaction time t i of the
vehicle and the neighboring vehicles. The state of each vehicle consists of a five tuple
of l; t i
ð Þ i ¼ 1; 2; 3; 4
ð
Þthat can be computed from the data illustrated in Fig. 1.
Finally, the most important safety in autonomous driving is used to evaluate the
chosen actions. We define the reward function as follows:
r s ¼
À40
if collision
minðt 1 ; t 2 Þ if l ¼ 0 and d 1 [ 3 and d 2 [ 3
minðt 3 ; t 4 Þ if l ¼ 1 and d 3 [ 3 and d 4 [ 3
À5
else
8
> > <
> > :
ð1Þ
In the experiment, we will use function approximated RL based on neural network
to solve the above-mentioned MDP problem and compare it with the famous mobile
model Mobil [7] and expert method [10] to verify the validity of the MDP formulation.
3.2 Dynamic Coordination Graph for Multi-vehicle
In the dynamic environment, the dependencies between agents change rapidly as the
state changes. In order to apply the CG to the continuous domain and realize the
coordination among the agents, we propose an approach of constructing the dynamic
graph model based on evaluating the driving risk between each pairs of the vehicles.
A general highway scene is used to illustrate how to construct the coordination
relation between the two vehicles. The vehicle 1 is the lag vehicle, the vehicle 2 is the
lead vehicle, the speeds of vehicle 1 and vehicle 2 are v 1 and v 2 , respectively, and the
following distance between the two vehicles is d. The formula [19] for evaluating the
driving risk between two vehicles is as follows:
3
V
1
V
0
V
2
V
4
V
b v
v
f
1
d
1
d
4
d
2
d
Fig. 1. State of vehicle CAR0
Coordinated Learning for Lane Changing Based on Coordination …
601