126
F. Rathke et al.
−
b
c
q(b, c) log p(c|b) = −
M
j=1
c •, j
q c (c •, j )E q b [log p(c •, j |b)]
= −
M
j=1
N
c 1, j =1
q c (c 1, j )E q b [log p(c 1, j )|b)]
+
K
k=2
N
c k, j =1
N
c k−1, j =1
q c (c k, j , c k−1, j )E q b [log p(c k, j |c k−1, j , b])
(5.29)
Finally, using the notation introduced in (5.28), we can rewrite (5.29) in vectorized
form as
−
M
j=1
(q c;1, j )
T
ω 1, j +
K
k=2
q c;k∧k−1, j , , k, j
.
(5.30)
Third Summand log P(b) of J(q b , q c )
We use the relation
a
T Ba = tr(aa
T B) = =aa
T
, B,
valid for symmetric matrices, where ·, ·· denotes here the inner product over matrices. Making use of (5.26), we obtain
−
b
q b (b) log p(b)db = C +
1
2
−1
, + ¯
μ ¯
μ
T
− 2 ¯
μμ
T
+ μμ
T
.
(5.31)
Entropy Terms H[q b ] and H[q c ]
Finally, we make explicit the entropies of q b and q c . For the normal distribution q b
we have that
− H [q b ] =
b
q b (b) log q b (b)db = C −
1
2
log ||,
(5.32)
see for example [22, Eq. (A.20)]. For the negative entropy of q c , making use of the
structure of q c , we have
−H [q c ] =
M
j=1
K
k=1
c k, j
q c (c k, j ) log q c (c k, j )
+
K
k=2
c k−1, j
c k, j
q c (c k, j , c k−1, j ) log
q c (c k, j , c k−1, j )
q c (c k−1, j )q c (c k, j )
.