PROBABILISTIC PCA
In effect, this is a new matrix factorization model.
I With the SVD, we had X = USVT.
I We now approximate X ≈ WZ, where
I W is a d × K matrix. In different settings this is called a “factor loadings”
matrix, or a “dictionary.” It’s like the eigenvectors, but no orthonormality.
I The ith column of Z is called zi 2 RK. Think of it as a low-dimensional
representation of xi.
The generative process of Probabilistic PCA is
xi ∼ N(Wzi; σ2I); zi ∼ N(0; I):
In this case, we don’t know W or any of the zi.
THE LIKELIHOOD
Maximum likelihood
Our goal is to find the maximum likelihood solution of the matrix W under
the marginal distribution, i.e., with the zi vectors integrated out,
WML = arg max
W
ln p(x1; : : : ; xnjW) = arg max
W
nXi=1
ln p(xijW):
This is intractable because p(xijW) = N(xij0; σ2I + WWT);
N(xij0; σ2I + WWT) = 1
(2π)d2 jσ2I + WWTj1 2 e−
12
xT(σ2I+WWT)−1x
We can set up an EM algorithm that uses the vectors z1; : : : ; zn
EM FOR PROBABILISTIC PCA
Setup
The marginal log likelihood can be expressed using EM as
nXi=1
ln Z p(xi; zijW) dzi = Xn
i=1
Z q(zi) ln p(xqi;(zziij)W) dzi L
+
nXi=1
Z q(zi) ln p(zqij(xzii;)W) dzi KL
EM Algorithm: Remember that EM has two iterated steps
1. Set q(zi) = p(zijxi; W) for each i (making KL = 0) and calculate L
2. Maximize L with respect to W
Again, for this to work well we need that
I we can calculate the posterior distribution p(zijxi; W), and
I maximizing L is easy, i.e., we update W using a simple equation
THE ALGORITHM
EM for Probabilistic PCA
Given: Data x1:n, xi 2 Rd and model xi ∼ N(Wzi; σ2); zi ∼ N(0; I), z 2 RK
Output: Point estimate of W and posterior distribution on each zi
E-Step: Set each q(zi) = p(zijxi; W) = N(zijµi; Σi) where
Σi = (I + WTW=σ2)−1; µi = ΣiWTxi=σ2
M-Step: Update W by maximizing the objective L from the E-step
W = "Xi=n1 xiµT i # "σ2I + Xi=n1 (µiµT i + Σi)#−1
Iterate E and M steps until increase in Pn i=1 ln p(xijW) is “small.”
Comment:
I The probabilistic framework gives a way to learn K and σ2 as well.
Негізгі бет Ғылым және технология 14 3 Probabilistic PCA | Machine Learning
Пікірлер