Comparison of Principal Components

122

appendix E

comparison of principal components

Assume two random vectors, and , have the covariance matrices of and , respectively. Suppose the coefficients for the first principal components (PCs) of and are and , respectively, that is, are the first orthonormal eigenvectors of , and are the first orthonormal eigenvectors of . In order to compare these two sets of PCs, it is necessary to compare the two subspaces spanned by and , respectively. The following two theorems in [Krzanowski, 1979] propose one rigorous way to analyze it.

Proposition E.1 [Krzanowski, 1979]: Denote and , the minimum angle between an arbitrary vector in the subspace and another arbitrary vector in the subspace is given by , where is the largest eigenvalue of .

Proof:

Arbitrarily select one vector from , and represent it as , then the projection of onto is given by . Due to the geometry property of projection, to find the minimum angle between two arbitrary vectors in and is to find , such that the angle between and , say , is minimal.

From , and .

So minimizing is equivalent to maximizing , subject to . The closed form solution from the well known linear algebra fact is given by:

When is the first eigenvector of , , which is the largest eigenvalue of .

One point to note is that all eigenvalues, , of satisfy , which is verified by:

, where .

is the projection coefficients of onto , with being an orthonormal set, so

■

Proposition E.2 [Krzanowski, 1979]: , , and are defined as in Proposition E.1, Let , be the -th largest eigenvalue and corresponding eigenvector of , respectively. Take , then and form orthogonal vectors in and , respectively. The angle between the -th pair , is given by . Proposition E.1 shows that and give the two closest vectors when one is constrained to be in the subspace and the other in . It follows that and give directions, orthogonal to the previous ones, between which lies the next smallest angle between the subspaces.

Proof:

Arbitrarily select one vector from , which is orthogonal to , and represent it as , then the projection of onto is given by . The angle between and , say , satisfies

.

So we need to maximize , subject to , and . By the same Lagrange multiplier method we use for PCA construction in [Appendix B], it turns out that the optimal solution is

, when is the second eigenvector of .

It is also true that , because , and that , because .

Continuing in this way, the conclusion of this theorem is reached.

■

Let be the angle between and , i.e., , then , so we have

.

Thus the summation of the eigenvalues of equals the sum of squares of the cosines of the angles between each basis element of and each basis element of . This sum is invariant with respect to whichever basis you select for and . In more detail, let

, and ,

where , are orthogonal matrices, i.e., , and . If is the angle between and , then

.

So, this sum can be used as a measure of total similarity between the two subspaces. It can be checked that if , , and if , .