1
appendix D
Principal component analysis
Principal component analysis (PCA) is a statistical technique that can be constructed by several ways, one commonly cited of which is stated in this appendix. By stating a few directly useful properties of PCA for radar signal analysis, we by no means, tend to give an even superficial survey of this ever-growing topic. For deeper and more completecoverageof PCA and its applications, please refer to[Jolliffe, 2002].[p266~p272, Theodoridis and Koutroumbas, 2006; Chapter 8, Mardia et al., 1979; Chapter 11, Anderson, 2003] are also niceand shorter materials to explain some general properties of PCA. For simplifying the presentation, all the following properties of PCA are proved under the assumption that all eigenvalues of whichever covariance matrix concerned are positive and distinct.
One PCA construction: Assume a random vector ,taking values in , has a mean and covariance matrix of and , respectively. are ordered eigenvalues of , such that the -th eigenvalue of means the -th largest of them.Similarly, a vector is the -th eigenvector of when it corresponds to the -th eigenvalue of . To derive the form of principal components (PCs), consider the optimization problem of maximizing , subject to . The Lagrange multiplier method is used to solve this question.
.
Because is the eigenvalue of , with being the corresponding normalized eigenvector, is maximized by choosing to be the first eigenvector of . In this case, is named the first PC of , is the vector of coefficients for , and .
To find the second PC, , we need to maximize subject to being uncorrelated with . Because , this problem is equivalently set as maximizing , subject to , and . We still make use of the Lagrange multiplier method.
.
Because is the eigenvalue of , with being the corresponding normalized eigenvector, is maximized by choosing to be the second eigenvector of . In this case, is named the second PC of , is the vector of coefficients for , and . Continuing in this way, it can be shown that the -th PC is constructed by selecting to be the -th eigenvector of , and has variance of .The key result in regards to PCA is that the principal components are the only set of linear functions of original data that are uncorrelated and have orthogonal vectors of coefficients.
PropositionD.1[Jolliffe, 2002]: For any positive integer , let be an real matrix with orthonormal columns, i.e., , and . Then the trace of covariance matrix of is maximized by taking , where is the -th eigenvector of .
Proof:
Because is symmetric with all distinct eigenvalues, so is an orthonormal basis with being the -th eigenvector of , and we can represent the columns of as
, .
So we have
where , is an matrix.Then,, with being a diagonal matrix whose -th diagonal element is ,and the covariance matrix of is,
where is the -th row of . So,
.
Because, so, and the columns of are orthonormal. By the Gram-Schmidt method, can expand to , such that has its columns as an orthonormal basis of and contains as its first columns. is square shape, thus being an orthogonal matrix and having its rows as another orthonormal basis of . One row of is a part of one row of , so , . Considering the constraints , and the objective . We derive that is maximized if for , and for . When , straightforward calculation yields that is an all-zero matrix except , . This fulfills the maximization condition. Actually, by taking , where is any orthonormal basis of the subspace of , the maximization condition is also satisfied, thus yielding the same trace of covariance matrix of .
■
PropositionD.2[Jolliffe, 2002]: Suppose that we wish to approximatethe random vectorby its projection onto a subspace spanned by columns of , where is a real matrix with orthonormal columns, i.e., . If is the residual variance foreach component of, then is minimized if , where are the first eigenvectors of . In other words, the trace of convariance matrix of is minimized if . When , which is a commonly applied preprocessing step in data analysis methods, this property is saying that is minimized if .
Proof:
The projection of a random vector onto a subspace spanned by columns of is . Then the residual vector is , which has a convariance matrix
.
Then,
.
Also, we know
.
The last equation comes from the fact that has orthonormal columns.
So,
.
To minimize , it suffices to maximize . This can be done by choosing , where are the first eigenvectors of , according to PropositionD.1stated above.
■