Matrix Algebra

Singular Value Decomposition

Singular values. Let

be an $m\times n$ matrix. Then the $n\times n$ square matrix $A^T A$

is symmetric and it has an orthogonal diagonalization

$\displaystyle A^T A = U D U^T$

where the diagonal matrix

has the eigenvalues $\lambda_i$ of the symmetric matrix $A^T A$

and an orthogonal matrix $U = [\mathbf{u}_1 \cdots \mathbf{u}_n]$ with the corresponding eigenvectors which are orthonormal. Moreover, we obtain

$\displaystyle \Vert A\mathbf{u}_i\Vert^2 = \mathbf{u}_i^T A^T A\mathbf{u}_i = \lambda_i\Vert\mathbf{u}_i\Vert = \lambda_i$

and observe that $\lambda_i$ 's are all nonnegative (that is, $\lambda_i\ge 0$ ). The square root

$\displaystyle \sigma_i = \sqrt{\lambda_i} = \Vert A\mathbf{u}_i\Vert$

of the strictly positive eigenvalue of $A^T A$

is called the singular value.

Orthogonal basis for column spaces. Suppose $\{\mathbf{u}_1,\ldots,\mathbf{u}_n\}$ is an orthonormal basis forming an orthogonal diagonalization of the symmetric matrix $A^T A$ , and that $A^T A$ has $r$ strictly positive eigenvalues $\lambda_1,\ldots,\lambda_r$ . Then $\{A\mathbf{u}_1,\ldots,A\mathbf{u}_r\}$ becomes an orthogonal basis for the column space $\mathrm{col}\,A$ and $\mathrm{rank} A = r$ .

Singular value decomposition. Let be an $m\times n$ matrix. Suppose that $\{\mathbf{u}_1,\ldots,\mathbf{u}_n\}$ is an orthonormal basis forming an orthogonal diagonalization of the symmetric matrix $A^T A$ , and that

$\displaystyle \mathbf{w}_1=A\mathbf{u}_1/\Vert A\mathbf{u}_1\Vert,\ldots,\mathbf{w}_r=A\mathbf{u}_r/\Vert A\mathbf{u}_r\Vert$

is an orthonormal basis for the column space $\mathrm{col}\,A$ . Then we have

$\displaystyle A = W_r D U_r^T$

where $W_r = [\mathbf{w}_1\cdots\mathbf{w}_r]$ is the $m\times r$ matrix, $U_r = [\mathbf{u}_1\cdots\mathbf{u}_r]$ is the $n\times r$ matrix, and

is the $r\times r$ diagonal matrix with singular values $\sigma_1>0,\ldots,\sigma_r>0$ . This factorization is called a (reduced) singular value decomposition (or SVD) of

Matlab/Octave. A general form of singular value decomposition uses an $m\times m$ orthogonal matrix and an $n\times n$ orthogonal matrix which extend respectively $W_r$ and $U_r$ . Then the singular value decomposition becomes

$\displaystyle A = W \Sigma U^T$

where

$\displaystyle \Sigma = \begin{bmatrix} D & O \\ O & O \end{bmatrix}$

is the $m\times n$ matrix which includes the $r\times r$ diagonal matrix

and

denotes a matrix of all zero entries. In matlab/octave [W,S,U] = svd(A) performs a singular value decomposition of A.

EXAMPLE 3. Find a singular value decomposition for

$A = \begin{bmatrix} 4 & 11 & 14 \\ 8 & 7 & -2 \end{bmatrix}$
$A = \begin{bmatrix} 1 & -1 \\ -2 & 2 \\ 2 & -2 \end{bmatrix}$