Singular Value Decomposition Example

Paul Skoufranis
March 11, 2010
We shall explore a concept known as the singular value decomposition of a matrix.
Denition Let A be an n m matrix. The singular values of A are the square roots of the eigenvalues of
m m matrix A
T
A listed according to their algebraic multiplicity. We should note that the eigenvalues of
A
T
A will always be positive numbers so it makes sense to take the square root. It is customary to denote
the singular values by
1
,
2
, . . .,
m
with
1

2

m
.
Theorem Any n m matrix A can be written in the form A = UV
T
where U is an orthogonal n n
matrix, V is an orthogonal mm matrix, and is an n m matrix whose rst r diagonal entries are the
non-zero singular values
1
,
2
, . . .,
r
of A and whose other entries are all zeros. The expression UV
T
is
known as the Singular Value Decomposition of A.
Now we shall go through an example of computing the Singular Value Decomposition (SVD) of a ma-
trix A, why it works, and how we can use the SVD to determine the action of A on the unit circle x
2
+y
2
= 1.
Example Compute the SVD of the matrix
A =
_
_
1 1
1 1

3 0
_
_
and use this decomposition to describe the action of A on the unit circle.
Solution: The rst step in nding the SVD of A is to nd the singular values of A. To do this, we
simply need to compute the eigenvalues of A
T
A. Thus let us rst compute A
T
A;
A
T
A =
_
1 1

3
1 1 0
_
_
_
1 1
1 1

3 0
_
_
=
_
5 2
2 2
_
Therefore the characteristic polynomial of A
T
A is
f
A
T
A
() = det(A
T
AI) = (5 )(2 ) 4 =
2
7 + 6 = ( 1)( 6)
Therefore the eigenvalues of A
T
A are 1 and 6. Thus the singular values of A are
1
= 1 and
2
=

6 (in
general, at this step you would take the square root of each eigenvalue to get a singular value and include
it am() times). This tells us that the matrix will be the 3 2 matrix
=
_
_

6 0
0 1
0 0
_
_
where we place
1
=

6 in the rst column and

2
= 1 in the second column since

6 > 1 (normally, if

1

2

r
> 0 are your non-zero singular values (including repetitions), we would place these
entries along the diagonal of and place zeros elsewhere).
Our next goal is to determine the matrix V . We know that since A
T
A is a symmetric matrix, A
T
A will
have an orthonormal basis of eigenvectors. It turns out that the matrix V will be 22 matrix whose columns
are an orthonormal eigenbasis for A
T
A. Therefore we compute the eigenspaces of A
T
A. We notice that
E
6
= ker(A
T
A6I) = ker
__
1 2
2 4
__
= ker
__
1 2
0 0
__
= span
_
1

5
_
2
1
__
and
E
1
= ker(A
T
AI) = ker
__
4 2
2 1
__
= ker
__
1
1
2
0 0
__
= span
_
1

5
_
1
2
__
Let
v
1
=
1

5
_
2
1
_
and v
2
=
1

5
_
1
2
_
Then we let V be the 2 2 matrix
V =
_
v
1
v
2

=
_
2

5
1

5
2

5
_
where we place the eigenvector for the eigenvalue 6 in the rst column and the eigenvector for the eigenvalue
1 in the second column since we placed

6 in rst column of and

1 in the second column of .

Now our question is how do we nd the orthogonal 3 3 matrix U? Well, if we could nd an orthogonal
3 3 matrix U such that AV = U, we would be done since V
1
= V
T
as V is an orthogonal matrix. Let
us consider the actions of AV and on the standard basis e
1
and e
2
of R
2
. Well

_
1
0
_
=
1
_
_
1
0
0
_
_

_
0
1
_
=
2
_
_
0
1
0
_
_
and
AV
_
1
0
_
= Av
1
AV
_
0
1
_
= Av
2
Therefore, if U were an orthogonal 3 3 matrix such that AV = U and u
1
, u
2
, and u
3
were the columns
of U, then {u
1
, u
2
, u
3
} would be an orthonormal basis of R
3
and
Av
1
= AV
_
1
0
_
= U
_
1
0
_
=

6U
_
_
1
0
0
_
_
=
1
u
1
and
Av
2
= AV
_
0
1
_
= U
_
0
1
_
=

1U
_
_
0
1
0
_
_
=
2
u
2
Thus we should let
u
1
=
1

1
Av
1
=
1
6
_
_
1 1
1 1

3 0
_
_
_
1

5
_
2
1
__
=
1

30
_
_
3
3
2

3
_
_
and
u
2
=
1

2
Av
2
=
1
1
_
_
1 1
1 1

3 0
_
_
_
1

5
_
1
2
__
=
1

5
_
_
1
1

3
_
_
2
If we check, we see that {u
1
, u
2
} are indeed orthonormal vectors. The question is, why does this work?
Well, just using the facts that u
1
=
1
1
Av
1
, u
2
=
1
2
Av
2
, A
T
Av
1
=
2
1
v
1
(since v
1
is an eigenvector for A
T
A
with eigenvalue
2
1
), A
T
Av
2
=
2
2
v
2
(v
2
is an eigenvector for A
T
A with eigenvalue
2
2
), and {v
1
, v
2
} is an
orthonormal set, we see that
u
1
u
1
=
_
1

1
Av
1
_

_
1

1
Av
1
_
=
_
1

1
Av
1
_
T
_
1

1
Av
1
_
=
1

2
1
v
T
1
A
T
Av
1
=
1

2
1
v
T
1
(A
T
Av
1
)
=
1

2
1
v
T
1
(
2
1
v
1
)
= v
T
1
v
1
= v
1
v
1
= 1
and
u
2
u
2
=
_
1

2
Av
2
_

_
1

2
Av
2
_
=
_
1

2
Av
2
_
T
_
1

2
Av
2
_
=
1

2
2
v
T
2
A
T
Av
2
=
1

2
2
v
T
2
(A
T
Av
2
)
=
1

2
2
v
T
2
(
2
2
v
2
)
= v
T
2
v
2
= v
2
v
2
= 1
and
u
1
u
2
=
_
1

1
Av
1
_

_
1

2
Av
2
_
=
_
1

1
Av
1
_
T
_
1

2
Av
2
_
=
1

2
v
T
1
A
T
Av
2
=
1

2
v
T
1
(A
T
Av
2
)
=
1

2
v
T
1
(
2
2
v
2
)
=

2

1
v
T
1
v
2
=

2

1
v
1
v
2
= 0
3
which is want we wanted.
However, to nd an orthogonal matrix using u
1
and u
2
, we need a third orthonormal vector as we need
U to be a 3 3 matrix and we only have two orthonormal vectors (in some cases, you will not need this
step). To nd u
3
, write
u
3
=
_
_
a
b
c
_
_
Then we want a
2
+ b
2
+ c
2
= 1 so u
3
is unit length, and we want
0 = u
1
u
3
=
1

30
(3a + 3b + 2

3c)
and
0 = u
2
u
3
=
1

5
(a b +

3c)
By solving a system of linear equations, we obtain that

_
_
1
1
0
_
_
where R is the set of all solutions to the equations 0 =
1

30
(3a +3b +2

3c) and 0 =
1

5
(a b +

3c).
Therefore, if we let u
3
=
1

2
_
_
1
1
0
_
_
, u
3
is a normal vector that is orthogonal to u
1
and u
2
. Thus {u
1
, u
2
, u
3
}
is an orthonormal basis for R
3
. Then, if we let
U =
_
u
1
u
2
u
3

=
_

_
3

30

1

5
1

2
3

30

1

5

1

2
2

10

5
0
_

_
then U is an orthogonal matrix and A = UV
T
as desired. If you are not condent in your computations,
you could easily check your answer by multiplying this out.
Now we desire to determine the action of A on the unit circle. To see this, we will rst apply V
T
to the
unit circle, then to that image, and then we will apply U. The following is a diagram representing the
actions of each map:
4
In the diagram, we start with the top left diagram. The vector v
1
and its images are shown in blue and the
vector v
2
and its images are shown in red. First we know that V take e
1
to v
1
and e
2
to v
2
. Since V
T
= V
1
as V is an orthogonal matrix, V
T
takes v
1
to e
1
and v
2
to e
2
. Then adds a z-axis and dilates the x-axis by

6. Then U rotates the whole diagram in an awkward manner (you probably do not need to know exactly
what this rotation is but, it takes the x-axis to the line through u
1
, it takes the y-axis to the line through
u
2
, and it takes the z-axis to the line through u
3
). It is hard to see what the curve in the nal diagram looks
like. The blue line goes to the point

30
(3, 3, 2

3) and the red line goes to

1

5
(1, 1,

3).
5