{ }
( ) max : ( ) A A = e
2
Geometric interpretation
Matrix expands or shrinks any vector lying in
direction of eigenvector by scalar factor.
Expansion or contraction factor is given by
corresponding eigenvalue .
3
Characteristic Polynomial
Equation of is equivalent to:
Which has non zero solution X if and only if its matrix is singular.
Eigenvalues of A are roots of of characteristic polynomial:
A matrix always has n eigenvalues. But may not be real or
distinct.
Complex eigenvalues of real matrix occur in complex conjugate
pairs.
( ) X 0 A I =
AX X =
i
det( ) 0 A I =
n n
4
Coefficients of Characteristic Polynomial
We can write characteristic polynomial as:
Where pk are sum of all kbyk principal minor of A.
For finding of Characteristic Polynomial coefficients, A Krylov
method is introduced.
5
1 2
1 2
( ) ( 1) ( ... )
n n n n
n
P p p p
= + + + +
1 11 22
...
( 1) det( )
nn
n
n
P a a a traceA
P A
= + + + =
=
Krylov method
CayleyHamilton theorem: every square matrix satisfies
its own characteristic equation.
Coefficients will find by sloving of below linear
systems:
6
Characteristic Polynomial, Continued
Computing eigenvalues using characteristic polynomial is not
recommended because of:
Work in computing coefficients of characteristic
polynomial.
Sensitivity of coefficients of characteristic polynomial.
Work in solving for roots of characteristic polynomial.
Characteristic polynomial is powerful theoretically tool but
usually not useful computationally.
7
Some notes
The problem of finding eigenvalues cannot be easier
than finding roots of polynomials.
If the degree of the polynomial is larger than 5,the roots
cannot be found by analyzing method.
So all algorithms of eigenproblem will be iterative
essentially .
8
Multiplicity and Diagonalizability
Multiplicity is number root appears when polynomial is
written as product of linear factors.
Eigenvalues of multiplicity 1 is simple
Defective matrix has eigenvalue of multiplicity k>1
with fewer than k linearly independent corresponding
eigenvector
Nondefective matrix A has n linearly independent
eigenvectors , so it is diagonalizable:
Where X is non singular matrix of eigenvectors.
9
1
X AX D
=
1 2
( , ,... )
n
D diag =
Relevant Properties of Matrices
10
Properties of Eigenvalue Problems
Properties of eigenvalue problem affecting choice of algorithm:
Are all eigenvalues needed or only a few?
Are only eigenvalues needed, or are corresponding eigenvectors
also needed?
Is matrix real or complex?
Is matrix relatively small and dense or large and sparse?
Does matrix have any special properties such as symmetry or is it
general matrix?
11
Transformation
Shift: If and is any scalar, then
, So eigenvalues of shifted
matrix are shifted eigenvalues of original matrix
Inversion: If A is nonsingular and with
Then and .
Powers: If then .So
eigenvalues of power matrix are same power of
eigenvalues of original matrix
AX X =
o
( ) ( ) A I X X o o =
AX X =
0 X =
0 =
1
1
( ) A X X
=
AX X =
k k
A X X =
12
Similarity Transformation
B is similar to A if there is nonsingular matrix
such that
A and B have same eigenvalues
If y is eigenvector of B, Then x=Ty is
eigenvector of A
1
B T AT
=
1
( ) (Ty) By y T ATy y A Ty
= = =
13
Forms Attainable by Similarity
Computations of Jordan form have two numerical problems:
1)It is a discontinuous function of A. So any rounding error can
change it completely.
2)It cannot be computed stably generally.
14
Roughly classification of eigenproblem
Direct methods
Used on dense matrices
Cost O( ) operation to compute all of the eigenvalues and
eigenvectors.
Iterative methods
Usually applied to sparse matrices or matrices for which
matrixvector multiplication is the only convenient operation to
perform.
Convergence properties depend strongly on the matrix entries.
Provide approximations only to a subset of the eigenvalues and
eigenvectors.
3
n
15
Canonical Forms and similarity transformation
Most of our algorithms transforming the matrix A into
simpler (canonical forms)
From canonical forms, it is easy to compute its
eigenvalues and eigenvectors.
The two must common canonical forms are:
Jordan Form
Schur Form
Jordan form is useful theoretically but it is hard to
compute in a numerically stable fashion.
Matrix in Jordan or Schur form is triangular.
16
Jordan and Schur Canonical Forms
Given A, there exist a non singular S such that
J is jordan canonical form and is block diagonal.
Schur: there exist a unitary matrix Q such that
T is upper triangular and the eigenvalues of A are the
diagonal entries of A
1
S AS J
=
*
Q AQ T =
17
1 1 2 2
( ( ), ( ),..., ( ))
n n n k k
J diag J J J =
Matrix reduction
We seek algorithms with lowest computational
complexity.
We will initially reduce the matrix A to upper
Hessenberg form where A is not symmetric.
If A is symmetric we will reduce it to tridiagonal
form.
Computational cost will reduce from to .
3
( ) O n
2
( ) O n
18
Direct Methods for Symmetric matrices
Tridiagonal QR iteration
Rayleigh Quotient iteration
Divide and conquer (Fastest available
algorithm)
Bisection and inverse iteration
Jacobi method
19
Direct Methods for Symmetric matrices ,
continued
The fastest available algorithm for symmetric
eigenproblem ( finding all eigenvalues and all
eigenvectores of a large dense or banded symmetric
matrix) is divide and conquer method.
Jacobi algorithm can find tiny eigenvalues more
accurately than alternative algorithms like divide and
conquer although sometimes more slowly.
All the algorithms except Rayleigh and Jacobis method
assume that the matrix has first been reduced to
tridiagonal form.
20
Comparison of direct methods for the
symmetric Eigenproblem
Tridiagonal QR iteration
Find all eigenvalues and optionally eigenvectors.
It is fastest method to find all the eigenvalues.
Cost operation for finding eigenvalues :
Cost operation for finding eigenvectors:
This method is fastest for small matrices: n<25
Matlab Command: eig
2
( ) O n
3
(6n ) O
21
Tridiagonal QR ITERATION
QR algorithm phases for non symmetric eigenproblem:
1.Given A, find an orthogonal Q so that is upper Hessenberg
2. Apply QR iteration to H to get a sequence of upper Hessenberg matrices
converging to real Schur form.
QR algorithm phases for non symmetric eigenproblem:
1.Given , find an orthogonal Q so that is tridiagonal.
2. Apply QR iteration to T to get a sequence of Tridiagonal matrices
converging to diagonal form.
Finding all eigenvalues and eigenvectors of T costs
The eigenvector Total cost to find the eigenvalues of A is
The total cost to find all eigenvalues and eigenvectors of A is
T
QAQ H =
T
A A =
T
QAQ H =
3 2
6 ( ) n o n +
3 2
4
( )
3
n o n +
3 2
2
8 ( )
3
n o n +
22
Rayliegh quotint iteration
Given with
When stopping criterion is satisfied, is within
tolerance of an eigenvalue of A
0
x
0
2
1 x =
i
23
Divide and Conquer
This method is fastest available now for computation of all
eigenvalues and eigenvectors of tridiagonal matrix whose
dimension is larger than 25.
This method express symmetric matrix T as:
24
Divide and Conquer, Continued
Suppose we have eigen decomposition of T1 and T2
The eigenvalues of T are the same as those of similar matrix
We suppose:
The eigenvalues of T are the roots of so=called secular equation
T
D uu +
1
2
0
0
D
A
(
=
(
A
1
det(D uu ) det((D I)(I (D ) )).
T T
I uu
+ = +
1
det(I (D ) ) 0
T
uu
+ =
2
1 1
1
det(I (D ) ) 1 (D ) 1 ( )
n
T T
i
i
i
u
uu u u f
d
=
+ = + = +
( ) 0 f =
25
Divide and Conquer, Continued
We can find a version of Newtons method that converges fast to
each root .
If is an eigenvalue of , Then is its
eigenvector. Since is diagonal, this cost O(n) flops to
compute. On the other hand, finding all n eigenvalues costs
o
T
D uu +
1
( ) D I u o
( ) D I o
2
( ) O n
26
Divide and Conquer, Algorithm
Algorithm:
Computational Complexity:
In practice, c is usually much less than due to deflation phenomenon
27
Divide and Conquer, Deflation
So far we have assumed that the di are distinct and ui.
Otherwise, Secular equation will have k<n asymptotes
and roots.
If di=di+1 or ui=0, then di is also an eigenvalue of
THIS PROCESS IS CALLED DEFLATION
Using of deflation caused matrix multiplication is fast.
This mean that ith column of is ei (no work is needed for it)
( ) 0 f =
T
D uu +
Q'
28
Bisection and inverse Iteration
To find only k eigenvalues at cost O(nk), Bisection method is an
applicable method.
Sylvesters Inertia Theorem: Let A be symmetric and X be
nonsingular. Then A and have the same inertia.
Inertia(A) = where are the number of negative, zero
and positive eigenvalues of A, respectively.
We can use Gaussian elimination to factorize , where
L is nonsingular and D diagonal.
Inertia(AzI)= Inertia (D)
The number of eigenvalues in the interval [z1,z2) equals ( number of
eigenvalues of A <z2  number of eigenvalues of A<z1 )
T
X AX
( , , ) u t , , u t
T
A zI LDL =
29
Bisection Algorithm
30
Bisection and inverse Iteration
Factorization of AzI is quite simple for tridiagonal matrix.
To compute eigenvectors once we have computed eigenvalues,
we can use Inverse iteration.
Since we can use accurate eigenvalues as shift,convergence
usually takes one or two iteration
31
Jacobi method
One of oldest methods for computing eigenvalues is Jacobi
method which uses similarity transformation based on plane
rotation.
Sequence of plane rotation chosen to annihilate symmetric pairs
of matrix entries, eventually converging to diagonal form.
Choice of plane rotation slightly more complicated than in
Givens method for QR factorization.
To annihilate given off diagonal pair, Choose c and s so that
is diagonal
32
Jacobi method, continued
Starting with symmetric matrix A0=A, each iteration has form
Where is plane rotation chosen to annihilate a symmetric pair
of entries in Ak
Pane rotation repeatedly applied from both side in symmetric
sweeps through matrix until of diagonal is reduced to within
tolerance of zero.
resulting diagonal matrix orthogonally similar to original matrix,
so diagonal entries are eigenvalues and eigenvectors are given by
product of plane rotations.
1
T
k k k k
A J A J
+
=
k
J
33
Jacobi method, continued
Jacobi method is reliable, simple to program, and capable of high
accuracy, but converges rather slowly and difficult to generalize
beyond symmetric matrices
Except for small problem, more modern methods usually
requires 5 to 10 times less work than Jacobi.
One source of inefficiency is that previously annihilated entries
can subsequently become nonzero again, thereby requiring
repeated annihilation.
Newer methods such as QR iteration preserve zero entries
introduced into matrix.
34
Rootfinding methods
Our Goal is numerical approximation of the zeros of a realvalued
function of one variable. These methods are usually iterative.
The aim is to generate a sequence of values such that
( ) k
x
35
Geometric approach to Rootfinding
Bisection Method
Chord method
Secant method
False Position(Regula Falsi) method
Newton method
Increasing computational complexity
36
Bisection Method
Theorem of zeros for continuous function: Given a continuous
function such that f(a)f(b) <0 then such
that
The Speed of convergence of the bisection method:
:[a, b] f ( , ) a b o  e
( ) 0 f o =
0
2
k
k
b a
k
I = >
37
Bisection Method
Absolute error at step k:
If we want , Then
In Particular, to gain a significant figure in the accuracy of the
approximation of the root, one needs bisection.
( ) (k) k
e x o =
( ) ( )
0 lim 0
2
k k
k
k
b a
e k e
s > =
( ) m
x o c s
log(b a/ )
0.6931
m
c
>
3.32
38
Bisection Method
First two step of bisection method Convergence history
Iteration
A
b
s
o
l
u
t
e
e
r
r
o
r
39
Introduction to Chord, Secant, Regula Falsi
and Newton's method
Expanding f in a Taylor series around .
Linearized version :
Basic equation for these methods:
Given , determine by solving
is suitable approximation of
We consider 4 particular choices of
o
( )
( ) 0 f(x) ( ) f x f o o ' = = +
( 1) ( ) 1 ( )
( )
k k k
k
x x q f x
+
=
( ) k
x
( 1) k
x
+
( ) ( 1) ( ) 1
( ) ( ) 0
k k k
k
f x x x q
+
+ =
k
q
( )
( )
k
f x
'
k
q
40
Introduction to Chord,Secant, Regula Falsi
and Newtonss method
First step of chord method
First three step of secant method
41
Chord method
Given an initial value
The sequence converges to the root
(b) f(a)
0
k
f
q q k
b a
= = >
(0)
x
( 1) (0) ( )
( )
(b) (a)
k k
b a
x x f x
f f
+
=
{ }
( ) k
x
42
Secant method
Giving two initial values
We need an extra point and the corresponding value than chord
method. The benefit due to the increase in the computational cost
is higher speed of convergence of the secant method.
The order of sequence convergence in this method can be of to
1.63.
(k) (k 1)
( ) ( 1)
( ) f(x )
0
k
k k
f x
q k
x x
= >
( 1) (0)
, x x
( ) ( 1)
( 1) ( ) (k)
(k) (k 1)
( ) 0
( ) f(x )
k k
k k
x x
x x f x k
f x
= >
43
Regual Falsi method
being the maximum index less than k such that
The above iteration terminates at the mth step such that
The regula falsi method, though of the same complexity as the
secant method, has linear convergence order
Unlike the secant method, the iterates generated by this
algorithm are all contained within the starting interval
( ) ( )
( 1) ( ) (k)
(k) (k )
( ) 0
( ) f(x )
k k
k k
x x
x x f x k
f x
'
+
'
= >
(k) (k )
( ). ( ) 0 f x f x
'
<
k '
(m)
( ) f x c <
( 1) (0)
, x x
(
44
Regual Falsi method
45
Newtons method
Newtons method requires the two functional evaluations
and .
The increasing Computational compensated for bye a higher
order of convergence.
It can be proven that it is more convenient to employ the secant
method whenever the number of floating point operations to
evaluate are about twice those needed for evaluating f
(k)
( ) 0
k
q f x k ' = >
( )
( 1) ( )
(k)
( )
0
( )
k
k k
f x
x x k
f x
+
= >
'
f '
( )
( )
k
f x
( )
( )
k
f x '
46
Newtons method
The first two steps of Newtons method Convergence history
1) Chord method
2) Bisection method
3) Secant method
4)Newtons method
47
Fixed point iteration for Nonlinear Equation
This method is based on the fact that for a given , it is
always possible to transform the problem into a
equivalent such that whenever
So approximating the zeros of a function has become the
problem of finding the fixed points of
(fixed point iteration, Picard iteration, Functional iteration)
:[ , ] f a b
( ) 0 f x =
( ) 0 x x  = ( )  o o =
( ) 0 f o =

( 1) ( )
( ) 0
k k
x x k 
+
= >
48
Fixed point iteration for Nonlinear Equation
The Choice of is not unique. For instance, any function of the
form , Where F is a continuous function such
that
Sufficient condition in order for the fixed point method to
converge to the root:
1)

( ) ( ( )) x x F f x  = +
(0) 0 F =
49
Fixed point iteration for Nonlinear Equation
is called the asymptotic convergence factor.
The asymptotic convergence rate:
2)
( )  o
'
1
log
( )
R
 o
=
'
50
Zeros of Algebraic Equations
Ndegree polynomial
The two property to localize the zeros of a polynomial:
0
( )
n
k
n k
k
P x a x
=
=
1
1
1
( ) ( ) ...( ) ,
k
k
n n k l
l
P x a x m x m m n o o
=
= =
51
Horner Method and deflation
Horners method is based on the observation that any polynomial
can be written as:
This method efficiently evaluates the polynomial at a point z
through synthetic division algorithm:
All the coefficients depend on z and
The Polynomial
Has degree n1 in the variable x and depends on the parameter z
through bk . It is called the associated polynomial of pn.
0 1 2
( ) ( ( ... ( 1 )...)).
n n n
P x a x a x a x a a x = + + + + +
1
1, 2,....0
n n
k k k
b a
b a b z k n n
+
=
= + =
k
b
0
( )
n
b p z =
1 1 2 n n 1
1
( ; z) b b x ... b x 1
n
n k k
k
q x b x
=
= + + + =
52
Horner Method and deflation
Deflation procedure for finding the roots of . For m=n,n1,..,1
1. Find a root r of pm using suitable approximation method
2.evaluate qm1(x;r)
3.let pm1=qm1
We can use following methods for step 1:
The Newton Horner Method
The Muller Method
0 n 1
( ) ( ) q (x; z)
n
P x b x z
= +
n
P
53
The Newton Horner Method
This deflation being helped by the fact that:
The approximation of as root of is carried out until all the
roots of have been computed.
1 1 1
( ; ) ( ) ( ; ) ( ) ( ; )
n n n n n
P q x z x z q x z p z q z z
' ' '
= + =
( ) ( )
( 1) ( ) ( )
( ) ( ) ( )
1
( ) ( )
( ) q ( ; )
k k
n j n j
k k k
j j j
k k k
n j n j j
p r p r
r r r
p r r r
+
= =
'
1
( ) ( ) ( )
n j n
P x x r P x
=
1
( )
n
P x
( )
n
P x
54
The Muller Method
Unlike Newtons or secant methods, Muller's method is able to
compute complex zeros of a given function f, even starting from
a real initial datum; moreover, its order of convergence is almost
quadratic.
55
The Muller Method
Given , the new point is determined by setting
This method extends the secant method, substituting the linear
polynomial with a second degree polynomial.
(0) (1) (2)
, , x x x
(3)
x
(3)
2
(x ) 0 p =
(2) (2) (2) (1) (2) (1) (2) (1) (0)
2
( ) f(x ) (x x ) f[x , x ] (x x )(x x ) f[x , x , x ] p x = + +
( ) ( ) ( , ) ( , )
[ , ] , [ , , ]
f f f f
f f
q t q q
q q t
q t
= =
(2) (2) (2) 2 (2) (1) (0)
2
( ) f(x ) (x x ) (x x ) f[x , x , x ] p x e = + +
(2) (1) (2) (1) (2) (1) (0)
(2) (1) (2) (0) (0) (1)
f[x , x ] (x x ) f[x , x , x ]
f[x , x ] f[x , x ] f[x , x ]
e = +
= +
56
The Muller Method
{ }
( )
( 1) ( )
0.5
2 ( ) ( ) ( 1) ( 2)
2 ( )
4 ( ) ( , , )
k
k k
k k k k
f x
x x
f x f x x x e e
+
=
( ) ( ) k k
e x o =
( 1)
( )
1 ( )
, 1.84
6 ( )
k
k p
k
e
f
Lim p
f
e
o
o
+
'''
=
'
57
Bibliography
1) Demmel ,J. Applied Numerical Linear Algebra, MIT 1996.
2) Quarteroni, A & Sacco.R and Saleri.F . Numerical
mathematics. Springer. 2000.
3) Linear algebra: numerical methods,2000
4) Heath. M.T, Scientific Computing: An Introductory Survey,
Chapter 4 Eigenvalue Problems, University of Illinois at
UrbanaChampaign, 2002.
5) Kubcek.M, Janovska.D, Dubcova.M . numerical methods
and algorithms, 2005.
6) Wilkinson.J.H, The algebraic eigenvalue problem,Oxford,
1965
58