Numerical Methods in Chemical Engineering

4 Iterative methods
4 Iterative methods............................................................................................1 4.1 Overview ..................................................................................................1 Steady heat conduction in a slab (Laplace's Equation).....................2 Solving the equations using an elimination method .........................6

4.1.1 4.1.2 4.2

Iterative Methods ...................................................................................10 The Jacobi method ..........................................................................11 Convergence of the Jacobi method .................................................14

4.2.1 4.2.2 4.3

Summary ................................................................................................17

4.1 Overview

In this lecture we will look at iterative methods that can be used for very large systems of equations.

To generate a suitable system of linear equations, we will solve Laplace's equation for steady state heat conduction in a rectangular geometry.

Useful reading:

Chapter 2 of "Numerical Methods for Chemical Engineers with Matlab Applications" by Constantinides and Mostoufi

-1-

Numerical Methods in Chemical Engineering 4.1.1 Steady heat conduction in a slab (Laplace's Equation) The equation which governs temperature in a slab of material is
T = 2 T , t

(4-1)

where is the thermal diffusivity. Here we will only consider the steady state problem, i.e.
2T = 0 ,

(4-2)

which, in two dimensions and Cartesian co-ordinates can be written as:

2 T 2T + =0 x 2 y 2

(4-3)

We are going to solve this problem on the following domain, T = Tb2

T = Tb3

T = Tb4

y x T = Tb1

-2-

Numerical Methods in Chemical Engineering To proceed, we place a grid over our solution domain, and track the temperature at each point on the grid. For simplicity, we will use a regular, equally spaced grid (x = y).
(Ny-1) Nx +1

j = Ny

.. j=3 j=2 j=1 TNx+1 T1 i=1 TNx+2 T2 i=2 TNx+3 T3 i=3 ... ... T2Nx TNx i = Nx

The index of the general node is k = i +Nx (j 1) So that Ti,j = T k= i +Nx (j 1) (4-4)

We can use finite differences to approximate (4-3) at a general node. Thus, we need estimates of the second differentials at node k. We can assume a piece-wise linear profile in temperature, i.e.

-3-

Numerical Methods in Chemical Engineering

Ti + 1 T Ti Ti - 1 x i-1 i x i+1 x

Then we can estimate the second derivative as

T x 2
2

T x

i +1 / 2

T x

i 1 / 2

(T

i +1, j

x Ti , j )

(T

i, j

Ti 1, j ) x

Ti +1, j 2Ti , j + Ti 1, j

x 2

(4-5)

Making a similar approximation for the second differential in the y direction allows us to write an approximate form of (4-3) as: Ti+1, j 2Ti , j + Ti1, j Ti , j +1 2Ti , j + Ti , j 1 + =0 x 2 y 2 (or in terms of the node indices)
Tk +1 2Tk + Tk 1 Tk + Nx 2Tk + Tk Nx + =0 x 2 y 2

(4-6)

(4-7)

For this example lets use an equal spaced grid (x = y)

Tk Nx + Tk 1 4Tk + Tk +1 + Tk + Nx = 0 (or Tk = (Tk Nx + Tk 1 + Tk +1 + Tk + Nx ) / 4 )

(4-8)

The temperature at a grid point is equal to the average of the surrounding temperatures.

-4-

Numerical Methods in Chemical Engineering For the nodes on the boundary, we have a very simple equation:
Tk ,boundary = some fixed temperature,

(4-9)

because, we want to specify the boundary temperatures.

Equations (4-8) for each node and (4-9) for the boundary nodes are linear, and can be written as a matrix equation AT = b (4-10)

The matrix A and right hand side vector b for Nx = Ny = 5 is written out below A
1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 -4 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 -4 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 -4 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 -4 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 -4 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 -4 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 -4 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 -4 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 -4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 Tb1 Tb1 Tb1 Tb1 Tb1 Tb3 0 0 0 Tb4 Tb3 0 0 0 Tb4 Tb3 0 0 0 Tb4 Tb2 Tb2 Tb2 Tb2 Tb2

-5-

Numerical Methods in Chemical Engineering A plot produced by spy(A) for Nx = Ny = 10

0 10 20 30 40 50 60 70 80 90 100 0 20 40 nz = 356 60 80 100

We can see that this sparse matrix has structure, but is not tri-diagonal. There are offset bands. It is these off-set bands which can cause trouble.

4.1.2 Solving the equations using an elimination method

The system equation A T = b we set up in the previous section can be solved in Matlab using the \ command. The solution for Nx = Ny = 10 is shown below for Tb1 = 10 and Tb2 = Tb3 = Tb4 = 0.

-6-

Numerical Methods in Chemical Engineering

10

6 j

5 i

10

10

0 10

6 5 4 3 4 5 6

10

2 0 0 1

Figure 1: Solution to the Laplace Equation over a square domain. Nx = Ny = 10. All boundaries are at T = 0 except for nodes with j = 10 which is at T = 10.

-7-

Numerical Methods in Chemical Engineering We can see what happens when we do Gaussian elimination on A by doing LU decomposition (Using the Matlab routine lu).
[L,U ,P] = lu(A) subplot(1,2,1) spy(L) subplot(1,2,2) spy(U)
L 0 20 40 60 80 100 0 50 nz = 627 100 0 20 40 60 80 100 0 50 nz = 599 100 U

When we LU factorise the matrix A we produce matrices, which are less sparse than the original matrices. Now we have filled in all the zero elements between the central band and the offset diagonal. If we do Gaussian elimination on a matrix like A, we will need to store more elements as the algorithm proceeds. If we had solved a 3D problem, we would have had another offset diagonal band even further from the central diagonal band.

Thus, the matrix produced by elimination takes up more memory than the original matrix. For Matlab this is not a problem it just allocates extra memory. Other languages (where you control memory management) would be less forgiving. The other clever thing Matlab does is attempt to reorder the equations in order to move elements closer to the diagonal.

-8-

Numerical Methods in Chemical Engineering

This problem is sometimes called "fill in". "Fill in" means that when we solve partial differential equations, we will often want to use a method other than e.g. Gaussian elimination. Iterative methods provide an alternative way of solving a set of linear equations.

Typically, we will only use iterative methods when we have large, sparse systems of equations, where the structure of the sparse matrix is not conducive to the use of special methods (e.g. tridiagonal). We will mainly encounter these sorts of systems of equations when we try to solve PDE's in anything other than 1D geometry.

-9-

Numerical Methods in Chemical Engineering

4.2 Iterative Methods

These were particularly important in the early days of computing, when memory was limited. In fact, many can be implemented without reference to matrices. These methods include the Jacobi method, Gauss-Sidel method, Successive over-relaxation etc.... Here we will look at just one method to give us a feel for iterative methods. Matlab1 also has routines for iteratively solving sets of sparse linear equations. These include:
Function Description

bicg gmres bicgstab

Bi-conjugate gradient method Generalised minimum residuals method Bi-conjugate gradient method (stabilised)

See the Matlab helpfile for a description of these functions. The methods used are also described in NUMERICAL RECIPES IN FORTRAN 77: THE ART OF SCIENTIFIC COMPUTING, by Press W.H, Teukolsky S.A, Vetterling W.T and Flannery B.P. (2001), Numerical Recipes in Fortran 77: The art of scientific computing, Cambridge University Press. Chapter 2.7. Available online at www.NR.com

- 10 -

Numerical Methods in Chemical Engineering 4.2.1 The Jacobi method In our previous example, the equation derived from node k was Tk Nx + Tk 1 4Tk + Tk +1 + Tk + Nx = 0 which can be re-arranged as
Tk = Tk Nx + Tk 1 + Tk +1 + Tk + Nx 4

(4-11)

(4-12)

In the Jacobi scheme the iteration proceeds as follows:

Starting with an initial guess for the values of T at each node, we form a new updated value using (4-12) and store it in a new vector Tnew
Tk ,new = Tk Nx ,old + Tk 1,old + Tk +1,old + Tk + Nx ,old
4

(4-13)

We then repeat this for all other nodes. Once we have cycled through all nodes, we use the new value as a guess, and repeat.

- 11 -

Numerical Methods in Chemical Engineering set Told = a guess

Calculate the new node temperature from the guessed values

Have all nodes been updated? Move to next node NO YES Is Max (Tnew Told) < tolerance? YES Report the answer as Tnew NO set Told = Tnew

The advantage here is that only two vectors need to be stored the old temperature and the updated temperature. performed using matrices: This operation can also be

- 12 -

Numerical Methods in Chemical Engineering Consider the matrix A, which will generally have the form

* *

A=
*

* * *

* * * * * * * * * * *

* *

We can split this into a diagonal matrix, D , and another matrix S

* * * * * * *

* * * * * * * *

* * * * *

A=

Then we can solve AT = b by;

(D + S) T DT D Tnew Tnew

=b = b-ST = b - S Told = D-1(b - S Told )

(4-14) (4-15) (4-16) (4-17)

- 13 -

Numerical Methods in Chemical Engineering 4.2.2 Convergence of the Jacobi method If we define the error at the kth iteration as ek = Tk - T Tk+1 = D-1(b - S Tk) ek+1+ T= D-1(b - S ek -ST) D ek+1 = - S ek ek+1 = - D-1S ek (4-19) (4-20) (4-18)

Substituting this into (4-17), we find that the error at the next iteration is

The eigenvalues of D-1S must always have a modulus less than unity to ensure that |ek+1| <| ek|. We can express the vector of errors in terms of the eigenvectors of D-1S

= a1 u1 + a2 u2 + a3 u3 + a4 u4

....

Then ek+1 = - D-1S (a1 u1 + a2 u2 + a3 u3 + a4 u4....) = - a1 D-1S u1 - a2 D-1S u2 - a3 D-1S u3 - a4 D-1S u4 .... = - a1 1 u1 - a2 2 u2 - a3 3u3 - a4 4 u4 ....

Thus, the magnitude of the error will grow on each iteration if any of the eigenvalues () have a complex modulus greater than unity. So we need to find the largest magnitude of eigenvalue for the composite matrix D-1S.

- 14 -

Numerical Methods in Chemical Engineering To proceed further we need Gershgorin's theorem.

Gershgorin's theorem states

for a square matrix and row k, an eigenvalue is located on the complex plane within a radius equal to the sum of the moduli of the off-diagonal elements of that row. i.e.
mk , k mk ,1 + mk , 2 + K + mk , N 2 + mk , N 1 + mk , N

(4-21)

Im() Radius = sum of moduli of off-diagonal elements

mk,k

Re()

The eigenvalue, k must be in this circle

- 15 -

Numerical Methods in Chemical Engineering If we apply this to our Jacobi iteration, the off-diagonal elements of row k of D-1S are 1/ak,k times the off-diagonal elements of our original matrix A, whilst the diagonal element is zero. This implies
k a k ,k = a k ,1 + a k , 2 + K + a k , N 2 + a k , N 1 + a k , N

For |k| < 1

a k ,1 + a k , 2 + K + a k , N 2 + a k , N 1 + a k , N < a k ,k

a
jk

, j < a k ,k

i.e. for the Jacobi method to be stable - the size of the diagonal element must be larger than the sum of the moduli of the other elements in the row. Such a matrix is said to be Diagonally dominant. It turns out, that for the sorts of banded matrices you find when solving PDE's, the following condition is required for stability,

a
jk

, j a k ,k

- 16 -

Numerical Methods in Chemical Engineering

4.3 Summary

We have seen that the solution of partial differential equations can produce sparse systems of linear equations. The structure of the sparse system may not be conducive to solution by a direct method, such as Gaussian Elimination (even if the Gaussian elimination has been modified to take advantage of the sparsity). Typically, the matrix produced by the direct method may be less sparse than that for the original system of equations. This problem will occur when you try to solve a PDE in more than one spatial dimension. If your computer has sufficient memory, a direct method will be more efficient. However, if you have a very large system of equations, an iterative method becomes attractive (e.g. heat flow in 3D with a very large number of grid points). The Jacobi method for iteratively solving a set of linear equations was introduced. This is the simplest of all iteration method. The Jacobi method is VERY INEFFICIENT and should not be used in practice. However, there are many methods which are based upon the Jacobi method (e.g. SOR method), which are used in practice. Matlab also has its own routines for the iterative solution to sparse linear systems, which are quite efficient. The exact choice of iteration method must depend on the structure of the sparse matrix, as iterative methods may only work for certain classes of matrix (e.g. diagonally dominant).

- 17 -