Iterative methods : In iterative methods, the process starts with an initial approximation to the unknown vector x of Ax = b and then this will be improved by an iterative process x⁽ⁿ⁺¹⁾ = Qx⁽ⁿ⁾ + C where x⁽ⁿ⁺¹⁾ and x⁽ⁿ⁾ are the values of x at (n+1) and (n) the iteratives respectively and Q and C are iterative matrix and constant vector of the sceme. There are various methods to generate Q and C.

• Jacobi method
• Gauss-Siedel method
• SOR method

Jacobi Method

Denote the first approximation to x_i by x_i⁽¹⁾, the second by x_i⁽²⁾, etc., and assume that n of them have been calculated, i.e., x_i⁽ⁿ⁾ is known for i =1(1)m. Then the Jacobi iterative method expresses the (n+1)th iterative values exclusively in terms of the nth iterative in the form 
 

	1/aii	{		i-1		m		},   i = 1(1)m.
xi(n+1)  =			bi -	S	aij xj(n) -	S	aij xj(n)
				j=1		j=i+1

or in the matrix notation, the coefficient matrix A of the system Ax = b, is split into D - L - U where D has the diagonal elements of A and L & U respectively have the lower diagonal and upper diagonal elements of A with a negative sign then the Jacobi scheme can be written as 
Dx = (L + U)x + b
or Dx⁽ⁿ⁺¹⁾ = (L+ U)x⁽ⁿ⁾ + b,
giving x⁽ⁿ⁺¹⁾ = D^-1(L+ U)x⁽ⁿ⁾ + D^-1b.
i.e., the Jacobi iterative matrix Q_Joc = D^-1(L+ U) and C_Joc = D^-1b
 

GAUSS - SEIDEL METHOD

In this method the (n+1)th iterative values are used as soon as they are available and the iterative scheme is defined by
 

	1/aii	{		i-1		m		},   i = 1(1)m.
xi(n+1) =			bi -	S	aij xj(n) -	S	aij xj(n)
				j=1		j=i+1

again in the matrix notation, the coefficient matrix A of the system Ax = b, is split into D - L - U where D has the diagonal elements of A and L & U respectively have the lower diagonal and upper diagonal elements of A with a negative sign then the Gauss-Seidel scheme can be written as 
Dx = (L + U)x + b
or (D-L)x⁽ⁿ⁺¹⁾ = Ux⁽ⁿ⁾ + b,
giving x⁽ⁿ⁺¹⁾ = (D-L)^-1 Ux⁽ⁿ⁾ + (D-L)^-1b.
i.e., the Jacobi iterative matrix Q_GS = (D-L)^-1 U and C_GS = (D-L)^-1b.
 

SUCCESSIVE OVER-RELAXATION METHOD

If the Gauss-Seidel iteration equations is written as
 

	xi(n)  + w/aii	{		i-1		m		},   i = 1(1)m.
xi(n+1) =			bi -	S	aij xj(n) -	S	aij xj(n)
				j=1		j=i+1

Or
 

	w/aii	{		i-1		m		} - (w-1)xi(n),   i = 1(1)m.
xi(n+1) =			bi -	S	aij xj(n) -	S	aij xj(n)
				j=1		j=i+1

Where the factor w, called the acceleration parameter or relaxation factor, generally lies in the range 1<w<2. The determination of the optimum value of w for maximum rate of convergence is again a matter of discussion and is not considered in this article. The value w = 1 gives the Gauss-Seidel iteration.

In the matrix notation if  d⁽ⁿ⁾ = x⁽ⁿ⁺¹⁾ - x⁽ⁿ⁾ then we have 

Dd₁⁽ⁿ⁾ = D(x⁽ⁿ⁺¹⁾ - x⁽ⁿ⁾) = Lx⁽ⁿ⁺¹⁾ + Ux⁽ⁿ⁾ + b - Dx⁽ⁿ⁾

d⁽ⁿ⁾ = wd₁⁽ⁿ⁾,

x⁽ⁿ⁺¹⁾ - x⁽ⁿ⁾ = wD^-1(Lx⁽ⁿ⁺¹⁾ + Ux⁽ⁿ⁾ + b - Dx⁽ⁿ⁾)

Therefore (I-wD^-1L)x⁽ⁿ⁺¹⁾ = { (1 - w)I +wD^-1U}x⁽ⁿ⁾ +wD^-1b.

Hence x⁽ⁿ⁺¹⁾ = (I-wD^-1L)^-1{ (1 - w)I +wD^-1U}x⁽ⁿ⁾ + (I-wD^-1L)^-1wD^-1b

That is Q_SOR = (I-wD^-1L)^-1{ (1 - w)I +wD^-1U} & C_SOR = (I-wD^-1L)^-1wD^-1b.

A necessary and sufficient condition for the convergence of iterative schemes

Consider any iterative scheme 

x⁽ⁿ⁺¹⁾ = Qx⁽ⁿ⁾ + C where Q is the iterative matrix and C is the constant vector of known values. If e⁽ⁿ⁾ is the error in the nth approximation to the exact solution then it can be written as e⁽ⁿ⁾ = x - x⁽ⁿ⁾.
Similarly e⁽ⁿ⁺¹⁾ = x - x⁽ⁿ⁺¹⁾
Therefore e⁽ⁿ⁺¹⁾ = Q e⁽ⁿ⁾
or e⁽ⁿ⁾ = Q e^(n-1) =  Q² e^(n-2) = . . . = Qⁿ e⁽⁰⁾
Since the sequence of approximations {x₁, x₂, . . ., x_n, . . . }converges to x as n tends to infinity if 
 

lim	e(n) = 0	Þ	lim	Qn = 0
n®¥			n®¥

If Q has m lineary independent Eigen vectors v_r (r =1 to m) then these m vectors can be used as a basis for any m dimensional space and hence any vector in this m dimentional space can be represented in terms of these m vectors. In particular, 
 

	m
e(0) =	S	crvr
	r =1

where c_r(r =1 to m) are scalers.
 

		m
Hence	e(1) = Qe(0) =	S	crQvr
		r =1

But Qv_r = l_rv_r by the definition of an eigen value, where l_r is the eigen value corresponding to v_r. Hence

	m
e(1) =	S	crlrvr
	r =1

Similarly

	m
e(n) =	S	crlrnvr
	r =1

Therefore e⁽ⁿ⁾ will tend to the null vector as n tends to infinity, for arbitrary e⁽⁰⁾ , if and only if |l_r|<1 for all r. In other words the iteration will converge for arbitrary x⁽⁰⁾ if and only if the spectral radius r(Q) of Q is less than one.
As a corollary to this result a sufficient condition for convergence is that ||Q||<1. To prove this we have that Qv_r = l_rv_r. Hence

||Qv_r||=||l_rv_r||=| l_r| ||v_r||. But for any matrix norm that is compatible with a vector norm ||v_r||, ||Qv_r|| < ||Q|| ||v_r||. Therefore |l_r| ||v_r|| < ||Q|| ||v_r||, so | l_r| < ||Q||,   s = 1(1)m. It follows from this that a sufficient condition for convergence is that ||Q||<1. It is not a necessary condition because the norm of Q can exceed one even when  r(Q)<1.