• Direct methods : These are the methods which can find the solution of the system in a finite 

number of steps known apriori. Some of the important direct methods are 

1. Elimination methods
2. Decomposition methods

Gauss Elimination Method : 

There are  two basic steps in this elimination method. They are 
 

• Forward elimination 
• Back substitution. 

In forward elimination the augmented matrix (the elements of the vector b has joined with the coefficient matrix A  as (n+1)th column) and is denoted by A|b is converted into upper diagonal form by making  use of matrix row transformations (one can also convert into a lower triangular form in which case the process is called backward elimination). 

Then by starting with the last row of the upper triangular matrix (first row for lower triangular matrix) the unknown quatity is obtained by back (forward) substitution. 

For example consider the following n algebraic linear equations in n unknowns x ₁, x ₂ , . . ., x_n as 

or in the matrix notation Ax = b

Forward elimination procedure  : 

Use the row transformation 

R₂ ® R₂ - R ₁*a₂₁/a₁₁means the elements of the second row are replaced by the second row elements subtracted with the first row elements  multiplied with the coefficient of the first element of the second row and divided with the diagonal elements of the first row. 

This will make the elements in the second row first column as zero. Similarly the remaining (n-2 ) rows are also replaced with corresponding row transformations so that the elements below the diagonal element in the first column become zero. 

Now the elements of the second now can be used to make the elements below the diagonal element of the second column, zero. Here the elements of the first row cann't be used since that will change the zeros in the first column to non zero again. 

By applying similar procedure for the remaining columns of the augmented matrix A|b(except the last column), the coefficent matrix part of the augmented matrix A|b will become upper diagonal. (A similar procedure can be applied from the last row to make the coefficient matrix A as lower diagonal). Now the last row of the augmented matrix has only two non-zero terms (the coefficient of x _n and b_n). This can be used to find x_n. Once x_n is known, the value of x_n is substituted in (n-1)^th row and obtain the x_n-1. As we continue this process until first row gives unknown quatities x₁, x ₂, . . ., x _n. 

Example : Consider the simple example : 

Forward elimination : 

R₂ ® R₂ - 2/1 R ₁
R₃ ® R₃ - 3/1 R ₁

A|b =		1	1	1	6
		0	1	2	8
		0	1	-1	-1

R₃ ® R₃ -1/1 R ₂
 

A|b =		1	1	1	6
		0	1	2	8
		0	0	-3	-9

Back Substitution :
From the last row
-3x₃ = -9 Þ x₃ = 3

From the second row x₂ + 2x₃ = 8
x₂ = 8 - 2x₃ = 2

Now from the first row x ₁+ x₂ + x ₃ = 6

x₁ = 6 - 2 - 3 =1 Þ x = (1, 2, 3) ^TThe same procedure can be extended to the system of any order n provided the system has an unique solution. 

Pivoting : The main draw back of the above elimination process is division by the diagonal term while converting the augmented matrix into upper triangular form. If the diagonal element is zero or a vanishingly very small then the elements of the rows below this diagonal become very large in magnitude and difficult to handle because of the finite storage capacity of the computers. Alternative is to convert the system such that the element which has large magnitude in that column comes at the pivotal position i.e., the diagonal position. 

Partial Pivoting : If only row interchanging is used to bring the element of large magnitude of the pivotal column to the pivotal position at each step of diagonalization then such a process is called partial pivoting. In this process the matrix may have larger element in non-pivotal column(the column where the pivot is there) but the largest element in the pivotal column only brought to pivotal (or diagonal) position in this process by making use of row transformations. 

Complete Pivoting : In this process the largest element(in magnitude) of the whole coefficient matrix A is first brought at 1x 1 position of the coefficeint matrix and then leaving the first row and first column, the largest among the remaining elements is brought to the pivotal 2 x2 position and so on by using both row and column transformations, is called complete pivoting. During row transformations the last column of the augmented matrix also has to be considered but this column is not considered to find the largest element in magnitude. Since the column transformations are also allowed in this process, there will be a change in the position of the individual elements of the unknown vector X. Hence in the end the elements of the unknown vector X has to be rearranged by applying inverse column transformations in reverse order to all the column transformations preformed. 

Example : Consider x ₁+ x₂ + x₃ = 6, 3x ₁ + 3x ₂ + 4x ₃ = 20, 2x₁+ x ₂ + 3x₃ = 13Partial pivoting : Since the largest element in the first column is at 3x₁ which is not in the pivotal position, perform the row transformation  R₁ « R ₂. Now the system is 

R₂ ® R₂ - 1/3 R ₁
R₃ ® R₃ - 2/3 R ₁

Since this is a '0' at the pivotal position i.e., at second row second column apply R ₂«R₃ (interchange rows tow and three) 
 

A|b =		3	3	4	20
		0	-1	1/3	-1/3
		0	0	-1/3	-2/3

Now the augmented matrix is in diagonal form(the part of coefficient matrix A). 

Back substitution : From the last row : -1/3x₃ = -2/3 Þ x ₃ = 2
Now from the second row 
-x₂ = -1/3 - 1/3x₂ Þ x ₂ =1
and from the first row 3x₁ = 20 - 3x₁ - 4x(2) Þ x₁ =3
Hence the solution is X = (3 1 2)^T

Complete Pivoting :

The given system is 

3x₁+ 3x₂ + 4x ₃ = 20
x₁+ x₂ + x₃ = 6
2x₁+ x₂ + 3x₃ = 13
 Since the largest element in magnitude is at first row third column perform the column transformation C₁ « C₃ (interchange first and third columns) then the augmented matrix is 

Augmented matrix is =		4	3	3	20
		1	1	1	6
		3	1	2	13

(please note the order of the individual elements of the unknown vector x is now (x₃ x₂ x ₁) ^T
Perform R₂ ® R₂ - 1/4R ₁
R₃ ® R₃ - 3/4R ₁
 

A|b =		4	3	3	20
		0	1/4	1/4	-2
		0	-5/4	-1/4	3/5

Now the elment with the largest magnitude is in the third row(leaving the first row aside) 
Perform R₂« R₃
 

A|b =		4	3	3	20
		0	-5/4	-1/4	-2
		0	1/4	1/4	1

R₃ ® R₃ - (-1/5) R ₂
 

A|b =		4	3	3	20
		0	-5/4	-1/4	-2
		0	0	1/5	3/5

From the last row 1/5x ₁ = 3/5 Þ x₁ =3

From the second row -5/4x ₂ = -2 +1/4 x 3 Þ x ₂=1

4x₃ = 20-3x₂-3x ₁ = 20 -3 -9 = 8 Þ x ₃ = 2
 

COMPLEXITY OR OPERATIONAL COUNT
 
 

First step (division by first pivot)	:	n
second step (division by second pivot)	:	n-1
. . .	: : :	. . .
nth step(division by nth pivot)	:	1

Total number of divisions = Sn = n(n+1) / 2

no. of multiplications:
 

First step :	second equ	:	n
	third equ	:	n
	. . .	: : :	. . .
	nth equ	:	n
	Total	:	n(n-1)

Similarly for second, third and so on
ÞTotal no of multiplications in Forward elimination = Sn(n-1) = S(n² - n) = (n/3) (n+1)(n-1)

No. of multiplications in back substitutions
 

(n-1)th equation	:	1
(n-2)th equation	:	2
. . .	: : :	. . .
first equation	:	n-1

ÞTotal no of multiplications in back substitution = S(n-1) = (n/2)(n-1)

Total multiplications = (n/3)(n+1)(n-1) + (n/2)(n-1) = (n/6) (n-1)(2n+5)

Operational count = Total no of divisions and multiplications

= (n/2)(n+1) + (n/6)(n-1)(2n+5) = (n/3)(n2 +3n -1)

for very large 'n' the operational count is @ n³/3 or the complexity of the Gauss elimination is O(n³/3)

No. of additions and subtractions = (n/6)(n-1)(2n+5).

Similarly operational count for cholesky method is (1/6)(n³+9n²+2n).
 
 

Ill Conditioned Systems :

During computation it is not possible to store the numbers exactly in the computer but prone to some round off errors. If dA is the error in A and db is the error in b then the equation Ax = b is actually solved for (A + dA)= b + db.
(or)
|| - x|| < ||(A + dA)^-1 - A^-1|| ||b|| + ||(A + dA)^-1|| ||db||

where ||.|| is any matrix norm

This gives
|| - x|| < (||A^-1dA|| ||db||) (||A^-1|| / (1 - ||A^-1dA||))

|| - x|| < [ ||db|| / ||b||   +  ||dA|| / ||dA|| ] ( k(A) / (1 - ||A^-1dA||)))

where k(A) = ||A^-1|| ||A^-1|| is called the condition number of the matrix A. If k(A) is small (close to one) small change in A and b leads to small changes in x where as for large values of k(A)  a small change in A or b (or both) leads to large changes in x. The systems for which k(A) is large are called ill conditioned systems.

Example : consider 2.1x +1.8y = 2.1 and 6.2x + 5.3y = 6.2

eigen value of A are 74.18 and l₂ = 0.000012
if we use || ||₂ for the norm then k(A) = Ö(l₁/l₂) = 2472.73
That is the given system is ill-conditioned.