Linear Algebra

Vectors
A vector is an ordered set of scalars. We mostly used column vectors (n rows, 1 column) by
convention. Vector dimension equals the number of elements. Geometrically, a vector represents
a point in space. Vectors have both direction and norm (magnitude)

A = [1;2]

Linear Equations (Ax = b)

Three possible solution scenarios:
Unique solution:
When rank(A) = n (for n × n matrix)
Zero solutions:
When rank(A) < n and rank([A,b]) ≠ rank(A)
Infinite solutions:
When rank(A) < n and rank([A,b]) = rank(A)

Rank:
Maximum number of linear independent rows/columns: The rank tells you how many of these
vectors are truly "going in different directions" and are not just combinations of the others.

Eigenvalues and Eigenvectors

For a square matrix A: An eigenvector 𝑣 is a non-zero vector that, when multiplied by A, results
in a scalar multiple of itself: 𝐴𝑣 = 𝜆𝑣. The scalar λ is the eigenvalue corresponding to that
eigenvector. They reveal fundamental properties about transformations represented by the matrix.
Eigenvalues determine stability, convergence, and behavior of systems modeled by matrices.

In MATLAB, [V,D] = eig(A) returns:

V: Matrix with eigenvectors as columns
D: Diagonal matrix with eigenvalues on diagonal

Matrix Operations in MATLAB

inv(A): Find inverse of matrix A
A\b: Solve Ax = b (left division - equivalent to inv(A)*b)
b/A: Right division (equivalent to b*inv(A))
rank(A): Determine rank of matrix
size(A): Return dimensions (rows, columns)
cond(A): Calculate condition number
eig(A): Find eigenvalues
[V,D] = eig(A): Get both eigenvalues (diagonal of D) and eigenvectors (columns of V)
norm(x): Calculate vector norm

Condition Number
Ratio of largest to smallest eigenvalue of a matrix. High condition number indicates an ill-
conditioned matrix. Ill-conditioned matrices meaning small changes in input cause large changes
in output.

Like try to solve these two sets of problems

𝑥 + 2𝑦 = 1 𝑎𝑛𝑑 2𝑥 + 3.999𝑦 = 𝟐. 𝟎𝟎𝟏 will give an answer as 𝑥 = 3 𝑎𝑛𝑑 𝑦 = −1

Whereas,

𝑥 + 2𝑦 = 1 𝑎𝑛𝑑 2𝑥 + 3.999𝑦 = 𝟐 will give an answer as 𝑥 = 1 𝑎𝑛𝑑 𝑦 = 0

General Form of a 3x3 System of Linear Equations

Let's represent a general system of three linear equations with three variables (𝑥₁, 𝑥₂, 𝑥₃) using
general coefficients and constants.

The System of Equations

The general form is written as:
𝑎₁₁𝑥₁ + 𝑎₁₂𝑥₂ + 𝑎₁₃𝑥₃ = 𝑏₁𝑎₂₁𝑥₁ + 𝑎₂₂𝑥₂ + 𝑎₂₃𝑥₃ = 𝑏₂𝑎₃₁𝑥₁ + 𝑎₃₂𝑥₂ + 𝑎₃₃𝑥₃ = 𝑏₃
𝑎₂₁𝑥₁ + 𝑎₂₂𝑥₂ + 𝑎₂₃𝑥₃ = 𝑏₂𝑎₃₁𝑥₁ + 𝑎₃₂𝑥₂ + 𝑎₃₃𝑥₃ = 𝑏₃
𝑎₃₁𝑥₁ + 𝑎₃₂𝑥₂ + 𝑎₃₃𝑥₃ = 𝑏₃

Where:
𝑥₁, 𝑥₂, 𝑥₃ are the variables.
aᵢⱼ represents the coefficient of the j-th variable (xⱼ) in the i-th equation.
bᵢ represents the constant term in the i-th equation.

We can represent it in following form

𝐴𝑥 = 𝐵
𝐴11 𝐴12 𝐴13 𝑥1 𝑏1
[𝐴21 𝐴22 𝐴23 ] [𝑥2 ] = [𝑏2 ]
𝐴31 𝐴32 𝐴33 𝑥3 𝑏3
Using MATLAB’s linear algebra:
𝑥 = 𝒊𝒏𝒗(𝐴) ∗ 𝑏
𝑥 = 𝐴\𝑏
Gauss Elimination (Algorithm)
Create matrix 𝐴 𝐴𝑢𝑔 = 𝐴𝑏 = [𝐴 | 𝐵]
In each step
If 𝐴𝑏(𝑘, 𝑘) is a pivot element, then Using pivot element we can create zeros in pivot column like
in the 𝑘 𝑡ℎ column. Suppose 𝑖 𝑡ℎ row is just below the 𝑘 𝑡ℎ 𝑟𝑜𝑤 then we can make zeros in that
column using following form
𝑅𝑖 = 𝑅𝑖 − factor∗ 𝑅𝑘
Where
𝐴𝑏(𝑖,𝑘)
Factor =
𝐴𝑏(𝑘,𝑘)
To use loops and for any size of the matrix
𝑘=1 𝐴11 𝐴12 𝐴13 𝑏1
𝐴𝑏 = [ 𝑘 = 2 𝐴21 𝐴22 𝐴23 𝑏2 ]
𝑘 = 𝑛 − 1 𝐴31 𝐴32 𝐴33 𝑏3

Outer loop (k): Iterates through each pivot position from the first row to the second-to-last
row (k = 1 to n-1). We stop at n-1 because the last row doesn't need any rows below it to be
eliminated.

Inner loop (i): For each pivot position k, we iterate through all rows below it (i = k+1
to n). This loop creates zeros beneath each pivot element.

Elimination factor: We calculate factor = Ab(i,k) / Ab(k,k) which is the factor needed
to make the element at position (i,k) zero.

Row operation: We replace row i by subtracting factor times row k from it:
Ab(i,:) = Ab(i,:) - factor * Ab(k,:).

This creates a zero at position (i,k) while maintaining the system's equivalence. This process
systematically transforms the augmented matrix Ab into row echelon form, allowing us to solve
the system using back substitution.
𝐴11 𝐴12 𝐴13 𝑏1
𝐴𝑏 = [ 0 𝐴22 𝐴23 𝑏2 ]
0 0 𝐴33 𝑏3
Converted into Ax=b form
𝐴11 𝐴12 𝐴13 𝑥1 𝑏1
[ 0 𝐴22 𝐴23 ] [𝑥2 ] = [𝑏2 ]
0 0 𝐴33 𝑥3 𝑏3
This is equal to
𝐴11 𝑥1 + 𝐴12 𝑥2 + 𝐴13 𝑥3 = 𝑏1
𝐴22 𝑥2 + 𝐴23 𝑥2 = 𝑏2
𝐴33 𝑥3 = 𝑏3
Use Back Substitution
𝑏(3) 𝐴𝑏(3, 𝑒𝑛𝑑)
𝑥3 = =
𝐴𝑏(3,3) 𝐴𝑏(3,3)
 𝑏(2) − 𝐴𝑏(2,3)𝑥3
𝑥2 =
𝐴𝑏(2,2)
𝑏(1) − 𝐴𝑏(1,2)𝑥2 − 𝐴𝑏(1,3)𝑥3
𝑥1 =
𝐴𝑏(1,1)
This 𝑥1 can be written as
𝑥2
(𝑏(1) − [𝐴𝑏(1, 2) 𝐴𝑏(1,3)] [𝑥 ])
3
𝑥1 =
𝐴𝑏(1,1)
OR
𝑥2
(𝐴𝑏(1, 𝑒𝑛𝑑) − [𝐴𝑏(1, 2) 𝐴𝑏(1,3)] [𝑥 ])
3
𝑥1 =
𝐴𝑏(1,1)

In MATLAB this becomes:

x(i) = (Ab(i,end) - Ab(i,i+1:n) * x(i+1:n)) / Ab(i,i);

LU Decomposition
Matrix A can be written as a product of the lower and upper triangular matrix
𝐴 = 𝐿𝑈
U is the matrix obtained after gauss elimination, and L is the below matrix
1 0 … 0
𝑚
𝐿 = [ 2,1 1 … :]
𝑚3,1 𝑚3,2 … :
m is the multiplier

Gauss Siedel Method

𝑗=𝑘−1 𝑗=𝑛
𝑏𝑘 − (∑𝑗=1 𝐴𝑘,𝑗 𝑥𝑗𝑖+1 + ∑𝑗=𝑘+1 𝐴𝑘,𝑗 𝑥𝑗𝑖 )
𝑥𝑘𝑖+1 =
𝐴𝑘,𝑘

Consider the Examples

If we take first example (left side) we need to iterate like this

x=[0;0]
The following iteration needs to be perform multiple times
for i = 1:20
x(1) = 1 - 2*x(2)
x(2) = -4 + x(1)
end
but we observe that this is not converging as the diagonal are not dominating. In the second case
(taking the right side):

for i = 1:20
x(1) = 4 + x(2)
x(2) = 0.5*(1-x(1))
end
This is converging.

Let us Solve the Following equation:

𝑥₁ + 𝑥₂ + 𝑥₃ = 4
2𝑥₁ + 𝑥₂ + 3𝑥₃ = 7
3𝑥1 + 4𝑥2 − 2𝑥₃ = 9
we need to rearrange the equation as first equation 2 then 3 then 1. This arrangement is to get
diagonally dominant matrix.
A = [ Ab(2,:); Ab(3,:); Ab(1,:)];
Now apply Gauss Siedel Method,
First initialize the number of variables n
n=3;
x= zeros(n,1); % initial solutions
err = zeros(n,1); % to calculate error
k: will iterate for the n number of variables
j: will iterate for the number of columns.

𝑗=𝑘−1 𝑗=𝑛
𝑏𝑘 − (∑𝑗=1 𝐴𝑘,𝑗 𝑥𝑗𝑖+1 + ∑𝑗=𝑘+1 𝐴𝑘,𝑗 𝑥𝑗𝑖 )
𝑥𝑘𝑖+1 =
𝐴𝑘,𝑘

How to write it in the MATLAB program

x(k) = (A(k,end) - A(k, 1:k-1) * x(1:k-1) - A(k, k+1:n) * x(k+1:n) )/ A(k,k);

Outer loop: This loop runs for a fixed number of iterations (25 in this case) to allow the Gauss-
Seidel method to converge to a solution.
For iterations = 1:50
for k = 1:n
% apply gauss Seidel method
end
Display the solution or iteration or error using disp or fprintf
end