Linear Algebra

TL;DR

Linear algebra is about solving systems of linear equations using vectors and matrices. It provides powerful tools to understand, manipulate, and solve problems in engineering, often involving multiple interacting variables. Mastering these concepts helps you model and analyze complex systems efficiently.

1. The Mental Model

Imagine you have several machines working together, each with inputs and outputs that affect the others. Linear algebra gives you a structured way to represent these relationships and figure out what the combined output will be, or what inputs you need to achieve a specific output.

2. The Core Material

Linear algebra primarily deals with vectors and matrices.

Vectors

A vector is essentially an ordered list of numbers. You can think of it as a point in space, or an arrow pointing from the origin to that point. In engineering, vectors often represent quantities like forces, velocities, or even parameters of a system.

• Column Vector: $\begin{pmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{pmatrix}$
• Row Vector: $\begin{pmatrix} x_1 & x_2 & \dots & x_n \end{pmatrix}$

You can add and subtract vectors, and multiply them by a scalar (a single number), which scales their length.

Matrices

A matrix is a rectangular array of numbers. It's a powerful tool for representing transformations, systems of equations, and data.

• An $m \times n$ matrix has $m$ rows and $n$ columns.
• Example: A $2 \times 3$ matrix: $\begin{pmatrix} a & b & c \\ d & e & f \end{pmatrix}$

Matrix Operations

• Addition/Subtraction: Matrices must have the same dimensions. You add/subtract corresponding elements.
• Scalar Multiplication: Multiply every element by the scalar.
• Matrix Multiplication: This is crucial. If you multiply an $m \times n$ matrix by an $n \times p$ matrix, the result is an $m \times p$ matrix. The number of columns in the first matrix must equal the number of rows in the second.
  • The element in row $i$, column $j$ of the product is the dot product of row $i$ from the first matrix and column $j$ from the second.
  • Important: Matrix multiplication is generally not commutative ($AB \neq BA$).

Systems of Linear Equations

Photo by Vitaly Gariev on Pexels

One of the most direct applications is solving systems of linear equations. A system like:
$a_{11}x_1 + a_{12}x_2 = b_1$
$a_{21}x_1 + a_{22}x_2 = b_2$

Can be written in matrix form as $A\mathbf{x} = \mathbf{b}$, where:
$A = \begin{pmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{pmatrix}$, $\mathbf{x} = \begin{pmatrix} x_1 \\ x_2 \end{pmatrix}$, $\mathbf{b} = \begin{pmatrix} b_1 \\ b_2 \end{pmatrix}$

Determinants and Inverses

Photo by Sergey Meshkov on Pexels

• Determinant (det(A) or |A|): A scalar value that can be computed from the elements of a square matrix. It tells you important things about the matrix, like whether the system $A\mathbf{x} = \mathbf{b}$ has a unique solution. A non-zero determinant means a unique solution exists.
• Inverse Matrix ($A^{-1}$): For a square matrix $A$, if $det(A) \neq 0$, then its inverse $A^{-1}$ exists such that $AA^{-1} = A^{-1}A = I$ (where $I$ is the identity matrix). The inverse allows you to solve $A\mathbf{x} = \mathbf{b}$ by $\mathbf{x} = A^{-1}\mathbf{b}$.

Eigenvalues and Eigenvectors

Photo by Shubham Dhage on Pexels

These describe the fundamental behavior of linear transformations.
* An eigenvector of a matrix $A$ is a non-zero vector $\mathbf{v}$ that, when multiplied by $A$, only changes by a scalar factor.
* The scalar factor is called the eigenvalue ($\lambda$).
* The relationship is expressed as $A\mathbf{v} = \lambda\mathbf{v}$.
* Eigenvalues and eigenvectors are critical in stability analysis, vibrations, and many other dynamic systems in engineering.

Here's how these concepts link together:

graph TD
    A["System of Linear Equations"] --> B["Represent as A x = b"]
    B --> C["A is Coefficient Matrix"]
    B --> D["x is Unknowns Vector"]
    B --> E["b is Constants Vector"]
    C --> F{"Square Matrix A?"}
    F -- Yes --> G["Calculate Determinant (det(A))"]
    F -- No --> H["Gaussian Elimination / Row Operations"]
    G -- det(A) != 0 --> I["Find Inverse Matrix (A^-1)"]
    G -- det(A) == 0 --> J["No unique solution / Infinite solutions"]
    I --> K["Solve for x: x = A^-1 b"]
    H --> K
    K --> L["Solution (x)"]
    C --> M["Eigenvalue Problem (A v = lambda v)"]
    M --> N["Find Eigenvalues (lambda)"]
    M --> O["Find Eigenvectors (v)"]
    N --> P["System Dynamics / Stability Analysis"]
    O --> P

3. Worked Example

Let's solve a system of linear equations using matrix inversion.

Consider the system:
$2x + 3y = 8$
$x - 2y = -3$

1. Write in matrix form $A\mathbf{x} = \mathbf{b}$:
   $A = \begin{pmatrix} 2 & 3 \\ 1 & -2 \end{pmatrix}$, $\mathbf{x} = \begin{pmatrix} x \\ y \end{pmatrix}$, $\mathbf{b} = \begin{pmatrix} 8 \\ -3 \end{pmatrix}$

2. Calculate the determinant of A:
   $det(A) = (2)(-2) - (3)(1) = -4 - 3 = -7$.
   Since $det(A) \neq 0$, an inverse exists, and a unique solution exists.

3. Find the inverse $A^{-1}$:
   For a $2 \times 2$ matrix $\begin{pmatrix} a & b \\ c & d \end{pmatrix}$, the inverse is $\frac{1}{ad-bc}\begin{pmatrix} d & -b \\ -c & a \end{pmatrix}$.
   $A^{-1} = \frac{1}{-7}\begin{pmatrix} -2 & -3 \\ -1 & 2 \end{pmatrix} = \begin{pmatrix} 2/7 & 3/7 \\ 1/7 & -2/7 \end{pmatrix}$

4. Solve for $\mathbf{x}$ using $\mathbf{x} = A^{-1}\mathbf{b}$:
   $\begin{pmatrix} x \\ y \end{pmatrix} = \begin{pmatrix} 2/7 & 3/7 \\ 1/7 & -2/7 \end{pmatrix} \begin{pmatrix} 8 \\ -3 \end{pmatrix}$
   $x = (2/7)(8) + (3/7)(-3) = 16/7 - 9/7 = 7/7 = 1$
   $y = (1/7)(8) + (-2/7)(-3) = 8/7 + 6/7 = 14/7 = 2$

   So, the solution is $x=1, y=2$.

4. Key Takeaways

• Vectors are ordered lists of numbers, representing points or directions in space.
• Matrices are rectangular arrays of numbers used to represent transformations or systems of equations.
• Matrix multiplication is a fundamental operation, but generally not commutative ($AB \neq BA$).
• The determinant of a square matrix indicates if a unique solution exists for $A\mathbf{x}=\mathbf{b}$ and if the inverse exists.
• The inverse matrix $A^{-1}$ allows you to directly solve $A\mathbf{x}=\mathbf{b}$ as $\mathbf{x} = A^{-1}\mathbf{b}$ when $A^{-1}$ exists.
• Eigenvalues and eigenvectors describe fundamental system behaviors, where a vector's direction is unchanged by a transformation, only its magnitude.
• Linear algebra is essential for modeling and solving complex engineering problems, from structural analysis to signal processing.

Common mistakes to avoid:
- Confusing row and column operations during Gaussian elimination or calculating inverses.
- Attempting to multiply matrices with incompatible dimensions.
- Assuming matrix multiplication is commutative.
- Forgetting that a determinant of zero means the inverse does not exist, and there isn't a unique solution.

5. Now Try It

Solve the following system of equations using matrix methods (finding the inverse):

$3x + y = 7$
$5x + 2y = 12$

What to do:
1. Write the system in the matrix form $A\mathbf{x} = \mathbf{b}$.
2. Calculate the determinant of matrix $A$.
3. Find the inverse of matrix $A$, $A^{-1}$.
4. Calculate $\mathbf{x} = A^{-1}\mathbf{b}$ to find the values of $x$ and $y$.

What success looks like: You should find that $x=2$ and $y=1$.