Fundamentals of Neural Networks and Machine Learning

TL;DR

Machine learning lets computers learn from data without explicit programming, with neural networks being a powerful subset inspired by the human brain. You'll understand the basic structure of a neural network and how it learns to make predictions or classify data. This foundation is key to seeing how AI can aggregate these networks later.

1. The Mental Model

Think of machine learning as teaching a kid to identify cats by showing them many pictures, rather than writing down a long list of rules. Neural networks are like a group of these "kids" (neurons) connected together, where each kid learns a small part of the cat's features, and they all work together to make the final "cat or not cat" decision.

2. The Core Material

Machine learning (ML) is a field of AI that allows systems to learn from data. Instead of being explicitly programmed for every task, ML models identify patterns and make decisions or predictions based on the data they've been trained on.

What is Machine Learning?

Photo by Ann H on Pexels

At its heart, ML is about building algorithms that can learn from data. There are generally three types:

• Supervised Learning: You provide the algorithm with labeled data, meaning both the input and the correct output are known. The algorithm learns to map inputs to outputs. Think of predicting house prices (output) based on features like size and location (input).
• Unsupervised Learning: You provide unlabeled data, and the algorithm tries to find inherent structures, patterns, or groupings within it. Clustering customers into different segments based on their purchasing behavior is an example.
• Reinforcement Learning: An agent learns to make decisions by performing actions in an environment and receiving rewards or penalties. Training a robot to navigate a maze is a good example.

Neural Networks: Inspired by the Brain

Photo by Google DeepMind on Pexels

Neural networks (NNs), or Artificial Neural Networks (ANNs), are a specific type of ML algorithm inspired by the structure and function of the human brain. They're particularly good at finding complex patterns in data.

The Neuron: The Basic Unit

The fundamental building block of a neural network is the neuron (or perceptron). Imagine it as a tiny decision-maker.

Here's how a single neuron typically works:

1. Inputs (x): It receives one or more inputs, which are numerical values.
2. Weights (w): Each input is multiplied by a weight. Weights represent the importance of each input.
3. Summation: All weighted inputs are summed up.
4. Bias (b): A bias value is added to this sum. The bias allows the neuron to activate even if all inputs are zero, or makes it harder to activate.
5. Activation Function (f): The sum (plus bias) then passes through an activation function. This function decides whether the neuron "fires" or not and introduces non-linearity, which is crucial for learning complex patterns. Common activation functions include ReLU, Sigmoid, and Tanh.

Here's a simple diagram to visualize this:

graph TD
    Input1["Input 1 (x1)"] --> Weighted1["x1 * w1"]
    Input2["Input 2 (x2)"] --> Weighted2["x2 * w2"]
    InputN["Input N (xn)"] --> WeightedN["xn * wn"]

    Weighted1 --> Summation
    Weighted2 --> Summation
    WeightedN --> Summation
    Bias["Bias (b)"] --> Summation

    Summation("Weighted Sum (Σ(x*w) + b)") --> ActivationFunction("Activation Function (f)")
    ActivationFunction --> Output["Output (y)"]

Layers in a Neural Network

Neural networks are usually organized into layers:

• Input Layer: Receives the raw data. The number of neurons here equals the number of features in your input data.
• Hidden Layer(s): These are the "thinking" layers. They perform computations and extract features from the input data. A network with more than one hidden layer is called a deep neural network.
• Output Layer: Produces the network's final prediction or classification. The number of neurons depends on the type of problem (e.g., 1 for binary classification, multiple for multi-class classification or regression).

How Neural Networks Learn: Backpropagation

The "learning" in a neural network happens through a process called training. When you provide the network with labeled data, it makes a prediction.

1. Forward Pass: Input data goes through the network, layer by layer, to produce an output.
2. Calculate Loss: The network's output is compared to the actual correct output (the label). The difference is measured by a loss function (e.g., Mean Squared Error for regression, Cross-Entropy for classification). The loss tells us how "wrong" the network's prediction was.
3. Backpropagation: This is the core learning step. The error (loss) is propagated backward through the network. During this process, the network figures out how much each weight and bias contributed to the error.
4. Update Weights and Biases: Based on the error contribution, the weights and biases are adjusted slightly to reduce the loss. This adjustment is guided by an optimizer (e.g., Gradient Descent).

This entire process (forward pass, loss calculation, backpropagation, weight update) is repeated many, many times over the training data in what are called epochs, gradually improving the network's ability to make accurate predictions.

3. Worked Example

Let's imagine a tiny neural network trying to predict if a student will pass an exam based on two inputs: hours studied and previous GPA.

Input data:
* Hours Studied (x1): 0.8 (e.g., 8 hours out of 10 max)
* Previous GPA (x2): 0.7 (e.g., 70% out of 100%)
* Target Output (y_true): 1 (Pass)

Let's use a very simple setup:
* One neuron.
* Initial weights: w1 = 0.5, w2 = 0.3
* Initial bias: b = 0.1
* Activation function: Sigmoid, defined as $f(z) = 1 / (1 + e^{-z})$

Step 1: Forward Pass

1. Weighted Sum (z):
   $z = (\text{x1} * \text{w1}) + (\text{x2} * \text{w2}) + \text{b}$
   $z = (0.8 * 0.5) + (0.7 * 0.3) + 0.1$
   $z = 0.40 + 0.21 + 0.1$
   $z = 0.71$

2. Activation Function:
   $y_{\text{predicted}} = \text{sigmoid}(z)$
   $y_{\text{predicted}} = 1 / (1 + e^{-0.71})$
   $y_{\text{predicted}} \approx 1 / (1 + 0.491) \approx 1 / 1.491 \approx 0.67$

So, the network currently predicts a 67% chance of passing.

Step 2: Calculate Loss

Let's use a simple Squared Error Loss for this example:
$\text{Loss} = (y_{\text{true}} - y_{\text{predicted}})^2$
$\text{Loss} = (1 - 0.67)^2 = (0.33)^2 \approx 0.1089$

This loss is the error we want to reduce.

Step 3: Backpropagation (Conceptual)

The key idea here is to figure out how much each weight (w1, w2) and bias (b) needs to change to make y_predicted closer to y_true. This involves calculating the gradients (how much the loss changes with respect to each weight/bias) using calculus. For instance, if increasing w1 subtly increases y_predicted and y_predicted is already too high, we'd decrease w1. If y_predicted is too low, we'd increase w1.

Step 4: Update Weights and Biases (Conceptual)

Using an optimizer (like Stochastic Gradient Descent), the weights and bias would be updated:

$w_{\text{new}} = w_{\text{old}} - (\text{learning_rate} * \text{gradient_of_loss_wrt_w})$
$b_{\text{new}} = b_{\text{old}} - (\text{learning_rate} * \text{gradient_of_loss_wrt_b})$

After one iteration (or "epoch" with this single example), the weights and bias would be slightly adjusted, hoping to produce a prediction closer to 1 (pass) on the next try. This process repeats for many iterations and many examples.

4. Key Takeaways

• Machine learning enables computers to learn from data patterns without explicit programming rules.
• Neural networks are a powerful subset of machine learning, loosely modeled after the human brain's structure.
• A single neuron takes weighted inputs, sums them with a bias, and passes the result through an activation function to produce an output.
• Neural networks consist of input, hidden, and output layers, with computations happening in the hidden layers.
• Networks learn by comparing their predictions to actual outcomes (loss), then adjusting weights and biases through backpropagation to minimize that loss.
• The activation function introduces non-linearity, allowing neural networks to learn complex, non-linear relationships in data.

Common Mistakes to Avoid:
- Don't confuse "AI" generally with "neural networks"; NNs are a type of AI approach.
- Don't expect a neural network to learn effectively if your data is noisy, insufficient, or poorly prepared.
- Avoid thinking of neurons as actual brain cells; they are mathematical approximations and simplifications.
- Don't skip understanding activation functions; they're critical for network capability.

5. Now Try It

Exercise:
Imagine you're designing a single neuron to decide if a fruit is "ripe" based on its "color score" (0.0-1.0, 0=green, 1=red) and "softness score" (0.0-1.0, 0=hard, 1=squishy).

Given:
* Input fruit: color_score = 0.9 (quite red), softness_score = 0.7 (quite soft)
* Assume current learned weights: w_color = 0.6, w_softness = 0.4
* Assume current bias: b = -0.2
* Use the Sigmoid activation function: f(z) = 1 / (1 + e^-z)

What to do:
Calculate the raw weighted sum z and then the neuron's final output y_predicted (ripe/not ripe probability) for this fruit. You can use an online calculator or