Neural Networks & Deep Learning Foundations

TL;DR

Neural Networks are computational models inspired by the brain, designed to learn patterns from data. Deep Learning uses neural networks with many layers (deep architectures) to solve complex problems like image recognition and natural language processing. Understanding their foundational components, like neurons and activation functions, is key to building and training these powerful models.

1. The Mental Model

Think of a neural network as a team of interconnected "decision-makers" (neurons). Each decision-maker takes information, processes it, and passes it on. By working together in layers, they can learn to recognize complex relationships and make sophisticated predictions.

2. The Core Material

At its heart, a neural network is a series of layers, each made up of individual neurons.

Neurons (Perceptrons)

Photo by Google DeepMind on Pexels

A neuron is the basic building block. It receives inputs, multiplies each input by a specific weight, sums these weighted inputs, adds a bias, and then applies an activation function to produce an output.

Here's the math for a single neuron:
$y = f(\sum_{i=1}^{n} (x_i \cdot w_i) + b)$

Where:
* $x_i$: input values
* $w_i$: weights associated with each input
* $b$: bias term
* $\sum$: summation
* $f$: activation function
* $y$: output of the neuron

Weights and Biases

Photo by Екатерина Глущенко on Pexels

Weights determine the strength of the connection between neurons. A higher weight means that input has a stronger influence on the neuron's output. Biases allow the activation function to shift, giving the neuron more flexibility to model patterns. During training, the network adjusts these weights and biases to minimize prediction errors.

Activation Functions

Photo by Shubham Dhage on Pexels

Activation functions introduce non-linearity into the network, allowing it to learn more complex patterns than a simple linear model could. Without them, stacking layers would just result in another linear function.

Common activation functions:
* Sigmoid: Squashes output to a range between 0 and 1. Good for binary classification output layers.
* ReLU (Rectified Linear Unit): $f(x) = \max(0, x)$. Very popular due to its computational efficiency and ability to mitigate vanishing gradient problems.
* Softmax: Used in the output layer for multi-class classification, converting outputs into probabilities that sum to 1.

Layers

Neural networks are organized into layers:
1. Input Layer: Receives the raw data. The number of neurons equals the number of features in your dataset.
2. Hidden Layers: One or more layers between the input and output. These layers learn increasingly complex representations of the data. "Deep Learning" refers to networks with many hidden layers.
3. Output Layer: Produces the network's final prediction. The number of neurons depends on the task (e.g., 1 for binary classification, multiple for multi-class classification or regression).

Here's how these components flow together in a simple network:

graph TD
    A["Input Layer"] --> B("Hidden Layer 1 (Neurons + Activations)")
    B --> C("Hidden Layer 2 (Neurons + Activations)")
    C --> D("Output Layer (Neurons + Activations)")

Forward Propagation

Photo by Google DeepMind on Pexels

This is the process where input data flows through the network, from the input layer, through hidden layers, to the output layer, making a prediction. Each neuron calculates its output based on its inputs, weights, bias, and activation function.

Backpropagation

After forward propagation, the network's prediction is compared to the actual target value, and an error (loss) is calculated. Backpropagation is the algorithm used to adjust the network's weights and biases to reduce this error. It works by propagating the error backward through the network, layer by layer, and updating parameters using an optimization algorithm (like Gradient Descent). Gradient Descent iteratively adjusts parameters in the direction that minimizes the loss function.

3. Worked Example

Let's trace a forward pass through a very simple neuron.

Suppose we have a single neuron with two inputs ($x_1=0.5$, $x_2=0.8$), corresponding weights ($w_1=0.3$, $w_2=0.7$), a bias ($b=-0.2$), and a ReLU activation function.

1. Calculate the weighted sum plus bias:
   $z = (x_1 \cdot w_1) + (x_2 \cdot w_2) + b$
   $z = (0.5 \cdot 0.3) + (0.8 \cdot 0.7) + (-0.2)$
   $z = 0.15 + 0.56 - 0.2$
   $z = 0.71 - 0.2$
   $z = 0.51$

2. Apply the ReLU activation function:
   $y = \max(0, z)$
   $y = \max(0, 0.51)$
   $y = 0.51$

So, the output of this neuron for the given inputs and parameters is 0.51. In a real network, this output would then become an input to the next layer's neurons.

4. Key Takeaways

• Neural networks learn by adjusting weights and biases to map inputs to desired outputs.
• Activation functions introduce non-linearity, allowing networks to learn complex relationships.
• Deep Learning refers to neural networks with multiple hidden layers, enabling them to learn hierarchical features.
• Forward propagation calculates the network's output, while backpropagation adjusts parameters based on error.
• Gradient Descent is a common optimization algorithm used with backpropagation to minimize the loss.
• Understanding the role of each component (neurons, weights, biases, activations, layers) is fundamental.

Common Mistakes to Avoid:
* Forgetting that activation functions are crucial for learning non-linear patterns.
* Confusing forward propagation (prediction) with backpropagation (learning/updating).
* Not understanding that weights and biases are the "learnable" parameters of the network.
* Thinking a deeper network is always better; sometimes simpler models perform just as well or better.

5. Now Try It

Task: Design a simple neural network architecture for a binary classification problem (e.g., predicting if an email is spam or not spam) with 10 input features.

Steps:
1. Determine the number of neurons in the input layer.
2. Propose a simple hidden layer structure (number of layers, number of neurons per layer, and their activation functions).
3. Specify the output layer structure and its activation function.
4. Briefly explain why you chose those specific activation functions for the hidden and output layers.

Success looks like: A clear description of the input, hidden, and output layers, including neuron counts and chosen activation functions, with a concise justification for the activation functions.