Conditional Probability and Bayes' Theorem

TL;DR

Conditional probability tells you the chance of one event happening given that another event has already occurred. Bayes' Theorem is a powerful way to update your beliefs about an event's probability when new evidence comes to light. These tools help you make more informed decisions by considering what you already know.

1. The Mental Model

Think of conditional probability as zooming in on a specific part of your sample space. Instead of looking at all possibilities, you're only considering the outcomes where a certain condition has already been met. Bayes' Theorem then lets you flip that around, figuring out a "cause" given an "effect."

2. The Core Material

Understanding Conditional Probability

Photo by DS stories on Pexels

Conditional probability is the probability of event A happening, given that event B has already happened. We write this as P(A|B), which you can read as "the probability of A given B."

The formula for conditional probability is:

P(A|B) = P(A and B) / P(B)

Where:
* P(A|B) is the probability of A occurring given B has occurred.
* P(A and B) is the probability that both A and B occur.
* P(B) is the probability of B occurring.

It's crucial that P(B) is not zero, because you can't condition on an impossible event.

Let's use a simple example: imagine you draw a card from a standard 52-card deck.
* Let A be the event of drawing a King.
* Let B be the event of drawing a Face Card (King, Queen, Jack).

What's P(King | Face Card)? You're no longer considering all 52 cards; you're only looking at the 12 face cards. Out of those 12, 4 are Kings. So, P(King | Face Card) = 4/12 = 1/3.

Using the formula:
* P(King and Face Card) = P(King) because all Kings are Face Cards. So, P(King and Face Card) = 4/52.
* P(Face Card) = 12/52.
* P(King | Face Card) = (4/52) / (12/52) = 4/12 = 1/3. The formula works!

Introducing Bayes' Theorem

Photo by Jonathan Borba on Pexels

Bayes' Theorem takes conditional probability a step further. It's particularly useful when you want to find P(A|B) but you might only know P(B|A) and the individual probabilities P(A) and P(B). It essentially tells you how to update the probability of a hypothesis (A) when you observe new evidence (B).

The formula for Bayes' Theorem is:

P(A|B) = [P(B|A) * P(A)] / P(B)

Let's break down the terms:
* P(A|B): Posterior Probability - This is what you want to find: the probability of your hypothesis A being true, given the evidence B. This is your "updated belief."
* P(B|A): Likelihood - The probability of observing the evidence B, given that your hypothesis A is true.
* P(A): Prior Probability - Your initial belief about the probability of hypothesis A being true, before you see any new evidence.
* P(B): Evidence Probability - The overall probability of observing the evidence B. You can often calculate this using the Law of Total Probability: P(B) = P(B|A)P(A) + P(B|not A)P(not A).

Here's a visual of how these ideas connect:

graph TD
    A["Event A (Hypothesis)"]
    B["Event B (Evidence)"]

    subgraph Conditional Probability
        A_and_B["P(A and B)"]
        P_B["P(B)"]
        P_A_given_B["P(A|B) = P(A and B) / P(B)"]
        A_and_B --> P_A_given_B
        P_B --> P_A_given_B
    end

    subgraph Bayes' Theorem
        P_A_given_B_Bayes["P(A|B)"]
        P_B_given_A["P(B|A) (Likelihood)"]
        P_A["P(A) (Prior)"]
        P_B_overall["P(B) (Evidence Probability)"]

        P_B_given_A --> P_A_given_B_Bayes
        P_A --> P_A_given_B_Bayes
        P_B_overall --> P_A_given_B_Bayes
        style P_A_given_B_Bayes fill:#f9f,stroke:#333,stroke-width:2px
    end

    classDef default fill:#eee,stroke:#333,stroke-width:1px;

Proportional Probability (Often Implied)

Photo by DS stories on Pexels

When dealing with conditional probability and Bayes' Theorem, you're implicitly working with proportional probabilities. The P(B) in the denominator of P(A|B) acts as a normalising factor. It ensures your conditional probability sums to 1. When you compare P(A|B) to P(A), you're seeing how the evidence B has proportionally changed your belief in A.

3. Worked Example

Let's say a rare disease (D) affects 1% of the population. A test (T) for this disease is quite good:
* If you have the disease, the test is positive 90% of the time (P(T+|D) = 0.90).
* If you don't have the disease, the test is positive 5% of the time (a false positive, P(T+|not D) = 0.05).

You just tested positive. What's the probability you actually have the disease? P(D|T+)?

Here's what we know:
* P(D) = 0.01 (Prior probability of having the disease)
* P(not D) = 1 - P(D) = 0.99
* P(T+|D) = 0.90 (Likelihood of a positive test if you have the disease)
* P(T+|not D) = 0.05 (Likelihood of a positive test if you don't have the disease)

We want to find P(D|T+). Using Bayes' Theorem:
P(D|T+) = [P(T+|D) * P(D)] / P(T+)

First, we need P(T+), the overall probability of testing positive. We can find this using the Law of Total Probability:
P(T+) = P(T+|D)P(D) + P(T+|not D)P(not D)
P(T+) = (0.90 * 0.01) + (0.05 * 0.99)
P(T+) = 0.009 + 0.0495
P(T+) = 0.0585

Now, plug this back into Bayes' Theorem:
P(D|T+) = (0.90 * 0.01) / 0.0585
P(D|T+) = 0.009 / 0.0585
P(D|T+) ≈ 0.1538

So, even with a positive test, there's only about a 15.38% chance that you actually have the disease. This often surprises people and highlights the importance of Bayes' Theorem, especially with rare conditions. The high rate of false positives in the general (healthy) population significantly impacts the posterior probability.

4. Key Takeaways

• Conditional probability, P(A|B), calculates the likelihood of A happening given B has already happened.
• The formula for conditional probability is P(A|B) = P(A and B) / P(B).
• Bayes' Theorem, P(A|B) = [P(B|A) * P(A)] / P(B), updates your belief in a hypothesis (A) given new evidence (B).
• P(A) is your prior belief, P(B|A) is the likelihood of evidence given the hypothesis, and P(A|B) is your updated posterior belief.
• The denominator P(B) is the total probability of the evidence occurring, often found using the Law of Total Probability.
• Bayes' Theorem is particularly powerful for understanding how new information changes the probability of an event, especially in medical testing or forensic analysis.

Common Mistakes to Avoid:
- Confusing P(A|B) with P(B|A): These are usually not the same thing. Don't swap them!
- Forgetting the denominator P(B) in Bayes' Theorem: This term is crucial for normalising your probability and often requires careful calculation.
- Ignoring the prior probability: Your initial belief P(A) significantly impacts the final P(A|B).
- Misinterpreting results: A low posterior probability even with a "good" test can occur if the prior probability of the condition is very low.

5. Now Try It

A factory produces light bulbs, and 2% are defective. There are two machines, M1 and M2, that produce these bulbs. Machine M1 produces 60% of the bulbs, and Machine M2 produces 40%. It's known that 3% of the bulbs from M1 are defective, and 1% of the bulbs from M2 are defective.

You pick a bulb at random, and it's defective. What's the probability it came from Machine M1? (i.e., find P(M1 | Defective)).

Success looks like: You should be able to clearly identify the prior probabilities, the likelihoods, calculate the overall probability of a defective bulb, and then apply Bayes' Theorem correctly to arrive at the final answer. You should get a probability that makes sense in the context of the problem (e.g., if M1 makes more bulbs and has a higher defect rate, you'd expect P(M1|Defective) to be higher than P(M2|Defective)).