Introduction to Hypothesis Testing and T-tests

TL;DR

Hypothesis testing helps you decide if your data supports a claim about a population, using a statistical test like the t-test. You start by assuming no effect (the null hypothesis) and see if your data strongly contradicts it. The t-test is perfect for comparing means of one or two small groups when you don't know the population standard deviation.

1. The Mental Model

Think of hypothesis testing as a legal trial. You assume the defendant (the "null hypothesis") is innocent. You gather evidence (your data) to see if it's strong enough to prove guilt "beyond a reasonable doubt" (a low p-value).

2. The Core Material

Hypothesis testing is a formal procedure to evaluate two mutually exclusive statements about a population using sample data. These statements are the null hypothesis ($H_0$) and the alternative hypothesis ($H_a$ or $H_1$).

Understanding Hypotheses

Photo by Linken Van Zyl on Pexels

• Null Hypothesis ($H_0$): This is the statement of no effect, no difference, or no relationship. It's the status quo assumption. For example, "There is no difference in average height between group A and group B."
• Alternative Hypothesis ($H_a$): This is what you're trying to find evidence for. It contradicts the null hypothesis. For example, "There is a difference in average height between group A and group B."

The Steps of Hypothesis Testing

Photo by Tara Winstead on Pexels

Here's the general process:

graph TD
    A["Formulate $H_0$ and $H_a$"] --> B["Choose a Significance Level ($\alpha$)"];
    B --> C["Collect Sample Data"];
    C --> D["Calculate Test Statistic (e.g., t-statistic)"];
    D --> E{"Compare p-value to $\alpha$?"};
    E -- "p-value < $\alpha$" --> F["Reject $H_0$ (Support $H_a$)"];
    E -- "p-value $\ge$ $\alpha$" --> G["Fail to Reject $H_0$ (Not enough evidence for $H_a$)"];

1. Formulate $H_0$ and $H_a$: Clearly define your null and alternative hypotheses.
2. Choose a Significance Level ($\alpha$): This is your threshold for "reasonable doubt." Commonly, $\alpha = 0.05$ (5%) is used, meaning you're willing to accept a 5% chance of incorrectly rejecting a true null hypothesis (a Type I error).
3. Collect Sample Data: Gather data through appropriate sampling methods.
4. Calculate Test Statistic: This is a value derived from your sample data, used to evaluate the hypotheses. The t-statistic is one such test statistic.
5. Calculate the p-value: The p-value is the probability of observing sample data as extreme as, or more extreme than, what you got, assuming the null hypothesis is true. A small p-value means your data is unlikely if $H_0$ is true.
6. Make a Decision:
   • If p-value < $\alpha$: You reject the null hypothesis. This suggests your data provides enough evidence to support the alternative hypothesis.
   • If p-value $\ge$ $\alpha$: You fail to reject the null hypothesis. This means your data doesn't provide enough evidence to support the alternative hypothesis. You don't "accept" the null; you just didn't find sufficient evidence against it.

Introduction to T-tests

Photo by adrian vieriu on Pexels

T-tests are a family of hypothesis tests used when you want to compare means, especially when:
* Your sample size is small (generally n < 30, though it can be used for larger samples too).
* You don't know the population standard deviation.
* Your data is approximately normally distributed.

There are a few types of t-tests:

1. One-Sample T-test

Compares the mean of a single sample to a known or hypothesized population mean.
* Use case: Is the average weight of a new potato variety different from the standard 500g?

2. Independent Samples T-test (or Two-Sample T-test)

Compares the means of two independent groups.
* Use case: Is there a difference in average test scores between students taught with method A and students taught with method B?

3. Paired Samples T-test

Compares the means of two related groups or measurements from the same subjects under two different conditions.
* Use case: Is there a difference in blood pressure before and after taking a specific medication?

The formula for a general t-statistic looks something like this:

$t = \frac{\text{sample mean} - \text{hypothesized population mean}}{\text{standard error of the mean}}$

The "standard error of the mean" accounts for the variability in your sample. For different t-tests, this standard error calculation varies slightly.

3. Worked Example

Let's do an independent samples t-test using Python.

Scenario: A company wants to know if a new fertilizer (Fertilizer B) increases corn yield compared to their old fertilizer (Fertilizer A). They test each fertilizer on 10 different plots of land and record the yield in bushels per acre.

• Fertilizer A yields (bushels/acre): [105, 110, 108, 112, 107, 115, 109, 106, 111, 113]
• Fertilizer B yields (bushels/acre): [118, 120, 115, 122, 119, 125, 117, 121, 116, 123]

1. Formulate Hypotheses:
* $H_0$: There is no difference in average corn yield between Fertilizer A and Fertilizer B ($\mu_A = \mu_B$).
* $H_a$: Fertilizer B leads to a higher average corn yield than Fertilizer A ($\mu_B > \mu_A$). (This is a one-tailed test because we're specifically looking for higher yield from B).

2. Choose Significance Level: Let's set $\alpha = 0.05$.

3. Collect Data: (Already given)

4. Calculate Test Statistic (and p-value) using Python:

import numpy as np
from scipy import stats

fertilizer_A = np.array([105, 110, 108, 112, 107, 115, 109, 106, 111, 113])
fertilizer_B = np.array([118, 120, 115, 122, 119, 125, 117, 121, 116, 123])

# Perform independent t-test. The `equal_var=False` argument is often safer
# if you're not sure if the variances of the two groups are equal (Welch's t-test).
# stats.ttest_ind returns a two-tailed p-value by default.
t_statistic, p_value_two_tailed = stats.ttest_ind(fertilizer_B, fertilizer_A, equal_var=False)

# Since our H_a is one-tailed (B > A), we need to halve the p-value.
# We also need to ensure the t-statistic is in the expected direction (positive for B > A).
# If t_statistic were negative, a one-tailed test for B > A would mean p=1,
# as the effect is in the opposite direction.
if t_statistic > 0:
    p_value_one_tailed = p_value_two_tailed / 2
else:
    p_value_one_tailed = 1 - (p_value_two_tailed / 2) # Or just 1 if the t-stat was negative for a "greater than" H_a

print(f"Mean Fertilizer A: {np.mean(fertilizer_A):.2f}")
print(f"Mean Fertilizer B: {np.mean(fertilizer_B):.2f}")
print(f"T-statistic: {t_statistic:.3f}")
print(f"One-tailed P-value: {p_value_one_tailed:.4f}")

Output (will vary slightly based on scipy version, but similar numbers):

Mean Fertilizer A: 109.60
Mean Fertilizer B: 119.60
T-statistic: 7.071
One-tailed P-value: 0.0000

5. Make a Decision:
Our one-tailed p-value is approximately 0.0000, which is much less than our $\alpha$ of 0.05.

Conclusion: We reject the null hypothesis. There is statistically significant evidence to conclude that Fertilizer B leads to a higher average corn yield compared to Fertilizer A.

4. Key Takeaways

• Hypothesis testing helps you decide between a "no effect" scenario ($H_0$) and your research claim ($H_a$) based on data.
• The p-value tells you how likely your observed data is if the null hypothesis were true.
• A p-value less than your significance level ($\alpha$) leads you to reject $H_0$.
• T-tests are for comparing means when you don't know the population standard deviation, especially with smaller samples.
• The three main types of t-tests are one-sample, independent samples, and paired samples.
• Always check if your alternative hypothesis requires a one-tailed or two-tailed p-value interpretation.

Common Mistakes to Avoid:
- Confusing "failing to reject $H_0$" with "accepting $H_0$": You simply don't have enough evidence to disprove it, not that it's definitively true.
- Ignoring the context of the p-value: A statistically significant result doesn't always mean practically significant.
- Using a t-test when assumptions aren't met: For example, using it on highly skewed data without transformation, or for comparing more than two groups (ANOVA would be better).
- Misinterpreting p-value as the probability $H_0$ is true: It's the probability of observing your data (or more extreme) given $H_0$ is true.
- Incorrectly applying one-tailed vs. two-tailed tests: Use one-tailed only when you have a specific directional hypothesis before seeing the data.

5. Now Try It

Imagine a new teaching method claims to improve student scores. You have a class of 25 students taught with the new method, whose average score on a standardizedtest was 82 with a standard deviation of 8. The known national average for this test using the old method is 78.

Your task:
1. Formulate the null and alternative hypotheses to test if the new method significantly improves scores.
2. Choose an appropriate significance level ($\alpha$).
3. Decide which type of t-test is suitable.
4. Mentally (or with a calculator if you're keen!) consider what a low p-value would tell you about the new teaching method.

Success looks like: Clearly stating $H_0$ and $H_a$, identifying the one-sample t-test, and understanding that a p-value less than $\alpha$ would support the claim of improved scores.