Types of Two-Sample T-tests

TL;DR

Two-sample t-tests compare the means of two independent groups. You choose the right t-test based on whether the variances of the two groups are equal or unequal. Welch's t-test is generally safer when you're unsure about variance equality.

1. The Mental Model

Imagine you have two separate piles of data, like exam scores from two different classes. You want to know if the average score from one class is truly different from the average score of the other. T-tests help you answer that question.

2. The Core Material

When comparing the means of two independent groups, there are mainly two types of two-sample t-tests you'll encounter: the Pooled (Student's) t-test and Welch's t-test. The key difference between them lies in how they handle the assumption about the population variances of the two groups.

Pooled (Student's) Two-Sample T-test

Photo by This And No Internet 25 on Pexels

This test assumes that the population variances of the two groups are equal. This means you're assuming that the spread of data in both groups is roughly the same. If this assumption holds, you can "pool" (combine) the variance information from both samples to get a more robust estimate of the common population variance.

The formula for the pooled t-statistic involves calculating a pooled standard deviation ($s_p$) and then using it to find the t-value:

$t = \frac{(\bar{x}_1 - \bar{x}_2)}{s_p \sqrt{\frac{1}{n_1} + \frac{1}{n_2}}}$

Where:
* $\bar{x}_1$ and $\bar{x}_2$ are the sample means.
* $n_1$ and $n_2$ are the sample sizes.
* $s_p$ is the pooled standard deviation.

The degrees of freedom for this test are $df = n_1 + n_2 - 2$.

Welch's Two-Sample T-test

Photo by Stéf -b. on Pexels

Welch's t-test is more robust because it does not assume equal population variances. This is a big advantage because, in real-world data, the variances between two groups are often not equal. If the variances are actually equal, Welch's test still performs well. If they are unequal, Welch's test is much more accurate than the pooled t-test. Because of its flexibility, many statisticians recommend using Welch's t-test as the default.

The formula for the Welch's t-statistic is:

$t = \frac{(\bar{x}_1 - \bar{x}_2)}{\sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}}$

Where:
* $s_1^2$ and $s_2^2$ are the sample variances.

The degrees of freedom for Welch's test are calculated using a more complex formula called the Satterthwaite approximation, which results in fractional degrees of freedom (e.g., 23.7 instead of 24). Most software will calculate this for you automatically.

Choosing Between the Tests

Photo by Tara Winstead on Pexels

How do you decide which one to use?

graph TD
    A["Are you comparing means of two *independent* groups?"] -->|Yes| B{Do you assume equal variances?}
    A -->|No| C["Consider paired t-test or ANOVA"]
    B -->|Yes, assume equal variances| D["Use Pooled (Student's) t-test"]
    B -->|No, don't assume / unsure| E["Use Welch's t-test (Recommended)"]

While you can use tests like Levene's test or F-test to check for equal variances, these tests aren't always reliable, especially with small sample sizes. A common practice is to just use Welch's t-test by default because it doesn't require the assumption of equal variances and performs well even if variances are equal.

Paired T-test (Briefly)

Photo by Jan Kopřiva on Pexels

It's important to distinguish these from a paired t-test. A paired t-test is used when you have two measurements from the same individuals or matched pairs (e.g., before and after treatment scores for the same person). The two-sample t-tests discussed above are for independent groups.

3. Worked Example

Let's say you're a teacher and want to see if a new teaching method (Group 1) results in different exam scores than the old method (Group 2). You collect the following hypothetical scores:

Group 1 (New Method): 85, 88, 90, 82, 87 (n1=5)
Group 2 (Old Method): 75, 78, 80, 72, 77 (n2=5)

First, let's calculate the basic statistics for each group:

• Group 1:

  • Mean ($\bar{x}_1$): (85+88+90+82+87)/5 = 86.4
  • Variance ($s_1^2$): 8.8 (calculated using the formula $\sum(x_i - \bar{x})^2 / (n-1)$)
  • Standard Deviation ($s_1$): $\sqrt{8.8} \approx 2.966$

• Group 2:

  • Mean ($\bar{x}_2$): (75+78+80+72+77)/5 = 76.4
  • Variance ($s_2^2$): 9.3 (calculated similarly)
  • Standard Deviation ($s_2$): $\sqrt{9.3} \approx 3.050$

Now, let's perform a Welch's t-test (since we're not assuming equal variances, and it's generally a good default).

Using the Welch's t-test formula:
$t = \frac{(\bar{x}_1 - \bar{x}_2)}{\sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}}$
$t = \frac{(86.4 - 76.4)}{\sqrt{\frac{8.8}{5} + \frac{9.3}{5}}}$
$t = \frac{10}{\sqrt{1.76 + 1.86}}$
$t = \frac{10}{\sqrt{3.62}}$
$t = \frac{10}{1.9026} \approx 5.256$

So, the calculated t-statistic is approximately 5.256. Most statistical software would then calculate the degrees of freedom (around 7.9 for this example) and give you a p-value. A t-value this large, especially with 5 samples per group, would almost certainly lead to a very small p-value, indicating a statistically significant difference between the two teaching methods.

4. Key Takeaways

• Two-sample t-tests compare the means of two independent groups.
• The Pooled (Student's) t-test assumes the population variances of the two groups are equal.
• Welch's t-test does not assume equal population variances and is more robust.
• Many statisticians recommend using Welch's t-test as a default because it works well even if variances are equal and is more accurate if they're not.
• A paired t-test is used for dependent (matched) samples, not independent groups.
• Statistical software handles the complex calculations for degrees of freedom and p-values automatically.

Common Mistakes to Avoid

• Using a two-sample t-test for paired data: If your data comes from the same subjects measured twice, you need a paired t-test.
• Blindly assuming equal variances: It's safer to use Welch's t-test unless you have a very strong theoretical reason to assume equal variances.
• Misinterpreting significance: A significant p-value only tells you there's a statistical difference, not necessarily a practically important one.
• Small sample sizes: T-tests perform better with larger sample sizes; small samples can lead to less reliable results.

5. Now Try It

You're analyzing the effectiveness of two different fertilizers (Fertilizer A and Fertilizer B) on plant growth. You measure the height of 10 plants treated with Fertilizer A and 12 plants treated with Fertilizer B. You calculate the mean height for Fertilizer A to be 25 cm (variance = 5 cm$^2$) and for Fertilizer B to be 22 cm (variance = 10 cm$^2$).

Your task: Calculate the t-statistic using Welch's t-test to determine if there's a difference in average plant height between the two fertilizers.

What success looks like: You'll have a single t-value, indicating how many standard errors the means are apart. You don't need to find the p-value or degrees of freedom for this exercise, just the t-statistic itself.