Introduction to Comparative Statistical Tests

TL;DR

Comparative statistical tests help you figure out if differences observed between groups or conditions are likely real or just due to chance. They're essential for drawing valid conclusions from your biometry data, allowing you to quantify uncertainty and make informed decisions. Choosing the right test depends on your data type, experimental design, and specific research question.

1. The Mental Model

Imagine you've measured something (like plant height or gene expression) in two different groups. Comparative tests help you decide if the average difference you see between those groups is big enough to say they're truly different, or if it's just random variation.

2. The Core Material

In biometry, you'll often want to compare things. Maybe you're comparing the effectiveness of two different fertilizers on crop yield, or the average weight of two animal populations. Comparative statistical tests provide a framework to formally assess these differences.

The core idea revolves around hypothesis testing:
* Null Hypothesis (H₀): There is no significant difference between the groups you're comparing. Any observed difference is due to random chance.
* Alternative Hypothesis (H₁): There is a significant difference between the groups. The observed difference is unlikely to be due to chance.

You collect data, run a statistical test, and get a p-value. The p-value tells you the probability of observing your data (or more extreme data) if the null hypothesis were true.

• If p-value < α (your chosen significance level, usually 0.05), you reject H₀. This means the difference is considered statistically significant.
• If p-value ≥ α, you fail to reject H₀. This means you don't have enough evidence to claim a significant difference.

Key Factors for Choosing a Test

Photo by Mike Hayward on Pexels

Choosing the right comparative test is crucial. Here are the main things to consider:

1. Type of Data:

   • Continuous (Interval/Ratio): Measurements like height, weight, temperature. Can take any value within a range.
   • Categorical (Nominal/Ordinal): Groups or categories.
     • Nominal: No inherent order (e.g., male/female, species A/B/C).
     • Ordinal: Has an order, but unequal intervals (e.g., small/medium/large, disease stage I/II/III).

2. Number of Groups/Conditions: Are you comparing two groups, or more than two?

3. Independence of Samples:

   • Independent (Unpaired): Data from one group doesn't influence the other (e.g., comparing two different groups of plants).
   • Dependent (Paired): Data points are related (e.g., measuring the same plants before and after treatment, or matched pairs).

4. Assumptions: Many tests have underlying assumptions about your data (e.g., normality, equal variances). If these aren't met, you might need a non-parametric alternative.

Common Comparative Tests (and when to use them)

Photo by Ivan S on Pexels

graph TD
    A["Comparative Test Selection"] --> B{"Compare 2 Groups or >2?"}
    B -- "2 Groups" --> C{"Data Type?"}
    B -- ">2 Groups" --> F{"Data Type?"}

    C -- "Continuous" --> D{"Samples Independent/Paired?"}
    D -- "Independent" --> D1{"Normal?"}
    D1 -- "Yes" --> D2["Independent t-test"]
    D1 -- "No" --> D3["Mann-Whitney U test"]
    D -- "Paired" --> D4{"Normal?"}
    D4 -- "Yes" --> D5["Paired t-test"]
    D4 -- "No" --> D6["Wilcoxon Signed-Rank test"]

    C -- "Categorical" --> E{"Expected Counts < 5?"}
    E -- "No" --> E1["Chi-squared test"]
    E -- "Yes" --> E2["Fisher's Exact test"]

    F -- "Continuous" --> G{"Normal & Equal Variance?"}
    G -- "Yes" --> G1["ANOVA"]
    G -- "No" --> G2["Kruskal-Wallis test"]

    F -- "Categorical" --> H["Chi-squared test (or exact tests)"]

• t-tests: Used to compare the means of two groups with continuous data.
  • Independent t-test: For two independent groups (e.g., treatment vs. control).
  • Paired t-test: For two related groups (e.g., before vs. after a treatment on the same subjects).
  • Assumes: Data is normally distributed, and independent t-test assumes equal variances.
• ANOVA (Analysis of Variance): Used to compare the means of more than two groups with continuous data. It tells you if there's any significant difference among the group means. If you find a difference, you'll often follow up with post-hoc tests (like Tukey's HSD) to see which specific groups differ.
  • Assumes: Data is normally distributed and variances are equal.
• Chi-squared test ($\chi^2$): Used to compare categorical data, looking at counts or frequencies in different categories. It checks if observed frequencies differ significantly from expected frequencies.
• Non-parametric tests: These are alternatives when your data doesn't meet the assumptions of parametric tests (like normality).
  • Mann-Whitney U test: Non-parametric alternative to the independent t-test.
  • Wilcoxon Signed-Rank test: Non-parametric alternative to the paired t-test.
  • Kruskal-Wallis test: Non-parametric alternative to one-way ANOVA.
  • Fisher's Exact Test: Used for categorical data, especially when expected counts are small, where Chi-squared might be unreliable.

3. Worked Example

Let's say you're a plant scientist testing two new fertilizers, "Fertilizer A" and "Fertilizer B", against a "Control" group (no fertilizer). You grow 10 plants for each group and measure their final height in cm.

Your data might look something like this:

Here's how you'd approach it:

1. Research Question: Do the different fertilizers (or no fertilizer) affect plant height?
2. Data Type: Plant height is continuous.
3. Number of Groups: You have three groups (Control, Fertilizer A, Fertilizer B).
4. Test Choice: Since you have continuous data and more than two independent groups, ANOVA is the appropriate primary test. You'd first check for normality and equal variances. If satisfied, you'd run ANOVA.

Using R:

# Sample data (for demonstration)
control <- c(15.2, 14.8, 16.1, 15.5, 15.0, 14.9, 16.0, 15.3, 15.7, 15.4)
fert_A <- c(22.1, 23.5, 21.9, 24.0, 22.8, 23.1, 22.5, 23.8, 22.9, 23.3)
fert_B <- c(20.5, 21.1, 19.8, 22.0, 20.3, 21.5, 20.9, 21.7, 20.1, 21.8)

# Combine into a single data frame
plant_data <- data.frame(
  height = c(control, fert_A, fert_B),
  group = factor(rep(c("Control", "Fertilizer A", "Fertilizer B"), each = 10))
)

# Run ANOVA
anova_result <- aov(height ~ group, data = plant_data)
summary(anova_result)

The summary(anova_result) output would show a p-value for the group effect. If this p-value is less than 0.05 (e.g., p < 2e-16 ***), you would reject the null hypothesis and conclude there's a significant difference in plant height among the groups. You'd then likely run a post-hoc test (like Tukey's HSD) to see which specific fertilizer group(s) differ from each other or the control.

# If ANOVA is significant, run post-hoc test (e.g., Tukey HSD)
TukeyHSD(anova_result)

This would show pairwise comparisons (e.g., "Fertilizer A vs. Control", "Fertilizer B vs. Control", "Fertilizer A vs. Fertilizer B") and their respective adjusted p-values, telling you exactly where the differences lie.

4. Key Takeaways

• Comparative statistical tests help you determine if observed differences between groups are statistically significant or just random chance.
• The null hypothesis (H₀) assumes no difference, while the alternative hypothesis (H₁) assumes there is a difference.
• The p-value indicates the probability of observing your data if the null hypothesis were true; typically, p < 0.05 leads to rejecting H₀.
• Your choice of test depends critically on your data type (continuous vs. categorical), the number of groups, and whether samples are independent or paired.
• T-tests compare two group means, ANOVA compares more than two group means, and Chi-squared tests compare categorical frequencies.
• Non-parametric tests are used when your data don't meet the assumptions of parametric tests, like normality.

Common Mistakes to Avoid:
- Using a t-test for more than two groups: Thisleads to an inflated Type I error rate (false positives). Always use ANOVA for three or more groups with continuous data.
- Ignoring assumptions: Applying parametric tests (like t-tests or ANOVA) to non-normally distributed data without checking assumptions or using non-parametric alternatives can lead to incorrect conclusions.
- Confusing correlation with causation: Comparative tests show association/difference, but they don't automatically imply one variable causes another without proper experimental design.
- Misinterpreting p-values: A non-significant p-value doesn't prove the null hypothesis is true; it just means you don't have enough evidence to reject it.

5. Now Try It

You're studying the effect of three different diets (Diet X, Diet Y, and Control) on the weight gain (in grams) of lab mice over a month. You have 8 mice per diet group. After one month, you record their weight gain.

1. State your null and alternative hypotheses for this experiment.
2. Identify the appropriate statistical test to determine if there's a significant difference in weight gain among the three diet groups. Justify your choice based on data type and number of groups.
3. What assumption(s) would you need to check before running this test?
4. Imagine your chosen test yields a p-value of 0.001. What would be your conclusion regarding the diets' effects on weight gain?

Success looks like: You correctly identify the hypotheses and the primary test (and a potential follow-up test), recognize key assumptions, and interpret the p-value in context.