Statistical Testing & Error
Type I, type II, power, and picking the right test
Hypothesis testing and the two errors
Null and alternative. The null hypothesis (H0) states that there is no difference or no association. The alternative hypothesis (H1) states that a difference exists. A statistical test tries to reject H0. H0 is never proven true, so the only possible conclusions are "reject" or "fail to reject". Failing to reject is not the same as proving there is no effect.
Type I error (α). A type I error rejects a null that is actually true, concluding that a difference exists when it does not. It is a false positive, such as deciding a drug works when it truly does not. Alpha is the significance level set before the study, conventionally 0.05, which accepts a 5% chance of a false positive. If p is less than α, H0 is rejected.
Type II error (β). A type II error fails to reject a null that is actually false, so a real difference is missed. It is a false negative, as when a truly effective drug shows no significant benefit in an underpowered trial. Beta is conventionally set at 0.20, a 20% chance of a false negative. The usual cause is a sample that is too small.
| Decision | H0 actually true (no real effect) | H0 actually false (real effect) |
|---|---|---|
| Reject H0 | Type I error (α): false positive | Correct: power (1 − β) |
| Fail to reject H0 | Correct decision | Type II error (β): false negative |
Power
Definition. Power = 1 − β, the probability of detecting an effect that really exists. Studies are conventionally designed for power of at least 0.80.
- Larger sample size (n): the lever the investigator controls most easily, and the usual way to boost power.
- Larger effect size: big differences are easier to detect than small ones.
- Larger α: a looser significance threshold raises power at the cost of more type I errors.
- Lower variance: more precise measurement makes a real difference stand out.
A "negative" small study may simply be underpowered. When a real effect is missed (for example p = 0.20 for a drug that truly works), think type II error from inadequate sample size.
p-values and confidence intervals
The p-value. The p-value is the probability of obtaining results at least as extreme as those observed if the null hypothesis were true. A p below 0.05 is called statistically significant and leads to rejecting H0. A p of 0.03 means a 3% chance of data this extreme under the null. It does not mean a 3% chance that the null is true or a 97% chance that the drug works. The p-value says nothing about effect size or clinical importance, and a tiny, trivial difference can be highly significant in a huge study.
Confidence intervals. A 95% confidence interval (CI) gives a range of plausible values for the true effect. If a study were repeated many times, 95% of the intervals built this way would contain the true value. A CI for a mean is roughly the mean ± 1.96 × SEM. A larger sample shrinks the SEM and narrows the CI.
- Difference (difference in means, absolute risk difference): a CI that includes 0 is not significant.
- Ratio (relative risk, odds ratio, hazard ratio): a CI that includes 1 is not significant.
- A 95% CI that excludes the null value corresponds to p below 0.05, so the CI and p-value tell the same story. A weight-loss difference of 2.0 kg with a 95% CI of −0.5 to 4.5 kg is therefore not significant.
Choosing a statistical test
| Test | Outcome data | Groups compared | Example |
|---|---|---|---|
| t-test | Continuous, normally distributed | Means of 2 groups | Mean blood pressure, drug vs placebo |
| Paired t-test | Continuous | 2 matched measurements | Same patients before and after treatment |
| ANOVA | Continuous | Means of 3 or more groups | Mean cholesterol across four diets |
| Chi-square | Categorical (counts, proportions) | 2 or more groups | Percentage cured in each arm |
| Fisher exact | Categorical | Small expected cell counts | Rare outcome in a small trial |
| Correlation (r) | Two continuous variables | Strength of linear association | Height vs weight |
Correlation. The correlation coefficient r measures the strength and direction of a linear relationship. It runs from −1 (perfect negative) through 0 (no linear relationship) to +1 (perfect positive). The closer |r| is to 1, the stronger the relationship. The sign gives only the direction. r² is the proportion of variance in one variable explained by the other.
Correlation is not causation. Confounding and reverse causation can both create an association. Causal inference draws on experimental evidence (RCTs) and on the Bradford Hill criteria, which include strength, consistency, temporality, dose-response (biological gradient), plausibility and coherence. Temporality, with the cause preceding the effect, is the one criterion that is essential.
Describing data: central tendency and the normal curve
Mean, median, mode. The mean is the arithmetic average and is pulled by outliers. The median is the middle value and resists outliers. The mode is the most frequent value. For skewed data with extreme values, such as hospital lengths of stay with a few 40-day admissions, the median best represents a typical case.
Skew. The mean is dragged toward the tail, and the tail's direction names the skew. In a positive (right) skew, mean > median > mode. In a negative (left) skew the order reverses: mean < median < mode.
Normal (Gaussian) distribution. The curve is symmetric and bell-shaped, and the mean, median and mode coincide.
SD vs SEM. The standard deviation describes the spread of individual values. The standard error of the mean, SEM = SD/√n, describes how precisely the sample mean estimates the population mean, and it shrinks as n grows.