e-Statistics

Multiple Comparisons

When the null hypothesis $H_0: \alpha_1 = \cdots = \alpha_k$ is rejected, the question arises on which pairs of $\alpha_i$ 's are really different. Such investigation is carried out by an analysis of all pairwise differences, called Multiple comparisons.

The data can be arranged in a form of grouped data which is "grouped by" the column of categorical variable indicating factor levels.

Summary statistics must be calculated one level at a time.

Here we construct the confidence intervals simultaneously for all pairwise differences $\theta_{ij} = \alpha_i - \alpha_j$ . Then the point estimate of $\theta_{ij}$ and the standard error $\sigma_{ij}$ are obtained respectively by $\hat{\theta}_{ij} = \bar{X}_{i\cdot} - \bar{X}_{j\cdot}$ and $\hat{\sigma}_{ij}^2 = MS_{\mbox{error}}\times\left(\frac{1}{n_i} + \frac{1}{n_j}\right)$ . Various methods are proposed to find a critical point $\rho$ so that we can obtain the confidence intervals

$\displaystyle \left( \hat{\theta}_{ij} - \hat{\sigma}_{ij} \rho,\: \hat{\theta}_{ij} - \hat{\sigma}_{ij} \rho \right)$ for every pairwise difference $\theta_{ij}$

in which overall probability is no less than $1 - \alpha =$

Tukey's method. Tukey introduced a studentized range distribution

$\displaystyle W = \max_{1 \le i, j \le k} \frac{\vert Z_i - Z_j\vert}{\sqrt{\chi^2 / (n - k)}}$
where 's are independent standard normal and $\chi^2$ is $\chi^2$ -distribution with degrees of freedom, independent of 's. Then we use

$\displaystyle \rho_{ij} = q(\alpha) \sqrt{\displaystyle\sum_{c=1}^k \frac{1}{n_c} \left/ k \left(\frac{1}{n_i} + \frac{1}{n_j}\right)\right.}$
where $q(\alpha)$ is given as the $(1-\alpha)$ -th percentile of the distribution for .
Scheffe's method. As a special case of Scheffe's S Method, we can obtain

$\displaystyle \rho = \sqrt{(k-1) F_{\alpha,k-1,n-k}}$
Bonferroni's method. The Boole's inequality implies that we can choose $\rho = t_{\beta/2,n - k}$ with $\beta = \alpha\left/ \binom{k}{2}\right.$ . Here $t_{\alpha,n}$ is the $(1-\alpha)$ -th percentile for student -distribution with degrees of freedom.

The significance tests for pairwise differences $\theta_{ij} = \alpha_i - \alpha_j$ are then performed in the following manners: If the confidence interval for $\theta_{ij}$ does not contain zero, then we reject “ $\alpha_i = \alpha_j$ .” The larger the critical point $\rho$ is, the harder it is to reject “ $\alpha_i = \alpha_j$ ” (that is, the more conservative).

Remark on simultaneity. Whether we should conduct the analysis of variance (AOV) before multiple comparisons (MC) is a little sensitive issue, since it creates simultaneity of AOV and MC. However, because of the duality between the AOV and the Scheffe's S Method, a systematic approach popular among statistician requires the AOV in order to proceed with the MC. Also note that when we attempt different multiple comparison procedures (for example, Scheffe's and Tukey–Kramer's methods), naturally we do not discuss simultaneity of these procedures and understandably their conclusions may be inconsistent (for example, Scheffe's method may not detect any significance while Tukey–Kramer's method indicates significances for some pairs).