Rank-Biserial Correlation

The standardized effect size reported for the wilcox_TOST procedure is the rank-biserial correlation. This is a fairly intuitive measure of effect size which has the same interpretation of the common language effect size (Kerby 2014). However, instead of assuming normality and equal variances, the rank-biserial correlation calculates the number of favorable (positive) and unfavorable (negative) pairs based on their respective ranks.

For the two sample case, the correlation is calculated as the proportion of favorable pairs minus the unfavorable pairs.

r_biserial = f_pairs − u_pairs

For the one sample or paired samples cases, the correlation is calculated with ties (values equal to zero) not being dropped. This provides a conservative estimate of the rank biserial correlation.

It is calculated in the following steps wherein z represents the values or difference between paired observations:

1. Calculate signed ranks:

r_j = −1 ⋅ sign(z_j) ⋅ rank(|z_j|) 2. Calculate the positive and negative sums:

$$ R_{+} = \sum_{1\le i \le n, \space z_i > 0}r_j $$

$$ R_{-} = \sum_{1\le i \le n, \space z_i < 0}r_j $$ 3. Determine the smaller of the two rank sums:

$$ T = min(R_{+}, \space R_{-}) $$

$$ S = \begin{cases} -4 & R_{+} \ge R_{-} \\ 4 & R_{+} < R_{-} \end{cases} $$ 4. Calculate rank-biserial correlation:

$$ r_{biserial} = S \cdot | \frac{\frac{T - \frac{(R_{+} + R_{-})}{2}}{n}}{n + 1} | $$

Confidence Intervals

The Fisher approximation is used to calculate the confidence intervals.

For paired samples, or one sample, the standard error is calculated as the following:

$$ SE_r = \sqrt{ \frac {(2 \cdot nd^3 + 3 \cdot nd^2 + nd) / 6} {(nd^2 + nd) / 2} } $$

wherein, nd represents the total number of observations (or pairs).

For independent samples, the standard error is calculated as the following:

$$ SE_r = \sqrt{\frac {(n1 + n2 + 1)} { (3 \cdot n1 \cdot n2)}} $$

The confidence intervals can then be calculated by transforming the estimate.

r_z = atanh(r_biserial)

Then the confidence interval can be calculated and back transformed.

r_CI = tanh(r_z ± Z_{(1 − α/2)} ⋅ SE_r)

Conversion to other effect sizes

Two other effect sizes can be calculated for non-parametric tests. First, there is the concordance probability, which is also known at the c-statistic, c-index, or probability of superiority¹. The c-statistic is converted from the correlation using the following formula:

$$ c = \frac{(r_{biserial} + 1)}{2} $$

The Wilcoxon-Mann-Whitney odds (O’Brien and Castelloe 2006), also known as the “Generalized Odds Ratio” (Agresti 1980), is calculated by converting the c-statistic using the following formula:

WMW_odds = e^logit(c)

Either effect size is available by simply modifying the ses argument for the wilcox_TOST function.

# Rank biserial
wilcox_TOST(formula = extra ~ group,
                      data = sleep,
                      paired = FALSE,
                      ses = "r",
                      eqb = .5)

## 
## Wilcoxon rank sum test with continuity correction
## 
## The equivalence test was non-significant W = 20.000, p = 8.94e-01
## The null hypothesis test was non-significant W = 25.500, p = 6.93e-02
## NHST: don't reject null significance hypothesis that the effect is equal to zero 
## TOST: don't reject null equivalence hypothesis
## 
## TOST Results 
##            Test Statistic p.value
## NHST                 25.5   0.069
## TOST Lower           34.0   0.894
## TOST Upper           20.0   0.013
## 
## Effect Sizes 
##                           Estimate               C.I. Conf. Level
## Median of Differences       -1.346       [-3.4, -0.1]         0.9
## Rank-Biserial Correlation   -0.490 [-0.7493, -0.1005]         0.9

# Odds

wilcox_TOST(formula = extra ~ group,
                      data = sleep,
                      paired = FALSE,
                      ses = "o",
                      eqb = .5)

## 
## Wilcoxon rank sum test with continuity correction
## 
## The equivalence test was non-significant W = 20.000, p = 8.94e-01
## The null hypothesis test was non-significant W = 25.500, p = 6.93e-02
## NHST: don't reject null significance hypothesis that the effect is equal to zero 
## TOST: don't reject null equivalence hypothesis
## 
## TOST Results 
##            Test Statistic p.value
## NHST                 25.5   0.069
## TOST Lower           34.0   0.894
## TOST Upper           20.0   0.013
## 
## Effect Sizes 
##                       Estimate             C.I. Conf. Level
## Median of Differences  -1.3464     [-3.4, -0.1]         0.9
## WMW Odds                0.3423 [0.1433, 0.8173]         0.9

# Concordance

wilcox_TOST(formula = extra ~ group,
                      data = sleep,
                      paired = FALSE,
                      ses = "c",
                      eqb = .5)

## 
## Wilcoxon rank sum test with continuity correction
## 
## The equivalence test was non-significant W = 20.000, p = 8.94e-01
## The null hypothesis test was non-significant W = 25.500, p = 6.93e-02
## NHST: don't reject null significance hypothesis that the effect is equal to zero 
## TOST: don't reject null equivalence hypothesis
## 
## TOST Results 
##            Test Statistic p.value
## NHST                 25.5   0.069
## TOST Lower           34.0   0.894
## TOST Upper           20.0   0.013
## 
## Effect Sizes 
##                       Estimate             C.I. Conf. Level
## Median of Differences   -1.346     [-3.4, -0.1]         0.9
## Concordance              0.255 [0.1254, 0.4497]         0.9

Basics of the Calculative Approach

In this section, I will quickly detail the calculative approach that underlies the Brunner-Munzel test in TOSTER.

A studentized test statistic can be calculated as:

$$ t = \sqrt{N} \cdot \frac{\hat p -p_{null}}{s} $$ Wherein the default null hypothesis p_null is typically 0.5 (50% probability of superiority is the default null), and σ_N refers the rank-based Brunner-Munzel standard error. The null can be modified therefore allowing for equivalence testing directly based on the relative effect. Additionally, for paired samples the probability of superiority is based on a hypothesis of exchangability and is not based on the differences scores³.

For more details on the calculative approach, I suggest reading Karch (2021). At small sample sizes, it is recommended that the permutation version of the test (perm = TRUE) be used rather than the basic test statistic approach.

Example

The interface for the function is very similar to the t.test function. The brunner_munzel function itself does not allow for equivalence tests, but you can set an alternative hypothesis for “two.sided”, “less”, or “greater”.

# studentized test
brunner_munzel(formula = extra ~ group,
                      data = sleep,
                      paired = FALSE)

## Sample size in at least one group is small. Permutation test (perm = TRUE) is highly recommended.

## 
##  two-sample Brunner-Munzel test
## 
## data:  extra by group
## t = -2.1447, df = 16.898, p-value = 0.04682
## alternative hypothesis: true relative effect is not equal to 0.5
## 95 percent confidence interval:
##  0.01387048 0.49612952
## sample estimates:
## p(X>Y) + .5*P(X=Y) 
##              0.255

# permutation
brunner_munzel(formula = extra ~ group,
                      data = sleep,
                      paired = FALSE,
               perm = TRUE)

## 
##  two-sample Brunner-Munzel permutation test
## 
## data:  extra by group
## t-observed = -2.1447, df = 16.898, p-value = 0.0482
## alternative hypothesis: true relative effect is not equal to 0.5
## 95 percent confidence interval:
##  0.00826466 0.49593164
## sample estimates:
## p(X>Y) + .5*P(X=Y) 
##              0.255

The simple_htest function allows TOST tests using a Brunner-Munzel test by setting the alternative to “equivalence” or “minimal.effect”. The equivalence bounds, based on the relative effect, can be set with the mu argument.

# permutation based Brunner-Munzel test of equivalence
simple_htest(formula = extra ~ group,
             test = "brunner",
             data = sleep,
             paired = FALSE,
             alternative = "equ",
             mu = .7,
             perm = TRUE)

## 
##  two-sample Brunner-Munzel permutation test
## 
## data:  extra by group
## t-observed = -3.8954, df = 16.898, p-value = 0.9974
## alternative hypothesis: equivalence
## null values:
## relative effect relative effect 
##             0.3             0.7 
## 90 percent confidence interval:
##  0.05824864 0.45564662
## sample estimates:
## p(X>Y) + .5*P(X=Y) 
##              0.255

Two Sample Algorithm

1. Form B bootstrap data sets from x* and y* wherein x* is sampled with replacement from x̃₁, x̃₂, ...x̃_n and y* is sampled with replacement from ỹ₁, ỹ₂, ...ỹ_n

2. t is then evaluated on each sample, but the mean of each sample (y or x) and the overall average (z) are subtracted from each

$$ t(z^{*b}) = \frac {(\bar x^*-\bar x - \bar z) - (\bar y^*-\bar y - \bar z)}{\sqrt {sd_y^*/n_y + sd_x^*/n_x}} $$

3. An approximate p-value can then be calculated as the number of bootstrapped results greater than the observed t-statistic from the sample.

$$ p_{boot} = \frac {\#t(z^{*b}) \ge t_{sample}}{B} $$

The same process is completed for the one sample case but with the one sample solution for the equation outlined by t(z^*b). The paired sample case in this bootstrap procedure is equivalent to the one sample solution because the test is based on the difference scores.

Example

We can use the sleep data to see the bootstrapped results. Notice that the plots show how the re-sampling via bootstrapping indicates the instability of Hedges’s d_z.

data('sleep')

test1 = boot_t_TOST(formula = extra ~ group,
                      data = sleep,
                      paired = TRUE,
                      eqb = .5,
                    R = 999)


print(test1)

## 
## Bootstrapped Paired t-test
## 
## The equivalence test was non-significant, t(9) = -2.777, p = 1e+00
## The null hypothesis test was significant, t(9) = -4.062, p = 0e+00
## NHST: reject null significance hypothesis that the effect is equal to zero 
## TOST: don't reject null equivalence hypothesis
## 
## TOST Results 
##                 t df p.value
## t-test     -4.062  9 < 0.001
## TOST Lower -2.777  9       1
## TOST Upper -5.348  9 < 0.001
## 
## Effect Sizes 
##               Estimate     SE               C.I. Conf. Level
## Raw             -1.580 0.3699 [-2.9073, -1.0601]         0.9
## Hedges's g(z)   -1.174 0.6937   [-1.405, 1.0716]         0.9
## Note: studentized bootstrap ci method utilized.

plot(test1)

log_TOST

For example, we could compare whether the cars of different transmissions are “equivalent” with regards to gas mileage. We can use the default equivalence bounds (eqb = 1.25).

log_TOST(
  mpg ~ am,
  data = mtcars
)

## 
## Log-transformed Welch Two Sample t-test
## 
## The equivalence test was non-significant, t(23.96) = -1.363, p = 9.07e-01
## The null hypothesis test was significant, t(23.96) = -3.826, p = 8.19e-04
## NHST: reject null significance hypothesis that the effect is equal to one 
## TOST: don't reject null equivalence hypothesis
## 
## TOST Results 
##                 t    df p.value
## t-test     -3.826 23.96 < 0.001
## TOST Lower -1.363 23.96   0.907
## TOST Upper -6.288 23.96 < 0.001
## 
## Effect Sizes 
##                  Estimate      SE               C.I. Conf. Level
## log(Means Ratio)  -0.3466 0.09061 [-0.5017, -0.1916]         0.9
## Means Ratio        0.7071      NA   [0.6055, 0.8256]         0.9

Note, that the function produces t-tests similar to the t_TOST function, but provides two effect sizes. The means ratio on the log scale (the scale of the test statistics), and the means ratio. The means ratio is missing standard error because the confidence intervals and estimate are simply the log scale results exponentiated.

Bootstrap + Log

However, it has been noted in the statistics literature that t-tests on the logarithmic scale can be biased, and it is recommended that bootstrapped tests be utilized instead. Therefore, the boot_log_TOST function can be utilized to perform a more precise test.

boot_log_TOST(
  mpg ~ am,
  data = mtcars,
  R = 499
)

## 
## Bootstrapped Log Welch Two Sample t-test
## 
## The equivalence test was non-significant, t(23.96) = -1.363, p = 9.48e-01
## The null hypothesis test was significant, t(23.96) = -3.826, p = 0e+00
## NHST: reject null significance hypothesis that the effect is equal to 1 
## TOST: don't reject null equivalence hypothesis
## 
## TOST Results 
##                 t    df p.value
## t-test     -3.826 23.96 < 0.001
## TOST Lower -1.363 23.96   0.948
## TOST Upper -6.288 23.96 < 0.001
## 
## Effect Sizes 
##                  Estimate      SE               C.I. Conf. Level
## log(Means Ratio)  -0.3466 0.08521 [-0.5058, -0.1564]         0.9
## Means Ratio        0.7071 0.05992    [0.603, 0.8552]         0.9
## Note: studentized bootstrap ci method utilized.