Variances Assumed Unequal: Cohen’s d(av)

For this calculation, the denominator is simply the square root of the average variance.

$$ s_{av} = \sqrt \frac {s_{1}^2 + s_{2}^2}{2} $$

The SMD, Cohen’s d(av), is then calculated as the following:

$$ d_{av} = \frac {\bar{x}_1 - \bar{x}_2} {s_{av}} $$

Note: the x with the bar above it (pronounced as “x-bar”) refers the the means of group 1 and 2 respectively.

The degrees of freedom for Cohen’s d(av), derived from Delacre et al. (2021), is the following:

$$ df = \frac{(n_1-1)(n_2-1)(s_1^2+s_2^2)^2}{(n_2-1) \cdot s_1^4+(n_1-1) \cdot s_2^4} $$

The non-centrality parameter (λ) is calculated as the following:

1. Under the Cousineau and Goulet-Pelletier (2021) method (smd_ci = "goulet"):

$$ \lambda = d_{av} \times \sqrt{\frac{n_1 \cdot n_2(\sigma^2_1+\sigma^2_2)}{2 \cdot (n_2 \cdot \sigma^2_1+n_1 \cdot \sigma^2_2)}} $$

2. Under all other methods (nct, t, or z):

$$ \lambda = \frac{2 \cdot (n_2 \cdot \sigma_1^2 + n_1 \cdot \sigma_2^2)} {n_1 \cdot n_2 \cdot (\sigma_1^2 + \sigma_2^2)} $$ The standard error (σ) of Cohen’s d(av) is calculated as the following from Bonett (2009)³:

$$ \sigma_{SMD} = \sqrt{d^2 \cdot (\frac{s_1^4}{n_1-1}+\frac{s_2^4}{n_2-1}) \div 8 \cdot s_{av}^4 + (\frac{s_1^2}{n_1-1}+\frac{s_2^2}{n_2-1}) \div s_{av}^2} $$

Variances Assumed Equal: Cohen’s d

For this calculation, the denominator is simply the pooled standard deviation.

$$ s_{p} = \sqrt \frac {(n_{1} - 1)s_{1}^2 + (n_{2} - 1)s_{2}^2}{n_{1} + n_{2} - 2} $$

$$ d = \frac {\bar{x}_1 - \bar{x}_2} {s_{p}} $$

The degrees of freedom for Cohen’s d is the following:

df = n₁ + n₂ − 2

The non-centrality parameter (λ) is calculated as the following:

1. Under the Cousineau and Goulet-Pelletier (2021) method (smd_ci = "goulet"):

$$ \lambda = d \cdot \sqrt \frac{\tilde n}{2} $$

wherein, ñ is the harmonic mean of the 2 sample sizes which is calculated as the following:

$$ \tilde n = \frac{2 \cdot n_1 \cdot n_2}{n_1 + n_2} $$

2. Under all other methods (nct, t, or z):

$$ \lambda = \frac{1}{n_1} +\frac{1}{n_2} $$

The standard error (σ) of Cohen’s d is calculated as the following:

1. Under the Cousineau and Goulet-Pelletier (2021) method (smd_ci = "goulet"):

$$ \sigma_{SMD} = \sqrt{\frac{df}{df-2} \cdot \frac{2}{\tilde n} (1+d^2 \cdot \frac{\tilde n}{2}) -\frac{d^2}{J}} $$

2. Under all other methods (nct, t, or z) it is calculated using the “unbiased” estimate from Viechtbauer (2007) (table 1; equation 26)⁴:

$$ \sigma_{SMD} = \sqrt{\frac{1}{n_1} + \frac{1}{n_2} + (1 - \frac{df-2}{df \cdot J^2}) \cdot d_s^2} $$

Glass’s Delta

For this calculation, the denominator is simply the standard deviation of one of the groups (x for glass = "glass1", or y for glass = "glass2".

$$ s_{c} = SD_{control \space group} $$

$$ d = \frac {\bar{x}_1 - \bar{x}_2} {s_{c}} $$

The degrees of freedom for Glass’s delta is the following:

df = n_c − 1

The non-centrality parameter (λ) is calculated as the following:

1. Under the Cousineau and Goulet-Pelletier (2021) method (smd_ci = "goulet"):

$$ \lambda = d \cdot \sqrt \frac{\tilde n}{2} $$

wherein, ñ is the harmonic mean of the 2 sample sizes which is calculated as the following:

$$ \tilde n = \frac{2 \cdot n_1 \cdot n_2}{n_1 + n_2} $$

2. Under all other methods (nct, t, or z):

$$ \lambda = \frac{1}{n_T} + \frac{s_c^2}{n_c \cdot s_c^2} $$

The standard error (σ) of Glass’s delta is calculated as the following:

1. Under the Cousineau and Goulet-Pelletier (2021) method (smd_ci = "goulet"):

$$ \sigma_{SMD} = \sqrt{\frac{df}{df-2} \cdot \frac{2}{\tilde n} (1+d^2 \cdot \frac{\tilde n}{2}) -\frac{d^2}{J}} $$

2. Under all other methods⁵ (nct, t, or z) derived from Bonett (2008):

$$ \sigma_{SMD} = \sqrt{\frac{s_e^2 \div s_c^2}{n_e-1}+ \frac{1}{n_c-1}+ \frac{d^2}{2 \cdot (n_c-1)}} $$

Cohen’s d(z): Change Scores

For this calculation, the denominator is the standard deviation of the difference scores which can be calculated from the standard deviations of the samples and the correlation between the paired samples.

$$ s_{diff} = \sqrt{sd_1^2 + sd_2^2 - 2 \cdot r_{12} \cdot sd_1 \cdot sd_2} $$

The SMD, Cohen’s d(z), is then calculated as the following:

$$ d_{z} = \frac {\bar{x}_1 - \bar{x}_2} {s_{diff}} $$

The degrees of freedom for Cohen’s d(z) is the following:

1. Under the Cousineau and Goulet-Pelletier (2021) method (smd_ci = "goulet"):

df = 2 ⋅ (N_pairs − 1)

2. Under all other methods (nct, t, or z):

df = N − 1

The non-centrality parameter (λ) is calculated as the following:

1. Under the Cousineau and Goulet-Pelletier (2021) method (smd_ci = "goulet"):

$$ \lambda = d_{z} \cdot \sqrt \frac{N_{pairs}}{2 \cdot (1-r_{12})} $$

2. Under all other methods (nct, t, or z):

$$ \lambda = \frac{1}{n} $$

The standard error (σ) of Cohen’s d(z) is calculated as the following:

1. Under the Cousineau and Goulet-Pelletier (2021) method (smd_ci = "goulet"):

$$ \sigma_{SMD} = \sqrt{\frac{df}{df-2} \cdot \frac{2 \cdot (1-r_{12})}{n} \cdot (1+d^2 \cdot \frac{n}{2 \cdot (1-r_{12})}) -\frac{d^2}{J^2}} \space \times \space \sqrt {2 \cdot (1-r_{12})} $$

2. Under all other methods (nct, t, or z) it calculated using the “unbiased” estimate from Viechtbauer (2007) (table 1; equation 26)⁶:

$$ \sigma_{SMD} = \sqrt{\frac{1}{N} + (1 - \frac{df-2}{df \cdot J^2}) \cdot d_z^2} $$

Cohen’s d(rm): Correlation Corrected

For this calculation, the same values for the same calculations above is adjusted for the correlation between measures. As Goulet-Pelletier and Cousineau (2018) mention, this is useful for when effect sizes are being compared for studies that involve between and within subjects designs.

First, the standard deviation of the difference scores are calculated

$$ s_{diff} = \sqrt{sd_1^2 + sd_2^2 - 2 \cdot r_{12} \cdot sd_1 \cdot sd_2} $$

The SMD, Cohen’s d(rm), is then calculated with a small change to the denominator⁷:

$$ d_{rm} = \frac {\bar{x}_1 - \bar{x}_2}{s_{diff}} \cdot \sqrt {2 \cdot (1-r_{12})} $$

The degrees of freedom for Cohen’s d(rm) is the following:

1. Under the Cousineau and Goulet-Pelletier (2021) method (smd_ci = "goulet"):

df = 2 ⋅ (N_pairs − 1)

2. Under all other methods (nct, t, or z):

df = N − 1

The non-centrality parameter (λ) is calculated as the following:

1. Under the Cousineau and Goulet-Pelletier (2021) method (smd_ci = "goulet"):

$$ \lambda = d_{rm} \cdot \sqrt \frac{N_{pairs}}{2 \cdot (1-r_{12})} $$

2. Under all other methods (nct, t, or z):

$$ \lambda = \frac{1}{n} $$

The standard error (σ) of Cohen’s d(rm) is calculated as the following:

$$ \sigma_{SMD} = \sqrt{\frac{df}{df-2} \cdot \frac{2 \cdot (1-r_{12})}{n} \cdot (1+d_{rm}^2 \cdot \frac{n}{2 \cdot (1-r_{12})}) -\frac{d_{rm}^2}{J^2}} $$

Glass’s Delta

For this calculation, the denominator is simply the standard deviation of one of the groups (x for glass = "glass1", or y for glass = "glass2".

$$ s_{c} = SD_{control \space condition} $$

$$ d = \frac {\bar{x}_1 - \bar{x}_2} {s_{c}} $$

The degrees of freedom for Glass’s delta is the following:

1. Under the Cousineau and Goulet-Pelletier (2021) method (smd_ci = "goulet"):

df = 2 ⋅ N − 1

2. Under all other methods (nct, t, or z):

df = N − 1

The non-centrality parameter (λ) is calculated as the following:

1. Under the Cousineau and Goulet-Pelletier (2021) method (smd_ci = "goulet"):

$$ \lambda = d \cdot \sqrt{\frac{N}{2 \cdot (1 - r_{12})}} $$

2. Under all other methods (nct, t, or z):

$$ \lambda = \frac{1}{N} $$

The standard error (σ) of Glass’s delta is calculated as the following derived⁸ from Bonett (2008):

$$ \sigma_{SMD} = \sqrt{\frac{s_{diff}^2}{s_c^2 \cdot df} + \frac{d^2}{2 \cdot df}} $$

Cousineau and Goulet-Pelletier (2021) Method (smd_ci = “goulet”)

Calculate confidence intervals around λ.

$$ t_L = t_{(1/2-(1-\alpha)/2,\space df, \space \lambda)} \\ t_U = t_{(1/2+(1-\alpha)/2,\space df, \space \lambda)} $$

Then transform back to the SMD.

$$ d_L = \frac{t_L}{\lambda} \cdot d \\ d_U = \frac{t_U}{\lambda} \cdot d $$

The non-central t-method (smd_ci = “nct”)

This method is commonly reported in the literature and in other R packages (e.g., effectsize). However, it is not without criticism (Spanos 2021; Viechtbauer 2007).

In thise method, you calculate the non-centrality parameters necessary to form confidence intervals wherein the observed t-statistic (t_obs)is utilized.

$$ t_L = t_{(1-alpha,\space df, \space t_{obs})} \\ t_U = t_{(alpha,\space df, \space t_{obs})} $$ The confidence intervals can then be constructed using the non-centrality parameter and the bias correction.

$$ d_L = t_L \cdot \sqrt{\lambda} \cdot J \\ d_U = t_U \cdot \sqrt{\lambda} \cdot J $$

Central t-method

This is less commonly used though it might perform alright in some scenarios, and as Bonett (2008) and Viechtbauer (2007) report it might perform adequately under with an adequately large sample size.

The limits of the t-distribution at the given alpha-level and degrees of freedom (qt(1-alpha,df)) are multiplied by the standard error of the calculated SMD.

$$ CI = SMD \space \pm \space t_{(1-\alpha,df)} \cdot \sigma_{SMD} $$

Normal method

This method will often be the most liberal estimate for the confidence interval. But as Bonett (2008) and Viechtbauer (2007) report, it might perform adequately under with an adequately large sample size. The use of the normal distribution can be justified by the fact that SMDs are asympototically normal.

The limits of the z-distribution at the given alpha-level (qnorm(1-alpha)) are multiplied by the standard error of the calculated SMD.

$$ CI = SMD \space \pm \space z_{(1-\alpha)} \cdot \sigma_{SMD} $$

Raw Data

The above results are only based on an approximating the differences between the SMDs. If the raw data is available, then the optimal solution is the bootstrap the results. This can be accomplished with the boot_compare_smd function.

For this example, we will simulate some data.

set.seed(4522)
diff_study1 = rnorm(25,.95)
diff_study2 = rnorm(50)
boot_test = boot_compare_smd(x1 = diff_study1,
                             x2 = diff_study2,
                             paired = TRUE)

boot_test
#> 
#>  Bootstrapped Differences in SMDs (paired)
#> 
#> data:  Bootstrapped
#> z (observed) = 2.887, p-value = 0.006003
#> alternative hypothesis: true difference in SMDs is not equal to 0
#> 95 percent confidence interval:
#>  0.3161207 1.4831351
#> sample estimates:
#> difference in SMDs 
#>          0.8058872

# Table of bootstrapped CIs
knitr::kable(boot_test$df_ci, digits = 4)

library(ggplot2)

list_res = boot_test$boot_res
df1 = data.frame(study = c(rep("original",length(list_res$smd1)),
                           rep("replication",length(list_res$smd2))),
                 smd = c(list_res$smd1,list_res$smd2))
ggplot(df1,
            aes(fill = study, color =smd, x = smd))+ 
 geom_histogram(aes(y=..density..), alpha=0.5, 
                position="identity")+
 geom_density(alpha=.2) +
  labs(y = "", x = "SMD (bootstrapped estimates)") +
  theme_classic()
#> `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.

df2 = data.frame(diff = list_res$d_diff)
ggplot(df2,
            aes(x = diff))+ 
 geom_histogram(aes(y=..density..), alpha=0.5, 
                position="identity")+
 geom_density(alpha=.2) +
  labs(y = "", x = "Difference in SMDs (bootstrapped estimates)") +
  theme_classic()
#> `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.