Package 'SimplyAgree'

Title: Flexible and Robust Agreement and Reliability Analyses
Description: Reliability and agreement analyses often have limited software support. Therefore, this package was created to make agreement and reliability analyses easier for the average researcher. The functions within this package include simple tests of agreement, agreement analysis for nested and replicate data, and provide robust analyses of reliability. In addition, this package contains a set of functions to help when planning studies looking to assess measurement agreement.
Authors: Aaron Caldwell [aut, cre]
Maintainer: Aaron Caldwell <[email protected]>
License: GPL (>= 3)
Version: 0.4.0.9000
Built: 2026-11-11 13:34:55 UTC
Source: https://github.com/arcaldwell49/simplyagree

Help Index


Assurance Probability for Limits of Agreement

Description

[Stable]

Usage

agree_assurance(
  conf.level = 0.95,
  assurance = 0.9,
  omega = NULL,
  pstar = 0.95,
  sigma = 1,
  n = NULL,
  m = 1,
  icc = 0
)

Arguments

conf.level

confidence level for the range of agreement (1 - alpha). The confidence level of the confidence interval of the range of agreement (tolerance interval). Default is 0.95.

assurance

target lower bound of the assurance probability (1 - gamma). The assurance probability is the probability that the study half-width will be less than omega. Common values are 0.80, 0.90, or 0.95.

omega

upper bound of assurance half-width. The sample size guarantees (assures) that 100(1 - gamma)% of interval half-widths will be less than this value. Can be specified in standard deviation units.

pstar

central proportion of the data distribution covered (P*). It is the proportion of observations that fall between the limits. For example, a value of 0.95 indicates that 95% of the variable's values fall between the limits. Must be between 0 and 1. Common values are 0.90 or 0.95.

sigma

population standard deviation of the paired differences. If the true value is unknown, omega can be specified in standard deviation units by setting sigma = 1. With clustered data (m > 1), this is the total SD of the differences (between- plus within-subject variance).

n

sample size (optional): the number of pairs, or the number of subjects when m > 1. If provided, the function will solve for omega rather than sample size.

m

number of paired measurements per subject (default 1, independent pairs). See the "Clustered data" section.

icc

intraclass correlation of the paired differences (between-subject share of the variance of the differences), in [0, 1). Ignored when m = 1. This is not the test-retest ICC of the measurements themselves.

Details

Calculate the sample size necessary for a confidence interval of the Bland-Altman range of agreement when the underlying data distribution is normal. This function uses the assurance probability criterion to determine the optimum sample size, based on the exact confidence interval method of Jan and Shieh (2018), which has been shown to be superior to approximate methods.

Overview

This function implements the exact method for determining sample size based on assurance probability for Bland-Altman limits of agreement, as described in Jan and Shieh (2018). The assurance probability criterion determines an N that guarantees with specified probability (1 - gamma) that the confidence interval half-width will be no more than a boundary value omega.

Technical Details

Suppose a study involves paired differences (X - Y) whose distribution is approximately N(mu, sigma^2). The range of agreement is defined as a confidence interval of the central portion of these differences, specifically the area between the 100(1-p)th and 100p-th percentiles, where p* = 2p - 1.

The exact two-sided, 100(1 - alpha)% confidence interval for the range of agreement is defined as:

Pr(theta_(1-p) < theta_hat_(1-p) and theta_hat_p < theta_p) = 1 - alpha

The equal-tailed tolerance interval recommended by Jan and Shieh (2018) is:

(X_bar - d, X_bar + d)

where d = g * S, g is the Odeh-Owen tolerance factor (tabulated as g” in Odeh and Owen (1980)), and S is the sample standard deviation.

Sample Size Determination

The sample size N is selected to satisfy: Pr(H <= omega) >= 1 - gamma

This leads to the expression:

psi(eta) >= 1 - gamma

where psi() is the CDF of a chi-square distribution with N-1 degrees of freedom and eta = (N-1) \* (omega/(g\*sigma))^2.

The method uses equal-tailed tolerance intervals based on the noncentral t-distribution to construct exact confidence intervals for the range of agreement. The tolerance factor g is calculated such that the interval maintains the specified confidence level under normality.

Jan and Shieh (2018) demonstrated through extensive simulations that this exact method should be adopted rather than the classical Bland-Altman approximate method.

Interpreting Results

Each subject produces two measurements (one for each method being compared). The sample size n returned is the number of subject pairs needed. The actual assurance probability may differ slightly from the target due to the discrete nature of sample size.

For dropout considerations, inflate the sample size using: N' = N / (1 - dropout_rate), always rounding up.

Value

An object of class "power.htest" containing the following components:

  • n: The required sample size (number of pairs, or of subjects when m > 1)

  • conf.level: The confidence level (1 - alpha)

  • assurance: The target assurance probability (1 - gamma)

  • actual.assurance: The actual assurance probability achieved (may differ slightly from target due to discrete nature of n)

  • omega: The upper bound of assurance half-width

  • pstar: The central proportion covered (P*)

  • sigma: The population standard deviation

  • g: The Odeh-Owen factor (g”) used to construct the tolerance interval, tabulated in Odeh and Owen (1980)

  • zp: The normal quantile for the central proportion

  • m, icc: The clustering inputs

  • n.obs: The total number of pairs, n * m

  • method: Description of the method used

  • note: Additional notes about the analysis

Assumptions

  • The paired differences are normally distributed

  • The variance is constant across the range of measurement

  • Pairs are independent, or (with m > 1) follow the balanced compound symmetry model in the "Clustered data" section

Clustered data

When each of n subjects contributes m paired measurements (repeated measures, or several pairs per subject), the pairs are not independent and treating n * m pairs as independent overstates precision. Setting m and icc accounts for this under a balanced one-way random-effects (compound symmetry) model for the differences:

  • The mean difference has variance sigma^2 \* DEFF / (n \* m), with design effect DEFF = 1 + (m - 1) \* icc, so the effective sample size for the mean is n \* m / DEFF.

  • The total variance is estimated as S^2 = MSB / m + (m - 1) / m \* MSW, whose distribution is a weighted sum of two independent chi-square variables with n - 1 and n(m - 1) degrees of freedom. This distribution is integrated numerically in place of the single chi-square of the independent case. It is more skewed than a Satterthwaite approximation suggests, which matters when icc is large.

Under these assumptions the calculation is exact (it agrees with simulation to Monte Carlo error), and m = 1 gives exactly the published method. The simple design effect 1 + (m - 1) \* icc applied to the whole sample size overstates the penalty, because the SD has a much smaller design effect, 1 + (m - 1) \* icc^2. Unbalanced designs, other correlation structures, or several levels of nesting are not covered; use simulation for those.

icc is the ICC of the differences, which can be estimated from pilot data with an intercept-only mixed model on the differences (e.g. lme4::lmer(diff ~ 1 + (1 | id))), and sigma is then the square root of the sum of the two variance components.

References

Jan, S.L. and Shieh, G. (2018). The Bland-Altman range of agreement: Exact interval procedure and sample size determination. Computers in Biology and Medicine, 100, 247-252. doi:10.1016/j.compbiomed.2018.06.020

Bland, J.M. and Altman, D.G. (2007). Agreement between methods of measurement with multiple observations per individual. Journal of Biopharmaceutical Statistics, 17(4), 571-582. doi:10.1080/10543400701329422

Mee, R.W. and Owen, D.B. (1983). Improved factors for one-sided tolerance limits for balanced one-way ANOVA random model. Journal of the American Statistical Association, 78(384), 901-905. doi:10.1080/01621459.1983.10477038

Odeh, R.E. and Owen, D.B. (1980). Tables for Normal Tolerance Limits, Sampling Plans, and Screening. Marcel Dekker, Inc., New York.

See Also

agree_expected_half() for sample size determination using expected half-width criterion, power_agreement_exact() for power analysis of agreement tests.

Examples

# Example: Planning a method comparison study
# Researchers want 95% confidence, 90% assurance that half-width
# will be within 2.5 SD units, covering central 95% of differences
agree_assurance(
  conf.level = 0.95,
  assurance = 0.90,
  omega = 2.5,
  pstar = 0.95,
  sigma = 1
)

# Clustered data: 4 pairs per subject, ICC of the differences 0.3
agree_assurance(
  assurance = 0.90,
  omega = 2.5,
  pstar = 0.95,
  m = 4,
  icc = 0.3
)

Agreement Coefficients

Description

[Superseded]

agree_coef() has been superseded by cac_test(), which returns one coefficient as an htest with a hypothesis test, summary() and plot() methods. agree_coef() still returns a table of all four coefficients.

agree_coef produces inter-rater reliability or "agreement coefficients" as described by Gwet: percent agreement, Gwet's AC1 (AC2 when weighted), Fleiss' kappa and Krippendorff's alpha.

Usage

agree_coef(
  wide = TRUE,
  col.names = NULL,
  measure,
  item,
  id,
  data,
  weighted = FALSE,
  conf.level = 0.95,
  weights = NULL,
  categories = NULL
)

Arguments

wide

Logical value (TRUE or FALSE) indicating if data is in a "wide" format. Default is TRUE.

col.names

If wide is equal to TRUE then col.names is a list of the column names containing the measurements for reliability analysis.

measure

Name of column containing the measurement of interest.

item

Name of column containing the items. If this is an inter-rater reliability study then this would indicate the rater (e.g., rater1, rater2, rater3, etc).

id

Column with subject identifier.

data

Data frame with all data.

weighted

Logical value (TRUE or FALSE) indicating whether to weight the responses. If TRUE (default is FALSE) then quadratic weights are utilized. This option should be set to TRUE for ordinal or continuous responses. Ignored when weights is given.

conf.level

the confidence level required. Default is 95%.

weights

Optional weighting scheme, as in cac_test(): e.g. "quadratic", "linear" or "ordinal", or a custom weight matrix. Overrides weighted.

categories

All possible rating categories, as in cac_test(). Defaults to the sorted observed ratings (or the levels of a factor).

Details

The estimates, standard errors and intervals are those of cac_test(); see there for details. Confidence limits are clipped to the possible range of each coefficient. When all ratings are in one category the chance-corrected coefficients are undefined and returned as NA.

Value

A data frame with one row per coefficient and the columns est (estimate), se (standard error), lower.ci and upper.ci.

References

Gwet, K.L. (2014, ISBN:978-0970806284). "Handbook of Inter-Rater Reliability", 4th Edition. Advanced Analytics, LLC.

Gwet, K. L. (2008). "Computing inter-rater reliability and its variance in the presence of high agreement," British Journal of Mathematical and Statistical Psychology, 61, 29-48.

See Also

cac_test()

Examples

data('reps')
agree_coef(data = reps, wide = TRUE, col.names = c("x","y"), weighted = TRUE)

Sample Size for Limits of Agreement Using Expected Half-Width

Description

[Stable]

Usage

agree_expected_half(
  conf.level = 0.95,
  delta = NULL,
  pstar = 0.95,
  sigma = 1,
  n = NULL,
  m = 1,
  icc = 0
)

Arguments

conf.level

confidence level for the range of agreement (1 - alpha). The confidence level of the confidence interval of the range of agreement (tolerance interval). Default is 0.95.

delta

target upper bound of expected half-width (delta). The sample size guarantees that the expected half-width of the confidence interval will be no more than this value. Can be specified in standard deviation units.

pstar

central proportion of the data distribution covered (P*). It is the proportion of observations that fall between the limits. For example, a value of 0.95 indicates that 95% of the variable's values fall between the limits. Must be between 0 and 1. Common values are 0.90 or 0.95.

sigma

population standard deviation of the paired differences. If the true value is unknown, delta can be specified in standard deviation units by setting sigma = 1. With clustered data (m > 1), this is the total SD of the differences (between- plus within-subject variance).

n

sample size (optional): the number of pairs, or the number of subjects when m > 1. If provided, the function will solve for a different parameter rather than sample size.

m

number of paired measurements per subject (default 1, independent pairs). See the "Clustered data" section.

icc

intraclass correlation of the paired differences (between-subject share of the variance of the differences), in [0, 1). Ignored when m = 1. This is not the test-retest ICC of the measurements themselves.

Details

Calculate the sample size necessary for a confidence interval of the Bland-Altman range of agreement when the underlying data distribution is normal. This function uses the expected half-width criterion to determine the optimum sample size, based on the exact confidence interval method of Jan and Shieh (2018), which has been shown to be superior to approximate methods.

Overview

This function implements the exact method for determining sample size based on expected half-width for Bland-Altman limits of agreement, as described in Jan and Shieh (2018). The expected half-width criterion determines an N that guarantees that the expected half-width of the confidence interval is less than a boundary value delta.

Technical Details

Suppose a study involves paired differences (X - Y) whose distribution is approximately N(mu, sigma^2). The range of agreement is defined as a confidence interval of the central portion of these differences, specifically the area between the 100(1-p)th and 100p-th percentiles, where p* = 2p - 1.

The exact two-sided, 100(1 - alpha)% confidence interval for the range of agreement is defined as:

Pr(theta_(1-p) < theta_hat_(1-p) and theta_hat_p < theta_p) = 1 - alpha

The equal-tailed tolerance interval recommended by Jan and Shieh (2018) is:

(X_bar - d, X_bar + d)

where d = g * S, g is the Odeh-Owen tolerance factor (tabulated as g” in Odeh and Owen (1980) and Hahn and Meeker (1991)), and S is the sample standard deviation.

Sample Size Determination

The sample size N is selected to satisfy: E(H) <= delta

where H is the half-width of the confidence interval. This leads to the expression:

g(P*, 1-alpha, N-1) / c <= delta / sigma

where c is a bias correction factor:

c = (Gamma((N-1)/2) * sqrt((N-1)/2)) / Gamma(N/2)

The expected half-width is E(H) = (g/c) * sigma. This accounts for the fact that the sample standard deviation S is a biased estimator of sigma, requiring correction when computing expected values.

The method uses equal-tailed tolerance intervals based on the noncentral t-distribution to construct exact confidence intervals for the range of agreement. The tolerance factor g is calculated such that the interval maintains the specified confidence level under normality.

Comparison with Assurance Probability Method

This function uses the expected half-width criterion, which ensures that E(H) <= delta. An alternative approach is the assurance probability criterion (see agree_assurance()), which ensures that Pr(H <= omega) >= 1 - gamma.

The expected half-width criterion:

  • Controls the average half-width across repeated sampling

  • Generally requires smaller sample sizes than assurance probability

  • Is appropriate when the average performance is of primary interest

The assurance probability criterion:

  • Provides a probability guarantee about the half-width

  • Generally requires larger sample sizes

  • Is appropriate when a stronger guarantee is needed for planning purposes

Interpreting Results

Each subject produces two measurements (one for each method being compared). The sample size n returned is the number of subject pairs needed. The actual expected half-width may differ slightly from the target due to the discrete nature of sample size.

For dropout considerations, inflate the sample size using: N' = N / (1 - dropout_rate), always rounding up.

Value

An object of class "power.htest" containing the following components:

  • n: The required sample size (number of pairs, or of subjects when m > 1)

  • conf.level: The confidence level (1 - alpha)

  • target.delta: The target upper bound of expected half-width

  • actual.delta: The expected half-width achieved (may be smaller than the target due to discrete nature of n)

  • pstar: The central proportion covered (P*)

  • sigma: The population standard deviation

  • g: The Odeh-Owen factor (g”) used to construct the tolerance interval, tabulated in Odeh and Owen (1980)

  • c: The bias correction factor, 1 / E(S / sigma), used in the expected half-width calculation

  • zp: The normal quantile for the central proportion

  • m, icc: The clustering inputs

  • n.obs: The total number of pairs, n * m

  • method: Description of the method used

  • note: Additional notes about the analysis

Clustered data

When each of n subjects contributes m paired measurements (repeated measures, or several pairs per subject), the pairs are not independent and treating n * m pairs as independent overstates precision. Setting m and icc accounts for this under a balanced one-way random-effects (compound symmetry) model for the differences:

  • The mean difference has variance sigma^2 \* DEFF / (n \* m), with design effect DEFF = 1 + (m - 1) \* icc, so the effective sample size for the mean is n \* m / DEFF.

  • The total variance is estimated as S^2 = MSB / m + (m - 1) / m \* MSW, whose distribution is a weighted sum of two independent chi-square variables with n - 1 and n(m - 1) degrees of freedom. This distribution is integrated numerically in place of the single chi-square of the independent case. It is more skewed than a Satterthwaite approximation suggests, which matters when icc is large.

Under these assumptions the calculation is exact (it agrees with simulation to Monte Carlo error), and m = 1 gives exactly the published method. The simple design effect 1 + (m - 1) \* icc applied to the whole sample size overstates the penalty, because the SD has a much smaller design effect, 1 + (m - 1) \* icc^2. Unbalanced designs, other correlation structures, or several levels of nesting are not covered; use simulation for those.

icc is the ICC of the differences, which can be estimated from pilot data with an intercept-only mixed model on the differences (e.g. lme4::lmer(diff ~ 1 + (1 | id))), and sigma is then the square root of the sum of the two variance components.

Assumptions

  • The paired differences are normally distributed

  • The variance is constant across the range of measurement

  • Pairs are independent, or (with m > 1) follow the balanced compound symmetry model in the "Clustered data" section

References

Jan, S.L. and Shieh, G. (2018). The Bland-Altman range of agreement: Exact interval procedure and sample size determination. Computers in Biology and Medicine, 100, 247-252. doi:10.1016/j.compbiomed.2018.06.020

Bland, J.M. and Altman, D.G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet, 327(8476), 307-310.

Hahn, G.J. and Meeker, W.Q. (1991). Statistical Intervals: A Guide for Practitioners. John Wiley & Sons, New York.

Bland, J.M. and Altman, D.G. (2007). Agreement between methods of measurement with multiple observations per individual. Journal of Biopharmaceutical Statistics, 17(4), 571-582. doi:10.1080/10543400701329422

Mee, R.W. and Owen, D.B. (1983). Improved factors for one-sided tolerance limits for balanced one-way ANOVA random model. Journal of the American Statistical Association, 78(384), 901-905. doi:10.1080/01621459.1983.10477038

Odeh, R.E. and Owen, D.B. (1980). Tables for Normal Tolerance Limits, Sampling Plans, and Screening. Marcel Dekker, Inc., New York.

See Also

agree_assurance() for sample size determination using assurance probability criterion, power_agreement_exact() for power analysis of agreement tests.

Examples

# Example 1: Reproduce Jan and Shieh (2018), page 251
# Expected half-width criterion with P* = 0.95
agree_expected_half(
  conf.level = 0.95,
  delta = 2.25 * 19.61,
  pstar = 0.95,
  sigma = 19.61
)
# Expected result: n = 155

# Example 2: Planning a method comparison study
# Researchers want 95% confidence with expected half-width
# no more than 2.4 SD units, covering central 90% of differences
agree_expected_half(
  conf.level = 0.95,
  delta = 2.4,
  pstar = 0.90,
  sigma = 1
)

# Example 3: Clustered data, 3 pairs per subject, ICC of the differences 0.2
# (n is then the number of subjects)
agree_expected_half(
  delta = 2.4,
  pstar = 0.90,
  m = 3,
  icc = 0.2
)

Tests for Absolute Agreement with Nested Data

Description

[Superseded]

Development on agree_nest() is complete, and for new code we recommend switching to agreement_limit(), which is easier to use, has more features, and still under active development.

agree_nest produces an absolute agreement analysis for data where there is multiple observations per subject but the mean varies within subjects as described by Zou (2013). Output mirrors that of agree_test but the CCC is Lin's estimator over all complete pairs with a delete-one-subject jackknife standard error; see ccc_test().

Usage

agree_nest(
  x,
  y,
  id,
  data,
  delta,
  agree.level = 0.95,
  conf.level = 0.95,
  TOST = TRUE,
  prop_bias = FALSE,
  ccc = TRUE
)

Arguments

x

Name of column with first measurement

y

Name of other column with the other measurement to compare to the first.

id

Column with subject identifier

data

Data frame with all data

delta

The threshold below which methods agree/can be considered equivalent, can be in any units. Equivalence Bound for Agreement.

agree.level

the agreement level required. Default is 95%. The proportion of data that should lie between the thresholds, for 95% limits of agreement this should be 0.95.

conf.level

the confidence level required. Default is 95%.

TOST

Logical indicator (TRUE/FALSE) of whether to use two one-tailed tests for the limits of agreement. Default is TRUE.

prop_bias

Logical indicator (TRUE/FALSE) of whether proportional bias should be considered for the limits of agreement calculations.

ccc

Calculate concordance correlation coefficient.

Value

Returns single simple_agree class object with the results of the agreement analysis.

  • loa: A data frame of the limits of agreement including the average difference between the two sets of measurements, the standard deviation of the difference between the two sets of measurements and the lower and upper confidence limits of the difference between the two sets of measurements.

  • h0_test: Decision from hypothesis test.

  • ccc.xy: Lin's concordance correlation coefficient over all complete pairs and confidence intervals, as in ccc_test(data_type = "nest").

  • call: the matched call.

  • var_comp: Table of Variance Components.

  • class: The type of simple_agree analysis.

References

Zou, G. Y. (2013). Confidence interval estimation for the Bland–Altman limits of agreement with multiple observations per individual. Statistical methods in medical research, 22(6), 630-642.

Lin, L. I. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics, 45(1), 255-268.

Examples

data('reps')
agree_nest(x = "x", y = "y", id = "id", data = reps, delta = 2)

Nonparametric Test for Limits of Agreement

Description

[Stable]

agree_np A non-parametric approach to limits of agreement. The hypothesis test is based on binomial proportions within the maximal allowable differences, and the limits are calculated with quantile regression.

Usage

agree_np(
  x,
  y,
  id = NULL,
  data,
  delta = NULL,
  prop_bias = FALSE,
  TOST = TRUE,
  agree.level = 0.95,
  conf.level = 0.95
)

Arguments

x

Name of column with first measurement.

y

Name of other column with the other measurement to compare to the first.

id

Column with subject identifier with samples are taken in replicates.

data

Data frame with all data.

delta

The threshold below which methods agree/can be considered equivalent and this argument is required. Equivalence Bound for Agreement or Maximal Allowable Difference.

prop_bias

Logical indicator (TRUE/FALSE) of whether proportional bias should be considered for the limits of agreement calculations.

TOST

Logical indicator (TRUE/FALSE) of whether to use two one-tailed tests for the limits of agreement. Default is TRUE.

agree.level

the agreement level required. Default is 95%. The proportion of data that should lie between the thresholds, for 95% limits of agreement this should be 0.95.

conf.level

the confidence level required. Default is 95%.

Value

Returns simple_agree object with the results of the agreement analysis.

  • loa: A data frame of the limits of agreement.

  • agree: A data frame of the binomial proportion of results in agreement.

  • h0_test: Decision from hypothesis test.

  • qr_mod: The quantile regression model.

  • call: The matched call

References

Bland, J. M., & Altman, D. G. (1999). Measuring agreement in method comparison studies. In Statistical Methods in Medical Research (Vol. 8, Issue 2, pp. 135–160). SAGE Publications. doi:10.1177/096228029900800204

Examples

data('reps')
agree_np(x = "x", y = "y", id = "id", data = reps, delta = 2)

Tests for Absolute Agreement with Replicates

Description

[Superseded]

Development on agree_reps() is complete, and for new code we recommend switching to agreement_limit(), which is easier to use, has more features, and still under active development.

agree_nest produces an absolute agreement analysis for data where there is multiple observations per subject but the mean does not vary within subjects as described by Zou (2013). Output mirrors that of agree_test but the CCC is estimated from variance components (Carrasco & Jover, 2003) with a delete-one-subject jackknife standard error; see ccc_test().

Usage

agree_reps(
  x,
  y,
  id,
  data,
  delta,
  agree.level = 0.95,
  conf.level = 0.95,
  prop_bias = FALSE,
  TOST = TRUE,
  ccc = TRUE
)

Arguments

x

Name of column with first measurement

y

Name of other column with the other measurement to compare to the first.

id

Column with subject identifier

data

Data frame with all data

delta

The threshold below which methods agree/can be considered equivalent, can be in any units. Equivalence Bound for Agreement.

agree.level

the agreement level required. Default is 95%. The proportion of data that should lie between the thresholds, for 95% limits of agreement this should be 0.95.

conf.level

the confidence level required. Default is 95%.

prop_bias

Logical indicator (TRUE/FALSE) of whether proportional bias should be considered for the limits of agreement calculations.

TOST

Logical indicator (TRUE/FALSE) of whether to use two one-tailed tests for the limits of agreement. Default is TRUE.

ccc

Calculate concordance correlation coefficient.

Value

Returns single list with the results of the agreement analysis.

  • loa: a data frame of the limits of agreement including the average difference between the two sets of measurements, the standard deviation of the difference between the two sets of measurements and the lower and upper confidence limits of the difference between the two sets of measurements.

  • h0_test: Decision from hypothesis test.

  • ccc.xy: The concordance correlation coefficient (variance components) and confidence intervals, as in ccc_test(data_type = "reps").

  • call: The matched call.

  • var_comp: Table of Variance Components.

  • class: The type of simple_agree analysis.

References

Zou, G. Y. (2013). Confidence interval estimation for the Bland–Altman limits of agreement with multiple observations per individual. Statistical methods in medical research, 22(6), 630-642.

Carrasco, JL & Jover, L. (2003). Estimating the generalized concordance correlation coefficient through variance components. Biometrics, 59(4), 849-858.

Carrasco, JL; Phillips, BR; Puig-Martinez, J; King, TS; Chinchilli, VM. (2013). Estimation of the concordance correlation coefficient for repeated measures using SAS and R. Computer Methods and Programs in Biomedicine, 109, 293-304.

Examples

data('reps')
agree_reps(x = "x", y = "y", id = "id", data = reps, delta = 2)

Tests for Absolute Agreement

Description

[Superseded]

Development on agree_test() is complete, and for new code we recommend switching to agreement_limit(), which is easier to use, has more features, and still under active development.

The agree_test function calculates a variety of agreement statistics. The hypothesis test of agreement is calculated by the method described by Shieh (2019). Bland-Altman limits of agreement, and confidence intervals, are also provided (Bland & Altman 1999; Bland & Altman 1986). In addition, the concordance correlation coefficient (CCC; Lin 1989) is additional part of the output.

Usage

agree_test(
  x,
  y,
  delta,
  conf.level = 0.95,
  agree.level = 0.95,
  TOST = TRUE,
  prop_bias = FALSE
)

Arguments

x

Vector with first measurement

y

Vector with second measurement

delta

The threshold below which methods agree/can be considered equivalent, can be in any units. Often referred to as the "Equivalence Bound for Agreement" or "Maximal Allowable Difference".

conf.level

the confidence level required. Default is 95%.

agree.level

the agreement level required. Default is 95%. The proportion of data that should lie between the thresholds, for 95% limits of agreement this should be 0.95.

TOST

Logical indicator (TRUE/FALSE) of whether to use two one-tailed tests for the limits of agreement. Default is TRUE.

prop_bias

Logical indicator (TRUE/FALSE) of whether proportional bias should be considered for the limits of agreement calculations.

Value

Returns single list with the results of the agreement analysis.

  • shieh_test: The TOST hypothesis test as described by Shieh.

  • ccc.xy: Lin's concordance correlation coefficient and confidence intervals.

  • s.shift: Scale shift from x to y.

  • l.shift: Location shift from x to y.

  • bias: a bias correction factor that measures how far the best-fit line deviates from a line at 45 degrees. No deviation from the 45 degree line occurs when bias = 1. See Lin 1989, page 258.

  • loa: Data frame containing the limits of agreement calculations

  • h0_test: Decision from hypothesis test.

  • call: the matched call

References

Shieh (2019). Assessing Agreement Between Two Methods of Quantitative Measurements: Exact Test Procedure and Sample Size Calculation, Statistics in Biopharmaceutical Research, doi:10.1080/19466315.2019.1677495

Bland, J. M., & Altman, D. G. (1999). Measuring agreement in method comparison studies. Statistical methods in medical research, 8(2), 135-160.

Bland, J. M., & Altman, D. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. The lancet, 327(8476), 307-310.

Lawrence, I., & Lin, K. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics, 255-268.

Examples

data('reps')
agree_test(x=reps$x, y=reps$y, delta = 2)

Limits of Agreement

Description

[Stable]

A function for calculating for Bland-Altman limits of agreement based on the difference between two measurements (difference = x-y). Please note that the package developer recommends reporting/using tolerance limits (see "tolerance_limit" function).

Usage

agreement_limit(
  x,
  y,
  id = NULL,
  data,
  data_type = c("simple", "nest", "reps"),
  loa_calc = c("mover", "blandaltman"),
  bound_type = c("iu", "joint"),
  agree.level = 0.95,
  alpha = 0.05,
  prop_bias = FALSE,
  log_tf = FALSE,
  log_tf_display = c("ratio", "sympercent"),
  lmer_df = c("satterthwaite", "asymptotic"),
  lmer_limit = 3000
)

Arguments

x

Name of column with first measurement

y

Name of other column with the other measurement to compare to the first.

id

Column with subject identifier. Default is "id" if no entry is provided.

data

Data frame with all data.

data_type

The type of data structure. Options include "simple" (all independent data points), "nest" (nested data) and "reps" (replicated data points).

loa_calc

The method by which the limits of agreement confidence intervals are calculated. Options are "mover" (Methods of Recovering Variances method) or "blandlatman" (Bland-Altman method).

bound_type

Which claim the confidence bounds of the limits of agreement (lower_loa_ci/upper_loa_ci) support. "iu" (default) gives a one-sided 1 - alpha bound on each limit (the outer ends of 1 - 2 * alpha two-sided intervals). These give an intersection-union test, at level alpha, of whether both limits of agreement lie within a maximal allowable difference, but are not a joint 1 - alpha interval for the limits. "joint" gives a one-sided 1 - alpha/2 bound on each limit (the outer ends of 1 - alpha two-sided intervals, the Bland-Altman convention), with at least 1 - alpha joint confidence for both limits. See details.

agree.level

the agreement level required. Default is 95%. The proportion of data that should lie between the thresholds, for 95% limits of agreement this should be 0.95.

alpha

The alpha-level for confidence levels.

prop_bias

Logical indicator (TRUE/FALSE) of whether proportional bias should be considered for the limits of agreement calculations. Note that a non-zero slope of the differences on the average can appear without any true proportional bias whenever the two methods have unequal measurement error variances, because cov(difference, average) = (var(x) - var(y)) / 2 (Bland & Altman, 1999); errors-in-variables methods such as dem_reg() or pb_reg() are better suited to assessing proportional bias.

log_tf

Calculate limits of agreement using log-transformed data.

log_tf_display

The type of presentation for log-transformed results. The differences between methods can be displayed as a "ratio" or "sympercent".

lmer_df

Degrees of freedom method, only matters for if data_type is "nest". Default is "satterthwaite". The "asymptotic" method is faster but more liberal.

lmer_limit

Sample size limit for degrees of freedom method. If number of observations exceeds this limit, then the "asymptotic" method is utilized.

Details

The limits of agreement (LoA) are calculated in this function are based on the method originally detailed by Bland & Atlman (1986 & 1999). The loa_calc allow users to specify the calculative method for the LoA which can be based on Bland-Altman (1999) (loa_calc = "blandaltman"), or by the more accurate MOVER method of Zou (2013) and Donner & Zou (2012) (loa_calc = "mover").

The confidence bounds on the limits of agreement answer one of two questions, set by bound_type:

  • "iu" (default): "Can I conclude agreement within plus or minus delta?" If lower_loa_ci > -delta and upper_loa_ci < delta, reject at level alpha that either limit of agreement lies outside plus or minus delta. conf.level (1 - alpha) applies to each side separately; both bounds hold together only about 1 - 2 * alpha of the time.

  • "joint": "Where are the limits of agreement?" With at least 1 - alpha confidence, both limits of agreement lie between lower_loa_ci and upper_loa_ci. Used as a test against plus or minus delta, the error rate is at most alpha/2 (conservative).

Because a one-sided bound on the (1 - agree.level)/2 percentile is a one-sided tolerance bound, these bounds are equal-tailed tolerance bounds; they target the same quantities as tolerance_limit(bound_type = "iu"). The bias confidence interval (lower.CL/upper.CL) is always a two-sided 1 - alpha interval.

Value

Returns single loa class object with the results of the agreement analysis.

  • loa: A data frame containing the Limits of Agreement.

  • call:The matched call.

References

MOVER methods:

Zou, G. Y. (2013). Confidence interval estimation for the Bland–Altman limits of agreement with multiple observations per individual. Statistical methods in medical research, 22(6), 630-642.

Donner, A., & Zou, G. Y. (2012). Closed-form confidence intervals for functions of the normal mean and standard deviation. Statistical Methods in Medical Research, 21(4), 347-359.

Bland & Altman methods:

Bland, J. M., & Altman, D. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet, 327(8476), 307-310.

Bland, J. M., & Altman, D. (1999). Measuring agreement in method comparison studies. Statistical methods in medical research, 8(2), 135-160.

Bland, J. M., & Altman, D. G. (1996). Statistics notes: measurement error proportional to the mean. BMJ, 313(7049), 106.

Examples

data('reps')

# Simple
agreement_limit(x = "x", y ="y", data = reps)

# Replicates
agreement_limit(x = "x", y ="y", data = reps, id = "id", data_type = "rep")

# Nested
agreement_limit(x = "x", y ="y", data = reps, id = "id", data_type = "nest")

Power Curve for Bland-Altman Limits of Agreement

Description

[Superseded]

This function calculates the power for the Bland-Altman method under varying parameter settings and for a range of sample sizes. It is superseded by power_agreement_limit(), which returns a power.htest object and can solve for the sample size, power, delta or the SD directly. This function remains available for drawing power curves.

Usage

blandPowerCurve(
  samplesizes = seq(10, 100, 1),
  mu = 0,
  SD,
  delta,
  conf.level = 0.95,
  agree.level = 0.95,
  m = 1,
  icc = 0
)

Arguments

samplesizes

vector of samples sizes at which to estimate power: the number of pairs, or the number of subjects when m > 1.

mu

mean of differences

SD

standard deviation of differences (the total SD when m > 1)

delta

The threshold below which methods agree/can be considered equivalent, can be in any units. Equivalence Bound for Agreement. More than one delta can be provided.

conf.level

the confidence level(s) required. Default is 95%. More than one confidence level can be provided.

agree.level

the agreement level(s) required. Default is 95%. The proportion of data that should lie between the thresholds, for 95% limits of agreement this should be 0.95. More than one confidence level can be provided.

m

number of paired measurements per subject (default 1, independent pairs).

icc

intraclass correlation of the paired differences, in [0, 1). Ignored when m = 1.

Details

Power follows the approximate method of Lu et al. (2016). With clustered data (m > 1), the standard error of the limits uses the effective sample size n \* m / (1 + (m - 1) \* icc) for the mean difference and a Satterthwaite df for the SD, under a balanced compound symmetry model (see agree_expected_half() for the model). Like the original method, this is an approximation; m = 1 gives the original results.

Value

A dataframe is returned containing the power analysis results. The results can then be plotted with the plot.powerCurve function.

References

Lu, M. J., et al. (2016). Sample Size for Assessing Agreement between Two Methods of Measurement by Bland-Altman Method. The international journal of biostatistics, 12(2), doi:10.1515/ijb-2015-0039

Examples

powerCurve <- blandPowerCurve(samplesizes = seq(10, 200, 1),
mu = 0,
SD = 3.3,
delta = 8,
conf.level = .95,
agree.level = .95)
# Plot the power curve
plot(powerCurve, type = 1)
# Find at what N power of .8 is achieved
find_n(powerCurve, power = .8)

# If the desired power is not found then
## Sample size range must be expanded

Methods for cac_htest objects

Description

Methods for objects returned by cac_test() and ord_kappa_test(). They print as standard "htest" objects; summary() describes the result in words and plot() shows the ratings.

Usage

## S3 method for class 'cac_htest'
summary(object, style = c("auto", "hypothesis", "estimation"), ...)

## S3 method for class 'summary.cac_htest'
print(x, ...)

## S3 method for class 'cac_htest'
plot(x, type = c("raters", "subjects", "effects"), ...)

Arguments

style

How summary() words the result: "auto" (default), "hypothesis", or "estimation"; see reli_htest-methods for the two styles.

...

Not used.

x, object

An object of class cac_htest.

type

For plot(), the plot to draw: "raters" (default), the distribution of each rater's ratings over the categories, which shows raters who use the scale differently; "subjects", the ratings of each subject, with subjects ordered by their mean category, where a single large point marks complete agreement; or, for ord_kappa_test() only, "effects", each rater's estimated effect (conditional mode) on the latent scale with a 95% interval, where positive values mark raters who give higher ratings.

Value

summary

An object of class summary.cac_htest: a list with the description in text, the style used, the S-value (s.value, when there is a p-value) and the parts used to build it. It prints as a paragraph.

plot

A ggplot2 plot (see type).

Examples

ratings <- data.frame(
  R1 = c(1, 2, 3, 3, 2, 1, 4, 1, 2, NA, NA, NA),
  R2 = c(1, 2, 3, 3, 2, 2, 4, 1, 2, 5, NA, NA),
  R3 = c(NA, 3, 3, 3, 2, 3, 4, 2, 2, 5, 1, 3),
  R4 = c(1, 2, 3, 3, 2, 4, 4, 1, 2, 5, 1, NA))
res <- cac_test(col.names = c("R1", "R2", "R3", "R4"), data = ratings,
                null.value = 0.4, alternative = "greater")
summary(res)
summary(res, style = "estimation")
plot(res)
plot(res, type = "subjects")

Chance-Corrected Agreement Coefficients for Categorical Ratings

Description

[Stable]

Estimates and tests one agreement coefficient for categorical (nominal or ordinal) ratings of the same subjects by two or more raters: percent agreement, Gwet's AC1 (AC2 when weighted), Fleiss' kappa, or Krippendorff's alpha. Missing ratings are allowed. For ordinal ratings from many raters, see also the model-based kappa in ord_kappa_test().

Usage

cac_test(
  x = NULL,
  data = NULL,
  measure = NULL,
  item = NULL,
  id = NULL,
  col.names = NULL,
  coef = c("gwet", "fleiss", "krippendorff", "pa"),
  weights = "unweighted",
  categories = NULL,
  conf.level = 0.95,
  null.value = NULL,
  alternative = c("two.sided", "greater", "less")
)

Arguments

x

A formula of the form values ~ rater | id. Leave NULL to use measure, item and id (long data) or col.names (wide data).

data

A data frame.

measure, item, id

Names of the columns with the ratings, the raters, and the subjects (long data).

col.names

Names of the columns with each rater's ratings (wide data, one row per subject).

coef

The coefficient: "gwet" (Gwet's AC1/AC2; default), "fleiss" (Fleiss' kappa), "krippendorff" (Krippendorff's alpha) or "pa" (percent agreement, not corrected for chance).

weights

How partial agreement between different categories is credited: "unweighted" (default; only identical ratings agree), "quadratic", "linear", "ordinal", "radical", "ratio", "circular" or "bipolar" (Gwet, 2014, ch. 3), or a custom q x q matrix with values in [0, 1] and 1 on the diagonal, ordered as categories. Use weights only for ordered categories.

categories

All possible rating categories, in order. Defaults to the levels of a factor, and otherwise to the sorted observed ratings. Give it when some categories of the scale were never used: Gwet's AC1/AC2 and the weights depend on the full set of categories.

conf.level

Confidence level of the interval (default 0.95).

null.value

The hypothesized value of the coefficient. NULL (default) gives no test.

alternative

The alternative hypothesis: "two.sided" (default), "greater" or "less". Without a null.value the interval is two-sided.

Details

All four coefficients share the observed agreement pap_a, the proportion of agreeing pairs of ratings within subjects (weighted when weights is used), averaged over the subjects with two or more ratings. The chance-corrected coefficients are (pa−pe)/(1−pe)(p_a - p_e)/(1 - p_e) and differ in the chance agreement pep_e: Fleiss' kappa and Krippendorff's alpha use the overall category proportions, so they are low when one category dominates even if raters rarely disagree (the "kappa paradox"); Gwet's AC1 uses the chance that a rater classifies at random and is stable in that situation (Gwet, 2008). Krippendorff's alpha also corrects for the number of pairs.

Standard errors come from Gwet's (2008, 2014) linearisation over subjects, which treats the subjects as a random sample and the raters as fixed. The interval and test use the t distribution with n - 1 degrees of freedom (n subjects; for Krippendorff's alpha, those with 2 or more ratings), and limits are clipped to the possible range of the coefficient. These match the irrCAC package (with an infinite population).

Ratings may be numeric, character or factor. Subjects with no ratings are dropped; subjects with one rating add to the category proportions but not to the observed agreement. Each subject can have at most one rating per rater.

Value

A list of class c("cac_htest", "htest") containing:

statistic

The t statistic (only with a null.value).

parameter

The degrees of freedom (only with a null.value).

p.value

The p-value (only with a null.value).

conf.int

The confidence interval for the coefficient.

estimate

The estimated coefficient.

stderr

The standard error of the coefficient.

null.value

The hypothesized value.

alternative

The alternative hypothesis.

method

A description of the coefficient.

data.name

A description of the data.

fit

Details: observed (pa) and chance (pe) agreement, the categories, the weight matrix, and the numbers of subjects and raters.

data

The ratings in long format.

settings

The settings used, for the methods.

References

Gwet, K. L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1), 29-48. doi:10.1348/000711006X126600

Gwet, K. L. (2014). Handbook of Inter-Rater Reliability (4th ed.). Advanced Analytics, LLC. ISBN 978-0970806284.

Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378-382. doi:10.1037/h0031619

Krippendorff, K. (1970). Estimating the reliability, systematic error and random error of interval data. Educational and Psychological Measurement, 30(1), 61-70. doi:10.1177/001316447003000105

See Also

cac_htest-methods for summary() and plot(); ord_kappa_test() for the model-based kappa for ordinal ratings.

Examples

# Gwet (2014): 12 subjects rated by 4 raters on a 5-point scale
ratings <- data.frame(
  R1 = c(1, 2, 3, 3, 2, 1, 4, 1, 2, NA, NA, NA),
  R2 = c(1, 2, 3, 3, 2, 2, 4, 1, 2, 5, NA, NA),
  R3 = c(NA, 3, 3, 3, 2, 3, 4, 2, 2, 5, 1, 3),
  R4 = c(1, 2, 3, 3, 2, 4, 4, 1, 2, 5, 1, NA))
cac_test(col.names = c("R1", "R2", "R3", "R4"), data = ratings)

# Quadratic weights, Fleiss' kappa, tested against 0.6
res <- cac_test(col.names = c("R1", "R2", "R3", "R4"), data = ratings,
                coef = "fleiss", weights = "quadratic",
                null.value = 0.6, alternative = "greater")
res
summary(res)
plot(res)

Methods for ccc_htest objects

Description

Methods for objects returned by ccc_test(). They print as standard "htest" objects; summary() describes the result in words, plot() shows the agreement between the two methods, and check() gives diagnostics.

Usage

## S3 method for class 'ccc_htest'
summary(object, style = c("auto", "hypothesis", "estimation"), ...)

## S3 method for class 'ccc_htest'
plot(x, type = c("identity", "ba"), ...)

## S3 method for class 'ccc_htest'
check(x, ...)

Arguments

style

How summary() words the result: "auto" (default), "hypothesis", or "estimation", as for icc_test() results (see reli_htest-methods).

...

Not used.

x, object

An object of class ccc_htest.

type

For plot(): "identity" (default), the measurements of y against x with the line of identity (perfect concordance) and a Deming regression line, coloured by subject for replicate or nested data; or "ba", the Bland-Altman plot of the differences against the means (from agreement_limit() with the same data type).

Details

summary() describes the estimate and its interval, what was estimated, the design, and how the inference was done, in the hypothesis or estimation style described in reli_htest-methods. For simple paired data it also reports Lin's decomposition of the CCC into precision (the Pearson correlation, r) and accuracy (the bias correction factor, C_b): CCC = r x C_b. A low CCC can come from either.

check() shows the normality and heteroskedasticity of the differences (as for agreement_limit() with the same data), and the linearity of the relationship between the methods: the residuals of y regressed on x against the fitted values, with a test of a quadratic term (Ramsey's RESET test). The CCC, through its precision component, summarises a linear relationship. Proportional bias is not shown: a slope different from 1 between the methods lowers the bias correction factor C_b, so it is part of what the CCC measures rather than a violation of its assumptions. For simple paired data check() also shows a Q-Q plot of the squared Mahalanobis distances of the pairs against a chi-square distribution with 2 degrees of freedom: Lin's standard error assumes the two measurements are bivariate normal, and points far above the line suggest they are not. For replicate or nested data the linearity test uses all pairs, which are correlated within subjects, so its p-value is approximate; the jackknife standard error makes few distributional assumptions, so these panels mainly help interpret the CCC.

Value

summary

An object of class summary.ccc_htest (which prints like summary.reli_htest): the description in text, the style used, the S-value (s.value, when there is a p-value), and the parts used to build it.

plot

A ggplot2 plot (see type).

check

Diagnostic plots combined with patchwork.

References

Lin, L. I. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics, 45(1), 255-268. doi:10.2307/2532051

Carrasco, J. L., & Jover, L. (2003). Estimating the generalized concordance correlation coefficient through variance components. Biometrics, 59(4), 849-858. doi:10.1111/j.0006-341x.2003.00099.x

Ramsey, J. B. (1969). Tests for specification errors in classical linear least-squares regression analysis. Journal of the Royal Statistical Society: Series B, 31(2), 350-371. doi:10.1111/j.2517-6161.1969.tb00796.x

Examples

data("reps")
res <- ccc_test(x = "x", y = "y", data = reps)
summary(res)
plot(res)
plot(res, type = "ba")
check(res)

Concordance Correlation Coefficient

Description

[Experimental]

Computes Lin's Concordance Correlation Coefficient (CCC) and performs a hypothesis test, returning an object of class "htest".

Usage

ccc_test(
  x,
  y = NULL,
  id = NULL,
  data = NULL,
  data_type = c("simple", "reps", "nest"),
  conf.level = 0.95,
  null.value = NULL,
  alternative = c("two.sided", "greater", "less"),
  boot = FALSE,
  boot_ci = c("stud", "perc", "bca"),
  replicates = 999
)

Arguments

x

Either a numeric vector of measurements for method 1, or a character string naming the column in data containing method 1 measurements.

y

Either a numeric vector of measurements for method 2, or a character string naming the column in data containing method 2 measurements.

id

A character string naming the column in data containing subject identifiers. Required when data_type is "reps" or "nest".

data

An optional data frame containing the variables named in x, y, and id.

data_type

Character string specifying the data structure: "simple" for paired data with one observation per subject (default), "reps" for replicate data where subjects are measured multiple times, "nest" for nested data structures.

conf.level

Confidence level for the interval, default is 0.95.

null.value

The CCC under the null hypothesis. The default, NULL, gives the estimate and interval without a test.

alternative

Character string specifying the alternative hypothesis: "two.sided" (default), "greater", or "less". Used only when null.value is given (without a test the interval is two-sided); a one-sided alternative gives a one-sided interval, as in icc_test().

boot

If TRUE, use a nonparametric bootstrap that resamples subjects for the interval and p-value. Default FALSE. Available only for data_type = "reps", where it is recommended with few subjects (fewer than about 30) or clearly non-normal data; see Details.

boot_ci

Bootstrap interval type: "stud" (studentized, default and recommended), "perc" (percentile) or "bca". Used only when boot = TRUE.

replicates

Number of bootstrap replicates (default 999).

Details

The concordance correlation coefficient (CCC) quantifies agreement between two measurements of the same quantity. It is the product of a precision component (Pearson's correlation) and an accuracy component (the bias correction factor), and together these capture how far the observed data depart from the line of perfect concordance (the 45-degree line through the origin). Its purpose is closely related to that of dem_reg(). By analogy, Lin's CCC is to Pearson's r what Deming regression is to ordinary least squares. Deming regression estimates the relationship (slope and intercept) and asks whether it deviates from the line of identity, whereas the CCC summarizes that deviation in a single index.

Simple data. Lin's (1989) estimator and asymptotic standard error are used, from the complete pairs; these match DescTools::CCC() and epiR::epi.ccc() (with ci = "z-transform").

Replicate data ("reps"; the true value is constant within a subject). The CCC is estimated from variance components (Carrasco & Jover, 2003): a linear mixed model with a fixed method effect and random subject and subject-by-method effects, fit by REML, gives

CCC=σs2σs2+σsm2+σm2+σe2CCC = \frac{\sigma^2_{s}}{\sigma^2_{s} + \sigma^2_{sm} + \sigma^2_{m} + \sigma^2_{e}}

where σm2\sigma^2_{m} is the variance between the method means. All measurements are used, so the replicates of the two methods need not be paired or equal in number. The estimate equals that of cccrm::ccc_vc(int = TRUE).

Nested data ("nest"; the true value may change within a subject, and each row is a matched pair). Lin's estimator is computed over all complete pairs, which keep their pairing.

For both, the subject is the independent unit. The standard error of Fisher's z is a delete-one-subject jackknife, so it allows for the correlation of measurements within a subject. (Treating every pair as independent understates the standard error.) The interval and test use a t distribution with (number of subjects - 1) degrees of freedom. At least 3 subjects are needed.

In simulations (10, 20 or 50 subjects; 2, 3 or 6 measurements per subject; CCC 0.5 to 0.95; with and without a method difference and 15% missing data), 95% jackknife intervals covered about 93-95% of the time with normal data, but only 88-91% (replicate data) and 90-93% (nested data) with skewed subject effects and heavy-tailed errors, at every number of subjects. Misses fall mostly above the interval: the CCC is biased downward with few subjects (by about 0.03-0.04 with 10 subjects).

Bias. The jackknife estimate of that bias, and the bias-corrected CCC, are kept in the jackknife element, and summary() mentions them when the bias is more than 0.01 or more than 10% of the interval width. They are a guide, not a replacement for the estimate. In the simulations the correction removed 80-90% of the average bias with 10 or 20 subjects, but the corrected value was closer to the true CCC in only 43-50% of data sets, its root mean squared error was slightly larger (up to 28% larger with non-normal data and 10 subjects), and with 10 subjects it went past 1 (and is then truncated) in about 5% of replicate data sets. It is most useful when CCCs from several small studies are averaged, where the bias would otherwise accumulate. Centring the interval on the corrected value did not improve its coverage, so the interval and test use the uncorrected estimate.

Bootstrap (replicate data). With boot = TRUE, subjects are resampled with replacement (a subject drawn twice counts as two subjects) and the CCC is re-estimated. The studentized interval (boot_ci = "stud") works on Fisher's z scale and divides each replicate by its own delete-one-subject jackknife standard error; the p-value inverts the interval (as in icc_test()) and is approximate. In the simulations above it raised coverage with 10 subjects from 0.90 to 0.94 (from 0.88 to 0.94 with non-normal data) and with 20 subjects from 0.92 to 0.94, with intervals of about the same width, so boot = TRUE is a good idea for replicate data with few subjects. With non-normal data and 6 replicates per subject some designs still covered only 0.88-0.89. The percentile and BCa intervals covered poorly (0.80-0.91 on average) and are offered only for comparison. Each replicate needs one model fit per subject (from the subjects' summary statistics rather than lme4), so the studentized bootstrap takes roughly 20 seconds with 10 subjects and a minute with 30 subjects at the default 999 replicates. For nested data no bootstrap interval improved on the jackknife.

The confidence interval and the hypothesis test use Fisher's Z-transformation of the CCC:

Z=0.5×log⁡(1+CCC1−CCC)Z = 0.5 \times \log\left(\frac{1 + CCC}{1 - CCC}\right)

By default (null.value = NULL) no test is done.

Relation to the ICC. For two methods, the CCC is close to the absolute-agreement intraclass correlation, ICC(A,1) (icc_test(type = "agreement")). Lin's estimator (simple and nested data) uses n rather than n - 1 in the variances and covariance, as Lin (1989) defined it and as other software computes it, so it is slightly below the ICC in small samples; the difference vanishes as n grows (Nickerson, 1997; Carrasco & Jover, 2003). The variance-components estimator (replicate data) uses REML variance components and so corresponds to the ICC form. ccc_test() is meant for method-comparison (agreement) studies; icc_test() for reliability studies (see its "Choosing the form" and "Related coefficients" sections).

Methods. summary() describes the result in words (including, for simple data, whether a low CCC comes from poor precision or poor accuracy), plot() draws the identity or Bland-Altman plot, and check() gives diagnostics; see ccc_htest-methods.

Value

A list with class c("ccc_htest", "htest") containing the following components:

statistic

The test statistic on the Fisher z scale (Z for simple data, t for reps/nest), when a test was requested (omitted with boot = TRUE).

parameter

When a test was requested: the number of pairs n for simple data, or the degrees of freedom df (number of subjects - 1) for reps/nest. The number of pairs or subjects is also in settings$n.

stderr

The standard errors for CCC and Z-transformed CCC (for reps/nest, the jackknife SE of z, and the SE of the CCC from it by the delta method). With boot = TRUE, also boot: the standard deviation of the bootstrap estimates.

p.value

The p-value for the test, when a test was requested.

conf.int

A confidence interval for the CCC.

estimate

The estimated CCC.

null.value

The specified hypothesized value of CCC, if any.

alternative

The alternative hypothesis.

method

A character string indicating the method used.

data.name

A character string giving the names of the data.

data

The data used (x, y, and id for reps/nest), for the methods.

components

For simple data, Lin's decomposition: the Pearson correlation r (precision) and the bias correction factor C_b (accuracy), with CCC = r x C_b. NULL for reps/nest.

jackknife

For reps/nest, the delete-one-subject jackknife estimate of the small-sample bias: a list with bias ((g - 1) times the mean leave-one-out estimate minus the estimate), estimate_bc (the bias-corrected CCC, estimate - bias, kept within -1 to 1) and clamped (TRUE when the correction went past 1 or -1). Not printed, and not used by the interval or test; see Details. NULL for simple data.

settings

The settings used, for the methods.

boot

Bootstrap details (NULL unless boot = TRUE): the interval type, the number of replicates requested, used and failed, and the bootstrap estimates.

References

Lin, L. I. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics, 45(1), 255-268.

Lin, L. I. (2000). A note on the concordance correlation coefficient. Biometrics, 56(1), 324-325.

Carrasco, J. L., & Jover, L. (2003). Estimating the generalized concordance correlation coefficient through variance components. Biometrics, 59(4), 849-858. doi:10.1111/j.0006-341x.2003.00099.x

Nickerson, C. A. E. (1997). A note on "A concordance correlation coefficient to evaluate reproducibility". Biometrics, 53(4), 1503-1507. doi:10.2307/2533516

See Also

ccc_htest-methods, icc_test(), agreement_limit()

Examples

# Simple paired data using vectors
x <- c(1, 2, 3, 4, 5)
y <- c(1.1, 2.2, 2.9, 4.1, 5.2)
ccc_test(x, y)

# Using data frame interface
data("reps")
# Simple data
ccc_test(x = "x", y = "y", data = reps)

# Test against specific null value
ccc_test(x = "x", y = "y", data = reps,
         null.value = 0.8, alternative = "greater")

# Replicate data
ccc_test(x = "x", y = "y", id = "id", data = reps, data_type = "reps")

# Replicate data with few subjects: studentized subject bootstrap

ccc_test(x = "x", y = "y", id = "id", data = reps, data_type = "reps",
         boot = TRUE)


# Nested data
ccc_test(x = "x", y = "y", id = "id", data = reps, data_type = "nest")

# Describe, plot and check
res <- ccc_test(x = "x", y = "y", data = reps)
summary(res)
plot(res)
check(res)

Deming Regression

Description

[Stable]

A function for fitting a straight line to two-dimensional data (i.e., X and Y) that are measured with error.

Usage

dem_reg(
  formula = NULL,
  data,
  id = NULL,
  x = NULL,
  y = NULL,
  conf.level = 0.95,
  weighted = FALSE,
  weights = NULL,
  error.ratio = 1,
  model = TRUE,
  keep_data = FALSE,
  ...
)

Arguments

formula

A formula of the form y ~ x specifying the model. If provided, takes precedence over x and y arguments.

data

Data frame with all data.

id

Column with subject identifier (optional).

x

Name of column with first measurement (deprecated in favor of formula interface).

y

Name of other column with the other measurement to compare to the first (deprecated in favor of formula interface).

conf.level

The confidence level required. Default is 95%.

weighted

Logical indicator (TRUE/FALSE) for whether to use weighted Deming regression. Default is FALSE.

weights

an optional vector of weights to be used in the fitting process. Should be NULL or a numeric vector.

error.ratio

Ratio of the measurement error variance of x to that of y, i.e., var(x)/var(y), where var() denotes the error (not total) variance of each method. Default is 1 (equal error variances). Values greater than 1 indicate that x is measured with more error than y. This argument is ignored if subject identifiers are provided.

model

Logical. If TRUE (default), the model frame is stored in the returned object. This is needed for methods like plot(), fitted(), residuals(), and predict() to work without supplying data. If FALSE, the model frame is not stored (saves memory for large datasets), but these methods will require a data argument.

keep_data

Logical indicator (TRUE/FALSE). If TRUE, the jacknife samples are returned; default is FALSE.

...

Additional arguments (currently unused).

Details

This function provides a Deming regression analysis wherein the sum of distances in both x and y direction is minimized. Deming regression, also known as error-in-variable regression, is useful in situations where both X & Y are measured with error. The use of Deming regression is beneficial when comparing to methods for measuring the same continuous variable.

Currently, the dem_reg function covers simple Deming regression and weighted Deming regression. Weighted Deming regression can be used by setting the weighted argument to TRUE. The weights can be provided by the user or can be calculated within function.

If the data are measured in replicates, then the measurement error can be directly derived from the data. This can be accomplished by indicating the subject identifier with the id argument. When the replicates are not available in the data, then the ratio of error variances, var(x)/var(y), can be provided with the error.ratio argument. When subject identifiers are provided, the error ratio is instead estimated from the replicates as the within-subject variance of x divided by the within-subject variance of y.

Value

The function returns a simple_eiv (eiv meaning "error in variables") object with the following components:

  • coefficients: Named vector of coefficients (intercept and slope).

  • residuals: Optimized residuals from the fitted model.

  • fitted.values: Estimated true Y values (Y-hat).

  • model_table: Data frame presenting the full results from the Deming regression analysis.

  • vcov: Variance-covariance matrix for slope and intercept.

  • df.residual: Residual degrees of freedom.

  • call: The matched call.

  • terms: The terms object used.

  • xlevels: (Only for models with factors) levels of factors.

  • model: The model frame.

  • x_vals: Original x values used in fitting.

  • y_vals: Original y values used in fitting.

  • x_hat: Estimated true X values.

  • y_hat: Estimated true Y values.

  • error.ratio: Error ratio used in fitting.

  • weighted: Whether weighted regression was used.

  • weights: Weights used in fitting.

  • conf.level: Confidence level used.

  • resamples: List containing resamples from jacknife procedure (if keep_data = TRUE).

Interface Change

The x and y arguments are deprecated. Please use the formula interface instead:

  • Old: dem_reg(x = "x_var", y = "y_var", data = df)

  • New: dem_reg(y_var ~ x_var, data = df)

References

Linnet, K. (1990) Estimation of the linear relationship between the measurements of two methods with proportional errors. Statistics in Medicine, 9, 1463-1473.

Linnet, K. (1993). Evaluation of regression procedures for methods comparison studies. Clinical chemistry, 39, 424-432.

Sadler, W.A. (2010). Joint parameter confidence regions improve the power of parametric regression in method-comparison studies. Accreditation and Quality Assurance, 15, 547-554.

Examples

# Peak flow: mini vs Wright meter (Bland & Altman, 1986)
data("ba1986")
fit <- dem_reg(mini1 ~ wright1, data = ba1986)
summary(fit)
plot(fit)

# Joint test of intercept = 0 and slope = 1
joint_test(fit)

# Known error ratio, var(x) / var(y)
dem_reg(mini1 ~ wright1, data = ba1986, error.ratio = 2)

# Replicate measurements: the error ratio is estimated from the replicates
data("reps")
dem_reg(y ~ x, data = reps, id = "id")

# Weighted Deming regression (error proportional to the level)
dem_reg(mini1 ~ wright1, data = ba1986, weighted = TRUE)

Simulate Deming Regression Power

Description

[Experimental]

Functions for conducting power analysis and sample size determination for Deming regression in method comparison studies. These functions help determine the sample size needed to detect specified biases (proportional and/or constant) between two measurement methods.

Estimates statistical power to detect deviations from the line of identity using simulated data with known properties.

Usage

deming_power_sim(
  n_sims = 1000,
  sample_size = 50,
  x_range = c(10, 100),
  x_dist = "uniform",
  actual_slope = 1.05,
  actual_intercept = 0,
  ideal_slope = 1,
  ideal_intercept = 0,
  y_var_params = list(beta1 = 1, beta2 = 0, J = 1, type = "constant"),
  x_var_params = list(beta1 = 1, beta2 = 0, J = 1, type = "constant"),
  weighted = FALSE,
  conf.level = 0.95,
  joint_method = c("sadler", "wald")
)

Arguments

n_sims

Number of simulation iterations. Default is 1000.

sample_size

Sample size (number of paired observations) per simulation.

x_range

Numeric vector of length 2 specifying min and max of X values (e.g., c(10, 100)).

x_dist

Character specifying distribution of X values: "uniform", "central", or "right_skewed".

actual_slope

Actual slope used to generate data (e.g., 1.05 for 5% proportional bias).

actual_intercept

Actual intercept used to generate data.

ideal_slope

Hypothesized slope to test against (typically 1 for identity line).

ideal_intercept

Hypothesized intercept to test against (typically 0 for identity line).

y_var_params

List with Y variance parameters: beta1, beta2, J, type. Type can be "constant", "proportional", or "power". For power function: sigma^2 = (beta1 + beta2*U)^J

x_var_params

List with X variance parameters (same structure as y_var_params).

weighted

Logical. Use weighted Deming regression? Default is FALSE.

conf.level

Confidence level for tests. Default is 0.95.

joint_method

The joint test, passed to joint_test() as method: "sadler" (default, Sadler's sum-of-squares F test) or "wald" (jackknife Wald ellipse, liberal in small samples).

Details

Power Analysis for Deming Regression

This function generates simulated datasets with specified error characteristics and tests whether confidence intervals and joint confidence regions detect deviations from hypothesized values. The joint confidence region typically provides higher statistical power, especially when the X-range is narrow (high slope-intercept correlation).

Each simulated data set is fitted with dem_reg() and tested with joint_test() and the marginal confidence intervals, exactly as an analysis would be, so the estimated power is the power of those tests. The error ratio passed to dem_reg() is the ratio of the two variance functions at the mean of the simulated x values. Fits that fail count as non-detections.

The variance functions allow flexible modeling of heteroscedastic errors:

  • constant: sigma^2 = beta1

  • proportional: sigma^2 = beta1 * U^2 (constant CV%)

  • power: sigma^2 = (beta1 + beta2*U)^J

Value

A list of class "deming_power" containing:

power_ci_slope

Power based on slope confidence interval

power_ci_intercept

Power based on intercept confidence interval

power_either_ci

Power when either CI detects difference

power_joint

Power based on joint confidence region

settings

List of simulation settings

advantage

Difference in power between joint region and CIs

References

Sadler, W.A. (2010). Joint parameter confidence regions improve the power of parametric regression in method-comparison studies. Accreditation and Quality Assurance, 15, 547-554.

Examples

# Simple example: detect 5% proportional bias with constant variance
# (use n_sims = 1000 or more in practice)
power_result <- deming_power_sim(
  n_sims = 200,
  sample_size = 50,
  x_range = c(10, 100),
  actual_slope = 1.05,
  ideal_slope = 1.0,
  y_var_params = list(beta1 = 25, beta2 = 0, J = 1, type = "constant"),
  x_var_params = list(beta1 = 20, beta2 = 0, J = 1, type = "constant")
)
print(power_result)

# More complex: heteroscedastic errors
power_result2 <- deming_power_sim(
  n_sims = 200,
  sample_size = 75,
  x_range = c(1, 100),
  actual_slope = 1.03,
  y_var_params = list(beta1 = 0.5, beta2 = 0.05, J = 2, type = "power"),
  x_var_params = list(beta1 = 0.4, beta2 = 0.04, J = 2, type = "power"),
  weighted = TRUE
)

Determine Required Sample Size for Deming Regression

Description

[Experimental]

Automatically determines the minimum sample size needed to achieve target statistical power for detecting specified bias in method comparison studies using Deming regression.

Usage

deming_sample_size(
  target_power = 0.9,
  initial_n = 20,
  max_n = 500,
  n_sims = 500,
  use_joint = NULL,
  step_size = 5,
  ...
)

Arguments

target_power

Desired statistical power (e.g., 0.80 or 0.90). Default is 0.90.

initial_n

Starting sample size for search. Default is 20.

max_n

Maximum sample size to try. Default is 500.

n_sims

Number of simulations per sample size tested. Default is 500.

use_joint

Which target ends the search. NULL (default) searches until both the joint-region and the confidence-interval targets are reached, so both sample sizes (and the reduction) are reported. TRUE stops at the joint-region target and FALSE at the confidence-interval target; the other sample size is then reported only if it was reached first.

step_size

Step size for sample size increments. Default is 5.

...

Additional arguments passed to deming_power_sim()

Details

This function performs a grid search over sample sizes to find the minimum N needed to achieve the target power. It tests both confidence interval and joint confidence region approaches, allowing comparison of required sample sizes.

Using joint confidence regions typically requires far fewer samples than confidence intervals when the measurement range is narrow (max:min ratio < 10:1).

Each power is a simulation estimate (Monte Carlo SE about 1.3 percentage points at 90% power with n_sims = 500), and the search takes the first N whose estimate reaches the target, which tends to pick N slightly too small. In simulations the true power at the returned N was close to the target on average but below 85% in up to 20% of runs. Use a larger n_sims (or check the chosen N with deming_power_sim()) when the sample size matters. N is never below initial_n.

Value

A list of class "deming_sample_size" containing:

n_required_ci

Required N for confidence intervals

n_required_joint

Required N for joint confidence region

target_power

Target power level

power_curve

Data frame with N and power for both methods

reduction_n

Sample size reduction using joint method

reduction_pct

Percentage reduction

Examples

# Determine N needed for 90% power to detect 5% bias with the joint
# region (use_joint = TRUE stops there; n_sims = 500 or more in practice)
sample_size_result <- deming_sample_size(
  target_power = 0.90,
  initial_n = 10,
  max_n = 200,
  n_sims = 200,
  step_size = 10,
  use_joint = TRUE,
  x_range = c(20, 200),
  actual_slope = 1.05,
  ideal_slope = 1.0,
  y_var_params = list(beta1 = 1, beta2 = 0.05, J = 2, type = "power"),
  x_var_params = list(beta1 = 0.8, beta2 = 0.04, J = 2, type = "power")
)

print(sample_size_result)
plot(sample_size_result)

Intraclass Correlation Coefficient Test

Description

[Stable]

Estimates one intraclass correlation coefficient (ICC), with a confidence interval and a test against a null value, and returns an object of class "htest". The ICC form is chosen with model, type and unit, following McGraw and Wong (1996).

Usage

icc_test(
  x = NULL,
  data = NULL,
  measure = NULL,
  item = NULL,
  id = NULL,
  col.names = NULL,
  model = c("twoway", "oneway"),
  type = c("consistency", "agreement"),
  unit = c("single", "average"),
  method = c("anova", "reml"),
  conf.level = 0.95,
  null.value = NULL,
  alternative = c("two.sided", "greater", "less"),
  boot = FALSE,
  boot_ci = c("auto", "perc", "bca", "stud", "basic"),
  replicates = 1999,
  se_method = c("jackknife", "influence", "normal")
)

Arguments

x

A formula of the form values ~ item | id, where values is the measurement, item identifies the trial, rater or occasion, and id identifies the subject. Use either x, the measure/item/id arguments, or col.names.

data

A data frame containing the variables.

measure, item, id

Column names (character) of the measurement, the item (trial, rater, or occasion) and the subject identifier, for data in long format.

col.names

Column names (character) of the items, for data in wide format (one row per subject).

model

"twoway" (default) when every subject is measured on the same set of items (e.g. test-retest trials), or "oneway" when each subject is measured by a different set of items (e.g. different raters).

type

"consistency" (default) excludes systematic differences between items from the ICC; "agreement" includes them (absolute agreement). Only "agreement" is defined for model = "oneway".

unit

"single" (default) for the reliability of a single measurement, or "average" for the reliability of the mean of all k items.

method

"anova" (default) estimates the variance components from the analysis of variance and requires complete, balanced data. "reml" fits a linear mixed model and allows missing measurements.

conf.level

Confidence level of the interval (default 0.95).

null.value

The ICC under the null hypothesis. The default, NULL, gives the estimate and interval without a test.

alternative

"two.sided" (default), "greater", or "less". Used only when null.value is given (without a test the interval is two-sided). A one-sided alternative gives a one-sided interval (see Details).

boot

If TRUE, use a nonparametric bootstrap that resamples subjects for the interval and p-value. Default FALSE.

boot_ci

Bootstrap interval type: "auto" (default), "perc" (percentile), "bca", "stud" (studentized), or "basic". "auto" uses "stud" for method = "anova" and "perc" for method = "reml" (and "perc" when se_method = "normal"); see Details. Used only when boot = TRUE.

replicates

Number of bootstrap replicates (default 1999).

se_method

How the standard error is estimated: "jackknife" (default; delete one subject at a time), "influence" (per-subject influence, or sandwich, estimate), or "normal" (normal theory). The SE is reported in stderr and is used by the studentized bootstrap; it does not change the F-based interval or test. See Details.

Details

ICC forms.. The different ICCs can be confusing. Therefore, I have summarized it here. McGraw and Wong (1996) and Shrout and Fleiss (1979) name the forms differently:

model type unit McGraw & Wong Shrout & Fleiss
oneway agreement single ICC(1) ICC(1,1)
oneway agreement average ICC(k) ICC(1,k)
twoway agreement single ICC(A,1) ICC(2,1)
twoway agreement average ICC(A,k) ICC(2,k)
twoway consistency single ICC(C,1) ICC(3,1)
twoway consistency average ICC(C,k) ICC(3,k)

Whether items are treated as random or fixed changes the interpretation but not the calculation (McGraw & Wong, 1996). Average-measure forms are not defined for McGraw and Wong's two-way mixed model with an interaction.

Choosing the form. Answer three questions in order: (1) is every subject measured by the same items? Yes: "twoway"; no, each subject by different items: "oneway". (2) Should systematic differences between items count as error? Yes, when measurements from different items will be used interchangeably or on their absolute values (for example compared with a cut-off): "agreement"; no, when only the subjects' relative standing matters or systematic differences are reported separately: "consistency". (3) Will one measurement or the mean of the k items be used in practice? "single" or "average". The "Reliability Analysis" vignette has a table of common designs (vignette("reliability_analysis", package = "SimplyAgree")).

Related coefficients. With complete data, Cronbach's (1951) alpha equals ICC(C,k), the average-measure consistency ICC (McGraw & Wong, 1996): it describes the mean of the k items, not a single item. For two methods, Lin's concordance correlation coefficient (ccc_test()) is essentially ICC(A,1); the estimators differ only in using n rather than n - 1 in the variances (Nickerson, 1997; Carrasco & Jover, 2003).

Default form. The default, ICC(C,1), follows Hopkins (2000) and Weir (2005): check for systematic change between trials separately (for example with the trials F test, or t.test() for two trials), then report the consistency ICC, which is tied to the standard error of measurement (MSE\sqrt{MS_E}). Others, such as Koo and Li (2016), recommend absolute agreement (type = "agreement") for test-retest studies. Which form is appropriate depends on whether systematic differences between trials should count as error.

Average-measure forms are computed as the Spearman-Brown projection of the single-measure form to all k items. With complete data this is identical to the McGraw and Wong formulas; with missing data it keeps the estimand the reliability of the mean of all k items.

Confidence intervals are the F-based intervals of Shrout and Fleiss (1979) and McGraw and Wong (1996, Table 7), with α/2\alpha/2 in each tail. They are exact for the one-way and consistency forms and approximate (Satterthwaite degrees of freedom) for the agreement forms. With alternative = "greater" the interval is one-sided, ⁠[L, 1]⁠, where L is the one-sided conf.level lower bound; with "less" it is ⁠[-Inf, U]⁠.

Tests. By default (null.value = NULL) no test is done. The result is the estimate and a two-sided interval when printed to the console. Testing against an ICC of 0, the default in many programs, is rarely informative for reliability. When null.value is given, the tests are the F tests of McGraw and Wong (1996, Table 8) of H0:ρ=ρ0H_0: \rho = \rho_0. They are exact for the one-way and consistency forms, and invert exactly to the interval. For the agreement forms the test and the interval are separate Satterthwaite approximations (degrees of freedom evaluated at ρ0\rho_0 for the test and at the estimate for the interval), so with few subjects they can disagree near the interval limits; McGraw and Wong note that their power cannot be determined exactly.

Missing data. method = "reml" estimates the variance components from a linear mixed model and uses degrees of freedom based on the observed data (subject df n−1n - 1, error df N−n−k+1N - n - k + 1, with N the number of observations). The intervals and p-values are then approximate and a warning is given. In simulations with 20 subjects and 3 trials, 95% intervals covered 0.93 to 0.97 with 15% of measurements missing and 0.91 to 0.95 with 30% missing; coverage depends on the design and the pattern of missingness. boot = TRUE is an alternative.

Bootstrap. With boot = TRUE, subjects are resampled with replacement (keeping all of each subject's measurements) and the model is refitted. The BCa interval uses a leave-one-subject-out jackknife for the acceleration. The studentized interval works on the scale z=0.5log⁡Fz = 0.5 \log F and divides each replicate by its own standard error from se_method; a normal-theory SE cannot be used because it depends only on the degrees of freedom, which would reduce the studentized interval to the basic interval on the z scale. The p-value is obtained by inverting the chosen interval (as in the TOSTER package), so it is approximate and consistent with the interval. With method = "reml" each replicate refits the mixed model, which can be slow.

For the two-way agreement forms, resampling subjects alone would treat the items as fixed and ignore the uncertainty from sampling the items, which under-covered badly in simulations (0.70 to 0.80 with an ICC of 0.9). Each replicate therefore also multiplies the item mean square by an independent χk−12/(k−1)\chi^2_{k-1}/(k-1) draw, its normal-theory sampling variation (a hybrid bootstrap developed for this package). Like the F interval, this treats the items as a random sample; when the items are fixed it is conservative.

The bootstrap is mainly useful when the data are clearly non-normal. In simulations (3 trials, 10 or 30 subjects, ICC 0.5 or 0.9), the F intervals covered about 0.92 to 0.97 with normal, complete data but only 0.80 to 0.93 with skewed or heavy-tailed data. With complete data, the studentized bootstrap with the jackknife SE covered 0.90 to 0.965 for the consistency ICC, including 0.94 to 0.955 with skewed data and 30 subjects, and was the best method overall; for the agreement ICC it was roughly tied with the percentile interval (0.89 to 0.955), whose lower bound was too high less often. With 20% of measurements missing and method = "reml", the studentized interval (which then needs the influence SE) was unreliable at moderate ICCs (as low as 0.80), while the percentile interval covered 0.87 to 0.96. Hence boot_ci = "auto", and a warning when "stud" is requested with method = "reml". BCa never improved on the percentile interval. Every method under-covered with 10 subjects, skewed data and an ICC of 0.9 (0.84 to 0.91).

Standard errors. stderr gives the standard error of z=0.5log⁡Fz = 0.5 \log F (the scale of the studentized bootstrap) and, by the delta method, of the ICC. The jackknife (default) deletes one subject at a time; with complete data it uses closed-form leave-one-out mean squares, and with method = "reml" it refits the model n times. The influence estimate is a sandwich estimate from each subject's influence on the mean squares (with missing data, an approximation from each subject's observed measurements); it is cheaper but too small with few subjects. The normal-theory SE depends only on the degrees of freedom. In simulations (3 trials, 20 to 40 subjects, up to 30% missing), the jackknife SE averaged 0.89 to 1.08 times the true sampling SD, the influence SE 0.69 to 1.00 (lowest with few subjects, skewed data, or data missing at random), and the normal-theory SE 0.87 to 1.04 with normal data but 0.60 to 0.80 with skewed or heavy-tailed data. For the agreement forms, the jackknife and influence SEs add the normal-theory variance of the item mean square, treating the items as random. The studentized bootstrap needs an SE estimated from each resample, so it cannot use "normal"; with method = "reml" it requires "influence", because the jackknife would refit the model n times in every replicate.

Interpreting the ICC. No qualitative labels (such as "good" or "excellent") are given: published cutoffs disagree, and the ICC depends on how much subjects differ from each other (Weir, 2005). To claim that reliability exceeds a threshold, test against it, e.g. ⁠null.value = 0.75, alternative = "greater"⁠.

Value

A list of class c("reli_htest", "htest") containing:

statistic

The F statistic (omitted for bootstrap results).

parameter

The numerator and denominator degrees of freedom (omitted for bootstrap results).

p.value

The p-value of the test.

conf.int

The confidence interval for the ICC.

estimate

The estimated ICC, named by its McGraw and Wong form.

stderr

Standard errors of the ICC (ICC) and of z=0.5log⁡Fz = 0.5 \log F (z), from se_method. With boot = TRUE, also boot: the standard deviation of the bootstrap ICC estimates.

null.value

The hypothesized ICC.

alternative

The alternative hypothesis.

method

A description of the ICC and the inference.

data.name

A description of the data.

fit

The fitted variance components, mean squares and model.

data

The data in long format.

settings

The settings used, for the methods.

boot

Bootstrap details (NULL unless boot = TRUE).

References

Carrasco, J. L., & Jover, L. (2003). Estimating the generalized concordance correlation coefficient through variance components. Biometrics, 59(4), 849-858. doi:10.1111/j.0006-341x.2003.00099.x

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334. doi:10.1007/BF02310555

Hopkins, W. G. (2000). Measures of reliability in sports medicine and science. Sports Medicine, 30(1), 1-15. doi:10.2165/00007256-200030010-00001

Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155-163. doi:10.1016/j.jcm.2016.02.012

McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30-46. doi:10.1037/1082-989X.1.1.30

Nickerson, C. A. E. (1997). A note on "A concordance correlation coefficient to evaluate reproducibility". Biometrics, 53(4), 1503-1507. doi:10.2307/2533516

Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420-428. doi:10.1037/0033-2909.86.2.420

Ukoumunne, O. C., Davison, A. C., Gulliford, M. C., & Chinn, S. (2003). Non-parametric bootstrap confidence intervals for the intraclass correlation coefficient. Statistics in Medicine, 22(24), 3805-3821. doi:10.1002/sim.1643

Weir, J. P. (2005). Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. Journal of Strength and Conditioning Research, 19(1), 231-240. doi:10.1519/15184.1

See Also

reli_htest-methods for summary(), plot() and check().

Examples

# Shrout & Fleiss (1979) example: 6 subjects rated by 4 judges
sf <- data.frame(J1 = c(9, 6, 8, 7, 10, 6), J2 = c(2, 1, 4, 1, 5, 2),
                 J3 = c(5, 3, 6, 2, 6, 4), J4 = c(8, 2, 8, 6, 9, 7))
icc_test(col.names = c("J1", "J2", "J3", "J4"), data = sf)

# Absolute agreement, tested against 0.5
icc_test(col.names = c("J1", "J2", "J3", "J4"), data = sf,
         type = "agreement", null.value = 0.5, alternative = "greater")

# Long data with a formula
sf_long <- data.frame(id = rep(1:6, times = 4),
                      judge = rep(c("J1", "J2", "J3", "J4"), each = 6),
                      score = c(sf$J1, sf$J2, sf$J3, sf$J4))
res <- icc_test(score ~ judge | id, data = sf_long)
summary(res)

Simple Agreement Analysis

Description

Simple Agreement Analysis

Usage

jmvagree(
  data,
  method1,
  method2,
  ciWidth = 95,
  agreeWidth = 95,
  testValue = 2,
  CCC = TRUE,
  plotbland = TRUE,
  plotcon = FALSE,
  plotcheck = FALSE,
  prop_bias = FALSE,
  xlabel = "Average of Both Methods",
  ylabel = "Difference between Methods"
)

Arguments

data

Data

method1

Name of column containing 1st Vector of data

method2

Name of column containing Vector of data

ciWidth

a number between 50 and 99.9 (default: 95), the width of confidence intervals

agreeWidth

a number between 50 and 99.9 (default: 95), the width of agreement limits

testValue

a number specifying the limit of agreement

CCC

TRUE or FALSE (default), produce CCC table

plotbland

TRUE or FALSE (default), for Bland-Altman plot

plotcon

TRUE or FALSE (default), for Bland-Altman plot

plotcheck

TRUE or FALSE (default), assumptions plots

prop_bias

TRUE or FALSE

xlabel

The label for the x-axis on the BA plot

ylabel

The label for the y-axis on the BA plot

Value

A results object containing:

results$text a html
results$blandtab a table
results$ccctab a table
results$plotba an image
results$plotcon an image
results$plotcheck an image

Tables can be converted to data frames with asDF or as.data.frame. For example:

results$blandtab$asDF

as.data.frame(results$blandtab)


Nested/Replicate Data Agreement Analysis

Description

Nested/Replicate Data Agreement Analysis

Usage

jmvagreemulti(
  data,
  method1,
  method2,
  id,
  ciWidth = 95,
  agreeWidth = 95,
  testValue = 2,
  CCC = TRUE,
  valEq = FALSE,
  plotbland = FALSE,
  plotcon = FALSE,
  prop_bias = FALSE,
  xlabel = "Average of Both Methods",
  ylabel = "Difference between Methods"
)

Arguments

data

Data

method1

Name of column containing 1st Vector of data

method2

Name of column containing Vector of data

id

Name of column containing subject identifier

ciWidth

a number between 50 and 99.9 (default: 95), the width of confidence intervals

agreeWidth

a number between 50 and 99.9 (default: 95), the width of agreement limits

testValue

a number specifying the limit of agreement

CCC

TRUE or FALSE (default), produce CCC table

valEq

.

plotbland

TRUE or FALSE (default), for Bland-Altman plot

plotcon

TRUE or FALSE (default), for Line of identity plot

prop_bias

TRUE or FALSE

xlabel

The label for the x-axis on the BA plot

ylabel

The label for the y-axis on the BA plot

Value

A results object containing:

results$text a preformatted
results$blandtab a table
results$ccctab a table
results$plotba an image
results$plotcon an image

Tables can be converted to data frames with asDF or as.data.frame. For example:

results$blandtab$asDF

as.data.frame(results$blandtab)


Deming Regression

Description

Deming Regression

Usage

jmvdeming(
  data,
  method1,
  method2,
  ciWidth = 95,
  testValue = 1,
  plotcon = FALSE,
  plotcheck = FALSE,
  weighted = FALSE,
  xlabel = "Method: 1",
  ylabel = "Method: 2"
)

Arguments

data

Data

method1

Name of column containing 1st Vector of data

method2

Name of column containing Vector of data

ciWidth

a number between 50 and 99.9 (default: 95), the width of confidence intervals

testValue

Ratio of the error variances, Method 1 (x) / Method 2 (y). Default is 1.

plotcon

TRUE or FALSE (default), for Bland-Altman plot

plotcheck

TRUE or FALSE (default), assumptions plots

weighted

TRUE or FALSE

xlabel

The label for the x-axis

ylabel

The label for the y-axis

Value

A results object containing:

results$text a html
results$demtab a table
results$plotcon an image
results$plotcheck an image

Tables can be converted to data frames with asDF or as.data.frame. For example:

results$demtab$asDF

as.data.frame(results$demtab)


Reliability Analysis

Description

Reliability Analysis

Usage

jmvreli(data, vars, ciWidth = 95, desc = FALSE, plots = FALSE)

Arguments

data

the data as a data frame

vars

a list of the column names containing the measurements for reliability analysis.

ciWidth

a number between 50 and 99.9 (default: 95), the width of confidence intervals

desc

TRUE or FALSE (default), provide table of variance components

plots

TRUE or FALSE (default), plot data

Value

A results object containing:

results$text a html
results$icctab a table
results$vartab a table
results$plots an image

Tables can be converted to data frames with asDF or as.data.frame. For example:

results$icctab$asDF

as.data.frame(results$icctab)


Joint Confidence Region for Intercept and Slope

Description

[Experimental]

joint_test() tests whether the estimated intercept and slope jointly fall within a confidence region around specified ideal values (typically intercept = 0 and slope = 1 for method comparison studies). plot_joint() draws that region, with the ideal point marked as enclosed or not.

Usage

joint_test(object, ...)

## S3 method for class 'simple_eiv'
joint_test(
  object,
  ideal_intercept = 0,
  ideal_slope = 1,
  conf.level = NULL,
  test_method = c("F", "asymptotic"),
  method = NULL,
  ...
)

plot_joint(x, ...)

## S3 method for class 'simple_eiv'
plot_joint(
  x,
  ideal_intercept = 0,
  ideal_slope = 1,
  conf.level = NULL,
  test_method = c("F", "asymptotic"),
  show_intervals = TRUE,
  n_points = 100,
  method = NULL,
  ...
)

Arguments

object, x

A simple_eiv object from dem_reg() or pb_reg().

...

Additional arguments (currently unused).

ideal_intercept

The hypothesized intercept value (default: 0).

ideal_slope

The hypothesized slope value (default: 1).

conf.level

Confidence level for the region and test. The default, NULL, uses the level the model was fitted with (object$conf.level).

test_method

Reference distribution for the Wald test and ellipse (method = "wald" only). "F" (default) uses F(2, n-2); "asymptotic" uses chi-squared(2), the large-sample approximation. The Sadler test always uses F(2, n-2).

method

Which joint region and test. NULL (default) uses "sadler" for Deming fits and "wald" for Passing-Bablok fits. "sadler" is the sum-of-squares region of Sadler (2010), available for dem_reg() fits only. "wald" is the ellipse from the variance-covariance matrix (the only option for pb_reg()). See Details.

show_intervals

Logical (plot_joint() only). If TRUE (default), also shows the model's marginal confidence intervals as a rectangle. They are shown only when conf.level matches the level the model was fitted with.

n_points

Number of points used to draw the region (plot_joint() only; default 100).

Details

Sadler region (Deming, default)

For a Deming fit with error ratio λ\lambda and weights wiw_i, let

S(a,b)=∑iwi(yi−a−bxi)21+λb2,S(a, b) = \sum_i \frac{w_i (y_i - a - b x_i)^2}{1 + \lambda b^2},

the weighted sum of squares that Deming regression minimises (Sadler, 2010, Eq. 11; Draper & Smith). With S^\hat S its minimum, the test of (a0,b0)(a_0, b_0) is

F=(S(a0,b0)−S^)/2S^/(n−2)F = \frac{(S(a_0, b_0) - \hat S) / 2}{\hat S / (n - 2)}

on F(2, n - 2), and the confidence region is the set of (a,b)(a, b) where FF does not exceed the critical value. No variance-covariance matrix is involved; only the error ratio (not the size of the errors) matters. The region is not an ellipse: for each slope its intercept limits have a closed form, and its slope limits are the roots of a quadratic, so plot_joint() draws it exactly. If the data are too imprecise for the slope to be bounded, the test is still reported but the region is not drawn.

In simulations with normal errors (n = 10 to 100, narrow and wide ranges, error ratios 1 and 4, constant and proportional errors, and replicate data with the error ratio estimated), the Sadler test rejected a true null hypothesis 4.3-5.9% of the time at a nominal 5%. It was at least as powerful as the Wald test at the same size.

The model frame must be stored (dem_reg(model = TRUE), the default). dem_reg(weighted = TRUE) uses fixed weights computed from the data, while Sadler reweighted iteratively; the region uses the fitted model's weights.

Wald ellipse

method = "wald" computes the Mahalanobis distance between the estimated coefficients and the hypothesized values, using the variance-covariance matrix of the estimates, against 2 F(2, n - 2) (or chi-squared(2) with test_method = "asymptotic").

For Deming regression the matrix is the delete-one jackknife. This test is liberal in small samples: in the same simulations it rejected a true null 7.8-9.7% of the time at n = 10, about 7.4% at n = 20 and 6.1% at n = 50 (up to 14% with test_method = "asymptotic"), because the jackknife variance is itself unstable. Prefer the Sadler test for Deming fits.

For Passing-Bablok regression the model must have been fitted with replicates > 0 and se_method = "bootstrap" or "jackknife". Both held 4.3-5.4% in simulations (n = 20 to 100). The Dufey (2020) matrix (se_method = "dufey") is not used for joint tests: it rejected a true null 8-11% of the time.

Value

plot_joint() returns a ggplot. joint_test() returns an object of class htest containing:

statistic

For method = "sadler", the F statistic; for method = "wald", the Mahalanobis distance.

parameter

Degrees of freedom. For the F-based tests, df1 and df2; for the asymptotic test, df (always 2).

p.value

The p-value for the test.

conf.int

The confidence level used.

estimate

Named vector of estimated intercept and slope.

null.value

Named vector of hypothesized intercept and slope.

alternative

Description of the alternative hypothesis.

method

Description of the test.

data.name

Name of the input object.

References

Sadler, W.A. (2010). Joint parameter confidence regions improve the power of parametric regression in method-comparison studies. Accreditation and Quality Assurance, 15, 547-554.

Draper, N.R., & Smith, H. (1966). Applied Regression Analysis. Wiley, New York.

Examples

data("reps")
fit <- dem_reg(y ~ x, data = reps)
joint_test(fit)
plot_joint(fit)

# The Wald ellipse from the jackknife variance-covariance matrix
joint_test(fit, method = "wald")

Limits of Agreement with Linear Mixed Effects

Description

[Deprecated]

loa_lme() is deprecated, and for new code we recommend switching to tolerance_limit(), which fits the same random-intercept model with model = "lme", gives analytic prediction and tolerance limits (no bootstrap), and is still under active development. It takes the two measurements (x and y) rather than their difference and average. Supplying condition gives each condition its own residual variance, as het_var = TRUE does here.

This function allows for the calculation of (parametric) bootstrapped limits of agreement when there are multiple observations per subject.

Usage

loa_lme(
  diff,
  avg,
  condition = NULL,
  id,
  data,
  type = c("perc", "norm", "basic"),
  conf.level = 0.95,
  agree.level = 0.95,
  replicates = 999,
  prop_bias = FALSE,
  het_var = FALSE
)

Arguments

diff

Column name of the data frame that includes the difference between the 2 measurements of interest.

avg

Column name of the data frame that includes the average of the 2 measurements of interest.

condition

Column name indicating different conditions subjects were tested under. This can be left missing if there are no differing conditions to be tested.

id

Column name indicating the subject/participant identifier

data

A data frame containing the variables within the model.

type

A character string representing the type of bootstrap confidence intervals. Only "norm", "basic", and "perc" currently supported. Bias-corrected and accelerated, bca, is the default. See ?boot::boot.ci for more details.

conf.level

The confidence level required. Default is 95%.

agree.level

The agreement level required. Default is 95%.

replicates

The number of bootstrap replicates. Passed on to the boot function. Default is 999.

prop_bias

Logical indicator (default is FALSE) of whether proportional bias should be considered for the limits of agreement calculations.

het_var

Logical indicator (default is FALSE) of whether to assume homogeneity of variance in each condition.

Value

Returns single list with the results of the agreement analysis.

  • var_comp: Table of variance components

  • loa: A data frame of the limits of agreement including the average difference between the two sets of measurements, the standard deviation of the difference between the two sets of measurements and the lower and upper confidence limits of the difference between the two sets of measurements.

  • call: The matched call.

References

Parker, R. A., Weir, C. J., Rubio, N., Rabinovich, R., Pinnock, H., Hanley, J., McLoughan, L., Drost, E.M., Mantoani, L.C., MacNee, W., & McKinstry, B. (2016). "Application of mixed effects limits of agreement in the presence of multiple sources of variability: exemplar from the comparison of several devices to measure respiratory rate in COPD patients". PLOS One, 11(12), e0168321. doi:10.1371/journal.pone.0168321

See Also

tolerance_limit()

Examples

data("recpre_long")
# Instead of loa_lme(), use:
tolerance_limit(data = recpre_long, x = "PM", y = "AM", id = "id",
                condition = "trial_condition", model = "lme")

Methods for loa_mermod objects

Description

Methods defined for objects returned from the deprecated loa_lme.

Usage

## S3 method for class 'loa_mermod'
print(x, ...)

## S3 method for class 'loa_mermod'
plot(
  x,
  x_label = "Average of Both Methods",
  y_label = "Difference Between Methods",
  geom = "geom_point",
  smooth_method = NULL,
  smooth_se = TRUE,
  ...
)

## S3 method for class 'loa_mermod'
check(x, ...)

Arguments

x

object of class loa_mermod.

...

further arguments passed through, see description of return value for details. loa_mixed.

x_label

Label for x-axis.

y_label

Label for y-axis.

geom

String naming the type of geometry to display the data points. Default is "geom_point". Other options include: "geom_bin2d", "geom_density_2d", "geom_density_2d_filled", and "stat_density_2d".

smooth_method

Smoothing method (function) to use, accepts either NULL or a character vector, e.g. "lm", "glm", "gam", "loess" or a function. Default is NULL, which will not include a trend line.

smooth_se

Display confidence interval around smooth?

Value

print

Prints short summary of the Limits of Agreement

plot

Returns a plot of the limits of agreement


Mixed Effects Limits of Agreement

Description

[Deprecated]

loa_mixed() is outdated, and for new code we recommend switching to tolerance_limit(), which are easier to use, have more features, and are still under active development.

This function allows for the calculation of bootstrapped limits of agreement when there are multiple observations per subject.

Usage

loa_mixed(
  diff,
  condition,
  id,
  data,
  plot.xaxis = NULL,
  delta,
  conf.level = 0.95,
  agree.level = 0.95,
  replicates = 1999,
  type = "bca"
)

Arguments

diff

column name of the data frame that includes the continuous measurement of interest.

condition

column name indicating different conditions subjects were tested under.

id

column name indicating the subject/participant identifier

data

A data frame containing the variables within the model.

plot.xaxis

column name indicating what to plot on the x.axis for the Bland-Altman plots. If this argument is missing or set to NULL then no plot will be produced.

delta

The threshold below which methods agree/can be considered equivalent, can be in any units. Equivalence Bound for Agreement.

conf.level

the confidence level required. Default is 95%.

agree.level

the agreement level required. Default is 95%.

replicates

the number of bootstrap replicates. Passed on to the boot function. Default is 1999.

type

A character string representing the type of bootstrap confidence intervals. Only "norm", "basic", "bca", and "perc" currently supported. Bias-corrected and accelerated, bca, is the default. See ?boot::boot.ci for more details.

Value

Returns single list with the results of the agreement analysis.

  • var_comp: Table of variance components

  • loa: a data frame of the limits of agreement including the average difference between the two sets of measurements, the standard deviation of the difference between the two sets of measurements and the lower and upper confidence limits of the difference between the two sets of measurements.

  • h0_test: Decision from hypothesis test.

  • bland_alt.plot: Simple Bland-Altman plot. Red line are the upper and lower bounds for shieh test; grey box is the acceptable limits (delta). If the red lines are within the grey box then the shieh test should indicate 'reject h0', or to reject the null hypothesis that this not acceptable agreement between x & y.

  • conf.level: Returned as input.

  • agree.level: Returned as input.

References

Parker, R. A., Weir, C. J., Rubio, N., Rabinovich, R., Pinnock, H., Hanley, J., McLoughan, L., Drost, E.M., Mantoani, L.C., MacNee, W., & McKinstry, B. (2016). "Application of mixed effects limits of agreement in the presence of multiple sources of variability: exemplar from the comparison of several devices to measure respiratory rate in COPD patients". Plos One, 11(12), e0168321. doi:10.1371/journal.pone.0168321


Methods for loa_mixed_bs objects

Description

Methods defined for objects returned from the loa_mixed functions.

Usage

## S3 method for class 'loa_mixed_bs'
print(x, ...)

## S3 method for class 'loa_mixed_bs'
plot(x, ...)

Arguments

x

object of class loa_mixed_bs as returned from loa_mixed

...

further arguments passed through, see description of return value for details. loa_mixed.

Value

print

Prints short summary of the Limits of Agreement

plot

Returns a plot of the limits of agreement


Methods for loa objects

Description

Methods defined for objects returned from the agreement_limit function.

Usage

## S3 method for class 'loa'
print(x, digits = 4, ...)

## S3 method for class 'loa'
plot(
  x,
  geom = c("geom_point", "geom_bin2d", "geom_density_2d", "geom_density_2d_filled",
    "stat_density_2d"),
  delta = NULL,
  ...
)

## S3 method for class 'loa'
check(x, ...)

Arguments

x

object of class loa as returned from a agreement_limit function.

digits

The number of digits to print.

...

further arguments passed through, see description of return value for details. agreement_limit.

geom

String naming the type of geometry to display the data points. Default is "geom_point". Other options include: "geom_bin2d", "geom_density_2d", "geom_density_2d_filled", and "stat_density_2d".

delta

The maximal allowable difference.

Value

print

Prints short summary of the Limits of Agreement.

plot

Returns a plot of the limits of agreement.

check

Returns plots testing the assumptions of a Bland-Altman analysis. P-values for the normality and heteroskedascity tests are provided as captions to the plot.


Minimal Detectable Change

Description

[Experimental]

Estimates the minimal detectable change (MDC), also called the smallest detectable or real difference, the "minimal difference" (Weir, 2005), or the repeatability coefficient: the smallest change in one subject's score between two occasions that exceeds measurement error with a given confidence. It is a scaled standard error of measurement (SEM), so it comes with a confidence interval and an optional test against a change that matters, and is returned as an object of class "htest".

Usage

mdc_test(
  x = NULL,
  data = NULL,
  measure = NULL,
  item = NULL,
  id = NULL,
  col.names = NULL,
  mdc.level = 0.95,
  sided = c("two.sided", "one.sided"),
  k_avg = 1,
  conf.level = 0.95,
  null.value = NULL,
  alternative = c("less", "greater", "two.sided"),
  ...
)

Arguments

x

A formula of the form values ~ item | id, or the result of reli_err_test() or reli_facet_test() with stat = "SEM", whose SEM is then used without refitting. Leave NULL to use measure, item and id (long data) or col.names (wide data).

data

A data frame containing the variables.

measure, item, id

Column names (character) of the measurement, the item (trial, rater, or occasion) and the subject identifier, for data in long format.

col.names

Column names (character) of the items, for data in wide format (one row per subject).

mdc.level

The confidence that a change of at least the MDC is not measurement error (default 0.95, the "MDC95"). This is not the confidence level of the interval, which is conf.level.

sided

"two.sided" (default) for a change in either direction, or "one.sided" when only a change in one stated direction (e.g. an improvement) is of interest. A one-sided MDC95 uses the same multiplier as the two-sided "MDC90".

k_avg

The number of measurements averaged at each occasion (default 1). Use it when the score used in practice is the mean of several trials: the MDC then shrinks by kavg\sqrt{k_{avg}}.

conf.level

Confidence level of the interval (default 0.95).

null.value

A change that matters, on the scale given by scale, to test against: by default (alternative = "less"), whether the MDC is below it, i.e. whether a change of that size can be distinguished from measurement error. It should come from an external anchor; see Details. The default, NULL, gives the estimate and interval without a test.

alternative

"less" (default), "greater", or "two.sided". The default tests whether the error is smaller than null.value. Used only when null.value is given; without a test the interval is two-sided.

...

Further arguments passed to reli_err_test() when x is not an existing result: scale, model, type, method, boot, boot_ci, replicates and se_method.

Details

Estimate. A change score is the difference between two measurements, each with error SEM, so its error SD is 2⋅SEM\sqrt{2} \cdot SEM, and

MDC=z⋅2⋅SEMkavg,MDC = \frac{z \cdot \sqrt{2} \cdot SEM}{\sqrt{k_{avg}}},

with zz the standard normal quantile for mdc.level (1.96 for a two-sided MDC95; Weir, 2005, eq. 13) and kavgk_{avg} the number of measurements averaged to give the score at each occasion (k_avg; 1 for single measurements). The mean of kavgk_{avg} measurements has error SEM/kavgSEM / \sqrt{k_{avg}}, so averaging shrinks the MDC by the same factor. The SEM is from reli_err_test(), with the same model, type and scale choices; the defaults give the typical error of Hopkins (2000), MSE\sqrt{MS_E}. With two trials the MDC equals the half-width of the limits of agreement of agreement_limit(), which also reports the bias between trials.

Interval and test. Because the MDC is a fixed multiple of the SEM, its interval is the SEM interval times that multiple, and the test of H0:MDC=H_0: MDC = null.value is the SEM test of null.value divided by it (a χ2\chi^2 test for the SEM, or the bootstrap with boot = TRUE). With scale = "log" the MDC is in sympercents; percent gives the equivalent percentage increase and decrease, which differ: 100(eMDC/100−1)100(e^{MDC/100} - 1) and 100(1−e−MDC/100)100(1 - e^{-MDC/100}).

Differences between raters or devices. The same reasoning applies to two measurements of one subject by different items, such as two raters or two devices, rather than two occasions: the MDC is the difference between them that measurement error alone would exceed only 5% of the time (for 95%). When the raters are interchangeable (any rater's score is used as it is, and raters are thought of as drawn from a pool), use type = "agreement", so that systematic differences between raters count as error; the consistency SEM leaves them out and understates how far apart two raters can be. For two particular raters, the limits of agreement of agreement_limit() report their bias and the spread around it separately. This is the MDC, not a multiple of the standard error of prediction (SEP): the SEP concerns a change from a subject's estimated true score (Weir, 2005), and 1.96⋅SEP1.96 \cdot SEP does not bound the difference between two raters.

Repeat measurements. When each rater measured each subject more than once, pass an SEM from reli_facet_test(): with facet = "intra" the MDC is for two measurements by the same rater (on two occasions, with occasion), with facet = "inter" and type = "agreement" for two measurements by different raters drawn from the pool. Averaging is then set in reli_facet_test() (k_avg, m_avg, r_avg), and k_avg here must be 1.

An individual threshold, not a group one. The MDC applies to a single individual's change between two measurements. It is not a threshold for an average (group) effect. The uncertainty in a mean change shrinks with the number of subjects (its standard error is about 2 SEM/n\sqrt{2} \, SEM / \sqrt{n}), so a mean effect well below the MDC can be estimated precisely and be clearly different from zero. Judge a mean effect by its own confidence interval. An effect smaller than the MDC is small relative to one person's measurement noise; that does not make it noise itself, and it does not show that the effect is absent!

Nor should the MDC be used to count "responders". An individual's observed change mixes their true response with within-person variation from occasion to occasion, so the proportion whose change exceeds the MDC reflects measurement noise as much as real differences between people: even if every subject had exactly the same true response, some would exceed the MDC and some would not (Senn, 2004). Real differences in response are estimated from the variance of the change scores in a treated group compared with a control group, or from replicated crossover designs (Atkinson & Batterham, 2015; Senn, 2016).

The MDC and the smallest worthwhile change. The MDC describes measurement noise. It answers the question: how large an observed change must be before it can be distinguished from error? Whether a change matters is a separate question that the reliability data cannot answer. There needs to be an external anchor, such as a clinically important difference or a change that alters competitive outcomes. Give that value as null.value (with the default alternative = "less") to test whether the MDC is below it.

A "smallest worthwhile change" (SWC) computed from the same reliability data cannot serve as that anchor. With SWC = 0.2 times the between-subject SD (Hopkins, 2004), SWC/SEM=0.2/1−ICCSWC / SEM = 0.2 / \sqrt{1 - ICC}, a restatement of the ICC. Hopkins (2004) offers it as a default (e.g. for fitness tests in team sports) when no relationship to a real outcome is available, and it should be treated as such a fallback.

Hopkins's (2004) SWC for solo athletes, 0.3 times the CV, is an external anchor, but only with the right CV: the within-athlete variability of top athletes' performance from one competition to the next, not the test's own typical error. Simulations (Hopkins, Hawley & Burke, 1999) showed that an enhancement of 0.3 of that CV gives a top athlete about one extra win in 10 events. That CV can be estimated with reli_err_test() (scale = "log") from repeated competition results, with competitions as items, and 0.3 times it given here as null.value. Applied to the test's own CV instead, the rule is circular: SWC/MDC=0.3/(1.962)≈0.11SWC / MDC = 0.3 / (1.96 \sqrt{2}) \approx 0.11 for every test. The anchor also rests on the one-in-10 convention and on elite fields of evenly matched athletes; for athletes further back in the field, Hopkins works out the worthwhile enhancement case by case.

Value

A list of class c("reli_htest", "htest"), as for reli_err_test(), containing:

statistic, parameter, p.value

The test of the SEM (equivalently of the MDC), when a null.value was given.

conf.int

The confidence interval for the MDC.

estimate

The MDC, on the scale given by scale.

stderr

Standard errors of the MDC and of its log.

null.value

The change tested against, if any.

alternative, method, data.name

As for an htest.

percent

With scale = "log", the MDC and its interval as a percentage increase and decrease.

sem

The SEM result the MDC was computed from.

fit, data, settings, boot

As for reli_err_test(); settings also holds the MDC settings and multiplier (mdc).

References

Atkinson, G., & Batterham, A. M. (2015). True and false interindividual differences in the physiological response to an intervention. Experimental Physiology, 100(6), 577-588. doi:10.1113/EP085070

Hopkins, W. G. (2000). Measures of reliability in sports medicine and science. Sports Medicine, 30(1), 1-15. doi:10.2165/00007256-200030010-00001

Hopkins, W. G. (2004). How to interpret changes in an athletic performance test. Sportscience, 8, 1-7.

Hopkins, W. G., Hawley, J. A., & Burke, L. M. (1999). Design and analysis of research on sport performance enhancement. Medicine & Science in Sports & Exercise, 31(3), 472-485. doi:10.1097/00005768-199903000-00018

Senn, S. (2004). Individual response to treatment: is it a valid assumption? BMJ, 329(7472), 966-968. doi:10.1136/bmj.329.7472.966

Senn, S. (2016). Mastering variation: variance components and personalised medicine. Statistics in Medicine, 35(7), 966-977. doi:10.1002/sim.6739

Weir, J. P. (2005). Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. Journal of Strength and Conditioning Research, 19(1), 231-240. doi:10.1519/15184.1

See Also

reli_err_test() for the SEM; agreement_limit() for the limits of agreement; reli_htest-methods for summary(), plot() and check().

Examples

# Weir (2005) data set A: 8 subjects, 2 trials
weir <- data.frame(t1 = c(146, 148, 170, 90, 157, 156, 176, 205),
                   t2 = c(140, 152, 152, 99, 145, 153, 167, 218))
mdc_test(col.names = c("t1", "t2"), data = weir)

# Can a change of 30 units be distinguished from measurement error?
res <- mdc_test(col.names = c("t1", "t2"), data = weir, null.value = 30)
summary(res)

# From an existing SEM, one-sided, as a percentage (log scale)
# in the result we can access the MDC on the percentage scale (converted from sympercent)
sem <- reli_err_test(col.names = c("t1", "t2"), data = weir, scale = "log")
res_log <- mdc_test(sem, sided = "one.sided")
res_log$percent

Model-Based Kappa for Ordinal Ratings from Many Raters

Description

[Experimental]

Estimates the model-based kappa κm\kappa_m of Nelson and Edwards (2015) and Nelson, Mitani and Edwards (2017), and please cite these papers if this function is utilized. This is a chance-corrected measure of exact agreement between raters who classify subjects on an ordinal scale. It is computed from an ordinal (cumulative probit) generalized linear mixed model with crossed random effects for subjects and raters, so it describes the populations of raters and subjects that were sampled, and it is not affected by the prevalence of the categories.

Requires the ordinal package.

Usage

ord_kappa_test(
  x = NULL,
  data = NULL,
  measure = NULL,
  item = NULL,
  id = NULL,
  col.names = NULL,
  categories = NULL,
  conf.level = 0.95,
  null.value = NULL,
  alternative = c("two.sided", "greater", "less"),
  group = NULL,
  group_type = c("rater", "subject")
)

Arguments

x

A formula of the form rating ~ rater | id. Leave NULL to use measure, item and id (long data) or col.names (wide data).

data

A data frame.

measure, item, id

Names of the columns with the ratings, the raters, and the subjects (long data).

col.names

Names of the columns with each rater's ratings (wide data, one row per subject).

categories

All categories of the ordinal scale, in order. Defaults to the levels of a factor, and otherwise to the sorted observed ratings. Character ratings need categories to set their order. The number of categories, C, enters κm\kappa_m through the chance agreement 1/C.

conf.level

Confidence level of the interval (default 0.95).

null.value

The hypothesized value of κm\kappa_m, in [0, 1). NULL (default) gives no test. Not used with group.

alternative

The alternative hypothesis: "two.sided" (default), "greater" or "less". Without a null.value the interval is two-sided.

group

Optional name of a column with a two-level rater or subject characteristic (e.g. experienced or inexperienced raters, younger or older patients), constant within each rater (or subject). The two groups are then compared; see Details. The order of the levels is that of a factor, and otherwise sorted. Requires long data.

group_type

Whether group describes the "rater"s (default) or the "subject"s.

Details

Model. The rating of subject i by rater j falls in category c when a latent variable falls between thresholds αc−1\alpha_{c-1} and αc\alpha_c:

P(Yij≤c∣ui,vj)=Φ(αc−ui−vj),P(Y_{ij} \le c \mid u_i, v_j) = \Phi(\alpha_c - u_i - v_j),

with subject effects ui∼N(0,σu2)u_i \sim N(0, \sigma^2_u) and rater effects vj∼N(0,σv2)v_j \sim N(0, \sigma^2_v). It is fitted by the Laplace approximation with ordinal::clmm().

Kappa. Observed agreement p0p_0 is the probability that two raters put the same subject in the same category. Chance agreement is taken at its minimum, pc=1/Cp_c = 1/C, reached when the thresholds split the latent scale into C equally likely categories, and p0p_0 is evaluated at those thresholds. Then

κm=p0−1/C1−1/C,\kappa_m = \frac{p_0 - 1/C}{1 - 1/C},

which depends only on C and ρ=σu2/(σu2+σv2+1)\rho = \sigma^2_u / (\sigma^2_u + \sigma^2_v + 1), the share of the latent variance due to differences between subjects. κm\kappa_m is 0 when ρ=0\rho = 0 and approaches 1 as ρ\rho approaches 1. Because the thresholds drop out, κm\kappa_m does not depend on how common each category is. The observed agreement at the fitted thresholds is also returned (fit$p0).

Inference. The standard error is from the delta method with the covariance matrix of the variance components from the model Hessian (Nelson et al., 2017). The interval and test are formed on the logit of ρ\rho and mapped back to κm\kappa_m, so the interval stays within [0, 1]; Nelson et al. (2017) used the Wald interval κ^m±1.96 se\hat\kappa_m \pm 1.96\, se, which can be found from estimate and stderr. The p-value is from a z test of ρ=ρ0\rho = \rho_0, where ρ0\rho_0 gives the null.value of κm\kappa_m.

Design. Raters and subjects are treated as random samples from their populations. Missing ratings are allowed. At least three raters are needed to estimate the rater variance (Nelson et al., 2017); with few raters or subjects the variance components, and so the interval, are imprecise.

Comparing groups. With group, the group enters the model as a fixed effect (shifting the thresholds, e.g. for a higher prevalence of severe ratings in older patients), and the rater (or subject) variance may differ between the two groups (Nelson et al., 2017). The result gives κm\kappa_m within each group, with an interval for their difference, and the test is the likelihood ratio test of a common variance. For rater groups, fit$subgroups also gives κm\kappa_m between a rater from each group; chance agreement is then minimised numerically over common thresholds, and the fixed effect matters. Subgroup intervals are formed on the logit scale.

Nelson et al. (2017) add a random slope for the group indicator and test its variance and correlation against a 50:50 mixture of chi-square distributions with 1 and 2 df. Because the group is constant within each rater (or subject), only the variance in each group is identified, not the intercept-slope covariance, and the Hessian of that model is singular. The equivalent model with a separate variance for each group is fitted instead: it has the same likelihood and the same kappas, with one parameter fewer. Equal variances lie inside its parameter space, so the likelihood ratio statistic is compared with a chi-square distribution with 1 df.

Value

A list of class c("cac_htest", "htest") containing:

statistic

The z statistic (only with a null.value), or with group, the likelihood ratio statistic.

p.value

The p-value (only with a null.value or group).

conf.int

The confidence interval for κm\kappa_m, or with group, for the difference between the groups (second minus first).

estimate

The estimated κm\kappa_m, or with group, one per group.

stderr

The delta-method standard error(s).

null.value

The hypothesized value.

alternative

The alternative hypothesis.

method

A description of the method.

data.name

A description of the data.

fit

The fitted model (model), the variance components (sigma2_subject, sigma2_rater), rho and its standard error, the thresholds, the observed agreement at the fitted thresholds (p0), the number of categories C, the rater conditional modes and their conditional variances (rater_effects), and the numbers of subjects and raters. With group, fit also has the null model (null_model), the likelihood ratio test (lrt), the fixed effect of the group (beta), one variance per group, and a table of subgroup kappas with standard errors and intervals (subgroups).

data

The ratings in long format.

settings

The settings used, for the methods.

References

Nelson, K. P., & Edwards, D. (2015). Measures of agreement between many raters for ordinal classifications. Statistics in Medicine, 34(23), 3116-3132. doi:10.1002/sim.6546

Nelson, K. P., Mitani, A. A., & Edwards, D. (2017). Assessing the influence of rater and subject characteristics on measures of agreement for ordinal ratings. Statistics in Medicine, 36(20), 3181-3199. doi:10.1002/sim.7323

See Also

cac_htest-methods for summary() and plot(); cac_test() for Gwet's AC1/AC2, Fleiss' kappa and Krippendorff's alpha.

Examples

if (requireNamespace("ordinal", quietly = TRUE)) {
  # Simulated ratings of 60 subjects by 8 raters on a 5-point scale
  set.seed(42)
  d <- expand.grid(id = 1:60, rater = paste0("R", 1:8))
  latent <- rnorm(60, sd = 1.5)[d$id] + rnorm(8, sd = 0.5)[d$rater] +
    rnorm(nrow(d))
  d$rating <- cut(latent, c(-Inf, -1.5, -0.5, 0.5, 1.5, Inf), labels = FALSE)
  res <- ord_kappa_test(rating ~ rater | id, data = d)
  res
  summary(res)
  plot(res, type = "raters")

  # Do the first 4 raters agree better than the other 4?
  d$exp <- ifelse(d$rater %in% c("R1", "R2", "R3", "R4"), "exp", "inexp")
  res_g <- ord_kappa_test(rating ~ rater | id, data = d, group = "exp")
  res_g
  res_g$fit$subgroups
}

Passing-Bablok Regression for Method Comparison

Description

[Experimental]

A robust, nonparametric method for fitting a straight line to two-dimensional data where both variables (X and Y) are measured with error. Particularly useful for method comparison studies.

Usage

pb_reg(
  formula,
  data,
  id = NULL,
  method = c("scissors", "symmetric", "invariant"),
  conf.level = 0.95,
  weights = NULL,
  error.ratio = 1,
  replicates = 0,
  se_method = c("bootstrap", "jackknife", "dufey"),
  model = TRUE,
  keep_data = TRUE,
  ...
)

Arguments

formula

A formula of the form y ~ x specifying the model.

data

Data frame with all data.

id

Column with subject identifier (optional). If provided, replicate measurements are averaged within each subject before fitting.

method

Method for Passing-Bablok estimation. Options are:

  • "scissors": Scissors estimator (1988) - most robust, scale invariant (default)

  • "symmetric": Original Passing-Bablok (1983) - symmetric around 45-degree line

  • "invariant": Scale-invariant method (1984) - adaptive reference line

conf.level

The confidence level required. Default is 95%.

weights

An optional vector of case weights to be used in the fitting process. Should be NULL or a numeric vector.

error.ratio

Must be 1 (the default). Values other than 1 are no longer supported and give an error; see "Measurement Error" in Details. Use dem_reg() when the error variances of x and y differ.

replicates

Number of resamples for confidence intervals and the variance-covariance matrix. For se_method = "bootstrap" this is the number of bootstrap resamples; for se_method = "jackknife" it is the number of random delete-d subsets. If 0 (default), analytical confidence intervals are used and no variance-covariance matrix is returned (unless se_method = "dufey"). Resampling is recommended for weighted data and 'invariant' or 'scissors' methods.

se_method

Method used to estimate the variance-covariance matrix and confidence intervals of the coefficients. Options are:

  • "bootstrap": Nonparametric pairs (case) bootstrap with percentile confidence intervals (default). Used when replicates > 0.

  • "jackknife": Delete-d jackknife with d = floor(n/2), using replicates random subsets (Shao & Wu, 1989). Used when replicates > 0.

  • "dufey": [Experimental] Analytic sandwich estimator of Dufey (2020). Does not require resampling (replicates is ignored). Only available for the "scissors" method without case weights. Not used by joint_test() or plot_joint() (see Details).

model

Logical. If TRUE (default), the model frame is stored in the returned object. This is needed for methods like plot(), fitted(), residuals(), and predict() to work without supplying data. If FALSE, the model frame is not stored (saves memory for large datasets), but these methods will require a data argument.

keep_data

Logical indicator (TRUE/FALSE). If TRUE, intermediate calculations are returned; default is FALSE.

...

Additional arguments (currently unused).

Details

Passing-Bablok regression is a robust nonparametric method that estimates the slope as the shifted median of all possible slopes between pairs of points. The intercept is then calculated as the median of y - slope*x. This method is particularly useful when:

  • Both X and Y are measured with error

  • You want a robust method not sensitive to outliers

  • The relationship is assumed to be linear

  • X and Y are highly positively correlated

Methods

Three Passing-Bablok methods are available:

"scissors" (default): The scissors estimator (1988), most robust and scale-invariant. Uses the median of absolute values of angles.

"symmetric": The original method (1983), symmetric about the y = x line. Uses the line y = -x as the reference for partitioning points.

"invariant": Scale-invariant method (1984). First finds the median angle of slopes below the horizontal, then uses this as the reference line.

Measurement Error

Passing-Bablok regression implicitly assumes that the ratio of the error variances of y and x equals the squared slope (equal error variances when the slope is 1). Earlier versions accepted error.ratio != 1 (or estimated it from replicates via id) and weighted the pairwise slopes by it. In simulations this biased the slope (by -0.07 to -0.43 with an error ratio of 4), the bias did not shrink with sample size, and joint tests then rejected a true identity line 81-100% of the time with n >= 50. error.ratio != 1 is therefore an error. When the error variances differ, use dem_reg() with the appropriate error.ratio (or id).

With id, replicate measurements are averaged within each subject and the subject means are analysed.

Weighting

Case weights can be provided via the weights argument. Case weights allow you to down-weight or up-weight specific observations in the analysis.

Standard Errors and Variance-Covariance Matrix

The analytical (Passing & Bablok, 1983) confidence intervals do not provide a covariance between the intercept and slope, so a variance-covariance matrix (needed by joint_test() and plot_joint()) is only returned when one of the following is used:

  • Pairs bootstrap (se_method = "bootstrap", replicates > 0): whole (x, y) observations are resampled with replacement and the model is refit. The variance-covariance matrix is the covariance of the bootstrap estimates and the confidence intervals are percentile intervals.

  • Delete-d jackknife (se_method = "jackknife", replicates > 0): the model is refit on replicates random subsets that each leave out d = floor(n/2) observations. The delete-1 jackknife is inconsistent for median-type estimators such as Passing-Bablok; deleting d observations with n/d→0\sqrt{n}/d \to 0 restores consistency (Shao & Wu, 1989). Confidence intervals are Wald-type intervals using a t(n-2) quantile.

  • Dufey (2020) (se_method = "dufey"): an analytic, distribution-free estimator for the equivariant ("scissors") Passing-Bablok estimator, based on the U-statistic variance of Kendall's tau and a sandwich estimator for the intercept. The slope interval is formed from order statistics of the pairwise slopes and the intercept interval is Wald-type. This option is experimental. In simulations its marginal intervals had close to nominal coverage for n >= 20, but a joint test of intercept and slope built from its variance-covariance matrix rejected a true null hypothesis 8-11% of the time at a nominal 5%. joint_test() and plot_joint() therefore refuse Dufey fits; use the pairs bootstrap or delete-d jackknife, which held 4.3-5.4% in the same simulations.

Resampling is particularly useful for:

  • Weighted regression (case weights)

  • Methods 'invariant' and 'scissors' (where analytical CI validity is uncertain)

  • Joint tests of intercept and slope

The method automatically:

  • Tests for high positive correlation using Kendall's tau

  • Tests for linearity using a CUSUM test

  • Computes confidence intervals (analytical or bootstrap)

Value

The function returns a simple_eiv object with the following components:

  • coefficients: Named vector of coefficients (intercept and slope).

  • residuals: Residuals from the fitted model.

  • fitted.values: Predicted Y values.

  • model_table: Data frame presenting the full results from the Passing-Bablok regression.

  • vcov: Variance-covariance matrix for slope and intercept (if resampling or se_method = "dufey" is used; otherwise NULL).

  • df.residual: Residual degrees of freedom.

  • call: The matched call.

  • terms: The terms object used.

  • model: The model frame.

  • x_vals: Original x values used in fitting.

  • y_vals: Original y values used in fitting.

  • weights: Case weights (if provided).

  • error.ratio: Always 1 (kept for compatibility).

  • conf.level: Confidence level used.

  • method: Character string describing the method.

  • method_num: Numeric method identifier (1, 2, or 3).

  • kendall_test: Results of Kendall's tau correlation test.

  • cusum_test: Results of CUSUM linearity test.

  • n_slopes: Number of slopes used in estimation.

  • boot: Resampling results (if replicates > 0 and se_method is "bootstrap" or "jackknife").

  • se_method: Method actually used for standard errors ("analytic", "bootstrap", "jackknife", or "dufey").

References

Passing, H., & Bablok, W. (1983). A New Biometrical Procedure for Testing the Equality of Measurements from Two Different Analytical Methods. Application of linear regression procedures for method comparison studies in Clinical Chemistry, Part I. Cclm, 21(11), 709-720. doi: 10.1515/cclm.1983.21.11.709

Passing, H., & Bablok, W. (1984). Comparison of Several Regression Procedures for Method Comparison Studies and Determination of Sample Sizes Application of linear regression procedures for method comparison studies in Clinical Chemistry, Part II. Clinical Chemistry and Laboratory Medicine, 22(6). doi: 10.1515/cclm.1984.22.6.431

Bablok, W., Passing, H., Bender, R., & Schneider, B. (1988). A General Regression Procedure for Method Transformation. Application of Linear Regression Procedures for Method Comparison Studies in Clinical Chemistry, Part III. Clinical Chemistry and Laboratory Medicine, 26(11). doi: 10.1515/cclm.1988.26.11.783

Dufey, F. (2020). Derivation of Passing-Bablok regression from Kendall's tau. The International Journal of Biostatistics, 16(2), 20190157. doi: 10.1515/ijb-2019-0157

Sen, P. K. (1968). Estimates of the regression coefficient based on Kendall's tau. Journal of the American Statistical Association, 63(324), 1379-1389. doi: 10.1080/01621459.1968.10480934

Shao, J., & Wu, C. F. J. (1989). A general theory for jackknife variance estimation. The Annals of Statistics, 17(3), 1176-1197. doi: 10.1214/aos/1176347263

Examples

# Peak flow: mini vs Wright meter (Bland & Altman, 1986)
data("ba1986")

# Analytic confidence intervals (scissors method, default)
fit <- pb_reg(mini1 ~ wright1, data = ba1986)
summary(fit)
plot(fit)

# Symmetric and scale-invariant methods
pb_reg(mini1 ~ wright1, data = ba1986, method = "symmetric")
pb_reg(mini1 ~ wright1, data = ba1986, method = "invariant")


# Bootstrap (or delete-d jackknife) variance-covariance matrix, needed for
# the joint test of intercept and slope; use more replicates in practice
fit_boot <- pb_reg(mini1 ~ wright1, data = ba1986, replicates = 199)
joint_test(fit_boot)
plot_joint(fit_boot)

fit_jack <- pb_reg(mini1 ~ wright1, data = ba1986,
                   se_method = "jackknife", replicates = 199)

# Case weights
w <- rep(1, nrow(ba1986))
w[1:3] <- 2
pb_reg(mini1 ~ wright1, data = ba1986, weights = w, replicates = 199)

Power Calculation for Exact Agreement/Tolerance Test

Description

[Stable]

Computes sample size, power, or other parameters for the exact method of assessing agreement between two measurement methods, as described in Shieh (2019). This method tests whether the central portion of paired differences falls within specified bounds.

Usage

power_agreement_exact(
  n = NULL,
  delta = NULL,
  mu = 0,
  sigma = NULL,
  p0_star = 0.95,
  power = NULL,
  alpha = 0.05,
  max_iter = 1000,
  m = 1,
  icc = 0
)

Arguments

n

Number of subject pairs (sample size), or the number of subjects when m > 1

delta

Maximum allowable difference bound (half-width of tolerance interval)

mu

Mean of paired differences

sigma

Standard deviation of paired differences. With clustered data (m > 1), the total SD of the differences.

p0_star

The coverage proportion (content) of the tolerance interval. Central proportion under null hypothesis (default = 0.95)

power

Target power (probability of rejecting false null)

alpha

Significance level (Type I error rate, default = 0.05, Confidence level = 1-alpha)

max_iter

Maximum iterations for gamma computation (default = 1000)

m

number of paired measurements per subject (default 1, independent pairs). See the "Clustered data" section.

icc

intraclass correlation of the paired differences (between-subject share of the variance of the differences), in [0, 1). Ignored when m = 1. This is not the test-retest ICC of the measurements themselves.

Details

This function implements the exact agreement test procedure of Shieh (2019) for method comparison studies. The test evaluates whether the central proportion of the distribution of paired differences lies within the interval [-delta, delta].

The null hypothesis is: H0: theta_(1-p) <= -delta or delta <= theta_p The alternative is: H1: -delta < theta_(1-p) and theta_p < delta

where p = (1 + p0_star)/2, and theta_p represents the 100p-th percentile of the paired differences.

Specify three of: n, delta, power, and sigma. The fourth will be calculated. If mu is not specified, it defaults to 0. A sample size can only be found when the proportion of differences within [-delta, delta] under the alternative (p1_star) exceeds p0_star; otherwise power cannot exceed alpha and an error is returned.

Tolerance Interval Interpretation:

The parameter p0_star represents the tolerance coverage proportion, i.e., the proportion of the population that must fall within the specified bounds [-delta, delta] under the null hypothesis. This is conceptually related to tolerance intervals, but formulated as a hypothesis test rather than an estimation problem.

The power is that of Shieh's test (reject H0 when xbar -/+ gamma \* SE lies within [-delta, delta]; for independent pairs this test is reported by agree_test()). It is not the probability that tolerance limits from tolerance_limit() fall within [-delta, delta]; that decision rule is less powerful. In simulations at a planned power of 0.80, it was about 0.66 for independent pairs and about 0.57 to 0.61 for clustered data (m = 3, icc = 0.1 to 0.6). Allow a larger sample if the analysis will be based on tolerance limits.

Note: This differs from Bland-Altman's "95% limits of agreement," which are confidence intervals for the 2.5th and 97.5th percentiles, not tolerance intervals.

Value

An object of class "power.htest", a list with components:

n

Sample size (pairs, or subjects when m > 1)

delta

Maximum allowable difference bound

mu

Mean of differences

sigma

Standard deviation of differences

p0_star

Central proportion (null hypothesis)

p1_star

Central proportion (alternative hypothesis)

alpha

Significance level

power

Power of the test

critical_value

Critical value for test statistic

m, icc

The clustering inputs

n.obs

Total number of pairs, n * m

method

Description of the method

note

Additional notes

Clustered test statistic

With m > 1, the test is Shieh's test with the standard error of the mean difference taken as S \* sqrt(DEFF / (n \* m)), i.e. with the design effect treated as known. No function in this package currently carries out that clustered test, so treat the clustered power as a planning approximation.

Clustered data

When each of n subjects contributes m paired measurements (repeated measures, or several pairs per subject), the pairs are not independent and treating n * m pairs as independent overstates precision. Setting m and icc accounts for this under a balanced one-way random-effects (compound symmetry) model for the differences:

  • The mean difference has variance sigma^2 \* DEFF / (n \* m), with design effect DEFF = 1 + (m - 1) \* icc, so the effective sample size for the mean is n \* m / DEFF.

  • The total variance is estimated as S^2 = MSB / m + (m - 1) / m \* MSW, whose distribution is a weighted sum of two independent chi-square variables with n - 1 and n(m - 1) degrees of freedom. This distribution is integrated numerically in place of the single chi-square of the independent case. It is more skewed than a Satterthwaite approximation suggests, which matters when icc is large.

Under these assumptions the calculation is exact (it agrees with simulation to Monte Carlo error), and m = 1 gives exactly the published method. The simple design effect 1 + (m - 1) \* icc applied to the whole sample size overstates the penalty, because the SD has a much smaller design effect, 1 + (m - 1) \* icc^2. Unbalanced designs, other correlation structures, or several levels of nesting are not covered; use simulation for those.

icc is the ICC of the differences, which can be estimated from pilot data with an intercept-only mixed model on the differences (e.g. lme4::lmer(diff ~ 1 + (1 | id))), and sigma is then the square root of the sum of the two variance components.

References

Shieh, G. (2019). Assessing Agreement Between Two Methods of Quantitative Measurements: Exact Test Procedure and Sample Size Calculation. Statistics in Biopharmaceutical Research, 12(3), 352-359. doi:10.1080/19466315.2019.1677495

Bland, J.M. and Altman, D.G. (2007). Agreement between methods of measurement with multiple observations per individual. Journal of Biopharmaceutical Statistics, 17(4), 571-582. doi:10.1080/10543400701329422

Examples

# Example 1: Find required sample size
power_agreement_exact(delta = 7, mu = 0.5, sigma = 2.5,
                      p0_star = 0.95, power = 0.80, alpha = 0.05)

# Example 2: Calculate power for given sample size
power_agreement_exact(n = 15, delta = 0.1, mu = 0.011,
                      sigma = 0.044, p0_star = 0.80, alpha = 0.05)

# Example 3: Find required delta for given power and sample size
power_agreement_exact(n = 50, mu = 0, sigma = 2.5,
                      p0_star = 0.95, power = 0.90, alpha = 0.05)

# Example 4: Clustered data, 3 pairs per subject, ICC of the differences 0.15
power_agreement_exact(delta = 7, mu = 0.5, sigma = 2.5,
                      p0_star = 0.95, power = 0.80, m = 3, icc = 0.15)

Power Calculation for Bland-Altman Limits of Agreement

Description

[Stable]

Computes the sample size, power, maximal allowable difference, or standard deviation for a method comparison study analyzed with the Bland-Altman limits of agreement, using the method of Lu et al. (2016). Agreement is concluded when the confidence intervals of both limits of agreement fall within [-delta, delta].

Usage

power_agreement_limit(
  n = NULL,
  delta = NULL,
  mu = 0,
  sigma = NULL,
  power = NULL,
  conf.level = 0.95,
  agree.level = 0.95,
  m = 1,
  icc = 0
)

Arguments

n

Number of pairs (sample size), or the number of subjects when m > 1.

delta

Maximal allowable difference: the methods agree if the confidence limits of the limits of agreement lie within [-delta, delta].

mu

Mean of the paired differences (default 0).

sigma

Standard deviation of the paired differences. With clustered data (m > 1), the total SD of the differences.

power

Target power.

conf.level

Confidence level of the confidence intervals of the limits of agreement (default 0.95).

agree.level

Agreement level of the limits of agreement: the proportion of differences the limits should contain (default 0.95).

m

number of paired measurements per subject (default 1, independent pairs). See the "Clustered data" section.

icc

intraclass correlation of the paired differences (between-subject share of the variance of the differences), in [0, 1). Ignored when m = 1. This is not the test-retest ICC of the measurements themselves.

Details

Specify three of n, delta, sigma, and power; the fourth is calculated.

The limits of agreement are mu +/- z \* sigma, with z the (1 + agree.level) / 2 quantile of the standard normal, and their confidence intervals are those of Bland and Altman (1999), with standard error sigma \* sqrt(1/n + z^2 / (2 (n - 1))). This is the interval from agreement_limit() with loa_calc = "blandaltman". The power is Lu et al. (2016), eq. 5:

power = 1 - T(t, n - 1, tau1) - T(t, n - 1, tau2),

where T is the CDF of the non-central t distribution, t is the 1 - (1 - conf.level) / 2 quantile of the central t distribution with n - 1 df, tau1 = (delta - mu - z \* sigma) / SE, and tau2 = (delta + mu - z \* sigma) / SE. Negative values are set to 0. These are the values reported by PASS (Chapter 427), which notes that the sample sizes in Table 1 of Lu et al. (2016), based on their approximate eq. 6, are sometimes slightly too large.

A sample size can only be found when the true limits of agreement lie within the bounds, i.e. |mu| + z \* sigma < delta; otherwise power goes to 0 as n increases and an error is returned.

With clustered data (m > 1), the standard error of the limits uses the effective sample size n \* m / (1 + (m - 1) \* icc) for the mean difference and a Satterthwaite df for the SD, under a balanced compound symmetry model (see agree_expected_half() for the model). This is an approximation; m = 1 gives the original results.

For a power curve over a range of sample sizes, see blandPowerCurve().

Value

An object of class "power.htest", a list with components:

n

Sample size (pairs, or subjects when m > 1)

delta

Maximal allowable difference

mu

Mean of differences

sigma

Standard deviation of differences

conf.level

Confidence level of the CIs of the limits of agreement

agree.level

Agreement level of the limits of agreement

power

Power of the procedure

m, icc

The clustering inputs

n.obs

Total number of pairs, n * m

method

Description of the method

note

Additional notes

References

Lu, M. J., et al. (2016). Sample Size for Assessing Agreement between Two Methods of Measurement by Bland-Altman Method. The International Journal of Biostatistics, 12(2). doi:10.1515/ijb-2015-0039

Bland, J. M., and Altman, D. G. (1999). Measuring agreement in method comparison studies. Statistical Methods in Medical Research, 8(2), 135-160. doi:10.1177/096228029900800204

NCSS (PASS Sample Size Software). Chapter 427: Bland-Altman Method for Assessing Agreement in Method Comparison Studies.

See Also

blandPowerCurve(), power_agreement_exact(), agree_expected_half(), agreement_limit()

Examples

# Sample size for 80% power (PASS Example 1: n = 60)
power_agreement_limit(delta = 7, mu = 0.5, sigma = 2.5, power = 0.80)

# Power at a given sample size (PASS Example 2)
power_agreement_limit(n = 201, delta = 2.5, mu = 0.2, sigma = 1)

# Largest SD for which 90% power is reached with 100 pairs
power_agreement_limit(n = 100, delta = 7, mu = 0.5, power = 0.90)

# Clustered data, 3 pairs per subject, ICC of the differences 0.15
power_agreement_limit(delta = 7, mu = 0.5, sigma = 2.5, power = 0.80,
                      m = 3, icc = 0.15)

Power and Sample Size for an Intraclass Correlation Test

Description

[Experimental]

Computes the power of the F test of an intraclass correlation coefficient (ICC) used by icc_test(), or solves for the number of subjects, the number of measurements per subject, the ICC under the alternative, or the significance level that gives a target power. Exactly one of n, k, rho1, power and sig.level must be NULL; that one is computed from the others, as in stats::power.t.test().

Usage

power_icc(
  n = NULL,
  k = 2,
  rho0 = 0,
  rho1 = NULL,
  sig.level = 0.05,
  power = NULL,
  model = c("oneway", "twoway"),
  unit = c("single", "average"),
  missing = 0,
  alternative = c("two.sided", "greater", "less"),
  method = c("exact", "approximate")
)

Arguments

n

Number of subjects.

k

Number of measurements (items: trials, raters or occasions) per subject. When solved for, the smallest whole number of at least 2 that reaches the target is returned.

rho0

The ICC under the null hypothesis (default 0).

rho1

The ICC under the alternative hypothesis.

sig.level

Significance level (Type I error rate; default 0.05).

power

Power of the test (1 minus the Type II error rate).

model

"oneway" (default) or "twoway". "twoway" means the consistency ICC, ICC(C,1) of icc_test(); the absolute agreement ICC is not supported (see Details).

unit

"single" (default) for the reliability of a single measurement, or "average" for the reliability of the mean of the k measurements (ICC(k) or ICC(C,k)), as in icc_test(). With "average", the ICC values are on the average-measure scale and k cannot be solved for. See Details.

missing

The expected proportion of measurements (subject-by-item cells) that will be missing, analysed with icc_test(method = "reml"). The default, 0, gives the exact complete-data calculation; values above 0 give an approximation (see Details). For subjects who will be lost entirely, reduce n instead.

alternative

"two.sided" (default), "greater", or "less", as in icc_test(). A design planned with the defaults matches the analysis run with the defaults.

method

"exact" (default) computes power from the exact distribution of the F statistic. "approximate" uses the closed form of Walter, Eliasziw and Donner (1998), available only for model = "oneway".

Details

Exact power. For the one-way and the two-way consistency ICC with complete data, the F ratio of the subject mean square to the error mean square is a scaled central F variable whatever the true ICC ρ\rho:

F∼1+(k−1)ρ1−ρFn−1,ν2,F \sim \frac{1 + (k-1)\rho}{1 - \rho} F_{n-1, \nu_2},

with ν2=n(k−1)\nu_2 = n(k-1) (one-way) or (n−1)(k−1)(n-1)(k-1) (two-way). The test of H0:ρ=ρ0H_0: \rho = \rho_0 (McGraw & Wong, 1996) divides the F ratio by the same factor at ρ0\rho_0, so its power is a single F probability (Donner & Eliasziw, 1987). Non-integer n is allowed, as in stats::power.t.test(); round up to the next whole number of subjects.

Approximate power uses the closed form of Walter, Eliasziw and Donner (1998),

n=1+2k(z1−α+z1−β)2(k−1)(log⁡C0)2,C0=1+kθ01+kθ1,θ=ρ1−ρ,n = 1 + \frac{2k (z_{1-\alpha} + z_{1-\beta})^2}{(k-1)(\log C_0)^2}, \quad C_0 = \frac{1 + k\theta_0}{1 + k\theta_1}, \quad \theta = \frac{\rho}{1 - \rho},

with α/2\alpha/2 in place of α\alpha for a two-sided test. Walter et al. (and the PASS software) write n for the number of measurements per subject and k for the number of subjects; here n is always the number of subjects and k the number of measurements per subject. The exact method is preferred; the approximation is provided to reproduce published tables.

Two-sided tests. When solving for rho1 with alternative = "two.sided", the solution above rho0 is returned; use alternative = "less" (with the significance level halved) for the one below.

Average-measure ICCs. The average-measure ICC is the Spearman-Brown projection of the single-measure ICC to k measurements, kρ/(1+(k−1)ρ)k\rho / (1 + (k-1)\rho), so a test about ICC(k) or ICC(C,k) is the same F test as the test about the corresponding single-measure ICC, and its interval is the projection of the single-measure interval. With unit = "average" the calculations are therefore exact, not approximations; k must be given because it defines the quantity being planned for.

Missing measurements (approximate). With missing = p > 0, the calculation uses the observed-data quantities that icc_test(method = "reml") uses for incomplete data: N=nk(1−p)N = nk(1 - p) observations, kˉ=N/n\bar{k} = N/n measurements per subject in the scale factor of the F ratio and in the interval, and error degrees of freedom N−n−k+1N - n - k + 1 (two-way) or N−nN - n (one-way). This is an approximation: the F distribution is exact only for complete data. In simulations of the REML analysis (one-way and two-way ICCs, 15 or 40 subjects, 3 or 5 items, 15% or 30% of measurements missing completely at random or 30% at random depending on the first measurement; 1000 replicates per setting), the approximate power was within 0.035 of the simulated power (Monte Carlo SE 0.016) and the approximate median interval width within 2.2% of the simulated median. The REML test itself was slightly liberal with 30% missing (type I error up to 0.069 missing completely at random and 0.073 at random, for a nominal 0.05), which the simulated power includes. A warning is given above 30% missing. The calculation assumes each subject keeps at least one measurement.

Planning other designs. These functions are exact only for the one-way and two-way consistency ICCs with complete, normally distributed data. They can still guide the planning of other ICC studies, as long as the direction of the error is kept in mind:

  • Absolute agreement ICC (ICC(A,1), and the CCC for two methods). Not supported exactly: its test has no exact F distribution, and because its denominator degrees of freedom depend on the sample, its power "cannot be determined exactly" (McGraw & Wong, 1996, footnote 3). When the items (trials, raters, methods) differ little systematically, ICC(A,1) is close to ICC(C,1) and the consistency calculation is a good guide. When they differ, the agreement test and interval also depend on the item mean square, which has only k - 1 degrees of freedom: power is lower and intervals wider than the consistency calculation says, most of all with 2 or 3 items. Treat the consistency result as a lower bound on the number of subjects; simulation would be needed for a precise answer.

  • Missing measurements. Use missing for measurements missing here and there (see below). For subjects who will be lost entirely, plan for the subjects who complete the study and recruit n/(1−d)n / (1 - d) for an expected dropout d.

  • Non-normal data. The exact calculations assume normality. With skewed or heavy-tailed data the F-based intervals are wider and cover less than planned (see icc_test() and reli_err_test() for simulations), so allow a margin or plan for the bootstrap with a larger sample.

Value

An object of class "power.htest", printed by stats::print.power.htest(), with elements n, k, rho0, rho1, sig.level, power, alternative, method and note.

References

Donner, A., & Eliasziw, M. (1987). Sample size requirements for reliability studies. Statistics in Medicine, 6(4), 441-448. doi:10.1002/sim.4780060404

McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30-46. doi:10.1037/1082-989X.1.1.30

Walter, S. D., Eliasziw, M., & Donner, A. (1998). Sample size and optimal designs for reliability studies. Statistics in Medicine, 17(1), 101-110. doi:10.1002/(SICI)1097-0258(19980115)17:1<101::AID-SIM727>3.0.CO;2-E

See Also

precision_icc() for the sample size for a confidence interval of a given width; icc_test() for the analysis.

Examples

# Subjects needed to show an ICC above 0.6 when it is 0.8, with 2 trials
power_icc(rho0 = 0.6, rho1 = 0.8, k = 2, power = 0.8,
          alternative = "greater")

# Power with 30 subjects and 3 trials
power_icc(n = 30, k = 3, rho0 = 0.5, rho1 = 0.75, alternative = "greater",
          model = "twoway")

# Trials needed with 20 subjects
power_icc(n = 20, k = NULL, rho0 = 0.5, rho1 = 0.8, power = 0.9,
          alternative = "greater")

Methods for powerCurve objects

Description

Methods defined for objects returned from the powerCurve function.

Usage

find_n(x, power = 0.8)

## S3 method for class 'powerCurve'
plot(x, ...)

Arguments

x

object of class powerCurve

power

Level of power (value between 0 and 1) for find_n to find the sample size.

...

further arguments passed through, see description of return value for details. blandPowerCurve.

Value

plot

Returns a plot of the limits of agreement (type = 1) or concordance plot (type = 2)

find_n

Find the smallest sample size at which the desired power is achieved, for each combination of settings. If the power is not reached within the sample sizes evaluated, N is NA (with a warning) and the range of sample sizes must be expanded.


Sample Size for the Precision of an Intraclass Correlation

Description

[Experimental]

Plans a reliability study so that the confidence interval for an intraclass correlation coefficient (ICC) from icc_test() is narrow enough. Computes the assurance (the probability that the interval is no wider than width), or solves for the number of subjects, the number of measurements per subject, or the width. Exactly one of n, k, width and assurance must be NULL.

Usage

precision_icc(
  n = NULL,
  k = 2,
  rho = NULL,
  width = NULL,
  conf.level = 0.95,
  assurance = 0.5,
  model = c("oneway", "twoway"),
  unit = c("single", "average"),
  missing = 0,
  method = c("exact", "approximate")
)

Arguments

n

Number of subjects.

k

Number of measurements (items: trials, raters or occasions) per subject. When solved for, the smallest whole number of at least 2 that reaches the target is returned.

rho

The anticipated ICC.

width

The full width of the confidence interval (upper minus lower limit). Zou (2012) uses the half-width ω\omega: width = 2 * omega.

conf.level

Confidence level of the interval (default 0.95).

assurance

The probability that the realised interval is no wider than width. The default, 0.5, plans for the median width.

model

"oneway" (default) or "twoway". "twoway" means the consistency ICC, ICC(C,1) of icc_test(); the absolute agreement ICC is not supported (see Details).

unit

"single" (default) for the reliability of a single measurement, or "average" for the reliability of the mean of the k measurements (ICC(k) or ICC(C,k)), as in icc_test(). With "average", the ICC values are on the average-measure scale and k cannot be solved for. See Details.

missing

The expected proportion of measurements (subject-by-item cells) that will be missing, analysed with icc_test(method = "reml"). The default, 0, gives the exact complete-data calculation; values above 0 give an approximation (see Details). For subjects who will be lost entirely, reduce n instead.

method

"exact" (default) computes the assurance from the exact distribution of the interval width. "approximate" uses the closed form of Zou (2012), available only for model = "oneway".

Details

Exact assurance. The two-sided F-based interval for the one-way or two-way consistency ICC (Searle's interval; Shrout & Fleiss, 1979; McGraw & Wong, 1996) is a function of the F ratio only, and the F ratio is a scaled central F variable for any true ICC (see power_icc()). The assurance is therefore the probability, under that distribution, of the F values whose interval is no wider than width. The width is largest at moderate F and shrinks towards both ends, so this set has two parts; the lower part (estimates near −1/(k−1)-1/(k-1)) is negligible in realistic designs. Mondal et al. (2024) recommend this interval for the one-way ICC.

Approximate sample size (Zou, 2012) for a Wald interval with Fisher's variance, half-width ω\omega and assurance 1−γ1 - \gamma:

n=1+[Az1−α/2+(Az1−α/2)2+4ωA∣B∣z1−γz1−α/2ω2k(k−1)]2,n = 1 + \left[\frac{A z_{1-\alpha/2} + \sqrt{(A z_{1-\alpha/2})^2 + 4 \omega A |B| z_{1-\gamma} z_{1-\alpha/2}}}{\omega\sqrt{2k(k-1)}}\right]^2,

with A=(1−ρ)(1+(k−1)ρ)A = (1-\rho)(1 + (k-1)\rho) and B=k−2+2ρ(1−k)B = k - 2 + 2\rho(1-k). It requires assurance of at least 0.5. Mondal et al. (2024) note that Zou used n−1n - 1 in deriving it and that another expression in his paper (the variance of the log-F statistic, not used here) is incorrect; the exact method avoids both issues. For a one-sided lower limit, the assurance that the lower limit exceeds a value ρ0\rho_0 is the power of the one-sided test: use power_icc(alternative = "greater").

Average-measure ICCs (unit = "average"): the interval for ICC(k) or ICC(C,k) has limits 1−a/F1 - a/F and 1−1/(bF)1 - 1/(bF) (with a and b the F quantiles), so its width, (a−1/b)/F(a - 1/b)/F, decreases with F and the assurance is a single F probability. width is on the average-measure scale. The approximate method is for single measures only.

Missing measurements (missing > 0): the interval width uses the observed-data quantities described in power_icc(), an approximation whose median widths were within 2.2% of simulated REML intervals with up to 30% of measurements missing.

The absolute agreement ICC is not supported exactly; see "Planning other designs" in power_icc() for using these calculations as a guide for it, for missing data, and for non-normal data.

Value

An object of class "power.htest", printed by stats::print.power.htest(), with elements n, k, rho, width, conf.level, assurance, method and note.

References

McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30-46. doi:10.1037/1082-989X.1.1.30

Mondal, D., Vanbelle, S., Cassese, A., & Candel, M. J. J. M. (2024). Review of sample size determination methods for the intraclass correlation coefficient in the one-way analysis of variance model. Statistical Methods in Medical Research, 33(3), 532-553. doi:10.1177/09622802231224657

Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420-428. doi:10.1037/0033-2909.86.2.420

Zou, G. Y. (2012). Sample size formulas for estimating intraclass correlation coefficients with precision and assurance. Statistics in Medicine, 31(29), 3972-3981. doi:10.1002/sim.5466

See Also

power_icc(); icc_test() for the analysis.

Examples

# Subjects for a median 95% CI width of 0.2 around an ICC of 0.8, 3 trials
precision_icc(rho = 0.8, k = 3, width = 0.2)

# ... with 90% assurance
precision_icc(rho = 0.8, k = 3, width = 0.2, assurance = 0.9)

# Zou (2012), Table 1: half-width 0.1, 80% assurance
precision_icc(rho = 0.6, k = 2, width = 0.2, assurance = 0.8,
              method = "approximate")

Measurement Error Test

Description

[Experimental]

Estimates one measurement error statistic, the standard error of measurement (SEM), the standard error of estimate (SEE), or the standard error of prediction (SEP), with a confidence interval and an optional test against an acceptable error, and returns an object of class "htest". The defaults give the typical error of Hopkins (2000): the SEM from a two-way model with systematic differences between items excluded, MSE\sqrt{MS_E}.

Usage

reli_err_test(
  x = NULL,
  data = NULL,
  measure = NULL,
  item = NULL,
  id = NULL,
  col.names = NULL,
  stat = c("SEM", "SEE", "SEP"),
  scale = c("raw", "log", "mean"),
  model = c("twoway", "oneway"),
  type = c("consistency", "agreement"),
  method = c("anova", "reml"),
  conf.level = 0.95,
  null.value = NULL,
  alternative = c("less", "greater", "two.sided"),
  boot = FALSE,
  boot_ci = c("auto", "perc", "bca", "stud", "basic"),
  replicates = 1999,
  se_method = c("jackknife", "influence", "normal")
)

Arguments

x

A formula of the form values ~ item | id, where values is the measurement, item identifies the trial, rater or occasion, and id identifies the subject. Use either x, the measure/item/id arguments, or col.names.

data

A data frame containing the variables.

measure, item, id

Column names (character) of the measurement, the item (trial, rater, or occasion) and the subject identifier, for data in long format.

col.names

Column names (character) of the items, for data in wide format (one row per subject).

stat

"SEM" (default), "SEE", or "SEP". See Details.

scale

The scale the statistic is reported on: "raw" (default, units of the measurement), "log" (sympercents: 100 times the statistic of the log-transformed measurements), or "mean" (percentage of the grand mean). The last two are coefficients of variation; see Details.

model

"twoway" (default) when every subject is measured on the same set of items (e.g. test-retest trials), or "oneway" when each subject is measured by a different set of items.

type

"consistency" (default) excludes systematic differences between items from the error; "agreement" includes them. Only "agreement" is defined for model = "oneway".

method

"anova" (default) estimates the variance components from the analysis of variance and requires complete, balanced data. "reml" fits a linear mixed model and allows missing measurements.

conf.level

Confidence level of the interval (default 0.95).

null.value

An acceptable error to test against, on the scale given by scale (units, sympercents, or percent of the mean). The default, NULL, gives the estimate and interval without a test.

alternative

"less" (default), "greater", or "two.sided". The default tests whether the error is smaller than null.value. Used only when null.value is given; without a test the interval is two-sided.

boot

If TRUE, use a nonparametric bootstrap that resamples subjects for the interval and p-value. Default FALSE.

boot_ci

Bootstrap interval type: "auto" (default), "perc" (percentile), "bca", "stud" (studentized), or "basic". With method = "anova", "auto" uses "bca" for the two-way agreement statistics and "stud" otherwise ("perc" when se_method = "normal"); with method = "reml" it uses "perc". See Details. Used only when boot = TRUE.

replicates

Number of bootstrap replicates (default 1999).

se_method

How the standard error of the log of the statistic is estimated: "jackknife" (default; delete one subject at a time, accurate in simulations but n refits with method = "reml"), "influence" (a per-subject sandwich estimate; cheap but too small with few subjects), or "normal" (normal theory; too small with skewed or heavy-tailed data, and cannot be used by the studentized bootstrap). It is reported in stderr and used by the studentized bootstrap; it does not change the analytic interval or test. For the two-way agreement statistics, the jackknife and influence SEs add the normal-theory variance of the item mean square, treating the items as random (as the hybrid bootstrap does). See icc_test() for the simulations.

Details

Choosing model and type. Choose them as for the ICC (see "Choosing the form" in icc_test()): type = "agreement" when systematic differences between items count as error, "consistency" when they are checked and reported separately. Report the error statistic that matches the ICC.

Statistics. With MSMS the mean squares of the subjects-by-items analysis of variance (n subjects, k items):

stat model type Estimate Analytic inference
SEM twoway consistency MSE\sqrt{MS_E} exact χ2\chi^2, df (n−1)(k−1)(n-1)(k-1)
SEM twoway agreement (MSJ+(n−1)MSE)/n\sqrt{(MS_J + (n-1) MS_E)/n} Satterthwaite χ2\chi^2
SEM oneway agreement MSW\sqrt{MS_W} exact χ2\chi^2, df n(k−1)n(k-1)
SEE any any SDICC(1−ICC)SD \sqrt{ICC(1 - ICC)} delta method, log scale
SEP any any SD1−ICC2SD \sqrt{1 - ICC^2} delta method, log scale

The consistency SEM is the typical error of Hopkins (2000) and the MSE\sqrt{MS_E} preferred by Weir (2005); the one-way SEM is the within-subject SD of Bland and Altman. The agreement SEM adds the variance between items to the error; with complete data it equals the one-way SEM, but its degrees of freedom differ.

SEE and SEP (Weir, 2005, eqs. 11 and 15) use the single-measure ICC of the same model and type (ICC(C,1), ICC(A,1) or ICC(1)). The SEE is the standard error of a subject's estimated true score; the SEP is used to judge whether a change from the estimated true score is real. Both depend on how much subjects differ from each other, so the SEM is generally the better description of measurement error.

Total SD. SD is the total SD matched to the denominator of the ICC: SD2=σsubjects2+σerror2SD^2 = \sigma^2_{subjects} + \sigma^2_{error} for consistency, plus σitems2\sigma^2_{items} for agreement. With this SD, SD1−ICCSD\sqrt{1 - ICC} equals the SEM exactly, which is why there is no separate ICC-based SEM. In mean squares (complete data): SD2=(MSB+(k−1)MSE)/kSD^2 = (MS_B + (k-1)MS_E)/k (consistency), (MSB+(k−1)MSE+k(MSJ−MSE)/n)/k(MS_B + (k-1)MS_E + k(MS_J - MS_E)/n)/k (agreement), and (MSB+(k−1)MSW)/k(MS_B + (k-1)MS_W)/k (one-way). It is not the SD of all the measurements, SStotal/(N−1)\sqrt{SS_{total}/(N-1)}, used by Weir (2005) and by reli_stats(), which estimates none of these variances; so SEE and SEP differ slightly from Weir's (for his data set A, SD 32.83 vs 31.74, SEE 7.35 vs 7.11, SEP 10.54 vs 10.20). A negative ICC estimate is treated as zero (SEE = 0, SEP = SD).

Intervals and tests. The SEM interval is the square root of the χ2\chi^2 interval for the error variance, and the test of H0:SEM=H_0: SEM = null.value is the χ2\chi^2 test of the variance, ν⋅SEM2/null2\nu \cdot SEM^2 / null^2. For the SEE and SEP the interval is a Wald interval on the log scale, with a delta-method SE from the normal-theory variances of the mean squares, and the test is the matching z test. With a one-sided alternative the interval is one-sided. method = "reml" with missing data uses degrees of freedom based on the observed data, as in icc_test(), and gives a warning.

Bootstrap. boot = TRUE resamples subjects, as in icc_test(), including the hybrid step for the two-way agreement statistics (the item mean square of each replicate is multiplied by an independent χk−12/(k−1)\chi^2_{k-1}/(k-1) draw). The studentized interval works on the log scale. The p-value inverts the interval and is approximate.

In simulations (3 trials, 10 or 30 subjects, ICC 0.5 or 0.9, normal or skewed and heavy-tailed data, complete data), the analytic intervals covered 0.93 to 0.97 with normal data for the consistency and one-way statistics, but only 0.79 to 0.90 with non-normal data (the χ2\chi^2 interval for a variance is sensitive to heavy tails). The studentized bootstrap was the best method for the consistency SEM, the SEP and the one-way SEM (0.89 to 0.96 throughout), and BCa for the agreement SEM and SEP (mean 0.91), hence boot_ci = "auto". The percentile interval was too narrow with 10 subjects (as low as 0.54). For the SEE the analytic interval was best with normal data (0.93 to 0.97); no bootstrap interval was reliable near an ICC of 0.5, where the SEE hardly changes with the ICC.

Agreement statistics with few items. The agreement SEM and SEP include the variance between items, which is estimated from only k - 1 degrees of freedom. When it is a large part of the error and k is small, every interval under-covers: with 3 trials and an item variance about as large as the error variance (ICC 0.9), the Satterthwaite interval covered 0.81 to 0.91 with normal data and BCa 0.91 to 0.95 (0.80 with 30 subjects and skewed data). Resampling subjects without the hybrid step covered only 0.39 to 0.63 in those settings. More items help more than more subjects.

Coefficient of variation. When the error grows with the size of the measurements, it is better expressed relative to them (Hopkins, 2000; Atkinson and Nevill, 1998; Weir, 2005). Two scales are available:

  • scale = "log" fits the model to the natural logarithm of the measurements (all must be positive) and reports 100×100 \times the statistic, in sympercents (Cole, 2000). Because this is a constant multiple of the log-scale statistic, the SEM keeps its exact (or Satterthwaite) χ2\chi^2 interval and test. A sympercent ss converts to the conventional CV% as 100exp⁡((s/100)2)−1100\sqrt{\exp((s/100)^2) - 1}; the two are nearly identical below about 15% (e.g. 10 sympercents is a CV of 10.03%). The conversion is monotone, so it can be applied to the interval limits too.

  • scale = "mean" reports 100×100 \times statistic / grand mean (the "pooled" CV). The interval and test use the delta method on the log scale, adding the uncertainty in the grand mean, whose normal-theory variance is MSB/(nk)MS_B/(nk) with items fixed (consistency, one-way) and (MSB+MSJ−MSE)/(nk)(MS_B + MS_J - MS_E)/(nk) with items random (two-way agreement).

In simulations like those below (population mean 5), the "mean" interval covered 0.93 to 0.96 with normal data and 30 subjects (except for the agreement SEM with few items, see below); leaving out the uncertainty in the grand mean lowered coverage by up to 0.05. With the bootstrap, the grand mean of each replicate of a two-way agreement statistic also gets the item mean's normal-theory variation, as in the hybrid step below.

null.value is on the same scale as the estimate. The log scale is usually preferable: it treats the error as proportional to each measurement rather than to the overall mean.

Minimal detectable change. The minimal detectable change (MDC), smallest real difference, or repeatability coefficient is a scaled SEM: 1.962SEM1.96 \sqrt{2} SEM for 95% (Weir, 2005, eq. 13). mdc_test() computes it, with its interval and a test, from the data or from an SEM result of this function. With two trials, agreement_limit() gives the same quantity as the half width of the limits of agreement, with the bias reported separately.

Value

A list of class c("reli_htest", "htest") containing:

statistic

The χ2\chi^2 (SEM) or z (SEE, SEP) statistic, when a test was requested without the bootstrap.

parameter

The degrees of freedom of the χ2\chi^2 test.

p.value

The p-value, when a test was requested.

conf.int

The confidence interval.

estimate

The estimate, named by stat.

stderr

Standard errors of the estimate (named by stat) and of its log (log), from se_method. With boot = TRUE, also boot: the standard deviation of the bootstrap estimates.

null.value

The acceptable error tested against, if any.

alternative

The alternative hypothesis.

method

A description of the statistic and the inference.

data.name

A description of the data.

fit

The fitted variance components, mean squares and model.

data

The data in long format.

settings

The settings used, for the methods.

boot

Bootstrap details (NULL unless boot = TRUE).

References

Atkinson, G., & Nevill, A. M. (1998). Statistical methods for assessing measurement error (reliability) in variables relevant to sports medicine. Sports Medicine, 26(4), 217-238. doi:10.2165/00007256-199826040-00002

Cole, T. J. (2000). Sympercents: symmetric percentage differences on the 100 log_e scale simplify the presentation of log transformed data. Statistics in Medicine, 19(22), 3109-3125. doi:10.1002/1097-0258(20001130)19:22<3109::AID-SIM558>3.0.CO;2-F

Hopkins, W. G. (2000). Measures of reliability in sports medicine and science. Sports Medicine, 30(1), 1-15. doi:10.2165/00007256-200030010-00001

Weir, J. P. (2005). Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. Journal of Strength and Conditioning Research, 19(1), 231-240. doi:10.1519/15184.1

See Also

icc_test(); reli_htest-methods for summary(), plot() and check().

Examples

# Weir (2005) data set A: 8 subjects, 2 trials
weir <- data.frame(t1 = c(146, 148, 170, 90, 157, 156, 176, 205),
                   t2 = c(140, 152, 152, 99, 145, 153, 167, 218))
reli_err_test(col.names = c("t1", "t2"), data = weir)

# Is the typical error below 10 units?
reli_err_test(col.names = c("t1", "t2"), data = weir, null.value = 10)

# Typical error as a coefficient of variation (sympercents)
reli_err_test(col.names = c("t1", "t2"), data = weir, scale = "log")

# Standard error of prediction
res <- reli_err_test(col.names = c("t1", "t2"), data = weir, stat = "SEP")
summary(res)

Reliability Across Facets: Repeats and Occasions

Description

[Experimental]

This function is experimental and may change; its arguments and name may change too even if its functionality is retained.

Estimates one reliability coefficient or one standard error of measurement (SEM) when each subject is measured more than once by each item (rater, device). For example, 3 operators each scanning 20 participants twice. The extra measurements separate error within a rater from disagreement between raters, which icc_test() and reli_err_test() cannot do with one measurement per subject and item. The measurements can be exchangeable repeats within each subject-by-item cell (the default), or shared occasions (occasion: every rater measured every subject on day 1 and again on day 2). The model is the generalizability-theory (G-study) model for that design. It's G-study capabilities are narrow, which is purposeful at this time. Returns an object of class "htest" with a confidence interval and an optional test.

Usage

reli_facet_test(
  x = NULL,
  data = NULL,
  measure = NULL,
  item = NULL,
  id = NULL,
  occasion = NULL,
  stat = c("ICC", "SEM"),
  facet = c("intra", "inter", "overall"),
  type = c("consistency", "agreement"),
  k_avg = 1,
  m_avg = 1,
  r_avg = 1,
  scale = c("raw", "log", "mean"),
  method = c("anova", "reml"),
  conf.level = 0.95,
  null.value = NULL,
  alternative = c("two.sided", "greater", "less"),
  ci_method = c("auto", "mls", "satterthwaite"),
  boot = FALSE,
  boot_ci = c("auto", "perc", "bca", "stud", "basic"),
  replicates = 1999,
  se_method = c("jackknife", "normal")
)

Arguments

x

A formula of the form values ~ item | id, with several rows per subject and item. Leave NULL to use measure, item and id. Wide data are not supported.

data

A data frame containing the variables.

measure, item, id

Column names (character) of the measurement, the item (trial, rater, or occasion) and the subject identifier, for data in long format.

occasion

NULL (default) when the measurements within each subject-by-item cell are exchangeable repeats, or the name of the column identifying shared occasions (e.g. day 1 and day 2, the same for every item). See "Nested or crossed".

stat

"ICC" (default), a reliability coefficient, or "SEM", the standard error of measurement. See Details.

facet

Which measurement varies: "intra" (default), the same item on another repeat or occasion (intra-rater, repeatability); "inter", another item (on the same occasion, with occasion); or "overall", another item on another occasion (the same as "inter" without occasion). See Details.

type

"consistency" (default) sets systematic differences between items (and occasions) aside, to be reported separately; "agreement" counts them as error, as in icc_test(). Without occasion, facet = "intra" has no systematic effects and type makes no difference.

k_avg, m_avg, r_avg

The planned score (D-study): the mean of k_avg items, m_avg occasions and r_avg repeats (defaults 1, a single measurement). k_avg must be 1 for facet = "intra" and m_avg must be 1 for facet = "inter" and without occasion; r_avg must be 1 for crossed data with one measurement per subject, item and occasion.

scale

For stat = "SEM", the scale it is reported on: "raw" (default), "log" (sympercents) or "mean" (percentage of the grand mean), as in reli_err_test(). Must be "raw" for stat = "ICC".

method

"anova" (default) for balanced data (every subject measured by every item, on every occasion, the same number of times), or "reml" (a linear mixed model) for missing cells or unequal numbers of repeats.

conf.level

Confidence level of the interval (default 0.95).

null.value

A value to test against, on the scale of the estimate. The default, NULL, gives the estimate and interval without a test.

alternative

"two.sided" (default), "greater" or "less". For an SEM, "less" tests whether the error is below an acceptable value. Used only when null.value is given.

ci_method

The analytic interval when it is not exact: "auto" (default), "mls" (modified large-sample) or "satterthwaite". See Details. Ignored when the interval is exact or boot = TRUE.

boot

If TRUE, use a nonparametric bootstrap that resamples subjects for the interval and p-value. Default FALSE.

boot_ci

Bootstrap interval type: "auto" (default), "perc", "bca", "stud" (studentized) or "basic". With method = "anova", "auto" uses "bca" for an agreement SEM that counts systematic differences between items or occasions and "stud" otherwise ("perc" when se_method = "normal"); with method = "reml" it uses "perc". See Details. Used only when boot = TRUE.

replicates

Number of bootstrap replicates (default 1999).

se_method

How the standard error on the working scale is estimated: "jackknife" (default; delete one subject at a time) or "normal" (normal theory, delta method). Reported in stderr and used by the studentized bootstrap.

Details

When considered nested versus crossed, ask whether "measurement 2" is the same event for every item. If each operator simply scanned each participant twice, in no shared order, the measurements are exchangeable repeats nested in the subject-by-item cell: leave occasion = NULL. If every operator measured everyone on day 1 and again on day 2, the days are shared occasions: give occasion. Treating occasions as repeats counts day-to-day shifts as repeat error, so the intra-rater error looks larger and cannot be split into a systematic shift (fixable) and noise. With occasion, the result (in nested) and summary() also give what the repeat model would have estimated.

For the model, all effects are random. Without occasion, with n subjects, k items and r repeats per cell,

ypir=μ+p+i+(pi)+epir;y_{pir} = \mu + p + i + (pi) + e_{pir};

with occasion (m occasions),

ypio=μ+p+i+o+(pi)+(po)+(io)+(pio,e),y_{pio} = \mu + p + i + o + (pi) + (po) + (io) + (pio, e),

where with one measurement per subject, item and occasion the three-way interaction and the error are not separable (the residual). Further repeats within each subject, item and occasion are allowed and add a separate repeat error. The ANOVA estimators are unconstrained: a negative variance component estimate is kept (as icc_test() keeps a negative ICC), so a coefficient can be negative. The component table in components (and summary()) shows each component with an interval and its share of the variance of a single measurement. The model is the unrestricted (random facets) one used by lme4 and by McGraw and Wong (1996); for the consistency forms the items may be regarded as fixed, which changes the interpretation but not the computation.

Following Brennan (2001), the facet that stays the same is part of the subject's score, and the facets that vary are error. For a planned score that is the mean of k′k' items (k_avg), m′m' occasions (m_avg) and r′r' repeats (r_avg), the error variance (SEM2^2) is:

facet Score Consistency error Agreement adds
intra, repeats σp2+σpi2\sigma^2_p + \sigma^2_{pi} σe2/r′\sigma^2_e/r' (nothing)
intra, occasions σp2+σpi2\sigma^2_p + \sigma^2_{pi} (σpo2+σpio,e2)/m′(\sigma^2_{po} + \sigma^2_{pio,e})/m' (σo2+σio2)/m′(\sigma^2_o + \sigma^2_{io})/m'
inter, repeats σp2\sigma^2_p σpi2/k′+σe2/(k′r′)\sigma^2_{pi}/k' + \sigma^2_e/(k'r') σi2/k′\sigma^2_i/k'
inter, occasions σp2+σpo2\sigma^2_p + \sigma^2_{po} (σpi2+σpio,e2)/k′(\sigma^2_{pi} + \sigma^2_{pio,e})/k' (σi2+σio2)/k′(\sigma^2_i + \sigma^2_{io})/k'
overall, occasions σp2\sigma^2_p σpi2/k′+σpo2/m′+σpio,e2/(k′m′)\sigma^2_{pi}/k' + \sigma^2_{po}/m' + \sigma^2_{pio,e}/(k'm') σi2/k′+σo2/m′+σio2/(k′m′)\sigma^2_i/k' + \sigma^2_o/m' + \sigma^2_{io}/(k'm')

(with further repeats within subject, item and occasion, the residual is split and its error part is also divided by r′r'). The coefficient (stat = "ICC") is score / (score + error). The intra-rater coefficient is the correlation between two measurements by the same item, pooled over items (it assumes the same error for every item; check() shows the within-cell SD by item). With occasions, the inter-rater coefficient is for raters on the same occasion, what most inter-rater studies report; "overall" is for any rater on any occasion, and its coefficients are the generalizability (Eρ2E\rho^2) and dependability (Φ\Phi) coefficients of G theory. In gauge R&R terms (repeats), σe2\sigma^2_e is repeatability and σi2+σpi2\sigma^2_i + \sigma^2_{pi} reproducibility.

It is important to know what averaging does for this function. Without occasion and with r′=rr' = r (the score averages as many repeats as the study had), the inter-rater consistency coefficient is the consistency ICC of the cell means, (F−1)/(F−1+k/k′)(F - 1)/(F - 1 + k/k') with F=MSP/MSPIF = MS_P/MS_{PI}, identical to icc_test() on the cell means; it is not the reliability of a single measurement unless r = 1. Averaging the repeats before calling icc_test() is therefore only right when the scores used in practice are also means of r repeats.

An estimand that is a multiple of one mean square (an SEM) or a function of the ratio of two (a coefficient) has an exact interval and test. Without occasion, the intra SEM (χ2\chi^2 on MSWMS_W) and the inter-rater consistency SEM and coefficient with r′=rr' = r. The others are approximate:

  • "mls": modified large-sample intervals for linear combinations of mean squares (Graybill & Wang, 1980; Ting et al., 1990; Burdick & Graybill, 1992). For a coefficient ρ\rho, the interval holds the values ρ0\rho_0 for which the MLS interval of σscore2−ρ0σtotal2\sigma^2_{score} - \rho_0 \sigma^2_{total} contains 0. The p-value inverts the interval, so there is no test statistic.

  • "satterthwaite": the SEM uses a χ2\chi^2 distribution with Satterthwaite df; a coefficient uses an F test of σscore2−ρ0σtotal2=0\sigma^2_{score} - \rho_0 \sigma^2_{total} = 0, the positive and negative parts each with Satterthwaite df (McGraw & Wong's device for ICC(A,1)), and the interval inverts it.

  • "auto" uses MLS, chosen in simulations (see "Simulations").

SEMs with scale = "mean" use a Wald interval on the log scale with a delta-method SE that includes the uncertainty of the grand mean.

When wanting bootstrapped results, boot = TRUE resamples subjects, each with all its measurements, as in icc_test(). Resampling subjects keeps the same items and occasions, so for statistics that count systematic differences between them as error, the mean squares of the effects without subjects (items, occasions, items by occasions) are also multiplied by independent χdf2/df\chi^2_{df}/df draws (the hybrid step). The p-value inverts the interval and is approximate.

When there is unbalanced data, method = "reml" fits the model with lme4 and keeps the analytic inference, with degrees of freedom from the observed data and the harmonic mean number of repeats in place of r, and gives a warning.

In simulations, repeats (no occasion): with 10 or 30 subjects, 2, 3 or 5 items, 2 or 3 repeats, four mixes of variance components and normal data (2000 data sets per setting), the exact intervals covered 0.940 to 0.951 and the MLS intervals at least 0.939 for every estimand (0.95 to 0.96 on average; slightly conservative for Φ\Phi), hence ci_method = "auto" uses MLS. The Satterthwaite intervals were similar for the intra-rater and consistency statistics but fell to 0.67 to 0.74 for the agreement SEM and coefficient with 2 items when systematic differences between items were the main error. With REML and unbalanced data (30 subjects, 20% of repeats missing, or 1 to 3 repeats per cell), the MLS intervals covered 0.937 to 0.969.

Occasions (2 or 3 items, 2 or 3 occasions, one measurement per cell, four mixes including a large day effect and large rater bias, 2000 data sets per setting): the MLS intervals covered at least 0.934 with 30 subjects and 0.928 with 10, for every facet, type and statistic; the Satterthwaite intervals fell to 0.69 to 0.79 for the agreement statistics.

Warning: there are uneven tails for agreement statistics. The agreement statistics include a component estimated from very few degrees of freedom (items, or occasions: k - 1 or m - 1). The MLS bound that this component drives is wide, and the other bound is close to nominal. With 30 subjects and 95% two-sided intervals, the agreement coefficients fell below their lower limit 0 to 1% of the time and above their upper limit about 3% (nominal 2.5% each; two-sided coverage up to 0.97 to 0.99 with occasions), and the agreement SEMs fell above their upper limit 0 to 0.6% of the time and below their lower limit 3 to 6%. So the usual one-sided tests are conservative: a coefficient greater than a threshold (alternative = "greater") and an SEM less than an acceptable error (alternative = "less"). The opposite one-sided tests can be somewhat liberal (up to about 6% instead of 2.5% for an agreement SEM greater than a threshold). With REML and 20% of cells missing, MLS covered 0.922 to 0.979. Treating the occasions as exchangeable repeats (leaving out occasion) gave intra-rater SEM intervals that covered the true agreement SEM only 0.24 to 0.92 of the time (0.04 to 0.87 for the consistency SEM, which it overstated by 30% on average).

With skewed, heavy-tailed data every analytic interval under-covered (0.78 to 0.90 with repeats). The studentized bootstrap was then the best interval for every statistic except the agreement SEM, for which BCa was best, hence boot_ci = "auto": with repeats it covered 0.90 to 0.93 for the intra-rater and consistency statistics with 30 subjects (2 or 3 items, 2 repeats, 300 data sets of 499 replicates per setting). As in reli_err_test(), agreement statistics with few items (or occasions) and large systematic differences between them under-covered with every method (0.69 to 0.85 with 2 or 3 items); more items help more than more subjects. Resampling subjects without the hybrid step covered as little as 0.15 there. With normal data the analytic intervals were better than any bootstrap interval. With occasions (2 days), the bootstrap gave the same "auto" choices and helped for the consistency statistics (0.92 to 0.93 with skewed data), but for the agreement statistics the MLS interval was as good or better even with skewed data, and every interval under-covered when a day effect on 2 days dominated the error.

Other software: Given the narrow range of application, I want highlight a few R packages. VCA (variance components for deeper nesting and laboratory precision designs), rptR (repeatability for non-Gaussian data), gtheoryr (crossed multi-facet G-studies and D-study design optimisation), SixSigma (ss.rr(), gauge R&R summaries) and MethComp (method comparison with linked replicates and method-specific variances) fit related models. This function adds intervals and tests for the coefficients and the intra- and inter-rater SEMs, and the link to mdc_test().

Value

A list of class c("reli_htest", "htest") containing:

statistic, parameter

The F (coefficients) or χ2\chi^2 (SEM) statistic and its df, when a test was requested with an exact or Satterthwaite method (z for SEMs with scale = "mean").

p.value

The p-value, when a test was requested.

conf.int

The confidence interval.

estimate

The estimate.

stderr

Standard errors of the estimate and on the working scale (z: −log⁡(1−ρ)/2-\log(1 - \rho)/2 for coefficients, the log for SEMs), from se_method; with boot = TRUE, also boot.

null.value, alternative, method, data.name

As for an htest.

components

The variance components with intervals and shares.

nested

With occasion and facet = "intra": the estimate the repeat model (occasions ignored) would give.

fit

The mean squares, degrees of freedom and fitted model.

data

The data in long format.

settings

The settings used, for the methods.

boot

Bootstrap details (NULL unless boot = TRUE).

References

Brennan, R. L. (2001). Generalizability Theory. Springer. doi:10.1007/978-1-4757-3456-0

Burdick, R. K., & Graybill, F. A. (1992). Confidence Intervals on Variance Components. Marcel Dekker.

Graybill, F. A., & Wang, C. M. (1980). Confidence intervals on nonnegative linear combinations of variances. Journal of the American Statistical Association, 75(372), 869-873. doi:10.1080/01621459.1980.10477565

McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30-46. doi:10.1037/1082-989X.1.1.30

Ting, N., Burdick, R. K., Graybill, F. A., Jeyaratnam, S., & Lu, T.-F. C. (1990). Confidence intervals on linear combinations of variance components that are unrestricted in sign. Journal of Statistical Computation and Simulation, 35(3-4), 135-143. doi:10.1080/00949659008811240

See Also

icc_test() and reli_err_test() for one measurement per subject and item; mdc_test(); reli_htest-methods for summary(), plot() and check().

Examples

# 12 participants, 3 operators, 2 scans each (exchangeable repeats)
set.seed(2)
scans <- expand.grid(scan = 1:2, operator = c("A", "B", "C"), id = 1:12)
subj <- rnorm(12, 50, 5)
bias <- c(A = 0, B = 1.5, C = -1)
scans$thickness <- subj[scans$id] + bias[scans$operator] +
  rnorm(nrow(scans), 0, 1.5)

# Intra-rater (repeatability) ICC and SEM
reli_facet_test(thickness ~ operator | id, data = scans)
reli_facet_test(thickness ~ operator | id, data = scans, stat = "SEM")

# Inter-rater agreement for a single scan by any operator
res <- reli_facet_test(thickness ~ operator | id, data = scans,
                       facet = "inter", type = "agreement")
summary(res)

# D-study: the mean of 2 scans by one operator
reli_facet_test(thickness ~ operator | id, data = scans, facet = "inter",
                type = "agreement", r_avg = 2)

# If scan 1 and scan 2 were shared days, the same for every operator
day1 <- reli_facet_test(thickness ~ operator | id, data = scans,
                        occasion = "scan", type = "agreement")
day1
day1$nested

# Smallest difference between two operators beyond measurement error
mdc_test(reli_facet_test(thickness ~ operator | id, data = scans,
                         stat = "SEM", facet = "inter",
                         type = "agreement"))

Methods for reli_htest objects

Description

Methods for objects returned by icc_test(), reli_err_test(), reli_facet_test() and mdc_test(). They print as standard "htest" objects; summary() describes the result in words, plot() shows the data, and check() gives model diagnostics.

Usage

## S3 method for class 'reli_htest'
summary(object, style = c("auto", "hypothesis", "estimation"), ...)

## S3 method for class 'summary.reli_htest'
print(x, ...)

## S3 method for class 'reli_htest'
plot(x, type = c("profile", "subjects", "items", "pairs", "components"), ...)

## S3 method for class 'reli_htest'
check(x, ...)

Arguments

style

How summary() words the result: "auto" (default), "hypothesis", or "estimation". "auto" uses "hypothesis" when the result has a p-value (a null.value was given) and "estimation" otherwise. See Details.

...

Not used.

x, object

An object of class reli_htest.

type

For plot(), the plot to draw: "profile" (default), each subject's measurements across the items in grey with the item means and their 95% confidence intervals (adjusted for subjects when measurements are missing); "subjects", the subjects ordered by their mean, with each subject's measurements, coloured by item, and their range, which compares the spread between subjects with the scatter within subjects (what the ICC summarises); "items", the deviation of each measurement from its subject's mean, by item, which shows systematic differences between items (error for an agreement ICC, but not for a consistency ICC); "pairs", each pair of items plotted against each other with the line of identity; or, for reli_facet_test() results, "components", the share of each variance component. For reli_facet_test() results the profile, items and pairs plots use the subject-by-item cell means.

Details

summary() describes the estimate and its interval, what was estimated, the design, and how the inference was done, in one of two styles:

  • "hypothesis" (Neyman-Pearson; as in the TOSTER package): the test decision at α=1−\alpha = 1 - conf.level, i.e. whether the null hypothesis can be rejected. Requires a p-value; without one, summary() falls back to "estimation" with a message.

  • "estimation" (following Greenland and colleagues): the interval is described as a compatibility interval, the range of values most compatible with the data under the model, with no decision and no α\alpha. If there is a p-value, it is converted to an S-value (surprisal), s=−log⁡2ps = -\log_2 p: the bits of information the data provide against the tested hypothesis, comparable to the surprise of getting s heads in a row from s tosses of a fair coin.

Qualitative labels such as "good" or "excellent" are never attached to an ICC; see icc_test() for why.

Diagnostics. The two-way model assumes that subject and item effects add. A subject-by-item interaction, for example items (raters, devices, occasions) that differ in scale, is absorbed into the error: it lowers the consistency ICC and inflates the SEM. check() tests for it with Mandel's (1961) column-regression test: each item gets its own slope on the estimated subject effects, and the k - 1 extra slopes are F-tested. The panel plots the residuals of the additive model against the estimated subject effect with one line per item, so items whose lines slope are those that differ in scale. Mandel's test includes the alternative of Tukey's (1949) one-degree-of-freedom test (scale growing with the item mean, the multiplicative pattern) and also detects items that differ in scale but not in location, which Tukey's test misses; for two items the two tests are identical. The covariates are built from the fitted additive model, as in Milliken and Graybill (1970), so the F test remains exact with missing measurements. In simulations (30 subjects, 3 items, 3 missing cells, additive normal data), its type I error was 0.0525 at the 0.05 level. It uses the fixed-effects fit (subjects and items as factors) even when the ICC came from REML, and it has little power with few subjects. A rising within-subject SD against the subject mean means the error grows with the size of the measurement, so a relative error (reli_err_test() with scale = "log") describes it better.

For reli_facet_test() results, check() shows the residuals of the full model (deviations from the cell means), the additivity panel on the subject-by-item cell means, the within-cell SD by item with Bartlett's test of equal repeat error across items (the intra-rater coefficient pools it over items), the within-cell SD against the cell mean, and, with occasions, each item's mean by occasion (shifts and rater-specific drift). summary() adds the table of variance components.

Value

summary

An object of class summary.reli_htest: a list with the description in text, the style used, the S-value (s.value, when there is a p-value) and the parts used to build it. It prints as a paragraph.

plot

A ggplot2 plot (see type).

check

Diagnostic plots combined with patchwork: normality and heteroskedasticity of the model residuals; for two-way models, additivity (residuals of the additive subjects + items model against the estimated subject effect, one line per item, with Mandel's test for non-additivity); and the within-subject SD against the subject mean (on the scale the model was fitted). See Details.

References

Mandel, J. (1961). Non-additivity in two-way analysis of variance. Journal of the American Statistical Association, 56(296), 878-888. doi:10.1080/01621459.1961.10482132

Milliken, G. A., & Graybill, F. A. (1970). Extensions of the general linear hypothesis model. Journal of the American Statistical Association, 65(330), 797-807. doi:10.1080/01621459.1970.10481125

Tukey, J. W. (1949). One degree of freedom for non-additivity. Biometrics, 5(3), 232-242. doi:10.2307/3001938

Greenland, S. (2019). Valid p-values behave exactly as they should: Some misleading criticisms of p-values and their resolution with S-values. The American Statistician, 73(sup1), 106-114. doi:10.1080/00031305.2018.1529625

Rafi, Z., & Greenland, S. (2020). Semantic and cognitive tools to aid statistical science: replace confidence and significance by compatibility and surprise. BMC Medical Research Methodology, 20, 244. doi:10.1186/s12874-020-01105-9

Examples

sf <- data.frame(J1 = c(9, 6, 8, 7, 10, 6), J2 = c(2, 1, 4, 1, 5, 2),
                 J3 = c(5, 3, 6, 2, 6, 4), J4 = c(8, 2, 8, 6, 9, 7))
res <- icc_test(col.names = c("J1", "J2", "J3", "J4"), data = sf)
summary(res)
plot(res)
plot(res, type = "subjects")
check(res)

# A test against a threshold, in both styles
res2 <- icc_test(col.names = c("J1", "J2", "J3", "J4"), data = sf,
                 null.value = 0.5, alternative = "greater")
summary(res2)
summary(res2, style = "estimation")

Reliability Statistics

Description

[Superseded]

reli_stats() and reli_aov() return the intraclass correlation coefficients (ICCs), the coefficient of variation (CV), and the standard errors of measurement (SEM), estimate (SEE) and prediction (SEP) described by Weir (2005), all at once.

These functions are superseded: they still work, but new code should use icc_test() for an ICC (with a test against a meaningful null value and optional bootstrap inference) and reli_err_test() for the SEM, SEE or SEP, including the CV via scale = "log" (sympercents) or scale = "mean". See Details for how the two sets of functions differ.

Usage

reli_stats(
  measure = NULL,
  item = NULL,
  id = NULL,
  data,
  wide = FALSE,
  col.names = NULL,
  se_type = c("MSE", "ICC1", "ICC2", "ICC3", "ICC1k", "ICC2k", "ICC3k"),
  cv_calc = c("MSE", "residuals", "SEM"),
  conf.level = 0.95,
  other_ci = FALSE,
  type = c("chisq", "perc", "norm", "basic"),
  replicates = 1999
)

reli_aov(
  measure = NULL,
  item = NULL,
  id = NULL,
  data,
  wide = FALSE,
  col.names = NULL,
  se_type = c("MSE", "ICC1", "ICC2", "ICC3", "ICC1k", "ICC2k", "ICC3k"),
  cv_calc = c("MSE", "residuals", "SEM"),
  conf.level = 0.95,
  other_ci = FALSE,
  type = c("chisq", "perc", "norm", "basic"),
  replicates = 1999
)

Arguments

measure

Name of column containing the measurement of interest.

item

Name of column containing the items. If this is a test-retest reliability study then this would indicate the time point (e.g., time1, time2, time3, etc.).

id

Column with subject identifier.

data

Data frame with all data.

wide

Logical value (TRUE or FALSE) indicating if data is in a "wide" format. Default is FALSE.

col.names

If wide is equal to TRUE then col.names is a list of the column names containing the measurements for reliability analysis.

se_type

Type of standard error calculation. The default, "MSE", uses the error mean square for the SEM, and ICC3 (ICC(C,1)) for the SEE and SEP. Otherwise, the SD of all the measurements and the named ICC are used for all three (see Details).

cv_calc

Coefficient of variation (CV) calculation: "MSE" (default), the square root of the error mean square divided by the grand mean, or "SEM", the SEM (see se_type) divided by the grand mean. "residuals" is no longer supported and is replaced by "MSE" with a warning.

conf.level

The confidence level required. Default is 95%. The ICC intervals are two-sided (see Details).

other_ci

Logical value (TRUE or FALSE) indicating whether to calculate confidence intervals for the CV, SEM, SEP, and SEE.

type

The type of confidence intervals for the CV, SEM, SEP, and SEE: "chisq" (default; analytic intervals, see Details), or a bootstrap interval, "perc" (percentile), "norm" (normal, with bias correction) or "basic".

replicates

The number of bootstrap replicates. Default is 1999.

Details

Estimation. Both functions use the engine of icc_test(): reli_stats() fits a linear mixed model by REML (crossed random effects for subjects and items) and allows missing measurements; reli_aov() uses the analysis of variance and keeps only subjects measured on every item (incomplete subjects are dropped with a message). With complete data the two give the same results (unless a variance component is estimated at zero). With missing data, reli_stats() uses degrees of freedom based on the observed data and warns that its intervals are approximate.

ICCs. The six forms of Shrout and Fleiss (1979) are returned, with the F-based intervals of McGraw and Wong (1996). The intervals are two-sided conf.level intervals, as in those papers (and in the psych, irr, SPSS and Stata implementations). Before version 0.4.0 the intervals used conf.level in each tail, so each limit was a one-sided conf.level bound and the pair covered only 1 - 2 * (1 - conf.level); the old output at conf.level = c is reproduced by conf.level = 1 - 2 * (1 - c) (e.g. old 0.95 = new 0.90). For a one-sided lower bound on a single ICC, use icc_test(alternative = "greater").

SEM, SEE and SEP. With se_type = "MSE", SEM = MSE\sqrt{MS_E} (Weir, 2005, preferred), and SEE = SDICC(1−ICC)SD\sqrt{ICC(1 - ICC)} and SEP = SD1−ICC2SD\sqrt{1 - ICC^2} with ICC3. With another se_type, the named ICC is used for all three, and SEM = SD1−ICCSD\sqrt{1 - ICC}. Here SD is the SD of all the measurements, SStotal/(N−1)\sqrt{SS_{total}/(N - 1)}, as in Weir (2005). reli_err_test() instead uses the SD matched to the ICC's denominator, so its SEE and SEP differ slightly. A negative ICC is treated as zero.

CV. The CV is the SEM (or MSE\sqrt{MS_E}) divided by the grand mean, as a proportion (printed as a percentage).

Intervals for the CV, SEM, SEE and SEP (other_ci = TRUE). With type = "chisq", the SEM from MSEMS_E has the exact χ2\chi^2 interval; every other statistic has a Wald interval on the log scale with a delta-method standard error from the normal-theory variances of the mean squares (and of the grand mean, for the CV). Before version 0.4.0 these used χ2\chi^2 intervals with the error degrees of freedom (and Vangel's interval for the CV), which do not apply to these statistics. The bootstrap types resample subjects (as icc_test() does, including its hybrid step for the item mean square); before version 0.4.0 reli_stats() used a parametric bootstrap.

Value

Returns single list with the results of the reliability analysis.

  • icc: Table of ICC results

  • lmer: The fitted model: a linear mixed model from lme4 (reli_stats()), or a linear model of subjects and items (reli_aov())

  • anova: Analysis of Variance table (mean squares reconstructed from the variance components for reli_stats())

  • var_comp: Table of Variance Components

  • n.id: Number of subjects/participants

  • n.item: Number of items/time points

  • cv: Coefficient of Variation (a one-row data frame)

  • SEM: Standard Error of Measurement (a one-row data frame)

  • SEE: Standard Error of the Estimate (a one-row data frame)

  • SEP: Standard Error of Prediction (a one-row data frame)

  • call: the matched call, with the settings used

References

Weir, J. P. (2005). Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. Journal of Strength and Conditioning Research, 19(1), 231-240. doi:10.1519/15184.1

Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420-428. doi:10.1037/0033-2909.86.2.420

McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30-46. doi:10.1037/1082-989X.1.1.30

See Also

icc_test(), reli_err_test()

Examples

data('reps')
reli_stats(data = reps, wide = TRUE, col.names = c("x","y"))

# The recommended replacements (reps has missing measurements, so REML)
icc_test(data = reps, col.names = c("x", "y"), method = "reml")
reli_err_test(data = reps, col.names = c("x", "y"), method = "reml")

reps

Description

A fake data set of a agreement study where both measures have replicates.

The data set published in the original Bland & Altman paper on agreement.

Usage

reps

ba1986

Format

A data frame with 20 rows with 3 variables

id

Subject identifier

x

X measurement

y

Y measurement

A data frame with 17 rows with 5 variables

id

Subject identifier

wright1

PERF measurement #1 using Wright device

wright2

PERF measurement #2 using Wright device

mini1

PERF measurement #1 using Mini device

mini2

PERF measurement #2 using Mini device

References

Bland, J. M., & Altman, D. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet, 327(8476), 307-310.


Methods for simple_agree objects

Description

Methods defined for objects returned from the agree functions.

Usage

## S3 method for class 'simple_agree'
print(x, ...)

## S3 method for class 'simple_agree'
plot(
  x,
  type = 1,
  x_name = "x",
  y_name = "y",
  geom = c("geom_point", "geom_bin2d", "geom_density_2d", "geom_density_2d_filled",
    "stat_density_2d"),
  smooth_method = NULL,
  smooth_se = TRUE,
  ...
)

check(x, ...)

## S3 method for class 'simple_agree'
check(x, ...)

Arguments

x

object of class simple_agree as returned from a function starting with 'agree'

...

further arguments passed through, see description of return value for details. agree_test.

type

Type of plot to output. Default (1) is Bland-Altman plot while type=2 will produce a line-of-identity plot.

x_name

Name/label for x values (first measurement)

y_name

Name/label for y values (second measurement)

geom

String naming the type of geometry to display the data points. Default is "geom_point". Other options include: "geom_bin2d", "geom_density_2d", "geom_density_2d_filled", and "stat_density_2d".

smooth_method

Smoothing method (function) to use, accepts either NULL or a character vector, e.g. "lm", "glm", "gam", "loess" or a function. Default is NULL, which will not include a trend line.

smooth_se

Display confidence interval around smooth?

Value

print

Prints short summary of the Limits of Agreement

plot

Returns a plot of the limits of agreement (type = 1) or concordance plot (type = 2)

check

Returns 2 plots, p_norm and p_het, testing the assumptions of a Bland-Altman analysis. P-values for the normality and heteroskedasticity tests are provided as captions to the plot.


Methods for simple_eiv objects

Description

Methods defined for objects returned from error-in-variables models (e.g., dem_reg, pb_reg). The joint confidence region for the intercept and slope is tested with joint_test() and plotted with plot_joint().

Usage

## S3 method for class 'simple_eiv'
print(x, ...)

## S3 method for class 'simple_eiv'
formula(x, ...)

## S3 method for class 'simple_eiv'
model.frame(formula, data = NULL, na.action = na.pass, ...)

## S3 method for class 'simple_eiv'
summary(object, ...)

## S3 method for class 'simple_eiv'
confint(object, parm, level = NULL, ...)

## S3 method for class 'simple_eiv'
plot(
  x,
  x_name = NULL,
  y_name = NULL,
  interval = c("none", "confidence"),
  level = NULL,
  n_points = 100,
  data = NULL,
  ...
)

## S3 method for class 'simple_eiv'
check(x, data = NULL, ...)

## S3 method for class 'simple_eiv'
vcov(object, ...)

## S3 method for class 'simple_eiv'
coef(object, ...)

## S3 method for class 'simple_eiv'
fitted(object, type = c("y", "x", "both"), data = NULL, ...)

## S3 method for class 'simple_eiv'
residuals(object, type = c("optimized", "x", "y", "raw_y"), data = NULL, ...)

## S3 method for class 'simple_eiv'
predict(
  object,
  newdata = NULL,
  interval = c("none", "confidence"),
  level = NULL,
  se.fit = FALSE,
  data = NULL,
  ...
)

Arguments

...

further arguments passed through.

formula

A simple_eiv object (for model.frame method)

data

Optional data frame. Required if model was fitted with model = FALSE and newdata is NULL.

na.action

Function for handling NA values (default: na.pass)

object, x

object of class simple_eiv from dem_reg or pb_reg function.

parm

A specification of which parameters are to be given confidence intervals, either a vector of numbers or a vector of names. If missing, all parameters are considered.

level

Confidence level for intervals (default uses the model's conf.level).

x_name

Name for x-axis label (optional).

y_name

Name for y-axis label (optional).

interval

Type of interval calculation. Can be "none" (default), or "confidence"

n_points

Number of points to use for drawing smooth interval bands (default = 100).

type

Type of residuals to return. Options are "optimized" (default), "x", "y", or "raw_y".

newdata

An optional data frame containing values of X at which to predict. If omitted, the fitted values are returned.

se.fit

Logical. If TRUE, standard errors of predictions are returned. Note: For Passing-Bablok regression without bootstrap, standard errors are not available and this argument is ignored with a warning.

Value

print

Prints short summary of the EIV regression model.

summary

Prints detailed summary.

plot

Returns a plot of the regression line and data.

check

Returns plots of residuals.

predict

Predicts Y values for new X values.

fitted

Extracts fitted values.

residuals

Extracts residuals.

coef

Extracts model coefficients.

vcov

Extracts variance-covariance matrix.


Methods for simple_reli objects

Description

Methods defined for objects returned from reli_stats() and reli_aov().

Usage

## S3 method for class 'simple_reli'
print(x, ...)

## S3 method for class 'simple_reli'
plot(x, ...)

## S3 method for class 'simple_reli'
check(x, ...)

Arguments

x

object of class simple_reli as returned from reli_stats() or reli_aov().

...

further arguments passed through, see description of return value for details.

Value

print

Prints a short summary of the reliability statistics.

plot

Returns a plot of the data points used in the reliability analysis.

check

Returns diagnostic plots (normality and heteroskedasticity) of the model residuals.


Data

Description

A dataset from a study on the reliability of human body temperature at different times of day before and after exercise.

Usage

temps

recpre_long

Format

A data frame with 60 rows and 10 variables:

id

Subject identifier

trial_num

order in which the experimental trial was completed

trial_condition

Environmental condition and metabolic heat production

tod

Time of Day

trec_pre

Rectal temperature before the beginning of the trial

trec_post

Rectal temperature at the end of the trial

trec_delta

Change in rectal temperature

teso_pre

Esophageal temperature before the beginning of the trial

teso_post

Esophageal temperature at the end of the trial

teso_delta

Change in esophageal temperature

An object of class tbl_df (inherits from tbl, data.frame) with 30 rows and 6 columns.

Source

Ravanelli N, Jay O. The Change in Core Temperature and Sweating Response during Exercise Are Unaffected by Time of Day within the Wake Period. Med Sci Sports Exerc. 2020 Dec 1. doi: 10.1249/MSS.0000000000002575. Epub ahead of print. PMID: 33273272.


Methods for tolerance_delta objects

Description

Methods defined for objects returned from the tolerance_delta function(s).

Usage

## S3 method for class 'tolerance_delta'
print(x, digits = 4, ...)

## S3 method for class 'tolerance_delta'
plot(
  x,
  geom = c("geom_point", "geom_bin2d", "geom_density_2d", "geom_density_2d_filled",
    "stat_density_2d"),
  delta = NULL,
  ...
)

## S3 method for class 'tolerance_delta'
check(x, ...)

Arguments

x

object of class tolerance_delta as returned from a agreement_limit function.

digits

The number of digits to print.

...

further arguments passed through, see description of return value for details. tolerance_limit.

geom

String naming the type of geometry to display the data points. Default is "geom_point". Other options include: "geom_bin2d", "geom_density_2d", "geom_density_2d_filled", and "stat_density_2d".

delta

The maximal allowable difference.

Value

print

Prints short summary of the tolerance limits, with the SD of the differences (coefficient of variation when log-transformed). If the SD differs between rows of the limits (e.g., by condition), it is shown as a column instead. For clustered models, the between- and within-subject SDs are also printed.

plot

Returns a plot of the tolerance limits.

check

Returns plots testing the assumptions of the model. P-values for the normality and heteroskedasticity tests are provided as captions to the plot.


Tolerance Limits from an Agreement Study

Description

[Stable]

A function for calculating tolerance limits for the difference between two measurements (difference = x-y). This is a procedure that should produce results similar to the Bland-Altman limits of agreement. See vignettes for more details.

Usage

tolerance_limit(
  data,
  x,
  y,
  id = NULL,
  condition = NULL,
  time = NULL,
  pred_level = 0.95,
  tol_level = 0.95,
  conf_level = 0.95,
  tol_method = c("analytic", "boot_cal"),
  bound_type = c("joint", "iu"),
  prop_bias = FALSE,
  log_tf = FALSE,
  log_tf_display = c("ratio", "sympercent"),
  model = c("gls", "lme"),
  cor_type = c("sym", "car1", "ar1", "none"),
  correlation = NULL,
  weights = NULL,
  keep_model = TRUE,
  replicates = 999
)

Arguments

data

A data frame containing the variables.

x

Name of the column for the first measurement.

y

Name of the column for the second measurement.

id

Name of the column for the subject ID. With model = "lme", two column names can be given for nested random intercepts, outer level first (e.g., c("golfer", "club") for clubs within golfers); see "Model assumptions" in details.

condition

Name of the column indicating different conditions subjects were tested under. This can be left missing if there are no differing conditions to be tested. Supplying a condition adds a separate residual variance for each condition (nlme::varIdent); see "Model assumptions" in details.

time

Name of the column indicating the time points. Only necessary if the data is from time series or repeated measures collection.

pred_level

Prediction level for the prediction interval, which is also the content (the proportion of differences to be covered, beta) for the tolerance limits. Default is 95%.

tol_level

Confidence level (gamma) for the tolerance limits. Default is 95%. See bound_type for what this confidence refers to.

conf_level

Confidence level for the confidence interval of the bias (lower.CL/upper.CL). Default is 95%. This does not affect the prediction or tolerance limits. For a two one-sided tests (TOST) procedure on the bias at level alpha, use conf_level = 1 - 2 * alpha (e.g., 0.90).

tol_method

Method for calculating the tolerance limits. Options are "analytic" (default) for closed-form limits and "boot_cal" (experimental) for the closed-form limits calibrated by a parametric bootstrap. The analytic limits are recommended; see details. Both target the interval set by bound_type. The previous names, "approx" and "perc", are deprecated aliases for "analytic" and "boot_cal".

bound_type

Which claim the tolerance limits (lower.TL/upper.TL) support. "joint" (default) gives a beta-content, gamma-confidence tolerance interval: with confidence tol_level, at least pred_level of all differences lie within the limits. "iu" gives equal-tailed bounds: a one-sided tol_level confidence bound on each of the (1 - pred_level)/2 and (1 + pred_level)/2 percentiles of the differences. The "iu" bounds are intended for an intersection-union test of agreement against a maximal allowable difference and are not a joint tol_level interval. See details.

prop_bias

Whether to include a proportional bias term in the model. Determines whether proportional bias should be considered for the prediction/tolerance limits calculations. Note that a slope of the differences on the average can appear without any true proportional bias when the two methods have unequal measurement error variances; see "Model assumptions" in details.

log_tf

Calculate limits of agreement using log-transformed data.

log_tf_display

The type of presentation for log-transformed results. The differences between methods can be displayed as a "ratio" or "sympercent".

model

The type of model for the differences. "gls" (default) is a marginal generalized least squares model (nlme::gls()) with the correlation structure set by cor_type. "lme" is a linear mixed model (nlme::lme()) with a random intercept for each id, which requires id; with two id columns it has nested random intercepts (outer level, and inner level within outer). With compound symmetry and no variance function both give the same fit; "lme" differs when serial correlation (cor_type = "ar1" or "car1") or a variance function (condition or weights) is added, because it keeps a persistent subject effect and applies the variance function to the residuals only. See "Model assumptions" in details.

cor_type

The type of correlation structure. "sym" is for Compound Symmetry, "car1" is for continuous autocorrelation structure of order 1, or "ar1" for autocorrelation structure of order 1. The autoregressive options ("ar1" and "car1") have no persistent subject effect; see "Model assumptions" in details. With model = "lme", "sym" fits the random intercept alone, "ar1" and "car1" add serial correlation of the residuals within id on top of the random intercept, and "none" is not allowed.

correlation

an optional corStruct object describing the within-group correlation structure that overrides the default setting. See the documentation of corClasses for a description of the available corStruct classes. If a grouping variable is to be used, it must be specified in the form argument to the corStruct constructor. Defaults to NULL. With model = "lme", this is the correlation of the residuals within the innermost level of id (on top of the random intercepts) and must be grouped by it using the internal names: id for one level (e.g., nlme::corAR1(form = ~ time | id)) or id/id_2 for two (e.g., nlme::corAR1(form = ~ time | id/id_2)).

weights

an optional varFunc object or one-sided formula describing the within-group heteroskedasticity structure that overrides the default setting. If given as a formula, it is used as the argument to varFixed, corresponding to fixed variance weights. See the documentation on varClasses for a description of the available varFunc classes. Variance covariates must use the internal column names (avg, condition, x, y, time) or fitted(.). Currently varIdent, varFixed, varExp, and varPower are supported. If the variance depends on avg, the limits are reported at the minimum, median, and maximum of avg even when prop_bias = FALSE. With model = "lme", the variance function applies to the residuals only; the random-intercept variance is common to all observations.

keep_model

Logical indicator to retain the fitted model (gls or lme). Useful when working with large data and the model is very large.

replicates

The number of bootstrap replicates. Passed on to the boot function. Default is 999.

Details

The tolerance limits calculated in this function are based on the papers by Francq & Govaerts (2016), Francq, et al. (2019), and Francq, et al. (2020). The formulas, including how the degrees of freedom are determined for clustered data, are given in vignette("agreement_analysis", package = "SimplyAgree").

The output contains three kinds of interval, which support different claims:

  • Prediction interval (lower.PL/upper.PL): a beta-expectation tolerance interval. It is expected to contain a single future difference, from a new subject, with probability pred_level, averaged over repeated studies. It is not a confidence statement: any single prediction interval may cover less than pred_level of the differences.

  • Tolerance limits, bound_type = "joint" (lower.TL/upper.TL): a beta-content, gamma-confidence tolerance interval. With confidence tol_level, at least pred_level of all differences lie within the limits. If the limits lie within a maximal allowable difference of plus or minus delta, one can conclude (at level 1 - tol_level) that at least pred_level of the differences lie within plus or minus delta (Lin, 2000). The tails do not need to be split equally.

  • Tolerance limits, bound_type = "iu": a one-sided tol_level bound on the lower and on the upper (1 -/+ pred_level)/2 percentile of the differences, i.e., on the limits of agreement. If both bounds lie within plus or minus delta, one can reject, at level 1 - tol_level, that either limit of agreement lies outside plus or minus delta (intersection-union test). Both bounds hold together only about 1 - 2 * (1 - tol_level) of the time, so they should not be reported as a joint tol_level interval. These target the same quantities as the default confidence bounds from agreement_limit() (numerically close but not identical, because the models differ).

The "joint" and "iu" limits test different hypotheses (coverage versus each tail), so they do not nest: the "iu" bounds can be wider than the "joint" limits, and a skewed distribution can pass one test and fail the other.

With tol_method = "analytic":

  • "joint" uses the approximation of Howe (1969) as described by Francq et al. (2020), with the standard error of prediction (SEP) in place of the sample standard deviation. For independent data this matches Howe's approximation exactly.

  • "iu" uses the exact noncentral t bound for independent data, and otherwise the MOVER bound of Zou (2013).

  • Both use a one-sided upper confidence bound for the residual standard deviation (SD). With independent data this is the usual chi-square bound. With compound symmetry (cor_type = "sym"), the between- and within-subject variance components are combined with the MOVER, as in agreement_limit() with data_type = "nest". With other correlation structures, the bound uses an effective degrees of freedom (SD.df) from the approximate covariance of the variance parameters.

The analytic limits are exact for independent data (without a variance function). In simulations (95% target), they were close to nominal with compound symmetry (about 0.95, with 10 to 20 subjects) and with AR(1) correlation (about 0.95). With condition (a separate residual variance per condition) combined with compound symmetry, the variance components for each condition are bounded using that condition's cluster sizes and degrees of freedom; coverage was then about 0.96. Other variance functions (weights) use the cluster sizes of the whole data set and have not been checked by simulation.

With tol_method = "boot_cal", the analytic limits are calibrated by a parametric bootstrap (Loh, 1987; Beran, 1987). New data are simulated from the fitted model (Francq et al., 2019), the model is refit to each replicate, and the analytic limits are computed for each replicate over a range of nominal confidence levels. The level is chosen at which the replicates achieve the target confidence for the fitted model:

  • "joint": a proportion tol_level of the replicate intervals contain at least pred_level of the fitted distribution of the differences.

  • "iu": separately for each bound, a proportion tol_level of the replicate bounds lie beyond the fitted limit of agreement.

The limits for the observed data are the analytic limits at the calibrated levels, which are returned as lower.TL.level and upper.TL.level. tol_method = "boot_cal" is experimental. The calibration is judged against the fitted model, so it treats the estimated correlation as known. With few subjects this adds its own error: in simulations with 10 to 15 subjects (95% target), coverage ranged from about 0.93 (compound symmetry, with or without condition) to about 0.96 (AR(1) correlation), while the analytic limits were 0.95 to 0.965 in the same settings. The analytic limits are therefore recommended; the bootstrap may be useful as a check for models whose analytic limits have not been evaluated (e.g., other variance functions or prop_bias = TRUE).

For clustered data (id supplied), all limits refer to a new difference from a new subject: the SD includes both the between- and within-subject variance.

Model assumptions

With model = "gls" (default), the model is a marginal (generalized least squares) model fit with nlme::gls(). For the default setup (cor_type = "sym" without a variance function), compound symmetry with a non-negative correlation gives the same likelihood as a random-intercept model, and targets the marginal distribution of a single difference from a randomly chosen subject. With model = "lme", the model is a linear mixed model fit with nlme::lme() with a random intercept for each id; with compound symmetry and no variance function it gives the same fit and limits. With model = "lme", the degrees of freedom for the bias are the containment degrees of freedom (subjects - 1 for the models fit here), because the Satterthwaite approximation for lme models in emmeans can fail. The other options carry assumptions that should be checked:

  • Autoregressive correlation (cor_type = "ar1" or "car1"): with model = "gls", the correlation between two measurements from the same subject decays towards zero as they get further apart in time. A persistent subject-specific bias (e.g., a subject by method interaction) instead makes all measurements from that subject equally correlated. When such an effect exists, the autoregressive structures miss most of the long-range correlation, so the standard error of the bias is too small and the limits are too narrow; the bootstrap does not correct this, because it simulates from the same model. In one simulation (20 subjects with 19 measurements each, and a random subject effect), the 95% confidence interval for the bias covered the true value 74% of the time with AR(1), versus 96% with compound symmetry. With model = "lme", the serial correlation is added to the residuals on top of a random intercept, which keeps the persistent subject effect. In simulations with a random subject effect and AR(1) residuals (15 subjects, 8 measurements each; 95% target), coverage of the joint tolerance limits was about 0.89 with ⁠model = "gls", cor_type = "ar1"⁠ and about 0.95 with ⁠model = "lme", cor_type = "ar1"⁠. Use the autoregressive options with model = "gls" only when no persistent subject effect is expected. The fits can be compared with AIC() on the returned models (a REML comparison is valid because the fixed effects are the same).

  • Variance functions with compound symmetry (condition, or weights together with cor_type = "sym"): in gls, the correlation applies to the standardized residuals, so the covariance between two measurements from the same subject is rho * sigma_i * sigma_j. The between-subject variance therefore scales with the variance function (e.g., it is larger in a condition with a larger residual SD), rather than being common to all conditions. The marginal variance of each condition, which drives the limits, is not directly affected, but the standard error of each condition's bias and the reported variance components are. With model = "lme", the variance function applies to the residuals only, and the between-subject variance is common to all conditions. In simulations where that was true (12 subjects; 95% target), the joint tolerance limits had coverage of about 0.96 with model = "gls" and about 0.97 with model = "lme".

  • Clustering levels: model = "gls" has a single grouping factor (id). With model = "lme", id can name two nested levels (e.g., id = c("golfer", "club") for shots with several clubs per golfer), giving random intercepts for the outer level and for the inner level within the outer level. Setting id to the inner level alone (e.g., a golfer-by-club identifier) should be avoided: it drops the correlation across inner levels within the same outer level, which understates the uncertainty. In simulations (15 subjects with 2 to 4 settings each; 95% target), the joint tolerance limits had coverage of about 0.88 to 0.91 with the inner level as id, versus about 0.96 with nested random intercepts. Grouping by the outer level only (id = "golfer") also gave close to nominal coverage (about 0.96), because the inner-level variance is then absorbed into the residual; the nested model additionally reports the inner-level variance component (SD.nested). More than two levels are not supported.

  • Variance components: the SD of a single difference splits into between-subject (SD.between), inner-level (SD.nested, with two id columns), and within-subject (SD.within) components. With model = "gls" and compound symmetry these are sqrt(rho) * SD and sqrt(1 - rho) * SD; with model = "lme", SD.between is the random-intercept SD and SD.within the residual SD. They can be used, for example, for the point estimate of limits for the mean of m measurements per subject, bias +/- z * sqrt(SD.between^2 + SD.within^2 / m) (without residual correlation), or for the between-subject floor of agreement. They are point estimates only.

  • Proportional bias (prop_bias = TRUE): a non-zero slope of the differences on the average can appear without any true proportional bias. Whenever the two methods have unequal measurement error variances, cov(difference, average) = (var(x) - var(y)) / 2 is not zero (Bland & Altman, 1999), so the slope should not be read as proportional bias on its own. Errors-in-variables methods such as dem_reg() or pb_reg() are better suited to assessing proportional bias.

Value

Returns single tolerance_delta class object with the results of the agreement analysis with a prediction interval and tolerance limits.

  • limits: A data frame containing the prediction/tolerance limits. Columns include bias (estimated mean difference), SEM (standard error of the bias), SD (residual standard deviation of a single difference at that row, from the variance function if one is in the model), SEP (standard error of prediction, sqrt(SD^2 + SEM^2)), SD.df (degrees of freedom of the residual variance), SD.upper (one-sided tol_level upper confidence bound for SD), SD.between/SD.nested/SD.within (between-subject, nested (inner level), and within-subject components of SD, under compound symmetry or with model = "lme"; NA otherwise, and SD.nested is NA without nesting), lower.CL/upper.CL (confidence limits for the bias), lower.PL/upper.PL (prediction limits), lower.TL/upper.TL (tolerance limits), and, for tol_method = "boot_cal", lower.TL.level/upper.TL.level (the calibrated nominal levels at which the analytic limits were computed).

  • model: The fitted gls or lme model; NULL if keep_model set to FALSE. The data (with the internal column names x, y, delta, avg, id (the row number if not supplied), and, if supplied, condition and time), correlation structure, and variance function are stored with the model, so update() and nlme::getData() can be used on it directly.

  • call: The matched call.

References

Francq, B. G., & Govaerts, B. (2016). How to regress and predict in a Bland–Altman plot? Review and contribution based on tolerance intervals and correlated‐errors‐in‐variables models. Statistics in mMdicine, 35(14), 2328-2358.

Francq, B. G., Lin, D., & Hoyer, W. (2019). Confidence, prediction, and tolerance in linear mixed models. Statistics in Medicine, 38(30), 5603-5622.

Francq, B. G., Berger, M., & Boachie, C. (2020). To tolerate or to agree: A tutorial on tolerance intervals in method comparison studies with BivRegBLS R Package. Statistics in Medicine, 39(28), 4334-4349.

Howe, W. G. (1969). Two-sided tolerance limits for normal populations—some improvements. Journal of the American Statistical Association, 64(326), 610-620.

Lin, L. I. (2000). Total deviation index for measuring individual agreement with applications in laboratory performance and bioequivalence. Statistics in Medicine, 19(2), 255-270.

Loh, W. Y. (1987). Calibrating confidence coefficients. Journal of the American Statistical Association, 82(397), 155-162.

Beran, R. (1987). Prepivoting to reduce level error of confidence sets. Biometrika, 74(3), 457-468.

Zou, G. Y. (2013). Confidence interval estimation for the Bland–Altman limits of agreement with multiple observations per individual. Statistical Methods in Medical Research, 22(6), 630-642.

Examples

data('reps')

# Simple
tolerance_limit(x = "x", y ="y", data = reps)

# Nested
tolerance_limit(x = "x", y ="y", data = reps, id = "id")

# Nested, random intercept (mixed) model
tolerance_limit(x = "x", y ="y", data = reps, id = "id", model = "lme")

# Nested random intercepts (e.g., measurements within settings within subjects)

reps2 = reps
reps2$setting = rep(1:2, length.out = nrow(reps2))
tolerance_limit(x = "x", y = "y", data = reps2, id = c("id", "setting"),
                model = "lme")