| Title: | Flexible and Robust Agreement and Reliability Analyses |
|---|---|
| Description: | Reliability and agreement analyses often have limited software support. Therefore, this package was created to make agreement and reliability analyses easier for the average researcher. The functions within this package include simple tests of agreement, agreement analysis for nested and replicate data, and provide robust analyses of reliability. In addition, this package contains a set of functions to help when planning studies looking to assess measurement agreement. |
| Authors: | Aaron Caldwell [aut, cre] |
| Maintainer: | Aaron Caldwell <[email protected]> |
| License: | GPL (>= 3) |
| Version: | 0.4.0.9000 |
| Built: | 2026-11-11 13:34:55 UTC |
| Source: | https://github.com/arcaldwell49/simplyagree |
agree_assurance( conf.level = 0.95, assurance = 0.9, omega = NULL, pstar = 0.95, sigma = 1, n = NULL, m = 1, icc = 0 )agree_assurance( conf.level = 0.95, assurance = 0.9, omega = NULL, pstar = 0.95, sigma = 1, n = NULL, m = 1, icc = 0 )
conf.level |
confidence level for the range of agreement (1 - alpha). The confidence level of the confidence interval of the range of agreement (tolerance interval). Default is 0.95. |
assurance |
target lower bound of the assurance probability (1 - gamma). The assurance probability is the probability that the study half-width will be less than omega. Common values are 0.80, 0.90, or 0.95. |
omega |
upper bound of assurance half-width. The sample size guarantees (assures) that 100(1 - gamma)% of interval half-widths will be less than this value. Can be specified in standard deviation units. |
pstar |
central proportion of the data distribution covered (P*). It is the proportion of observations that fall between the limits. For example, a value of 0.95 indicates that 95% of the variable's values fall between the limits. Must be between 0 and 1. Common values are 0.90 or 0.95. |
sigma |
population standard deviation of the paired differences. If the
true value is unknown, omega can be specified in standard deviation units
by setting sigma = 1. With clustered data ( |
n |
sample size (optional): the number of pairs, or the number of
subjects when |
m |
number of paired measurements per subject (default 1, independent pairs). See the "Clustered data" section. |
icc |
intraclass correlation of the paired differences (between-subject
share of the variance of the differences), in [0, 1). Ignored when |
Calculate the sample size necessary for a confidence interval of the Bland-Altman range of agreement when the underlying data distribution is normal. This function uses the assurance probability criterion to determine the optimum sample size, based on the exact confidence interval method of Jan and Shieh (2018), which has been shown to be superior to approximate methods.
This function implements the exact method for determining sample size based on assurance probability for Bland-Altman limits of agreement, as described in Jan and Shieh (2018). The assurance probability criterion determines an N that guarantees with specified probability (1 - gamma) that the confidence interval half-width will be no more than a boundary value omega.
Suppose a study involves paired differences (X - Y) whose distribution is approximately N(mu, sigma^2). The range of agreement is defined as a confidence interval of the central portion of these differences, specifically the area between the 100(1-p)th and 100p-th percentiles, where p* = 2p - 1.
The exact two-sided, 100(1 - alpha)% confidence interval for the range of agreement is defined as:
Pr(theta_(1-p) < theta_hat_(1-p) and theta_hat_p < theta_p) = 1 - alpha
The equal-tailed tolerance interval recommended by Jan and Shieh (2018) is:
(X_bar - d, X_bar + d)
where d = g * S, g is the Odeh-Owen tolerance factor (tabulated as g” in Odeh and Owen (1980)), and S is the sample standard deviation.
The sample size N is selected to satisfy: Pr(H <= omega) >= 1 - gamma
This leads to the expression:
psi(eta) >= 1 - gamma
where psi() is the CDF of a chi-square distribution with N-1 degrees of freedom and eta = (N-1) \* (omega/(g\*sigma))^2.
The method uses equal-tailed tolerance intervals based on the noncentral t-distribution to construct exact confidence intervals for the range of agreement. The tolerance factor g is calculated such that the interval maintains the specified confidence level under normality.
Jan and Shieh (2018) demonstrated through extensive simulations that this exact method should be adopted rather than the classical Bland-Altman approximate method.
Each subject produces two measurements (one for each method being compared). The sample size n returned is the number of subject pairs needed. The actual assurance probability may differ slightly from the target due to the discrete nature of sample size.
For dropout considerations, inflate the sample size using: N' = N / (1 - dropout_rate), always rounding up.
An object of class "power.htest" containing the following components:
n: The required sample size (number of pairs, or of subjects when m > 1)
conf.level: The confidence level (1 - alpha)
assurance: The target assurance probability (1 - gamma)
actual.assurance: The actual assurance probability achieved (may differ
slightly from target due to discrete nature of n)
omega: The upper bound of assurance half-width
pstar: The central proportion covered (P*)
sigma: The population standard deviation
g: The Odeh-Owen factor (g”) used to construct the tolerance
interval, tabulated in Odeh and Owen (1980)
zp: The normal quantile for the central proportion
m, icc: The clustering inputs
n.obs: The total number of pairs, n * m
method: Description of the method used
note: Additional notes about the analysis
The paired differences are normally distributed
The variance is constant across the range of measurement
Pairs are independent, or (with m > 1) follow the balanced
compound symmetry model in the "Clustered data" section
When each of n subjects contributes m paired measurements (repeated
measures, or several pairs per subject), the pairs are not independent and
treating n * m pairs as independent overstates precision. Setting m and
icc accounts for this under a balanced one-way random-effects
(compound symmetry) model for the differences:
The mean difference has variance sigma^2 \* DEFF / (n \* m), with design effect DEFF = 1 + (m - 1) \* icc, so the effective sample size for the mean is n \* m / DEFF.
The total variance is estimated as S^2 = MSB / m + (m - 1) / m \* MSW, whose distribution is a weighted sum of two independent chi-square variables with n - 1 and n(m - 1) degrees of freedom. This distribution is integrated numerically in place of the single chi-square of the independent case. It is more skewed than a Satterthwaite approximation suggests, which matters when icc is large.
Under these assumptions the calculation is exact (it agrees with simulation
to Monte Carlo error), and m = 1 gives exactly the published method. The
simple design effect 1 + (m - 1) \* icc applied to the whole sample size
overstates the penalty, because the SD has a much smaller design effect,
1 + (m - 1) \* icc^2. Unbalanced designs, other correlation structures, or
several levels of nesting are not covered; use simulation for those.
icc is the ICC of the differences, which can be estimated from pilot data
with an intercept-only mixed model on the differences (e.g.
lme4::lmer(diff ~ 1 + (1 | id))), and sigma is then the square root of
the sum of the two variance components.
Jan, S.L. and Shieh, G. (2018). The Bland-Altman range of agreement: Exact interval procedure and sample size determination. Computers in Biology and Medicine, 100, 247-252. doi:10.1016/j.compbiomed.2018.06.020
Bland, J.M. and Altman, D.G. (2007). Agreement between methods of measurement with multiple observations per individual. Journal of Biopharmaceutical Statistics, 17(4), 571-582. doi:10.1080/10543400701329422
Mee, R.W. and Owen, D.B. (1983). Improved factors for one-sided tolerance limits for balanced one-way ANOVA random model. Journal of the American Statistical Association, 78(384), 901-905. doi:10.1080/01621459.1983.10477038
Odeh, R.E. and Owen, D.B. (1980). Tables for Normal Tolerance Limits, Sampling Plans, and Screening. Marcel Dekker, Inc., New York.
agree_expected_half() for sample size determination using expected
half-width criterion, power_agreement_exact() for power analysis
of agreement tests.
# Example: Planning a method comparison study # Researchers want 95% confidence, 90% assurance that half-width # will be within 2.5 SD units, covering central 95% of differences agree_assurance( conf.level = 0.95, assurance = 0.90, omega = 2.5, pstar = 0.95, sigma = 1 ) # Clustered data: 4 pairs per subject, ICC of the differences 0.3 agree_assurance( assurance = 0.90, omega = 2.5, pstar = 0.95, m = 4, icc = 0.3 )# Example: Planning a method comparison study # Researchers want 95% confidence, 90% assurance that half-width # will be within 2.5 SD units, covering central 95% of differences agree_assurance( conf.level = 0.95, assurance = 0.90, omega = 2.5, pstar = 0.95, sigma = 1 ) # Clustered data: 4 pairs per subject, ICC of the differences 0.3 agree_assurance( assurance = 0.90, omega = 2.5, pstar = 0.95, m = 4, icc = 0.3 )
agree_coef() has been superseded by cac_test(), which returns one
coefficient as an htest with a hypothesis test, summary() and plot()
methods. agree_coef() still returns a table of all four coefficients.
agree_coef produces inter-rater reliability or "agreement coefficients" as described by Gwet: percent agreement, Gwet's AC1 (AC2 when weighted), Fleiss' kappa and Krippendorff's alpha.
agree_coef( wide = TRUE, col.names = NULL, measure, item, id, data, weighted = FALSE, conf.level = 0.95, weights = NULL, categories = NULL )agree_coef( wide = TRUE, col.names = NULL, measure, item, id, data, weighted = FALSE, conf.level = 0.95, weights = NULL, categories = NULL )
wide |
Logical value (TRUE or FALSE) indicating if data is in a "wide" format. Default is TRUE. |
col.names |
If wide is equal to TRUE then col.names is a list of the column names containing the measurements for reliability analysis. |
measure |
Name of column containing the measurement of interest. |
item |
Name of column containing the items. If this is an inter-rater reliability study then this would indicate the rater (e.g., rater1, rater2, rater3, etc). |
id |
Column with subject identifier. |
data |
Data frame with all data. |
weighted |
Logical value (TRUE or FALSE) indicating whether to weight the responses. If TRUE (default is FALSE) then quadratic weights are utilized. This option should be set to TRUE for ordinal or continuous responses. Ignored when |
conf.level |
the confidence level required. Default is 95%. |
weights |
Optional weighting scheme, as in |
categories |
All possible rating categories, as in |
The estimates, standard errors and intervals are those of cac_test();
see there for details. Confidence limits are clipped to the possible range
of each coefficient. When all ratings are in one category the
chance-corrected coefficients are undefined and returned as NA.
A data frame with one row per coefficient and the columns est
(estimate), se (standard error), lower.ci and upper.ci.
Gwet, K.L. (2014, ISBN:978-0970806284). "Handbook of Inter-Rater Reliability", 4th Edition. Advanced Analytics, LLC.
Gwet, K. L. (2008). "Computing inter-rater reliability and its variance in the presence of high agreement," British Journal of Mathematical and Statistical Psychology, 61, 29-48.
data('reps') agree_coef(data = reps, wide = TRUE, col.names = c("x","y"), weighted = TRUE)data('reps') agree_coef(data = reps, wide = TRUE, col.names = c("x","y"), weighted = TRUE)
agree_expected_half( conf.level = 0.95, delta = NULL, pstar = 0.95, sigma = 1, n = NULL, m = 1, icc = 0 )agree_expected_half( conf.level = 0.95, delta = NULL, pstar = 0.95, sigma = 1, n = NULL, m = 1, icc = 0 )
conf.level |
confidence level for the range of agreement (1 - alpha). The confidence level of the confidence interval of the range of agreement (tolerance interval). Default is 0.95. |
delta |
target upper bound of expected half-width (delta). The sample size guarantees that the expected half-width of the confidence interval will be no more than this value. Can be specified in standard deviation units. |
pstar |
central proportion of the data distribution covered (P*). It is the proportion of observations that fall between the limits. For example, a value of 0.95 indicates that 95% of the variable's values fall between the limits. Must be between 0 and 1. Common values are 0.90 or 0.95. |
sigma |
population standard deviation of the paired differences. If the
true value is unknown, delta can be specified in standard deviation units
by setting sigma = 1. With clustered data ( |
n |
sample size (optional): the number of pairs, or the number of
subjects when |
m |
number of paired measurements per subject (default 1, independent pairs). See the "Clustered data" section. |
icc |
intraclass correlation of the paired differences (between-subject
share of the variance of the differences), in [0, 1). Ignored when |
Calculate the sample size necessary for a confidence interval of the Bland-Altman range of agreement when the underlying data distribution is normal. This function uses the expected half-width criterion to determine the optimum sample size, based on the exact confidence interval method of Jan and Shieh (2018), which has been shown to be superior to approximate methods.
This function implements the exact method for determining sample size based on expected half-width for Bland-Altman limits of agreement, as described in Jan and Shieh (2018). The expected half-width criterion determines an N that guarantees that the expected half-width of the confidence interval is less than a boundary value delta.
Suppose a study involves paired differences (X - Y) whose distribution is approximately N(mu, sigma^2). The range of agreement is defined as a confidence interval of the central portion of these differences, specifically the area between the 100(1-p)th and 100p-th percentiles, where p* = 2p - 1.
The exact two-sided, 100(1 - alpha)% confidence interval for the range of agreement is defined as:
Pr(theta_(1-p) < theta_hat_(1-p) and theta_hat_p < theta_p) = 1 - alpha
The equal-tailed tolerance interval recommended by Jan and Shieh (2018) is:
(X_bar - d, X_bar + d)
where d = g * S, g is the Odeh-Owen tolerance factor (tabulated as g” in Odeh and Owen (1980) and Hahn and Meeker (1991)), and S is the sample standard deviation.
The sample size N is selected to satisfy: E(H) <= delta
where H is the half-width of the confidence interval. This leads to the expression:
g(P*, 1-alpha, N-1) / c <= delta / sigma
where c is a bias correction factor:
c = (Gamma((N-1)/2) * sqrt((N-1)/2)) / Gamma(N/2)
The expected half-width is E(H) = (g/c) * sigma. This accounts for the fact that the sample standard deviation S is a biased estimator of sigma, requiring correction when computing expected values.
The method uses equal-tailed tolerance intervals based on the noncentral t-distribution to construct exact confidence intervals for the range of agreement. The tolerance factor g is calculated such that the interval maintains the specified confidence level under normality.
This function uses the expected half-width criterion, which ensures that
E(H) <= delta. An alternative approach is the assurance probability criterion
(see agree_assurance()), which ensures that Pr(H <= omega) >= 1 - gamma.
The expected half-width criterion:
Controls the average half-width across repeated sampling
Generally requires smaller sample sizes than assurance probability
Is appropriate when the average performance is of primary interest
The assurance probability criterion:
Provides a probability guarantee about the half-width
Generally requires larger sample sizes
Is appropriate when a stronger guarantee is needed for planning purposes
Each subject produces two measurements (one for each method being compared). The sample size n returned is the number of subject pairs needed. The actual expected half-width may differ slightly from the target due to the discrete nature of sample size.
For dropout considerations, inflate the sample size using: N' = N / (1 - dropout_rate), always rounding up.
An object of class "power.htest" containing the following components:
n: The required sample size (number of pairs, or of subjects when m > 1)
conf.level: The confidence level (1 - alpha)
target.delta: The target upper bound of expected half-width
actual.delta: The expected half-width achieved (may be smaller than
the target due to discrete nature of n)
pstar: The central proportion covered (P*)
sigma: The population standard deviation
g: The Odeh-Owen factor (g”) used to construct the tolerance
interval, tabulated in Odeh and Owen (1980)
c: The bias correction factor, 1 / E(S / sigma), used in the expected
half-width calculation
zp: The normal quantile for the central proportion
m, icc: The clustering inputs
n.obs: The total number of pairs, n * m
method: Description of the method used
note: Additional notes about the analysis
When each of n subjects contributes m paired measurements (repeated
measures, or several pairs per subject), the pairs are not independent and
treating n * m pairs as independent overstates precision. Setting m and
icc accounts for this under a balanced one-way random-effects
(compound symmetry) model for the differences:
The mean difference has variance sigma^2 \* DEFF / (n \* m), with design effect DEFF = 1 + (m - 1) \* icc, so the effective sample size for the mean is n \* m / DEFF.
The total variance is estimated as S^2 = MSB / m + (m - 1) / m \* MSW, whose distribution is a weighted sum of two independent chi-square variables with n - 1 and n(m - 1) degrees of freedom. This distribution is integrated numerically in place of the single chi-square of the independent case. It is more skewed than a Satterthwaite approximation suggests, which matters when icc is large.
Under these assumptions the calculation is exact (it agrees with simulation
to Monte Carlo error), and m = 1 gives exactly the published method. The
simple design effect 1 + (m - 1) \* icc applied to the whole sample size
overstates the penalty, because the SD has a much smaller design effect,
1 + (m - 1) \* icc^2. Unbalanced designs, other correlation structures, or
several levels of nesting are not covered; use simulation for those.
icc is the ICC of the differences, which can be estimated from pilot data
with an intercept-only mixed model on the differences (e.g.
lme4::lmer(diff ~ 1 + (1 | id))), and sigma is then the square root of
the sum of the two variance components.
The paired differences are normally distributed
The variance is constant across the range of measurement
Pairs are independent, or (with m > 1) follow the balanced
compound symmetry model in the "Clustered data" section
Jan, S.L. and Shieh, G. (2018). The Bland-Altman range of agreement: Exact interval procedure and sample size determination. Computers in Biology and Medicine, 100, 247-252. doi:10.1016/j.compbiomed.2018.06.020
Bland, J.M. and Altman, D.G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet, 327(8476), 307-310.
Hahn, G.J. and Meeker, W.Q. (1991). Statistical Intervals: A Guide for Practitioners. John Wiley & Sons, New York.
Bland, J.M. and Altman, D.G. (2007). Agreement between methods of measurement with multiple observations per individual. Journal of Biopharmaceutical Statistics, 17(4), 571-582. doi:10.1080/10543400701329422
Mee, R.W. and Owen, D.B. (1983). Improved factors for one-sided tolerance limits for balanced one-way ANOVA random model. Journal of the American Statistical Association, 78(384), 901-905. doi:10.1080/01621459.1983.10477038
Odeh, R.E. and Owen, D.B. (1980). Tables for Normal Tolerance Limits, Sampling Plans, and Screening. Marcel Dekker, Inc., New York.
agree_assurance() for sample size determination using assurance
probability criterion, power_agreement_exact() for power analysis
of agreement tests.
# Example 1: Reproduce Jan and Shieh (2018), page 251 # Expected half-width criterion with P* = 0.95 agree_expected_half( conf.level = 0.95, delta = 2.25 * 19.61, pstar = 0.95, sigma = 19.61 ) # Expected result: n = 155 # Example 2: Planning a method comparison study # Researchers want 95% confidence with expected half-width # no more than 2.4 SD units, covering central 90% of differences agree_expected_half( conf.level = 0.95, delta = 2.4, pstar = 0.90, sigma = 1 ) # Example 3: Clustered data, 3 pairs per subject, ICC of the differences 0.2 # (n is then the number of subjects) agree_expected_half( delta = 2.4, pstar = 0.90, m = 3, icc = 0.2 )# Example 1: Reproduce Jan and Shieh (2018), page 251 # Expected half-width criterion with P* = 0.95 agree_expected_half( conf.level = 0.95, delta = 2.25 * 19.61, pstar = 0.95, sigma = 19.61 ) # Expected result: n = 155 # Example 2: Planning a method comparison study # Researchers want 95% confidence with expected half-width # no more than 2.4 SD units, covering central 90% of differences agree_expected_half( conf.level = 0.95, delta = 2.4, pstar = 0.90, sigma = 1 ) # Example 3: Clustered data, 3 pairs per subject, ICC of the differences 0.2 # (n is then the number of subjects) agree_expected_half( delta = 2.4, pstar = 0.90, m = 3, icc = 0.2 )
Development on agree_nest() is complete, and for new code we recommend
switching to agreement_limit(), which is easier to use, has more features,
and still under active development.
agree_nest produces an absolute agreement analysis for data where there is multiple observations per subject but the mean varies within subjects as described by Zou (2013). Output mirrors that of agree_test but the CCC is Lin's estimator over all complete pairs with a delete-one-subject jackknife standard error; see ccc_test().
agree_nest( x, y, id, data, delta, agree.level = 0.95, conf.level = 0.95, TOST = TRUE, prop_bias = FALSE, ccc = TRUE )agree_nest( x, y, id, data, delta, agree.level = 0.95, conf.level = 0.95, TOST = TRUE, prop_bias = FALSE, ccc = TRUE )
x |
Name of column with first measurement |
y |
Name of other column with the other measurement to compare to the first. |
id |
Column with subject identifier |
data |
Data frame with all data |
delta |
The threshold below which methods agree/can be considered equivalent, can be in any units. Equivalence Bound for Agreement. |
agree.level |
the agreement level required. Default is 95%. The proportion of data that should lie between the thresholds, for 95% limits of agreement this should be 0.95. |
conf.level |
the confidence level required. Default is 95%. |
TOST |
Logical indicator (TRUE/FALSE) of whether to use two one-tailed tests for the limits of agreement. Default is TRUE. |
prop_bias |
Logical indicator (TRUE/FALSE) of whether proportional bias should be considered for the limits of agreement calculations. |
ccc |
Calculate concordance correlation coefficient. |
Returns single simple_agree class object with the results of the agreement analysis.
loa: A data frame of the limits of agreement including the average difference between the two sets of measurements, the standard deviation of the difference between the two sets of measurements and the lower and upper confidence limits of the difference between the two sets of measurements.
h0_test: Decision from hypothesis test.
ccc.xy: Lin's concordance correlation coefficient over all complete pairs and confidence intervals, as in ccc_test(data_type = "nest").
call: the matched call.
var_comp: Table of Variance Components.
class: The type of simple_agree analysis.
Zou, G. Y. (2013). Confidence interval estimation for the Bland–Altman limits of agreement with multiple observations per individual. Statistical methods in medical research, 22(6), 630-642.
Lin, L. I. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics, 45(1), 255-268.
data('reps') agree_nest(x = "x", y = "y", id = "id", data = reps, delta = 2)data('reps') agree_nest(x = "x", y = "y", id = "id", data = reps, delta = 2)
agree_np A non-parametric approach to limits of agreement.
The hypothesis test is based on binomial proportions within the maximal allowable differences, and the limits are calculated with quantile regression.
agree_np( x, y, id = NULL, data, delta = NULL, prop_bias = FALSE, TOST = TRUE, agree.level = 0.95, conf.level = 0.95 )agree_np( x, y, id = NULL, data, delta = NULL, prop_bias = FALSE, TOST = TRUE, agree.level = 0.95, conf.level = 0.95 )
x |
Name of column with first measurement. |
y |
Name of other column with the other measurement to compare to the first. |
id |
Column with subject identifier with samples are taken in replicates. |
data |
Data frame with all data. |
delta |
The threshold below which methods agree/can be considered equivalent and this argument is required. Equivalence Bound for Agreement or Maximal Allowable Difference. |
prop_bias |
Logical indicator (TRUE/FALSE) of whether proportional bias should be considered for the limits of agreement calculations. |
TOST |
Logical indicator (TRUE/FALSE) of whether to use two one-tailed tests for the limits of agreement. Default is TRUE. |
agree.level |
the agreement level required. Default is 95%. The proportion of data that should lie between the thresholds, for 95% limits of agreement this should be 0.95. |
conf.level |
the confidence level required. Default is 95%. |
Returns simple_agree object with the results of the agreement analysis.
loa: A data frame of the limits of agreement.
agree: A data frame of the binomial proportion of results in agreement.
h0_test: Decision from hypothesis test.
qr_mod: The quantile regression model.
call: The matched call
Bland, J. M., & Altman, D. G. (1999). Measuring agreement in method comparison studies. In Statistical Methods in Medical Research (Vol. 8, Issue 2, pp. 135–160). SAGE Publications. doi:10.1177/096228029900800204
data('reps') agree_np(x = "x", y = "y", id = "id", data = reps, delta = 2)data('reps') agree_np(x = "x", y = "y", id = "id", data = reps, delta = 2)
Development on agree_reps() is complete, and for new code we recommend
switching to agreement_limit(), which is easier to use, has more features,
and still under active development.
agree_nest produces an absolute agreement analysis for data where there is multiple observations per subject but the mean does not vary within subjects as described by Zou (2013). Output mirrors that of agree_test but the CCC is estimated from variance components (Carrasco & Jover, 2003) with a delete-one-subject jackknife standard error; see ccc_test().
agree_reps( x, y, id, data, delta, agree.level = 0.95, conf.level = 0.95, prop_bias = FALSE, TOST = TRUE, ccc = TRUE )agree_reps( x, y, id, data, delta, agree.level = 0.95, conf.level = 0.95, prop_bias = FALSE, TOST = TRUE, ccc = TRUE )
x |
Name of column with first measurement |
y |
Name of other column with the other measurement to compare to the first. |
id |
Column with subject identifier |
data |
Data frame with all data |
delta |
The threshold below which methods agree/can be considered equivalent, can be in any units. Equivalence Bound for Agreement. |
agree.level |
the agreement level required. Default is 95%. The proportion of data that should lie between the thresholds, for 95% limits of agreement this should be 0.95. |
conf.level |
the confidence level required. Default is 95%. |
prop_bias |
Logical indicator (TRUE/FALSE) of whether proportional bias should be considered for the limits of agreement calculations. |
TOST |
Logical indicator (TRUE/FALSE) of whether to use two one-tailed tests for the limits of agreement. Default is TRUE. |
ccc |
Calculate concordance correlation coefficient. |
Returns single list with the results of the agreement analysis.
loa: a data frame of the limits of agreement including the average difference between the two sets of measurements, the standard deviation of the difference between the two sets of measurements and the lower and upper confidence limits of the difference between the two sets of measurements.
h0_test: Decision from hypothesis test.
ccc.xy: The concordance correlation coefficient (variance components) and confidence intervals, as in ccc_test(data_type = "reps").
call: The matched call.
var_comp: Table of Variance Components.
class: The type of simple_agree analysis.
Zou, G. Y. (2013). Confidence interval estimation for the Bland–Altman limits of agreement with multiple observations per individual. Statistical methods in medical research, 22(6), 630-642.
Carrasco, JL & Jover, L. (2003). Estimating the generalized concordance correlation coefficient through variance components. Biometrics, 59(4), 849-858.
Carrasco, JL; Phillips, BR; Puig-Martinez, J; King, TS; Chinchilli, VM. (2013). Estimation of the concordance correlation coefficient for repeated measures using SAS and R. Computer Methods and Programs in Biomedicine, 109, 293-304.
data('reps') agree_reps(x = "x", y = "y", id = "id", data = reps, delta = 2)data('reps') agree_reps(x = "x", y = "y", id = "id", data = reps, delta = 2)
Development on agree_test() is complete, and for new code we recommend
switching to agreement_limit(), which is easier to use, has more features,
and still under active development.
The agree_test function calculates a variety of agreement statistics. The hypothesis test of agreement is calculated by the method described by Shieh (2019). Bland-Altman limits of agreement, and confidence intervals, are also provided (Bland & Altman 1999; Bland & Altman 1986). In addition, the concordance correlation coefficient (CCC; Lin 1989) is additional part of the output.
agree_test( x, y, delta, conf.level = 0.95, agree.level = 0.95, TOST = TRUE, prop_bias = FALSE )agree_test( x, y, delta, conf.level = 0.95, agree.level = 0.95, TOST = TRUE, prop_bias = FALSE )
x |
Vector with first measurement |
y |
Vector with second measurement |
delta |
The threshold below which methods agree/can be considered equivalent, can be in any units. Often referred to as the "Equivalence Bound for Agreement" or "Maximal Allowable Difference". |
conf.level |
the confidence level required. Default is 95%. |
agree.level |
the agreement level required. Default is 95%. The proportion of data that should lie between the thresholds, for 95% limits of agreement this should be 0.95. |
TOST |
Logical indicator (TRUE/FALSE) of whether to use two one-tailed tests for the limits of agreement. Default is TRUE. |
prop_bias |
Logical indicator (TRUE/FALSE) of whether proportional bias should be considered for the limits of agreement calculations. |
Returns single list with the results of the agreement analysis.
shieh_test: The TOST hypothesis test as described by Shieh.
ccc.xy: Lin's concordance correlation coefficient and confidence intervals.
s.shift: Scale shift from x to y.
l.shift: Location shift from x to y.
bias: a bias correction factor that measures how far the best-fit line deviates from a line at 45 degrees. No deviation from the 45 degree line occurs when bias = 1. See Lin 1989, page 258.
loa: Data frame containing the limits of agreement calculations
h0_test: Decision from hypothesis test.
call: the matched call
Shieh (2019). Assessing Agreement Between Two Methods of Quantitative Measurements: Exact Test Procedure and Sample Size Calculation, Statistics in Biopharmaceutical Research, doi:10.1080/19466315.2019.1677495
Bland, J. M., & Altman, D. G. (1999). Measuring agreement in method comparison studies. Statistical methods in medical research, 8(2), 135-160.
Bland, J. M., & Altman, D. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. The lancet, 327(8476), 307-310.
Lawrence, I., & Lin, K. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics, 255-268.
data('reps') agree_test(x=reps$x, y=reps$y, delta = 2)data('reps') agree_test(x=reps$x, y=reps$y, delta = 2)
A function for calculating for Bland-Altman limits of agreement based on the difference between two measurements (difference = x-y).
Please note that the package developer recommends reporting/using tolerance limits (see "tolerance_limit" function).
agreement_limit( x, y, id = NULL, data, data_type = c("simple", "nest", "reps"), loa_calc = c("mover", "blandaltman"), bound_type = c("iu", "joint"), agree.level = 0.95, alpha = 0.05, prop_bias = FALSE, log_tf = FALSE, log_tf_display = c("ratio", "sympercent"), lmer_df = c("satterthwaite", "asymptotic"), lmer_limit = 3000 )agreement_limit( x, y, id = NULL, data, data_type = c("simple", "nest", "reps"), loa_calc = c("mover", "blandaltman"), bound_type = c("iu", "joint"), agree.level = 0.95, alpha = 0.05, prop_bias = FALSE, log_tf = FALSE, log_tf_display = c("ratio", "sympercent"), lmer_df = c("satterthwaite", "asymptotic"), lmer_limit = 3000 )
x |
Name of column with first measurement |
y |
Name of other column with the other measurement to compare to the first. |
id |
Column with subject identifier. Default is "id" if no entry is provided. |
data |
Data frame with all data. |
data_type |
The type of data structure. Options include "simple" (all independent data points), "nest" (nested data) and "reps" (replicated data points). |
loa_calc |
The method by which the limits of agreement confidence intervals are calculated. Options are "mover" (Methods of Recovering Variances method) or "blandlatman" (Bland-Altman method). |
bound_type |
Which claim the confidence bounds of the limits of agreement ( |
agree.level |
the agreement level required. Default is 95%. The proportion of data that should lie between the thresholds, for 95% limits of agreement this should be 0.95. |
alpha |
The alpha-level for confidence levels. |
prop_bias |
Logical indicator (TRUE/FALSE) of whether proportional bias should be considered for the limits of agreement calculations. Note that a non-zero slope of the differences on the average can appear without any true proportional bias whenever the two methods have unequal measurement error variances, because cov(difference, average) = (var(x) - var(y)) / 2 (Bland & Altman, 1999); errors-in-variables methods such as |
log_tf |
Calculate limits of agreement using log-transformed data. |
log_tf_display |
The type of presentation for log-transformed results. The differences between methods can be displayed as a "ratio" or "sympercent". |
lmer_df |
Degrees of freedom method, only matters for if data_type is "nest". Default is "satterthwaite". The "asymptotic" method is faster but more liberal. |
lmer_limit |
Sample size limit for degrees of freedom method. If number of observations exceeds this limit, then the "asymptotic" method is utilized. |
The limits of agreement (LoA) are calculated in this function are based on the method originally detailed by Bland & Atlman (1986 & 1999).
The loa_calc allow users to specify the calculative method for the LoA which can be based on Bland-Altman (1999) (loa_calc = "blandaltman"),
or by the more accurate MOVER method of Zou (2013) and Donner & Zou (2012) (loa_calc = "mover").
The confidence bounds on the limits of agreement answer one of two questions, set by bound_type:
"iu" (default): "Can I conclude agreement within plus or minus delta?" If lower_loa_ci > -delta and upper_loa_ci < delta, reject at level alpha that either limit of agreement lies outside plus or minus delta. conf.level (1 - alpha) applies to each side separately; both bounds hold together only about 1 - 2 * alpha of the time.
"joint": "Where are the limits of agreement?" With at least 1 - alpha confidence, both limits of agreement lie between lower_loa_ci and upper_loa_ci. Used as a test against plus or minus delta, the error rate is at most alpha/2 (conservative).
Because a one-sided bound on the (1 - agree.level)/2 percentile is a one-sided tolerance bound, these bounds are equal-tailed tolerance bounds; they target the same quantities as tolerance_limit(bound_type = "iu").
The bias confidence interval (lower.CL/upper.CL) is always a two-sided 1 - alpha interval.
Returns single loa class object with the results of the agreement analysis.
loa: A data frame containing the Limits of Agreement.
call:The matched call.
MOVER methods:
Zou, G. Y. (2013). Confidence interval estimation for the Bland–Altman limits of agreement with multiple observations per individual. Statistical methods in medical research, 22(6), 630-642.
Donner, A., & Zou, G. Y. (2012). Closed-form confidence intervals for functions of the normal mean and standard deviation. Statistical Methods in Medical Research, 21(4), 347-359.
Bland & Altman methods:
Bland, J. M., & Altman, D. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet, 327(8476), 307-310.
Bland, J. M., & Altman, D. (1999). Measuring agreement in method comparison studies. Statistical methods in medical research, 8(2), 135-160.
Bland, J. M., & Altman, D. G. (1996). Statistics notes: measurement error proportional to the mean. BMJ, 313(7049), 106.
data('reps') # Simple agreement_limit(x = "x", y ="y", data = reps) # Replicates agreement_limit(x = "x", y ="y", data = reps, id = "id", data_type = "rep") # Nested agreement_limit(x = "x", y ="y", data = reps, id = "id", data_type = "nest")data('reps') # Simple agreement_limit(x = "x", y ="y", data = reps) # Replicates agreement_limit(x = "x", y ="y", data = reps, id = "id", data_type = "rep") # Nested agreement_limit(x = "x", y ="y", data = reps, id = "id", data_type = "nest")
This function calculates the power for the Bland-Altman method under varying parameter settings and for a range of sample sizes.
It is superseded by power_agreement_limit(), which returns a power.htest
object and can solve for the sample size, power, delta or the SD directly.
This function remains available for drawing power curves.
blandPowerCurve( samplesizes = seq(10, 100, 1), mu = 0, SD, delta, conf.level = 0.95, agree.level = 0.95, m = 1, icc = 0 )blandPowerCurve( samplesizes = seq(10, 100, 1), mu = 0, SD, delta, conf.level = 0.95, agree.level = 0.95, m = 1, icc = 0 )
samplesizes |
vector of samples sizes at which to estimate power: the
number of pairs, or the number of subjects when |
mu |
mean of differences |
SD |
standard deviation of differences (the total SD when |
delta |
The threshold below which methods agree/can be considered equivalent, can be in any units. Equivalence Bound for Agreement. More than one delta can be provided. |
conf.level |
the confidence level(s) required. Default is 95%. More than one confidence level can be provided. |
agree.level |
the agreement level(s) required. Default is 95%. The proportion of data that should lie between the thresholds, for 95% limits of agreement this should be 0.95. More than one confidence level can be provided. |
m |
number of paired measurements per subject (default 1, independent pairs). |
icc |
intraclass correlation of the paired differences, in [0, 1).
Ignored when |
Power follows the approximate method of Lu et al. (2016). With clustered
data (m > 1), the standard error of the limits uses the effective sample
size n \* m / (1 + (m - 1) \* icc) for the mean difference and a
Satterthwaite df for the SD, under a balanced compound symmetry model (see
agree_expected_half() for the model). Like the original method, this is
an approximation; m = 1 gives the original results.
A dataframe is returned containing the power analysis results. The results can then be plotted with the plot.powerCurve function.
Lu, M. J., et al. (2016). Sample Size for Assessing Agreement between Two Methods of Measurement by Bland-Altman Method. The international journal of biostatistics, 12(2), doi:10.1515/ijb-2015-0039
powerCurve <- blandPowerCurve(samplesizes = seq(10, 200, 1), mu = 0, SD = 3.3, delta = 8, conf.level = .95, agree.level = .95) # Plot the power curve plot(powerCurve, type = 1) # Find at what N power of .8 is achieved find_n(powerCurve, power = .8) # If the desired power is not found then ## Sample size range must be expandedpowerCurve <- blandPowerCurve(samplesizes = seq(10, 200, 1), mu = 0, SD = 3.3, delta = 8, conf.level = .95, agree.level = .95) # Plot the power curve plot(powerCurve, type = 1) # Find at what N power of .8 is achieved find_n(powerCurve, power = .8) # If the desired power is not found then ## Sample size range must be expanded
Methods for objects returned by cac_test() and ord_kappa_test(). They
print as standard "htest" objects; summary() describes the result
in words and plot() shows the ratings.
## S3 method for class 'cac_htest' summary(object, style = c("auto", "hypothesis", "estimation"), ...) ## S3 method for class 'summary.cac_htest' print(x, ...) ## S3 method for class 'cac_htest' plot(x, type = c("raters", "subjects", "effects"), ...)## S3 method for class 'cac_htest' summary(object, style = c("auto", "hypothesis", "estimation"), ...) ## S3 method for class 'summary.cac_htest' print(x, ...) ## S3 method for class 'cac_htest' plot(x, type = c("raters", "subjects", "effects"), ...)
style |
How |
... |
Not used. |
x, object
|
An object of class |
type |
For |
summaryAn object of class summary.cac_htest: a list
with the description in text, the style used, the S-value
(s.value, when there is a p-value) and the parts used to build it. It
prints as a paragraph.
plotA ggplot2 plot (see type).
ratings <- data.frame( R1 = c(1, 2, 3, 3, 2, 1, 4, 1, 2, NA, NA, NA), R2 = c(1, 2, 3, 3, 2, 2, 4, 1, 2, 5, NA, NA), R3 = c(NA, 3, 3, 3, 2, 3, 4, 2, 2, 5, 1, 3), R4 = c(1, 2, 3, 3, 2, 4, 4, 1, 2, 5, 1, NA)) res <- cac_test(col.names = c("R1", "R2", "R3", "R4"), data = ratings, null.value = 0.4, alternative = "greater") summary(res) summary(res, style = "estimation") plot(res) plot(res, type = "subjects")ratings <- data.frame( R1 = c(1, 2, 3, 3, 2, 1, 4, 1, 2, NA, NA, NA), R2 = c(1, 2, 3, 3, 2, 2, 4, 1, 2, 5, NA, NA), R3 = c(NA, 3, 3, 3, 2, 3, 4, 2, 2, 5, 1, 3), R4 = c(1, 2, 3, 3, 2, 4, 4, 1, 2, 5, 1, NA)) res <- cac_test(col.names = c("R1", "R2", "R3", "R4"), data = ratings, null.value = 0.4, alternative = "greater") summary(res) summary(res, style = "estimation") plot(res) plot(res, type = "subjects")
Estimates and tests one agreement coefficient for categorical (nominal or
ordinal) ratings of the same subjects by two or more raters: percent
agreement, Gwet's AC1 (AC2 when weighted), Fleiss' kappa, or
Krippendorff's alpha. Missing ratings are allowed. For ordinal ratings
from many raters, see also the model-based kappa in ord_kappa_test().
cac_test( x = NULL, data = NULL, measure = NULL, item = NULL, id = NULL, col.names = NULL, coef = c("gwet", "fleiss", "krippendorff", "pa"), weights = "unweighted", categories = NULL, conf.level = 0.95, null.value = NULL, alternative = c("two.sided", "greater", "less") )cac_test( x = NULL, data = NULL, measure = NULL, item = NULL, id = NULL, col.names = NULL, coef = c("gwet", "fleiss", "krippendorff", "pa"), weights = "unweighted", categories = NULL, conf.level = 0.95, null.value = NULL, alternative = c("two.sided", "greater", "less") )
x |
A formula of the form |
data |
A data frame. |
measure, item, id
|
Names of the columns with the ratings, the raters, and the subjects (long data). |
col.names |
Names of the columns with each rater's ratings (wide data, one row per subject). |
coef |
The coefficient: |
weights |
How partial agreement between different categories is
credited: |
categories |
All possible rating categories, in order. Defaults to the levels of a factor, and otherwise to the sorted observed ratings. Give it when some categories of the scale were never used: Gwet's AC1/AC2 and the weights depend on the full set of categories. |
conf.level |
Confidence level of the interval (default 0.95). |
null.value |
The hypothesized value of the coefficient. |
alternative |
The alternative hypothesis: |
All four coefficients share the observed agreement
, the proportion of agreeing pairs of ratings within subjects
(weighted when weights is used), averaged over the subjects with two or
more ratings. The chance-corrected coefficients are
and differ in the chance agreement :
Fleiss' kappa and Krippendorff's alpha use the overall category
proportions, so they are low when one category dominates even if raters
rarely disagree (the "kappa paradox"); Gwet's AC1 uses the chance that a
rater classifies at random and is stable in that situation (Gwet, 2008).
Krippendorff's alpha also corrects for the number of pairs.
Standard errors come from Gwet's (2008, 2014) linearisation over subjects, which treats the subjects as a random sample and the raters as fixed. The interval and test use the t distribution with n - 1 degrees of freedom (n subjects; for Krippendorff's alpha, those with 2 or more ratings), and limits are clipped to the possible range of the coefficient. These match the irrCAC package (with an infinite population).
Ratings may be numeric, character or factor. Subjects with no ratings are dropped; subjects with one rating add to the category proportions but not to the observed agreement. Each subject can have at most one rating per rater.
A list of class c("cac_htest", "htest") containing:
statistic |
The t statistic (only with a |
parameter |
The degrees of freedom (only with a |
p.value |
The p-value (only with a |
conf.int |
The confidence interval for the coefficient. |
estimate |
The estimated coefficient. |
stderr |
The standard error of the coefficient. |
null.value |
The hypothesized value. |
alternative |
The alternative hypothesis. |
method |
A description of the coefficient. |
data.name |
A description of the data. |
fit |
Details: observed ( |
data |
The ratings in long format. |
settings |
The settings used, for the methods. |
Gwet, K. L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1), 29-48. doi:10.1348/000711006X126600
Gwet, K. L. (2014). Handbook of Inter-Rater Reliability (4th ed.). Advanced Analytics, LLC. ISBN 978-0970806284.
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378-382. doi:10.1037/h0031619
Krippendorff, K. (1970). Estimating the reliability, systematic error and random error of interval data. Educational and Psychological Measurement, 30(1), 61-70. doi:10.1177/001316447003000105
cac_htest-methods for summary() and plot();
ord_kappa_test() for the model-based kappa for ordinal ratings.
# Gwet (2014): 12 subjects rated by 4 raters on a 5-point scale ratings <- data.frame( R1 = c(1, 2, 3, 3, 2, 1, 4, 1, 2, NA, NA, NA), R2 = c(1, 2, 3, 3, 2, 2, 4, 1, 2, 5, NA, NA), R3 = c(NA, 3, 3, 3, 2, 3, 4, 2, 2, 5, 1, 3), R4 = c(1, 2, 3, 3, 2, 4, 4, 1, 2, 5, 1, NA)) cac_test(col.names = c("R1", "R2", "R3", "R4"), data = ratings) # Quadratic weights, Fleiss' kappa, tested against 0.6 res <- cac_test(col.names = c("R1", "R2", "R3", "R4"), data = ratings, coef = "fleiss", weights = "quadratic", null.value = 0.6, alternative = "greater") res summary(res) plot(res)# Gwet (2014): 12 subjects rated by 4 raters on a 5-point scale ratings <- data.frame( R1 = c(1, 2, 3, 3, 2, 1, 4, 1, 2, NA, NA, NA), R2 = c(1, 2, 3, 3, 2, 2, 4, 1, 2, 5, NA, NA), R3 = c(NA, 3, 3, 3, 2, 3, 4, 2, 2, 5, 1, 3), R4 = c(1, 2, 3, 3, 2, 4, 4, 1, 2, 5, 1, NA)) cac_test(col.names = c("R1", "R2", "R3", "R4"), data = ratings) # Quadratic weights, Fleiss' kappa, tested against 0.6 res <- cac_test(col.names = c("R1", "R2", "R3", "R4"), data = ratings, coef = "fleiss", weights = "quadratic", null.value = 0.6, alternative = "greater") res summary(res) plot(res)
Methods for objects returned by ccc_test(). They print as standard
"htest" objects; summary() describes the result in words,
plot() shows the agreement between the two methods, and check() gives
diagnostics.
## S3 method for class 'ccc_htest' summary(object, style = c("auto", "hypothesis", "estimation"), ...) ## S3 method for class 'ccc_htest' plot(x, type = c("identity", "ba"), ...) ## S3 method for class 'ccc_htest' check(x, ...)## S3 method for class 'ccc_htest' summary(object, style = c("auto", "hypothesis", "estimation"), ...) ## S3 method for class 'ccc_htest' plot(x, type = c("identity", "ba"), ...) ## S3 method for class 'ccc_htest' check(x, ...)
style |
How |
... |
Not used. |
x, object
|
An object of class |
type |
For |
summary() describes the estimate and its interval, what was estimated,
the design, and how the inference was done, in the hypothesis or
estimation style described in reli_htest-methods. For simple paired
data it also reports Lin's decomposition of the CCC into precision (the
Pearson correlation, r) and accuracy (the bias correction factor, C_b):
CCC = r x C_b. A low CCC can come from either.
check() shows the normality and heteroskedasticity of the differences
(as for agreement_limit() with the same data), and the linearity of the
relationship between the methods: the residuals of y regressed on x
against the fitted values, with a test of a quadratic term (Ramsey's
RESET test). The CCC, through its precision component, summarises a
linear relationship. Proportional bias is not shown: a slope different
from 1 between the methods lowers the bias correction factor C_b, so it
is part of what the CCC measures rather than a violation of its
assumptions. For simple paired data check() also shows a Q-Q plot of
the squared Mahalanobis distances of the pairs against a chi-square
distribution with 2 degrees of freedom: Lin's standard error assumes the
two measurements are bivariate normal, and points far above the line
suggest they are not. For replicate or nested data the linearity test
uses all pairs, which are correlated within subjects, so its p-value is
approximate; the jackknife standard error makes few distributional
assumptions, so these panels mainly help interpret the CCC.
summaryAn object of class summary.ccc_htest (which
prints like summary.reli_htest): the description in text, the
style used, the S-value (s.value, when there is a p-value), and the
parts used to build it.
plotA ggplot2 plot (see type).
checkDiagnostic plots combined with patchwork.
Lin, L. I. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics, 45(1), 255-268. doi:10.2307/2532051
Carrasco, J. L., & Jover, L. (2003). Estimating the generalized concordance correlation coefficient through variance components. Biometrics, 59(4), 849-858. doi:10.1111/j.0006-341x.2003.00099.x
Ramsey, J. B. (1969). Tests for specification errors in classical linear least-squares regression analysis. Journal of the Royal Statistical Society: Series B, 31(2), 350-371. doi:10.1111/j.2517-6161.1969.tb00796.x
data("reps") res <- ccc_test(x = "x", y = "y", data = reps) summary(res) plot(res) plot(res, type = "ba") check(res)data("reps") res <- ccc_test(x = "x", y = "y", data = reps) summary(res) plot(res) plot(res, type = "ba") check(res)
Computes Lin's Concordance Correlation Coefficient (CCC) and performs a
hypothesis test, returning an object of class "htest".
ccc_test( x, y = NULL, id = NULL, data = NULL, data_type = c("simple", "reps", "nest"), conf.level = 0.95, null.value = NULL, alternative = c("two.sided", "greater", "less"), boot = FALSE, boot_ci = c("stud", "perc", "bca"), replicates = 999 )ccc_test( x, y = NULL, id = NULL, data = NULL, data_type = c("simple", "reps", "nest"), conf.level = 0.95, null.value = NULL, alternative = c("two.sided", "greater", "less"), boot = FALSE, boot_ci = c("stud", "perc", "bca"), replicates = 999 )
x |
Either a numeric vector of measurements for method 1, or a character
string naming the column in |
y |
Either a numeric vector of measurements for method 2, or a character
string naming the column in |
id |
A character string naming the column in |
data |
An optional data frame containing the variables named in |
data_type |
Character string specifying the data structure:
|
conf.level |
Confidence level for the interval, default is 0.95. |
null.value |
The CCC under the null hypothesis. The default, |
alternative |
Character string specifying the alternative hypothesis:
|
boot |
If |
boot_ci |
Bootstrap interval type: |
replicates |
Number of bootstrap replicates (default 999). |
The concordance correlation coefficient (CCC) quantifies agreement between
two measurements of the same quantity. It is the product of a precision
component (Pearson's correlation) and an accuracy component (the bias
correction factor), and together these capture how far the observed data
depart from the line of perfect concordance (the 45-degree line through the
origin). Its purpose is closely related to that of dem_reg(). By analogy,
Lin's CCC is to Pearson's r what Deming regression is to ordinary least
squares. Deming regression estimates the
relationship (slope and intercept) and asks whether it deviates from the
line of identity, whereas
the CCC summarizes that deviation in a single index.
Simple data. Lin's (1989) estimator and asymptotic standard error are
used, from the complete pairs; these match DescTools::CCC() and
epiR::epi.ccc() (with ci = "z-transform").
Replicate data ("reps"; the true value is constant within a
subject). The CCC is estimated from variance components (Carrasco &
Jover, 2003): a linear mixed model with a fixed method effect and random
subject and subject-by-method effects, fit by REML, gives
where is the variance between the method means. All
measurements are used, so the replicates of the two methods need not be
paired or equal in number. The estimate equals that of
cccrm::ccc_vc(int = TRUE).
Nested data ("nest"; the true value may change within a subject,
and each row is a matched pair). Lin's estimator is computed over all
complete pairs, which keep their pairing.
For both, the subject is the independent unit. The standard error of Fisher's z is a delete-one-subject jackknife, so it allows for the correlation of measurements within a subject. (Treating every pair as independent understates the standard error.) The interval and test use a t distribution with (number of subjects - 1) degrees of freedom. At least 3 subjects are needed.
In simulations (10, 20 or 50 subjects; 2, 3 or 6 measurements per subject; CCC 0.5 to 0.95; with and without a method difference and 15% missing data), 95% jackknife intervals covered about 93-95% of the time with normal data, but only 88-91% (replicate data) and 90-93% (nested data) with skewed subject effects and heavy-tailed errors, at every number of subjects. Misses fall mostly above the interval: the CCC is biased downward with few subjects (by about 0.03-0.04 with 10 subjects).
Bias. The jackknife estimate of that bias, and the bias-corrected
CCC, are kept in the jackknife element, and summary() mentions them
when the bias is more than 0.01 or more than 10% of the interval width.
They are a guide, not a replacement for the estimate. In the simulations
the correction removed 80-90% of the average bias with 10 or 20 subjects,
but the corrected value was closer to the true CCC in only 43-50% of
data sets, its root mean squared error was slightly larger (up to 28%
larger with non-normal data and 10 subjects), and with 10 subjects it
went past 1 (and is then truncated) in about 5% of replicate data sets.
It is most useful when CCCs from several small studies are averaged,
where the bias would otherwise accumulate. Centring the interval on the
corrected value did not improve its coverage, so the interval and test
use the uncorrected estimate.
Bootstrap (replicate data). With boot = TRUE, subjects are
resampled with replacement (a subject drawn twice counts as two
subjects) and the CCC is re-estimated. The studentized interval
(boot_ci = "stud") works on Fisher's z scale and divides each replicate
by its own delete-one-subject jackknife standard error; the p-value
inverts the interval (as in icc_test()) and is approximate. In the
simulations above it raised coverage with 10 subjects from 0.90 to 0.94
(from 0.88 to 0.94 with non-normal data) and with 20 subjects from 0.92
to 0.94, with intervals of about the same width, so boot = TRUE is a
good idea for replicate data with few subjects. With non-normal data and
6 replicates per subject some designs still covered only 0.88-0.89. The
percentile and BCa intervals covered poorly (0.80-0.91 on average) and
are offered only for comparison. Each replicate needs one model fit per
subject (from the subjects' summary statistics rather than lme4), so the
studentized bootstrap takes roughly 20 seconds with 10 subjects and a
minute with 30 subjects at the default 999 replicates. For nested data
no bootstrap interval improved on the jackknife.
The confidence interval and the hypothesis test use Fisher's Z-transformation of the CCC:
By default (null.value = NULL) no test is done.
Relation to the ICC. For two methods, the CCC is close to the
absolute-agreement intraclass correlation, ICC(A,1)
(icc_test(type = "agreement")). Lin's estimator (simple and nested data)
uses n rather than n - 1 in the variances and covariance, as Lin (1989)
defined it and as other software computes it, so it is slightly below the
ICC in small samples; the difference vanishes as n grows (Nickerson, 1997;
Carrasco & Jover, 2003). The variance-components estimator (replicate
data) uses REML variance components and so corresponds to the ICC form. ccc_test() is meant for
method-comparison (agreement) studies; icc_test() for reliability
studies (see its "Choosing the form" and "Related coefficients"
sections).
Methods. summary() describes the result in words (including, for
simple data, whether a low CCC comes from poor precision or poor
accuracy), plot() draws the identity or Bland-Altman plot, and
check() gives diagnostics; see ccc_htest-methods.
A list with class c("ccc_htest", "htest") containing the
following components:
statistic |
The test statistic on the Fisher z scale ( |
parameter |
When a test was requested: the number of pairs |
stderr |
The standard errors for CCC and Z-transformed CCC (for
reps/nest, the jackknife SE of z, and the SE of the CCC from it by the
delta method). With |
p.value |
The p-value for the test, when a test was requested. |
conf.int |
A confidence interval for the CCC. |
estimate |
The estimated CCC. |
null.value |
The specified hypothesized value of CCC, if any. |
alternative |
The alternative hypothesis. |
method |
A character string indicating the method used. |
data.name |
A character string giving the names of the data. |
data |
The data used ( |
components |
For simple data, Lin's decomposition: the Pearson
correlation |
jackknife |
For reps/nest, the delete-one-subject jackknife
estimate of the small-sample bias: a list with |
settings |
The settings used, for the methods. |
boot |
Bootstrap details ( |
Lin, L. I. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics, 45(1), 255-268.
Lin, L. I. (2000). A note on the concordance correlation coefficient. Biometrics, 56(1), 324-325.
Carrasco, J. L., & Jover, L. (2003). Estimating the generalized concordance correlation coefficient through variance components. Biometrics, 59(4), 849-858. doi:10.1111/j.0006-341x.2003.00099.x
Nickerson, C. A. E. (1997). A note on "A concordance correlation coefficient to evaluate reproducibility". Biometrics, 53(4), 1503-1507. doi:10.2307/2533516
ccc_htest-methods, icc_test(), agreement_limit()
# Simple paired data using vectors x <- c(1, 2, 3, 4, 5) y <- c(1.1, 2.2, 2.9, 4.1, 5.2) ccc_test(x, y) # Using data frame interface data("reps") # Simple data ccc_test(x = "x", y = "y", data = reps) # Test against specific null value ccc_test(x = "x", y = "y", data = reps, null.value = 0.8, alternative = "greater") # Replicate data ccc_test(x = "x", y = "y", id = "id", data = reps, data_type = "reps") # Replicate data with few subjects: studentized subject bootstrap ccc_test(x = "x", y = "y", id = "id", data = reps, data_type = "reps", boot = TRUE) # Nested data ccc_test(x = "x", y = "y", id = "id", data = reps, data_type = "nest") # Describe, plot and check res <- ccc_test(x = "x", y = "y", data = reps) summary(res) plot(res) check(res)# Simple paired data using vectors x <- c(1, 2, 3, 4, 5) y <- c(1.1, 2.2, 2.9, 4.1, 5.2) ccc_test(x, y) # Using data frame interface data("reps") # Simple data ccc_test(x = "x", y = "y", data = reps) # Test against specific null value ccc_test(x = "x", y = "y", data = reps, null.value = 0.8, alternative = "greater") # Replicate data ccc_test(x = "x", y = "y", id = "id", data = reps, data_type = "reps") # Replicate data with few subjects: studentized subject bootstrap ccc_test(x = "x", y = "y", id = "id", data = reps, data_type = "reps", boot = TRUE) # Nested data ccc_test(x = "x", y = "y", id = "id", data = reps, data_type = "nest") # Describe, plot and check res <- ccc_test(x = "x", y = "y", data = reps) summary(res) plot(res) check(res)
A function for fitting a straight line to two-dimensional data (i.e., X and Y) that are measured with error.
dem_reg( formula = NULL, data, id = NULL, x = NULL, y = NULL, conf.level = 0.95, weighted = FALSE, weights = NULL, error.ratio = 1, model = TRUE, keep_data = FALSE, ... )dem_reg( formula = NULL, data, id = NULL, x = NULL, y = NULL, conf.level = 0.95, weighted = FALSE, weights = NULL, error.ratio = 1, model = TRUE, keep_data = FALSE, ... )
formula |
A formula of the form |
data |
Data frame with all data. |
id |
Column with subject identifier (optional). |
x |
Name of column with first measurement (deprecated in favor of formula interface). |
y |
Name of other column with the other measurement to compare to the first (deprecated in favor of formula interface). |
conf.level |
The confidence level required. Default is 95%. |
weighted |
Logical indicator (TRUE/FALSE) for whether to use weighted Deming regression. Default is FALSE. |
weights |
an optional vector of weights to be used in the fitting process. Should be NULL or a numeric vector. |
error.ratio |
Ratio of the measurement error variance of x to that of y, i.e., var(x)/var(y), where var() denotes the error (not total) variance of each method. Default is 1 (equal error variances). Values greater than 1 indicate that x is measured with more error than y. This argument is ignored if subject identifiers are provided. |
model |
Logical. If TRUE (default), the model frame is stored in the returned object.
This is needed for methods like |
keep_data |
Logical indicator (TRUE/FALSE). If TRUE, the jacknife samples are returned; default is FALSE. |
... |
Additional arguments (currently unused). |
This function provides a Deming regression analysis wherein the sum of distances in both x and y direction is minimized. Deming regression, also known as error-in-variable regression, is useful in situations where both X & Y are measured with error. The use of Deming regression is beneficial when comparing to methods for measuring the same continuous variable.
Currently, the dem_reg function covers simple Deming regression and weighted Deming regression.
Weighted Deming regression can be used by setting the weighted argument to TRUE.
The weights can be provided by the user or can be calculated within function.
If the data are measured in replicates, then the measurement error can be directly derived from the data. This can be accomplished by indicating the subject identifier with the id argument. When the replicates are not available in the data, then the ratio of error variances, var(x)/var(y), can be provided with the error.ratio argument. When subject identifiers are provided, the error ratio is instead estimated from the replicates as the within-subject variance of x divided by the within-subject variance of y.
The function returns a simple_eiv (eiv meaning "error in variables") object with the following components:
coefficients: Named vector of coefficients (intercept and slope).
residuals: Optimized residuals from the fitted model.
fitted.values: Estimated true Y values (Y-hat).
model_table: Data frame presenting the full results from the Deming regression analysis.
vcov: Variance-covariance matrix for slope and intercept.
df.residual: Residual degrees of freedom.
call: The matched call.
terms: The terms object used.
xlevels: (Only for models with factors) levels of factors.
model: The model frame.
x_vals: Original x values used in fitting.
y_vals: Original y values used in fitting.
x_hat: Estimated true X values.
y_hat: Estimated true Y values.
error.ratio: Error ratio used in fitting.
weighted: Whether weighted regression was used.
weights: Weights used in fitting.
conf.level: Confidence level used.
resamples: List containing resamples from jacknife procedure (if keep_data = TRUE).
The x and y arguments are deprecated. Please use the formula interface instead:
Old: dem_reg(x = "x_var", y = "y_var", data = df)
New: dem_reg(y_var ~ x_var, data = df)
Linnet, K. (1990) Estimation of the linear relationship between the measurements of two methods with proportional errors. Statistics in Medicine, 9, 1463-1473.
Linnet, K. (1993). Evaluation of regression procedures for methods comparison studies. Clinical chemistry, 39, 424-432.
Sadler, W.A. (2010). Joint parameter confidence regions improve the power of parametric regression in method-comparison studies. Accreditation and Quality Assurance, 15, 547-554.
# Peak flow: mini vs Wright meter (Bland & Altman, 1986) data("ba1986") fit <- dem_reg(mini1 ~ wright1, data = ba1986) summary(fit) plot(fit) # Joint test of intercept = 0 and slope = 1 joint_test(fit) # Known error ratio, var(x) / var(y) dem_reg(mini1 ~ wright1, data = ba1986, error.ratio = 2) # Replicate measurements: the error ratio is estimated from the replicates data("reps") dem_reg(y ~ x, data = reps, id = "id") # Weighted Deming regression (error proportional to the level) dem_reg(mini1 ~ wright1, data = ba1986, weighted = TRUE)# Peak flow: mini vs Wright meter (Bland & Altman, 1986) data("ba1986") fit <- dem_reg(mini1 ~ wright1, data = ba1986) summary(fit) plot(fit) # Joint test of intercept = 0 and slope = 1 joint_test(fit) # Known error ratio, var(x) / var(y) dem_reg(mini1 ~ wright1, data = ba1986, error.ratio = 2) # Replicate measurements: the error ratio is estimated from the replicates data("reps") dem_reg(y ~ x, data = reps, id = "id") # Weighted Deming regression (error proportional to the level) dem_reg(mini1 ~ wright1, data = ba1986, weighted = TRUE)
Functions for conducting power analysis and sample size determination for Deming regression in method comparison studies. These functions help determine the sample size needed to detect specified biases (proportional and/or constant) between two measurement methods.
Estimates statistical power to detect deviations from the line of identity using simulated data with known properties.
deming_power_sim( n_sims = 1000, sample_size = 50, x_range = c(10, 100), x_dist = "uniform", actual_slope = 1.05, actual_intercept = 0, ideal_slope = 1, ideal_intercept = 0, y_var_params = list(beta1 = 1, beta2 = 0, J = 1, type = "constant"), x_var_params = list(beta1 = 1, beta2 = 0, J = 1, type = "constant"), weighted = FALSE, conf.level = 0.95, joint_method = c("sadler", "wald") )deming_power_sim( n_sims = 1000, sample_size = 50, x_range = c(10, 100), x_dist = "uniform", actual_slope = 1.05, actual_intercept = 0, ideal_slope = 1, ideal_intercept = 0, y_var_params = list(beta1 = 1, beta2 = 0, J = 1, type = "constant"), x_var_params = list(beta1 = 1, beta2 = 0, J = 1, type = "constant"), weighted = FALSE, conf.level = 0.95, joint_method = c("sadler", "wald") )
n_sims |
Number of simulation iterations. Default is 1000. |
sample_size |
Sample size (number of paired observations) per simulation. |
x_range |
Numeric vector of length 2 specifying min and max of X values (e.g., c(10, 100)). |
x_dist |
Character specifying distribution of X values: "uniform", "central", or "right_skewed". |
actual_slope |
Actual slope used to generate data (e.g., 1.05 for 5% proportional bias). |
actual_intercept |
Actual intercept used to generate data. |
ideal_slope |
Hypothesized slope to test against (typically 1 for identity line). |
ideal_intercept |
Hypothesized intercept to test against (typically 0 for identity line). |
y_var_params |
List with Y variance parameters: beta1, beta2, J, type. Type can be "constant", "proportional", or "power". For power function: sigma^2 = (beta1 + beta2*U)^J |
x_var_params |
List with X variance parameters (same structure as y_var_params). |
weighted |
Logical. Use weighted Deming regression? Default is FALSE. |
conf.level |
Confidence level for tests. Default is 0.95. |
joint_method |
The joint test, passed to |
Power Analysis for Deming Regression
This function generates simulated datasets with specified error characteristics and tests whether confidence intervals and joint confidence regions detect deviations from hypothesized values. The joint confidence region typically provides higher statistical power, especially when the X-range is narrow (high slope-intercept correlation).
Each simulated data set is fitted with dem_reg() and tested with
joint_test() and the marginal confidence intervals, exactly as an analysis
would be, so the estimated power is the power of those tests. The error
ratio passed to dem_reg() is the ratio of the two variance functions at the
mean of the simulated x values. Fits that fail count as non-detections.
The variance functions allow flexible modeling of heteroscedastic errors:
constant: sigma^2 = beta1
proportional: sigma^2 = beta1 * U^2 (constant CV%)
power: sigma^2 = (beta1 + beta2*U)^J
A list of class "deming_power" containing:
power_ci_slope |
Power based on slope confidence interval |
power_ci_intercept |
Power based on intercept confidence interval |
power_either_ci |
Power when either CI detects difference |
power_joint |
Power based on joint confidence region |
settings |
List of simulation settings |
advantage |
Difference in power between joint region and CIs |
Sadler, W.A. (2010). Joint parameter confidence regions improve the power of parametric regression in method-comparison studies. Accreditation and Quality Assurance, 15, 547-554.
# Simple example: detect 5% proportional bias with constant variance # (use n_sims = 1000 or more in practice) power_result <- deming_power_sim( n_sims = 200, sample_size = 50, x_range = c(10, 100), actual_slope = 1.05, ideal_slope = 1.0, y_var_params = list(beta1 = 25, beta2 = 0, J = 1, type = "constant"), x_var_params = list(beta1 = 20, beta2 = 0, J = 1, type = "constant") ) print(power_result) # More complex: heteroscedastic errors power_result2 <- deming_power_sim( n_sims = 200, sample_size = 75, x_range = c(1, 100), actual_slope = 1.03, y_var_params = list(beta1 = 0.5, beta2 = 0.05, J = 2, type = "power"), x_var_params = list(beta1 = 0.4, beta2 = 0.04, J = 2, type = "power"), weighted = TRUE )# Simple example: detect 5% proportional bias with constant variance # (use n_sims = 1000 or more in practice) power_result <- deming_power_sim( n_sims = 200, sample_size = 50, x_range = c(10, 100), actual_slope = 1.05, ideal_slope = 1.0, y_var_params = list(beta1 = 25, beta2 = 0, J = 1, type = "constant"), x_var_params = list(beta1 = 20, beta2 = 0, J = 1, type = "constant") ) print(power_result) # More complex: heteroscedastic errors power_result2 <- deming_power_sim( n_sims = 200, sample_size = 75, x_range = c(1, 100), actual_slope = 1.03, y_var_params = list(beta1 = 0.5, beta2 = 0.05, J = 2, type = "power"), x_var_params = list(beta1 = 0.4, beta2 = 0.04, J = 2, type = "power"), weighted = TRUE )
Automatically determines the minimum sample size needed to achieve target statistical power for detecting specified bias in method comparison studies using Deming regression.
deming_sample_size( target_power = 0.9, initial_n = 20, max_n = 500, n_sims = 500, use_joint = NULL, step_size = 5, ... )deming_sample_size( target_power = 0.9, initial_n = 20, max_n = 500, n_sims = 500, use_joint = NULL, step_size = 5, ... )
target_power |
Desired statistical power (e.g., 0.80 or 0.90). Default is 0.90. |
initial_n |
Starting sample size for search. Default is 20. |
max_n |
Maximum sample size to try. Default is 500. |
n_sims |
Number of simulations per sample size tested. Default is 500. |
use_joint |
Which target ends the search. |
step_size |
Step size for sample size increments. Default is 5. |
... |
Additional arguments passed to deming_power_sim() |
This function performs a grid search over sample sizes to find the minimum N needed to achieve the target power. It tests both confidence interval and joint confidence region approaches, allowing comparison of required sample sizes.
Using joint confidence regions typically requires far fewer samples than confidence intervals when the measurement range is narrow (max:min ratio < 10:1).
Each power is a simulation estimate (Monte Carlo SE about 1.3 percentage
points at 90% power with n_sims = 500), and the search takes the first N
whose estimate reaches the target, which tends to pick N slightly too small.
In simulations the true power at the returned N was close to the target on
average but below 85% in up to 20% of runs. Use a larger n_sims (or check
the chosen N with deming_power_sim()) when the sample size matters. N is
never below initial_n.
A list of class "deming_sample_size" containing:
n_required_ci |
Required N for confidence intervals |
n_required_joint |
Required N for joint confidence region |
target_power |
Target power level |
power_curve |
Data frame with N and power for both methods |
reduction_n |
Sample size reduction using joint method |
reduction_pct |
Percentage reduction |
# Determine N needed for 90% power to detect 5% bias with the joint # region (use_joint = TRUE stops there; n_sims = 500 or more in practice) sample_size_result <- deming_sample_size( target_power = 0.90, initial_n = 10, max_n = 200, n_sims = 200, step_size = 10, use_joint = TRUE, x_range = c(20, 200), actual_slope = 1.05, ideal_slope = 1.0, y_var_params = list(beta1 = 1, beta2 = 0.05, J = 2, type = "power"), x_var_params = list(beta1 = 0.8, beta2 = 0.04, J = 2, type = "power") ) print(sample_size_result) plot(sample_size_result)# Determine N needed for 90% power to detect 5% bias with the joint # region (use_joint = TRUE stops there; n_sims = 500 or more in practice) sample_size_result <- deming_sample_size( target_power = 0.90, initial_n = 10, max_n = 200, n_sims = 200, step_size = 10, use_joint = TRUE, x_range = c(20, 200), actual_slope = 1.05, ideal_slope = 1.0, y_var_params = list(beta1 = 1, beta2 = 0.05, J = 2, type = "power"), x_var_params = list(beta1 = 0.8, beta2 = 0.04, J = 2, type = "power") ) print(sample_size_result) plot(sample_size_result)
Estimates one intraclass correlation coefficient (ICC), with a confidence
interval and a test against a null value, and returns an object of class
"htest". The ICC form is chosen with model, type and unit,
following McGraw and Wong (1996).
icc_test( x = NULL, data = NULL, measure = NULL, item = NULL, id = NULL, col.names = NULL, model = c("twoway", "oneway"), type = c("consistency", "agreement"), unit = c("single", "average"), method = c("anova", "reml"), conf.level = 0.95, null.value = NULL, alternative = c("two.sided", "greater", "less"), boot = FALSE, boot_ci = c("auto", "perc", "bca", "stud", "basic"), replicates = 1999, se_method = c("jackknife", "influence", "normal") )icc_test( x = NULL, data = NULL, measure = NULL, item = NULL, id = NULL, col.names = NULL, model = c("twoway", "oneway"), type = c("consistency", "agreement"), unit = c("single", "average"), method = c("anova", "reml"), conf.level = 0.95, null.value = NULL, alternative = c("two.sided", "greater", "less"), boot = FALSE, boot_ci = c("auto", "perc", "bca", "stud", "basic"), replicates = 1999, se_method = c("jackknife", "influence", "normal") )
x |
A formula of the form |
data |
A data frame containing the variables. |
measure, item, id
|
Column names (character) of the measurement, the item (trial, rater, or occasion) and the subject identifier, for data in long format. |
col.names |
Column names (character) of the items, for data in wide format (one row per subject). |
model |
|
type |
|
unit |
|
method |
|
conf.level |
Confidence level of the interval (default 0.95). |
null.value |
The ICC under the null hypothesis. The default, |
alternative |
|
boot |
If |
boot_ci |
Bootstrap interval type: |
replicates |
Number of bootstrap replicates (default 1999). |
se_method |
How the standard error is estimated: |
ICC forms.. The different ICCs can be confusing. Therefore, I have summarized it here. McGraw and Wong (1996) and Shrout and Fleiss (1979) name the forms differently:
model |
type |
unit |
McGraw & Wong | Shrout & Fleiss |
| oneway | agreement | single | ICC(1) | ICC(1,1) |
| oneway | agreement | average | ICC(k) | ICC(1,k) |
| twoway | agreement | single | ICC(A,1) | ICC(2,1) |
| twoway | agreement | average | ICC(A,k) | ICC(2,k) |
| twoway | consistency | single | ICC(C,1) | ICC(3,1) |
| twoway | consistency | average | ICC(C,k) | ICC(3,k) |
Whether items are treated as random or fixed changes the interpretation but not the calculation (McGraw & Wong, 1996). Average-measure forms are not defined for McGraw and Wong's two-way mixed model with an interaction.
Choosing the form. Answer three questions in order:
(1) is every subject measured by the same items? Yes: "twoway"; no,
each subject by different items: "oneway". (2) Should systematic
differences between items count as error? Yes, when measurements from
different items will be used interchangeably or on their absolute values
(for example compared with a cut-off): "agreement"; no, when only the
subjects' relative standing matters or systematic differences are
reported separately: "consistency". (3) Will one measurement or the mean
of the k items be used in practice? "single" or "average". The
"Reliability Analysis" vignette has a table of common designs
(vignette("reliability_analysis", package = "SimplyAgree")).
Related coefficients. With complete data, Cronbach's (1951) alpha
equals ICC(C,k), the average-measure consistency ICC (McGraw & Wong,
1996): it describes the mean of the k items, not a single item. For two
methods, Lin's concordance correlation coefficient (ccc_test()) is
essentially ICC(A,1); the estimators differ only in using n rather than
n - 1 in the variances (Nickerson, 1997; Carrasco & Jover, 2003).
Default form. The default, ICC(C,1), follows Hopkins (2000) and Weir
(2005): check for systematic change between trials separately (for
example with the trials F test, or t.test() for two trials), then report
the consistency ICC, which is tied to the standard error of measurement
(). Others, such as Koo and Li (2016), recommend absolute
agreement (type = "agreement") for test-retest studies. Which form is
appropriate depends on whether systematic differences between trials
should count as error.
Average-measure forms are computed as the Spearman-Brown projection of the single-measure form to all k items. With complete data this is identical to the McGraw and Wong formulas; with missing data it keeps the estimand the reliability of the mean of all k items.
Confidence intervals are the F-based intervals of Shrout and Fleiss
(1979) and McGraw and Wong (1996, Table 7), with in each
tail. They are exact for the one-way and consistency forms and
approximate (Satterthwaite degrees of freedom) for the agreement forms.
With alternative = "greater" the interval is one-sided, [L, 1], where L
is the one-sided conf.level lower bound; with "less" it is
[-Inf, U].
Tests. By default (null.value = NULL) no test is done. The result is
the estimate and a two-sided interval when printed to the console. Testing against an ICC of 0, the
default in many programs, is rarely informative for reliability. When
null.value is given, the tests are the F tests of McGraw and Wong (1996,
Table 8) of . They are exact for the one-way and consistency
forms, and invert exactly to the interval. For the agreement forms the
test and the interval are separate Satterthwaite approximations (degrees
of freedom evaluated at for the test and at the estimate for
the interval), so with few subjects they can disagree near the interval
limits; McGraw and Wong note that their power cannot be determined
exactly.
Missing data. method = "reml" estimates the variance components from
a linear mixed model and uses degrees of freedom based on the observed
data (subject df , error df , with N the
number of observations). The intervals and p-values are then approximate
and a warning is given. In simulations with 20 subjects and 3 trials,
95% intervals covered 0.93 to 0.97 with 15% of measurements missing and
0.91 to 0.95 with 30% missing; coverage depends on the design and the
pattern of missingness. boot = TRUE is an alternative.
Bootstrap. With boot = TRUE, subjects are resampled with replacement
(keeping all of each subject's measurements) and the model is refitted.
The BCa interval uses a leave-one-subject-out jackknife for the
acceleration. The studentized interval works on the scale
and divides each replicate by its own standard error
from se_method; a normal-theory SE cannot be used because it depends
only on the degrees of freedom, which would reduce the studentized
interval to the basic interval on the z scale. The p-value is obtained by inverting the chosen
interval (as in the TOSTER package), so it is approximate and consistent
with the interval. With method = "reml" each replicate refits the mixed
model, which can be slow.
For the two-way agreement forms, resampling subjects alone would treat the
items as fixed and ignore the uncertainty from sampling the items, which
under-covered badly in simulations (0.70 to 0.80 with an ICC of 0.9). Each
replicate therefore also multiplies the item mean square by an independent
draw, its normal-theory sampling variation (a
hybrid bootstrap developed for this package). Like the F interval, this
treats the items as a random sample; when the items are fixed it is
conservative.
The bootstrap is mainly useful when the data are clearly non-normal. In
simulations (3 trials, 10 or 30 subjects, ICC 0.5 or 0.9), the F
intervals covered about 0.92 to 0.97 with normal, complete data but only
0.80 to 0.93 with skewed or heavy-tailed data. With complete data, the
studentized bootstrap with the jackknife SE covered 0.90 to 0.965 for the
consistency ICC, including 0.94 to 0.955 with skewed data and 30
subjects, and was the best method overall; for the agreement ICC it was
roughly tied with the percentile interval (0.89 to 0.955), whose lower
bound was too high less often. With 20% of measurements missing and
method = "reml", the studentized interval (which then needs the
influence SE) was unreliable at moderate ICCs (as low as 0.80), while the
percentile interval covered 0.87 to 0.96. Hence boot_ci = "auto", and a
warning when "stud" is requested with method = "reml". BCa never
improved on the percentile interval. Every method under-covered with 10
subjects, skewed data and an ICC of 0.9 (0.84 to 0.91).
Standard errors. stderr gives the standard error of
(the scale of the studentized bootstrap) and, by the
delta method, of the ICC. The jackknife (default) deletes one subject at a
time; with complete data it uses closed-form leave-one-out mean squares,
and with method = "reml" it refits the model n times. The influence
estimate is a sandwich estimate from each subject's influence on the mean
squares (with missing data, an approximation from each subject's observed
measurements); it is cheaper but too small with few subjects. The
normal-theory SE depends only on the degrees of freedom. In simulations (3
trials, 20 to 40 subjects, up to 30% missing), the jackknife SE averaged
0.89 to 1.08 times the true sampling SD, the influence SE 0.69 to 1.00
(lowest with few subjects, skewed data, or data missing at random), and
the normal-theory SE 0.87 to 1.04 with normal data but 0.60 to 0.80 with
skewed or heavy-tailed data. For the agreement forms, the jackknife and
influence SEs add the normal-theory variance of the item mean square,
treating the items as random. The studentized bootstrap needs an SE
estimated from each resample, so it cannot use "normal"; with
method = "reml" it requires "influence", because the jackknife would
refit the model n times in every replicate.
Interpreting the ICC. No qualitative labels (such as "good" or
"excellent") are given: published cutoffs disagree, and the ICC depends on
how much subjects differ from each other (Weir, 2005). To claim that
reliability exceeds a threshold, test against it, e.g.
null.value = 0.75, alternative = "greater".
A list of class c("reli_htest", "htest") containing:
statistic |
The F statistic (omitted for bootstrap results). |
parameter |
The numerator and denominator degrees of freedom (omitted for bootstrap results). |
p.value |
The p-value of the test. |
conf.int |
The confidence interval for the ICC. |
estimate |
The estimated ICC, named by its McGraw and Wong form. |
stderr |
Standard errors of the ICC ( |
null.value |
The hypothesized ICC. |
alternative |
The alternative hypothesis. |
method |
A description of the ICC and the inference. |
data.name |
A description of the data. |
fit |
The fitted variance components, mean squares and model. |
data |
The data in long format. |
settings |
The settings used, for the methods. |
boot |
Bootstrap details ( |
Carrasco, J. L., & Jover, L. (2003). Estimating the generalized concordance correlation coefficient through variance components. Biometrics, 59(4), 849-858. doi:10.1111/j.0006-341x.2003.00099.x
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334. doi:10.1007/BF02310555
Hopkins, W. G. (2000). Measures of reliability in sports medicine and science. Sports Medicine, 30(1), 1-15. doi:10.2165/00007256-200030010-00001
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155-163. doi:10.1016/j.jcm.2016.02.012
McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30-46. doi:10.1037/1082-989X.1.1.30
Nickerson, C. A. E. (1997). A note on "A concordance correlation coefficient to evaluate reproducibility". Biometrics, 53(4), 1503-1507. doi:10.2307/2533516
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420-428. doi:10.1037/0033-2909.86.2.420
Ukoumunne, O. C., Davison, A. C., Gulliford, M. C., & Chinn, S. (2003). Non-parametric bootstrap confidence intervals for the intraclass correlation coefficient. Statistics in Medicine, 22(24), 3805-3821. doi:10.1002/sim.1643
Weir, J. P. (2005). Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. Journal of Strength and Conditioning Research, 19(1), 231-240. doi:10.1519/15184.1
reli_htest-methods for summary(), plot() and check().
# Shrout & Fleiss (1979) example: 6 subjects rated by 4 judges sf <- data.frame(J1 = c(9, 6, 8, 7, 10, 6), J2 = c(2, 1, 4, 1, 5, 2), J3 = c(5, 3, 6, 2, 6, 4), J4 = c(8, 2, 8, 6, 9, 7)) icc_test(col.names = c("J1", "J2", "J3", "J4"), data = sf) # Absolute agreement, tested against 0.5 icc_test(col.names = c("J1", "J2", "J3", "J4"), data = sf, type = "agreement", null.value = 0.5, alternative = "greater") # Long data with a formula sf_long <- data.frame(id = rep(1:6, times = 4), judge = rep(c("J1", "J2", "J3", "J4"), each = 6), score = c(sf$J1, sf$J2, sf$J3, sf$J4)) res <- icc_test(score ~ judge | id, data = sf_long) summary(res)# Shrout & Fleiss (1979) example: 6 subjects rated by 4 judges sf <- data.frame(J1 = c(9, 6, 8, 7, 10, 6), J2 = c(2, 1, 4, 1, 5, 2), J3 = c(5, 3, 6, 2, 6, 4), J4 = c(8, 2, 8, 6, 9, 7)) icc_test(col.names = c("J1", "J2", "J3", "J4"), data = sf) # Absolute agreement, tested against 0.5 icc_test(col.names = c("J1", "J2", "J3", "J4"), data = sf, type = "agreement", null.value = 0.5, alternative = "greater") # Long data with a formula sf_long <- data.frame(id = rep(1:6, times = 4), judge = rep(c("J1", "J2", "J3", "J4"), each = 6), score = c(sf$J1, sf$J2, sf$J3, sf$J4)) res <- icc_test(score ~ judge | id, data = sf_long) summary(res)
Simple Agreement Analysis
jmvagree( data, method1, method2, ciWidth = 95, agreeWidth = 95, testValue = 2, CCC = TRUE, plotbland = TRUE, plotcon = FALSE, plotcheck = FALSE, prop_bias = FALSE, xlabel = "Average of Both Methods", ylabel = "Difference between Methods" )jmvagree( data, method1, method2, ciWidth = 95, agreeWidth = 95, testValue = 2, CCC = TRUE, plotbland = TRUE, plotcon = FALSE, plotcheck = FALSE, prop_bias = FALSE, xlabel = "Average of Both Methods", ylabel = "Difference between Methods" )
data |
Data |
method1 |
Name of column containing 1st Vector of data |
method2 |
Name of column containing Vector of data |
ciWidth |
a number between 50 and 99.9 (default: 95), the width of confidence intervals |
agreeWidth |
a number between 50 and 99.9 (default: 95), the width of agreement limits |
testValue |
a number specifying the limit of agreement |
CCC |
|
plotbland |
|
plotcon |
|
plotcheck |
|
prop_bias |
|
xlabel |
The label for the x-axis on the BA plot |
ylabel |
The label for the y-axis on the BA plot |
A results object containing:
results$text |
a html | ||||
results$blandtab |
a table | ||||
results$ccctab |
a table | ||||
results$plotba |
an image | ||||
results$plotcon |
an image | ||||
results$plotcheck |
an image |
Tables can be converted to data frames with asDF or as.data.frame. For example:
results$blandtab$asDF
as.data.frame(results$blandtab)
Nested/Replicate Data Agreement Analysis
jmvagreemulti( data, method1, method2, id, ciWidth = 95, agreeWidth = 95, testValue = 2, CCC = TRUE, valEq = FALSE, plotbland = FALSE, plotcon = FALSE, prop_bias = FALSE, xlabel = "Average of Both Methods", ylabel = "Difference between Methods" )jmvagreemulti( data, method1, method2, id, ciWidth = 95, agreeWidth = 95, testValue = 2, CCC = TRUE, valEq = FALSE, plotbland = FALSE, plotcon = FALSE, prop_bias = FALSE, xlabel = "Average of Both Methods", ylabel = "Difference between Methods" )
data |
Data |
method1 |
Name of column containing 1st Vector of data |
method2 |
Name of column containing Vector of data |
id |
Name of column containing subject identifier |
ciWidth |
a number between 50 and 99.9 (default: 95), the width of confidence intervals |
agreeWidth |
a number between 50 and 99.9 (default: 95), the width of agreement limits |
testValue |
a number specifying the limit of agreement |
CCC |
|
valEq |
. |
plotbland |
|
plotcon |
|
prop_bias |
|
xlabel |
The label for the x-axis on the BA plot |
ylabel |
The label for the y-axis on the BA plot |
A results object containing:
results$text |
a preformatted | ||||
results$blandtab |
a table | ||||
results$ccctab |
a table | ||||
results$plotba |
an image | ||||
results$plotcon |
an image |
Tables can be converted to data frames with asDF or as.data.frame. For example:
results$blandtab$asDF
as.data.frame(results$blandtab)
Deming Regression
jmvdeming( data, method1, method2, ciWidth = 95, testValue = 1, plotcon = FALSE, plotcheck = FALSE, weighted = FALSE, xlabel = "Method: 1", ylabel = "Method: 2" )jmvdeming( data, method1, method2, ciWidth = 95, testValue = 1, plotcon = FALSE, plotcheck = FALSE, weighted = FALSE, xlabel = "Method: 1", ylabel = "Method: 2" )
data |
Data |
method1 |
Name of column containing 1st Vector of data |
method2 |
Name of column containing Vector of data |
ciWidth |
a number between 50 and 99.9 (default: 95), the width of confidence intervals |
testValue |
Ratio of the error variances, Method 1 (x) / Method 2 (y). Default is 1. |
plotcon |
|
plotcheck |
|
weighted |
|
xlabel |
The label for the x-axis |
ylabel |
The label for the y-axis |
A results object containing:
results$text |
a html | ||||
results$demtab |
a table | ||||
results$plotcon |
an image | ||||
results$plotcheck |
an image |
Tables can be converted to data frames with asDF or as.data.frame. For example:
results$demtab$asDF
as.data.frame(results$demtab)
Reliability Analysis
jmvreli(data, vars, ciWidth = 95, desc = FALSE, plots = FALSE)jmvreli(data, vars, ciWidth = 95, desc = FALSE, plots = FALSE)
data |
the data as a data frame |
vars |
a list of the column names containing the measurements for reliability analysis. |
ciWidth |
a number between 50 and 99.9 (default: 95), the width of confidence intervals |
desc |
|
plots |
|
A results object containing:
results$text |
a html | ||||
results$icctab |
a table | ||||
results$vartab |
a table | ||||
results$plots |
an image |
Tables can be converted to data frames with asDF or as.data.frame. For example:
results$icctab$asDF
as.data.frame(results$icctab)
joint_test() tests whether the estimated intercept and slope jointly fall
within a confidence region around specified ideal values (typically
intercept = 0 and slope = 1 for method comparison studies). plot_joint()
draws that region, with the ideal point marked as enclosed or not.
joint_test(object, ...) ## S3 method for class 'simple_eiv' joint_test( object, ideal_intercept = 0, ideal_slope = 1, conf.level = NULL, test_method = c("F", "asymptotic"), method = NULL, ... ) plot_joint(x, ...) ## S3 method for class 'simple_eiv' plot_joint( x, ideal_intercept = 0, ideal_slope = 1, conf.level = NULL, test_method = c("F", "asymptotic"), show_intervals = TRUE, n_points = 100, method = NULL, ... )joint_test(object, ...) ## S3 method for class 'simple_eiv' joint_test( object, ideal_intercept = 0, ideal_slope = 1, conf.level = NULL, test_method = c("F", "asymptotic"), method = NULL, ... ) plot_joint(x, ...) ## S3 method for class 'simple_eiv' plot_joint( x, ideal_intercept = 0, ideal_slope = 1, conf.level = NULL, test_method = c("F", "asymptotic"), show_intervals = TRUE, n_points = 100, method = NULL, ... )
object, x
|
A |
... |
Additional arguments (currently unused). |
ideal_intercept |
The hypothesized intercept value (default: 0). |
ideal_slope |
The hypothesized slope value (default: 1). |
conf.level |
Confidence level for the region and test. The default,
|
test_method |
Reference distribution for the Wald test and ellipse
( |
method |
Which joint region and test. |
show_intervals |
Logical ( |
n_points |
Number of points used to draw the region ( |
For a Deming fit with error ratio and weights , let
the weighted sum of squares that Deming regression minimises (Sadler, 2010,
Eq. 11; Draper & Smith). With its minimum, the test of
is
on F(2, n - 2), and the confidence region is the set of where
does not exceed the critical value. No variance-covariance matrix
is involved; only the error ratio (not the size of the errors) matters. The
region is not an ellipse: for each slope its intercept limits have a closed
form, and its slope limits are the roots of a quadratic, so plot_joint()
draws it exactly. If the data are too imprecise for the slope to be bounded,
the test is still reported but the region is not drawn.
In simulations with normal errors (n = 10 to 100, narrow and wide ranges, error ratios 1 and 4, constant and proportional errors, and replicate data with the error ratio estimated), the Sadler test rejected a true null hypothesis 4.3-5.9% of the time at a nominal 5%. It was at least as powerful as the Wald test at the same size.
The model frame must be stored (dem_reg(model = TRUE), the default).
dem_reg(weighted = TRUE) uses fixed weights computed from the data, while
Sadler reweighted iteratively; the region uses the fitted model's weights.
method = "wald" computes the Mahalanobis distance between the estimated
coefficients and the hypothesized values, using the variance-covariance
matrix of the estimates, against 2 F(2, n - 2) (or chi-squared(2) with
test_method = "asymptotic").
For Deming regression the matrix is the delete-one jackknife. This test is
liberal in small samples: in the same simulations it rejected a true
null 7.8-9.7% of the time at n = 10, about 7.4% at n = 20 and 6.1% at
n = 50 (up to 14% with test_method = "asymptotic"), because the jackknife
variance is itself unstable. Prefer the Sadler test for Deming fits.
For Passing-Bablok regression the model must have been fitted with
replicates > 0 and se_method = "bootstrap" or "jackknife". Both held
4.3-5.4% in simulations (n = 20 to 100). The Dufey (2020) matrix
(se_method = "dufey") is not used for joint tests: it rejected a true
null 8-11% of the time.
plot_joint() returns a ggplot. joint_test() returns an object
of class htest containing:
For method = "sadler", the F statistic; for
method = "wald", the Mahalanobis distance.
Degrees of freedom. For the F-based tests, df1 and
df2; for the asymptotic test, df (always 2).
The p-value for the test.
The confidence level used.
Named vector of estimated intercept and slope.
Named vector of hypothesized intercept and slope.
Description of the alternative hypothesis.
Description of the test.
Name of the input object.
Sadler, W.A. (2010). Joint parameter confidence regions improve the power of parametric regression in method-comparison studies. Accreditation and Quality Assurance, 15, 547-554.
Draper, N.R., & Smith, H. (1966). Applied Regression Analysis. Wiley, New York.
data("reps") fit <- dem_reg(y ~ x, data = reps) joint_test(fit) plot_joint(fit) # The Wald ellipse from the jackknife variance-covariance matrix joint_test(fit, method = "wald")data("reps") fit <- dem_reg(y ~ x, data = reps) joint_test(fit) plot_joint(fit) # The Wald ellipse from the jackknife variance-covariance matrix joint_test(fit, method = "wald")
loa_lme() is deprecated, and for new code we recommend switching to
tolerance_limit(), which fits the same random-intercept model with
model = "lme", gives analytic prediction and tolerance limits (no bootstrap),
and is still under active development. It takes the two measurements (x
and y) rather than their difference and average. Supplying condition
gives each condition its own residual variance, as het_var = TRUE does here.
This function allows for the calculation of (parametric) bootstrapped limits of agreement when there are multiple observations per subject.
loa_lme( diff, avg, condition = NULL, id, data, type = c("perc", "norm", "basic"), conf.level = 0.95, agree.level = 0.95, replicates = 999, prop_bias = FALSE, het_var = FALSE )loa_lme( diff, avg, condition = NULL, id, data, type = c("perc", "norm", "basic"), conf.level = 0.95, agree.level = 0.95, replicates = 999, prop_bias = FALSE, het_var = FALSE )
diff |
Column name of the data frame that includes the difference between the 2 measurements of interest. |
avg |
Column name of the data frame that includes the average of the 2 measurements of interest. |
condition |
Column name indicating different conditions subjects were tested under. This can be left missing if there are no differing conditions to be tested. |
id |
Column name indicating the subject/participant identifier |
data |
A data frame containing the variables within the model. |
type |
A character string representing the type of bootstrap confidence intervals. Only "norm", "basic", and "perc" currently supported. Bias-corrected and accelerated, bca, is the default. See ?boot::boot.ci for more details. |
conf.level |
The confidence level required. Default is 95%. |
agree.level |
The agreement level required. Default is 95%. |
replicates |
The number of bootstrap replicates. Passed on to the boot function. Default is 999. |
prop_bias |
Logical indicator (default is FALSE) of whether proportional bias should be considered for the limits of agreement calculations. |
het_var |
Logical indicator (default is FALSE) of whether to assume homogeneity of variance in each condition. |
Returns single list with the results of the agreement analysis.
var_comp: Table of variance components
loa: A data frame of the limits of agreement including the average difference between the two sets of measurements, the standard deviation of the difference between the two sets of measurements and the lower and upper confidence limits of the difference between the two sets of measurements.
call: The matched call.
Parker, R. A., Weir, C. J., Rubio, N., Rabinovich, R., Pinnock, H., Hanley, J., McLoughan, L., Drost, E.M., Mantoani, L.C., MacNee, W., & McKinstry, B. (2016). "Application of mixed effects limits of agreement in the presence of multiple sources of variability: exemplar from the comparison of several devices to measure respiratory rate in COPD patients". PLOS One, 11(12), e0168321. doi:10.1371/journal.pone.0168321
data("recpre_long") # Instead of loa_lme(), use: tolerance_limit(data = recpre_long, x = "PM", y = "AM", id = "id", condition = "trial_condition", model = "lme")data("recpre_long") # Instead of loa_lme(), use: tolerance_limit(data = recpre_long, x = "PM", y = "AM", id = "id", condition = "trial_condition", model = "lme")
Methods defined for objects returned from the deprecated loa_lme.
## S3 method for class 'loa_mermod' print(x, ...) ## S3 method for class 'loa_mermod' plot( x, x_label = "Average of Both Methods", y_label = "Difference Between Methods", geom = "geom_point", smooth_method = NULL, smooth_se = TRUE, ... ) ## S3 method for class 'loa_mermod' check(x, ...)## S3 method for class 'loa_mermod' print(x, ...) ## S3 method for class 'loa_mermod' plot( x, x_label = "Average of Both Methods", y_label = "Difference Between Methods", geom = "geom_point", smooth_method = NULL, smooth_se = TRUE, ... ) ## S3 method for class 'loa_mermod' check(x, ...)
x |
object of class |
... |
further arguments passed through, see description of return value
for details.
|
x_label |
Label for x-axis. |
y_label |
Label for y-axis. |
geom |
String naming the type of geometry to display the data points. Default is "geom_point". Other options include: "geom_bin2d", "geom_density_2d", "geom_density_2d_filled", and "stat_density_2d". |
smooth_method |
Smoothing method (function) to use, accepts either NULL or a character vector, e.g. "lm", "glm", "gam", "loess" or a function. Default is NULL, which will not include a trend line. |
smooth_se |
Display confidence interval around smooth? |
printPrints short summary of the Limits of Agreement
plotReturns a plot of the limits of agreement
loa_mixed() is outdated, and for new code we recommend
switching to tolerance_limit(), which are easier to use, have more features,
and are still under active development.
This function allows for the calculation of bootstrapped limits of agreement when there are multiple observations per subject.
loa_mixed( diff, condition, id, data, plot.xaxis = NULL, delta, conf.level = 0.95, agree.level = 0.95, replicates = 1999, type = "bca" )loa_mixed( diff, condition, id, data, plot.xaxis = NULL, delta, conf.level = 0.95, agree.level = 0.95, replicates = 1999, type = "bca" )
diff |
column name of the data frame that includes the continuous measurement of interest. |
condition |
column name indicating different conditions subjects were tested under. |
id |
column name indicating the subject/participant identifier |
data |
A data frame containing the variables within the model. |
plot.xaxis |
column name indicating what to plot on the x.axis for the Bland-Altman plots. If this argument is missing or set to NULL then no plot will be produced. |
delta |
The threshold below which methods agree/can be considered equivalent, can be in any units. Equivalence Bound for Agreement. |
conf.level |
the confidence level required. Default is 95%. |
agree.level |
the agreement level required. Default is 95%. |
replicates |
the number of bootstrap replicates. Passed on to the boot function. Default is 1999. |
type |
A character string representing the type of bootstrap confidence intervals. Only "norm", "basic", "bca", and "perc" currently supported. Bias-corrected and accelerated, bca, is the default. See ?boot::boot.ci for more details. |
Returns single list with the results of the agreement analysis.
var_comp: Table of variance components
loa: a data frame of the limits of agreement including the average difference between the two sets of measurements, the standard deviation of the difference between the two sets of measurements and the lower and upper confidence limits of the difference between the two sets of measurements.
h0_test: Decision from hypothesis test.
bland_alt.plot: Simple Bland-Altman plot. Red line are the upper and lower bounds for shieh test; grey box is the acceptable limits (delta). If the red lines are within the grey box then the shieh test should indicate 'reject h0', or to reject the null hypothesis that this not acceptable agreement between x & y.
conf.level: Returned as input.
agree.level: Returned as input.
Parker, R. A., Weir, C. J., Rubio, N., Rabinovich, R., Pinnock, H., Hanley, J., McLoughan, L., Drost, E.M., Mantoani, L.C., MacNee, W., & McKinstry, B. (2016). "Application of mixed effects limits of agreement in the presence of multiple sources of variability: exemplar from the comparison of several devices to measure respiratory rate in COPD patients". Plos One, 11(12), e0168321. doi:10.1371/journal.pone.0168321
Methods defined for objects returned from the loa_mixed functions.
## S3 method for class 'loa_mixed_bs' print(x, ...) ## S3 method for class 'loa_mixed_bs' plot(x, ...)## S3 method for class 'loa_mixed_bs' print(x, ...) ## S3 method for class 'loa_mixed_bs' plot(x, ...)
x |
object of class |
... |
further arguments passed through, see description of return value
for details.
|
printPrints short summary of the Limits of Agreement
plotReturns a plot of the limits of agreement
Methods defined for objects returned from the agreement_limit function.
## S3 method for class 'loa' print(x, digits = 4, ...) ## S3 method for class 'loa' plot( x, geom = c("geom_point", "geom_bin2d", "geom_density_2d", "geom_density_2d_filled", "stat_density_2d"), delta = NULL, ... ) ## S3 method for class 'loa' check(x, ...)## S3 method for class 'loa' print(x, digits = 4, ...) ## S3 method for class 'loa' plot( x, geom = c("geom_point", "geom_bin2d", "geom_density_2d", "geom_density_2d_filled", "stat_density_2d"), delta = NULL, ... ) ## S3 method for class 'loa' check(x, ...)
x |
object of class |
digits |
The number of digits to print. |
... |
further arguments passed through, see description of return value
for details.
|
geom |
String naming the type of geometry to display the data points. Default is "geom_point". Other options include: "geom_bin2d", "geom_density_2d", "geom_density_2d_filled", and "stat_density_2d". |
delta |
The maximal allowable difference. |
printPrints short summary of the Limits of Agreement.
plotReturns a plot of the limits of agreement.
checkReturns plots testing the assumptions of a Bland-Altman analysis. P-values for the normality and heteroskedascity tests are provided as captions to the plot.
Estimates the minimal detectable change (MDC), also called the smallest
detectable or real difference, the "minimal difference" (Weir, 2005), or
the repeatability coefficient: the smallest change in one subject's score
between two occasions that exceeds measurement error with a given
confidence. It is a scaled standard error of measurement (SEM), so it
comes with a confidence interval and an optional test against a change
that matters, and is returned as an object of class "htest".
mdc_test( x = NULL, data = NULL, measure = NULL, item = NULL, id = NULL, col.names = NULL, mdc.level = 0.95, sided = c("two.sided", "one.sided"), k_avg = 1, conf.level = 0.95, null.value = NULL, alternative = c("less", "greater", "two.sided"), ... )mdc_test( x = NULL, data = NULL, measure = NULL, item = NULL, id = NULL, col.names = NULL, mdc.level = 0.95, sided = c("two.sided", "one.sided"), k_avg = 1, conf.level = 0.95, null.value = NULL, alternative = c("less", "greater", "two.sided"), ... )
x |
A formula of the form |
data |
A data frame containing the variables. |
measure, item, id
|
Column names (character) of the measurement, the item (trial, rater, or occasion) and the subject identifier, for data in long format. |
col.names |
Column names (character) of the items, for data in wide format (one row per subject). |
mdc.level |
The confidence that a change of at least the MDC is not
measurement error (default 0.95, the "MDC95"). This is not the
confidence level of the interval, which is |
sided |
|
k_avg |
The number of measurements averaged at each occasion
(default 1). Use it when the score used in practice is the mean of
several trials: the MDC then shrinks by |
conf.level |
Confidence level of the interval (default 0.95). |
null.value |
A change that matters, on the scale given by |
alternative |
|
... |
Further arguments passed to |
Estimate. A change score is the difference between two measurements,
each with error SEM, so its error SD is , and
with the standard normal quantile for mdc.level (1.96 for a
two-sided MDC95; Weir, 2005, eq. 13) and the number
of measurements averaged to give the score at each occasion (k_avg;
1 for single measurements). The mean of measurements
has error , so averaging
shrinks the MDC by the same factor. The SEM is from reli_err_test(),
with the same model, type and scale choices; the defaults give the
typical error of Hopkins (2000), . With two trials the
MDC equals the half-width of the limits of agreement of
agreement_limit(), which also reports the bias between trials.
Interval and test. Because the MDC is a fixed multiple of the SEM, its
interval is the SEM interval times that multiple, and the test of
null.value is the SEM test of null.value divided by
it (a test for the SEM, or the bootstrap with boot = TRUE).
With scale = "log" the MDC is in sympercents; percent gives the
equivalent percentage increase and decrease, which differ:
and .
Differences between raters or devices. The same reasoning applies to
two measurements of one subject by different items, such as two raters or
two devices, rather than two occasions: the MDC is the difference between
them that measurement error alone would exceed only 5% of the time (for
95%). When the raters are interchangeable (any rater's score is used as
it is, and raters are thought of as drawn from a pool), use
type = "agreement", so that systematic differences between raters count
as error; the consistency SEM leaves them out and understates how far
apart two raters can be. For two particular raters, the limits of
agreement of agreement_limit() report their bias and the spread around
it separately. This is the MDC, not a multiple of the standard error of
prediction (SEP): the SEP concerns a change from a subject's estimated
true score (Weir, 2005), and does not
bound the difference between two raters.
Repeat measurements. When each rater measured each subject more than
once, pass an SEM from reli_facet_test(): with facet = "intra" the
MDC is for two measurements by the same rater (on two occasions, with
occasion), with facet = "inter" and type = "agreement" for two
measurements by different raters drawn from the pool. Averaging is then
set in reli_facet_test() (k_avg, m_avg, r_avg), and k_avg here
must be 1.
An individual threshold, not a group one. The MDC applies to a single
individual's change between two measurements. It is not a threshold for
an average (group) effect. The uncertainty in a mean change shrinks with
the number of subjects (its standard error is about
), so a mean effect well below the MDC can
be estimated precisely and be clearly different from zero. Judge a mean
effect by its own confidence interval. An effect smaller than the MDC is
small relative to one person's measurement noise; that does not make it
noise itself, and it does not show that the effect is absent!
Nor should the MDC be used to count "responders". An individual's observed change mixes their true response with within-person variation from occasion to occasion, so the proportion whose change exceeds the MDC reflects measurement noise as much as real differences between people: even if every subject had exactly the same true response, some would exceed the MDC and some would not (Senn, 2004). Real differences in response are estimated from the variance of the change scores in a treated group compared with a control group, or from replicated crossover designs (Atkinson & Batterham, 2015; Senn, 2016).
The MDC and the smallest worthwhile change. The MDC describes
measurement noise. It answers the question: how large an observed change must be before it can be
distinguished from error? Whether a change matters is a separate
question that the reliability data cannot answer. There needs to be an external
anchor, such as a clinically important difference or a change that
alters competitive outcomes. Give that value as null.value (with the
default alternative = "less") to test whether the MDC is below it.
A "smallest worthwhile change" (SWC) computed from the same reliability
data cannot serve as that anchor. With SWC = 0.2 times the
between-subject SD (Hopkins, 2004),
,
a restatement of the ICC. Hopkins (2004) offers it as a default (e.g. for
fitness tests in team sports) when no relationship to a real outcome is
available, and it should be treated as such a fallback.
Hopkins's (2004) SWC for solo athletes, 0.3 times the CV, is an external
anchor, but only with the right CV: the within-athlete variability of top
athletes' performance from one competition to the next, not the test's
own typical error. Simulations (Hopkins, Hawley & Burke, 1999) showed that
an enhancement of 0.3 of that CV gives a top athlete about one extra win
in 10 events. That CV can be estimated with reli_err_test()
(scale = "log") from repeated competition results, with competitions as
items, and 0.3 times it given here as null.value. Applied to the test's
own CV instead, the rule is circular:
for every test. The anchor also rests on the one-in-10 convention and on
elite fields of evenly matched athletes; for athletes further back in the
field, Hopkins works out the worthwhile enhancement case by case.
A list of class c("reli_htest", "htest"), as for
reli_err_test(), containing:
statistic, parameter, p.value
|
The test of the SEM (equivalently
of the MDC), when a |
conf.int |
The confidence interval for the MDC. |
estimate |
The MDC, on the scale given by |
stderr |
Standard errors of the MDC and of its log. |
null.value |
The change tested against, if any. |
alternative, method, data.name
|
As for an |
percent |
With |
sem |
The SEM result the MDC was computed from. |
fit, data, settings, boot
|
As for |
Atkinson, G., & Batterham, A. M. (2015). True and false interindividual differences in the physiological response to an intervention. Experimental Physiology, 100(6), 577-588. doi:10.1113/EP085070
Hopkins, W. G. (2000). Measures of reliability in sports medicine and science. Sports Medicine, 30(1), 1-15. doi:10.2165/00007256-200030010-00001
Hopkins, W. G. (2004). How to interpret changes in an athletic performance test. Sportscience, 8, 1-7.
Hopkins, W. G., Hawley, J. A., & Burke, L. M. (1999). Design and analysis of research on sport performance enhancement. Medicine & Science in Sports & Exercise, 31(3), 472-485. doi:10.1097/00005768-199903000-00018
Senn, S. (2004). Individual response to treatment: is it a valid assumption? BMJ, 329(7472), 966-968. doi:10.1136/bmj.329.7472.966
Senn, S. (2016). Mastering variation: variance components and personalised medicine. Statistics in Medicine, 35(7), 966-977. doi:10.1002/sim.6739
Weir, J. P. (2005). Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. Journal of Strength and Conditioning Research, 19(1), 231-240. doi:10.1519/15184.1
reli_err_test() for the SEM; agreement_limit() for the
limits of agreement; reli_htest-methods for summary(), plot() and
check().
# Weir (2005) data set A: 8 subjects, 2 trials weir <- data.frame(t1 = c(146, 148, 170, 90, 157, 156, 176, 205), t2 = c(140, 152, 152, 99, 145, 153, 167, 218)) mdc_test(col.names = c("t1", "t2"), data = weir) # Can a change of 30 units be distinguished from measurement error? res <- mdc_test(col.names = c("t1", "t2"), data = weir, null.value = 30) summary(res) # From an existing SEM, one-sided, as a percentage (log scale) # in the result we can access the MDC on the percentage scale (converted from sympercent) sem <- reli_err_test(col.names = c("t1", "t2"), data = weir, scale = "log") res_log <- mdc_test(sem, sided = "one.sided") res_log$percent# Weir (2005) data set A: 8 subjects, 2 trials weir <- data.frame(t1 = c(146, 148, 170, 90, 157, 156, 176, 205), t2 = c(140, 152, 152, 99, 145, 153, 167, 218)) mdc_test(col.names = c("t1", "t2"), data = weir) # Can a change of 30 units be distinguished from measurement error? res <- mdc_test(col.names = c("t1", "t2"), data = weir, null.value = 30) summary(res) # From an existing SEM, one-sided, as a percentage (log scale) # in the result we can access the MDC on the percentage scale (converted from sympercent) sem <- reli_err_test(col.names = c("t1", "t2"), data = weir, scale = "log") res_log <- mdc_test(sem, sided = "one.sided") res_log$percent
Estimates the model-based kappa of Nelson and Edwards
(2015) and Nelson, Mitani and Edwards (2017), and please cite these papers if this function is utilized.
This is a chance-corrected measure of
exact agreement between raters who classify subjects on an ordinal scale.
It is computed from an ordinal (cumulative probit) generalized linear mixed
model with crossed random effects for subjects and raters, so it describes
the populations of raters and subjects that were sampled, and it is not
affected by the prevalence of the categories.
Requires the ordinal package.
ord_kappa_test( x = NULL, data = NULL, measure = NULL, item = NULL, id = NULL, col.names = NULL, categories = NULL, conf.level = 0.95, null.value = NULL, alternative = c("two.sided", "greater", "less"), group = NULL, group_type = c("rater", "subject") )ord_kappa_test( x = NULL, data = NULL, measure = NULL, item = NULL, id = NULL, col.names = NULL, categories = NULL, conf.level = 0.95, null.value = NULL, alternative = c("two.sided", "greater", "less"), group = NULL, group_type = c("rater", "subject") )
x |
A formula of the form |
data |
A data frame. |
measure, item, id
|
Names of the columns with the ratings, the raters, and the subjects (long data). |
col.names |
Names of the columns with each rater's ratings (wide data, one row per subject). |
categories |
All categories of the ordinal scale, in order. Defaults to
the levels of a factor, and otherwise to the sorted observed ratings.
Character ratings need |
conf.level |
Confidence level of the interval (default 0.95). |
null.value |
The hypothesized value of |
alternative |
The alternative hypothesis: |
group |
Optional name of a column with a two-level rater or subject characteristic (e.g. experienced or inexperienced raters, younger or older patients), constant within each rater (or subject). The two groups are then compared; see Details. The order of the levels is that of a factor, and otherwise sorted. Requires long data. |
group_type |
Whether |
Model. The rating of subject i by rater j falls in category c when a
latent variable falls between thresholds and
:
with subject effects and rater effects
. It is fitted by the Laplace
approximation with ordinal::clmm().
Kappa. Observed agreement is the probability that two raters
put the same subject in the same category. Chance agreement is taken at
its minimum, , reached when the thresholds split the
latent scale into C equally likely categories, and is evaluated
at those thresholds. Then
which depends only on C and
, the share of the
latent variance due to differences between subjects. is 0
when and approaches 1 as approaches 1. Because
the thresholds drop out, does not depend on how common each
category is. The observed agreement at the fitted thresholds is also
returned (fit$p0).
Inference. The standard error is from the delta method with the
covariance matrix of the variance components from the model Hessian
(Nelson et al., 2017). The interval and test are formed on the logit of
and mapped back to , so the interval stays within
[0, 1]; Nelson et al. (2017) used the Wald interval
, which can be found from estimate and
stderr. The p-value is from a z test of , where
gives the null.value of .
Design. Raters and subjects are treated as random samples from their populations. Missing ratings are allowed. At least three raters are needed to estimate the rater variance (Nelson et al., 2017); with few raters or subjects the variance components, and so the interval, are imprecise.
Comparing groups. With group, the group enters the model as a fixed
effect (shifting the thresholds, e.g. for a higher prevalence of severe
ratings in older patients), and the rater (or subject) variance may differ
between the two groups (Nelson et al., 2017). The result gives
within each group, with an interval for their difference,
and the test is the likelihood ratio test of a common variance. For rater
groups, fit$subgroups also gives between a rater from
each group; chance agreement is then minimised numerically over common
thresholds, and the fixed effect matters. Subgroup intervals are formed
on the logit scale.
Nelson et al. (2017) add a random slope for the group indicator and test its variance and correlation against a 50:50 mixture of chi-square distributions with 1 and 2 df. Because the group is constant within each rater (or subject), only the variance in each group is identified, not the intercept-slope covariance, and the Hessian of that model is singular. The equivalent model with a separate variance for each group is fitted instead: it has the same likelihood and the same kappas, with one parameter fewer. Equal variances lie inside its parameter space, so the likelihood ratio statistic is compared with a chi-square distribution with 1 df.
A list of class c("cac_htest", "htest") containing:
statistic |
The z statistic (only with a |
p.value |
The p-value (only with a |
conf.int |
The confidence interval for |
estimate |
The estimated |
stderr |
The delta-method standard error(s). |
null.value |
The hypothesized value. |
alternative |
The alternative hypothesis. |
method |
A description of the method. |
data.name |
A description of the data. |
fit |
The fitted model ( |
data |
The ratings in long format. |
settings |
The settings used, for the methods. |
Nelson, K. P., & Edwards, D. (2015). Measures of agreement between many raters for ordinal classifications. Statistics in Medicine, 34(23), 3116-3132. doi:10.1002/sim.6546
Nelson, K. P., Mitani, A. A., & Edwards, D. (2017). Assessing the influence of rater and subject characteristics on measures of agreement for ordinal ratings. Statistics in Medicine, 36(20), 3181-3199. doi:10.1002/sim.7323
cac_htest-methods for summary() and plot(); cac_test() for
Gwet's AC1/AC2, Fleiss' kappa and Krippendorff's alpha.
if (requireNamespace("ordinal", quietly = TRUE)) { # Simulated ratings of 60 subjects by 8 raters on a 5-point scale set.seed(42) d <- expand.grid(id = 1:60, rater = paste0("R", 1:8)) latent <- rnorm(60, sd = 1.5)[d$id] + rnorm(8, sd = 0.5)[d$rater] + rnorm(nrow(d)) d$rating <- cut(latent, c(-Inf, -1.5, -0.5, 0.5, 1.5, Inf), labels = FALSE) res <- ord_kappa_test(rating ~ rater | id, data = d) res summary(res) plot(res, type = "raters") # Do the first 4 raters agree better than the other 4? d$exp <- ifelse(d$rater %in% c("R1", "R2", "R3", "R4"), "exp", "inexp") res_g <- ord_kappa_test(rating ~ rater | id, data = d, group = "exp") res_g res_g$fit$subgroups }if (requireNamespace("ordinal", quietly = TRUE)) { # Simulated ratings of 60 subjects by 8 raters on a 5-point scale set.seed(42) d <- expand.grid(id = 1:60, rater = paste0("R", 1:8)) latent <- rnorm(60, sd = 1.5)[d$id] + rnorm(8, sd = 0.5)[d$rater] + rnorm(nrow(d)) d$rating <- cut(latent, c(-Inf, -1.5, -0.5, 0.5, 1.5, Inf), labels = FALSE) res <- ord_kappa_test(rating ~ rater | id, data = d) res summary(res) plot(res, type = "raters") # Do the first 4 raters agree better than the other 4? d$exp <- ifelse(d$rater %in% c("R1", "R2", "R3", "R4"), "exp", "inexp") res_g <- ord_kappa_test(rating ~ rater | id, data = d, group = "exp") res_g res_g$fit$subgroups }
A robust, nonparametric method for fitting a straight line to two-dimensional data where both variables (X and Y) are measured with error. Particularly useful for method comparison studies.
pb_reg( formula, data, id = NULL, method = c("scissors", "symmetric", "invariant"), conf.level = 0.95, weights = NULL, error.ratio = 1, replicates = 0, se_method = c("bootstrap", "jackknife", "dufey"), model = TRUE, keep_data = TRUE, ... )pb_reg( formula, data, id = NULL, method = c("scissors", "symmetric", "invariant"), conf.level = 0.95, weights = NULL, error.ratio = 1, replicates = 0, se_method = c("bootstrap", "jackknife", "dufey"), model = TRUE, keep_data = TRUE, ... )
formula |
A formula of the form |
data |
Data frame with all data. |
id |
Column with subject identifier (optional). If provided, replicate measurements are averaged within each subject before fitting. |
method |
Method for Passing-Bablok estimation. Options are:
|
conf.level |
The confidence level required. Default is 95%. |
weights |
An optional vector of case weights to be used in the fitting process. Should be NULL or a numeric vector. |
error.ratio |
Must be 1 (the default). Values other than 1 are no longer
supported and give an error; see "Measurement Error" in Details. Use
|
replicates |
Number of resamples for confidence intervals and the
variance-covariance matrix. For |
se_method |
Method used to estimate the variance-covariance matrix and confidence intervals of the coefficients. Options are:
|
model |
Logical. If TRUE (default), the model frame is stored in the returned object.
This is needed for methods like |
keep_data |
Logical indicator (TRUE/FALSE). If TRUE, intermediate calculations are returned; default is FALSE. |
... |
Additional arguments (currently unused). |
Passing-Bablok regression is a robust nonparametric method that estimates the slope as the shifted median of all possible slopes between pairs of points. The intercept is then calculated as the median of y - slope*x. This method is particularly useful when:
Both X and Y are measured with error
You want a robust method not sensitive to outliers
The relationship is assumed to be linear
X and Y are highly positively correlated
Three Passing-Bablok methods are available:
"scissors" (default): The scissors estimator (1988), most robust and scale-invariant. Uses the median of absolute values of angles.
"symmetric": The original method (1983), symmetric about the y = x line. Uses the line y = -x as the reference for partitioning points.
"invariant": Scale-invariant method (1984). First finds the median angle of slopes below the horizontal, then uses this as the reference line.
Passing-Bablok regression implicitly assumes that the ratio of the error
variances of y and x equals the squared slope (equal error variances when
the slope is 1). Earlier versions accepted error.ratio != 1 (or estimated
it from replicates via id) and weighted the pairwise slopes by it. In
simulations this biased the slope (by -0.07 to -0.43 with an error ratio of
4), the bias did not shrink with sample size, and joint tests then rejected a
true identity line 81-100% of the time with n >= 50. error.ratio != 1 is
therefore an error. When the error variances differ, use dem_reg() with
the appropriate error.ratio (or id).
With id, replicate measurements are averaged within each subject and the
subject means are analysed.
Case weights can be provided via the weights argument. Case weights allow
you to down-weight or up-weight specific observations in the analysis.
The analytical (Passing & Bablok, 1983) confidence intervals do not provide a
covariance between the intercept and slope, so a variance-covariance matrix
(needed by joint_test() and plot_joint()) is only returned when one of the
following is used:
Pairs bootstrap (se_method = "bootstrap", replicates > 0): whole
(x, y) observations are resampled with replacement and the model is refit.
The variance-covariance matrix is the covariance of the bootstrap estimates
and the confidence intervals are percentile intervals.
Delete-d jackknife (se_method = "jackknife", replicates > 0): the
model is refit on replicates random subsets that each leave out
d = floor(n/2) observations. The delete-1 jackknife is inconsistent for
median-type estimators such as Passing-Bablok; deleting d observations with
restores consistency (Shao & Wu, 1989). Confidence
intervals are Wald-type intervals using a t(n-2) quantile.
Dufey (2020) (se_method = "dufey"): an analytic, distribution-free
estimator for the equivariant ("scissors") Passing-Bablok estimator, based on
the U-statistic variance of Kendall's tau and a sandwich estimator for the
intercept. The slope interval is formed from order statistics of the pairwise
slopes and the intercept interval is Wald-type. This option is
experimental. In simulations its marginal intervals had close to nominal
coverage for n >= 20, but a joint test of intercept and slope built from
its variance-covariance matrix rejected a true null hypothesis 8-11% of the
time at a nominal 5%. joint_test() and plot_joint() therefore refuse
Dufey fits; use the pairs bootstrap or delete-d jackknife, which held
4.3-5.4% in the same simulations.
Resampling is particularly useful for:
Weighted regression (case weights)
Methods 'invariant' and 'scissors' (where analytical CI validity is uncertain)
Joint tests of intercept and slope
The method automatically:
Tests for high positive correlation using Kendall's tau
Tests for linearity using a CUSUM test
Computes confidence intervals (analytical or bootstrap)
The function returns a simple_eiv object with the following components:
coefficients: Named vector of coefficients (intercept and slope).
residuals: Residuals from the fitted model.
fitted.values: Predicted Y values.
model_table: Data frame presenting the full results from the Passing-Bablok regression.
vcov: Variance-covariance matrix for slope and intercept (if resampling
or se_method = "dufey" is used; otherwise NULL).
df.residual: Residual degrees of freedom.
call: The matched call.
terms: The terms object used.
model: The model frame.
x_vals: Original x values used in fitting.
y_vals: Original y values used in fitting.
weights: Case weights (if provided).
error.ratio: Always 1 (kept for compatibility).
conf.level: Confidence level used.
method: Character string describing the method.
method_num: Numeric method identifier (1, 2, or 3).
kendall_test: Results of Kendall's tau correlation test.
cusum_test: Results of CUSUM linearity test.
n_slopes: Number of slopes used in estimation.
boot: Resampling results (if replicates > 0 and se_method is
"bootstrap" or "jackknife").
se_method: Method actually used for standard errors ("analytic",
"bootstrap", "jackknife", or "dufey").
Passing, H., & Bablok, W. (1983). A New Biometrical Procedure for Testing the Equality of Measurements from Two Different Analytical Methods. Application of linear regression procedures for method comparison studies in Clinical Chemistry, Part I. Cclm, 21(11), 709-720. doi: 10.1515/cclm.1983.21.11.709
Passing, H., & Bablok, W. (1984). Comparison of Several Regression Procedures for Method Comparison Studies and Determination of Sample Sizes Application of linear regression procedures for method comparison studies in Clinical Chemistry, Part II. Clinical Chemistry and Laboratory Medicine, 22(6). doi: 10.1515/cclm.1984.22.6.431
Bablok, W., Passing, H., Bender, R., & Schneider, B. (1988). A General Regression Procedure for Method Transformation. Application of Linear Regression Procedures for Method Comparison Studies in Clinical Chemistry, Part III. Clinical Chemistry and Laboratory Medicine, 26(11). doi: 10.1515/cclm.1988.26.11.783
Dufey, F. (2020). Derivation of Passing-Bablok regression from Kendall's tau. The International Journal of Biostatistics, 16(2), 20190157. doi: 10.1515/ijb-2019-0157
Sen, P. K. (1968). Estimates of the regression coefficient based on Kendall's tau. Journal of the American Statistical Association, 63(324), 1379-1389. doi: 10.1080/01621459.1968.10480934
Shao, J., & Wu, C. F. J. (1989). A general theory for jackknife variance estimation. The Annals of Statistics, 17(3), 1176-1197. doi: 10.1214/aos/1176347263
# Peak flow: mini vs Wright meter (Bland & Altman, 1986) data("ba1986") # Analytic confidence intervals (scissors method, default) fit <- pb_reg(mini1 ~ wright1, data = ba1986) summary(fit) plot(fit) # Symmetric and scale-invariant methods pb_reg(mini1 ~ wright1, data = ba1986, method = "symmetric") pb_reg(mini1 ~ wright1, data = ba1986, method = "invariant") # Bootstrap (or delete-d jackknife) variance-covariance matrix, needed for # the joint test of intercept and slope; use more replicates in practice fit_boot <- pb_reg(mini1 ~ wright1, data = ba1986, replicates = 199) joint_test(fit_boot) plot_joint(fit_boot) fit_jack <- pb_reg(mini1 ~ wright1, data = ba1986, se_method = "jackknife", replicates = 199) # Case weights w <- rep(1, nrow(ba1986)) w[1:3] <- 2 pb_reg(mini1 ~ wright1, data = ba1986, weights = w, replicates = 199)# Peak flow: mini vs Wright meter (Bland & Altman, 1986) data("ba1986") # Analytic confidence intervals (scissors method, default) fit <- pb_reg(mini1 ~ wright1, data = ba1986) summary(fit) plot(fit) # Symmetric and scale-invariant methods pb_reg(mini1 ~ wright1, data = ba1986, method = "symmetric") pb_reg(mini1 ~ wright1, data = ba1986, method = "invariant") # Bootstrap (or delete-d jackknife) variance-covariance matrix, needed for # the joint test of intercept and slope; use more replicates in practice fit_boot <- pb_reg(mini1 ~ wright1, data = ba1986, replicates = 199) joint_test(fit_boot) plot_joint(fit_boot) fit_jack <- pb_reg(mini1 ~ wright1, data = ba1986, se_method = "jackknife", replicates = 199) # Case weights w <- rep(1, nrow(ba1986)) w[1:3] <- 2 pb_reg(mini1 ~ wright1, data = ba1986, weights = w, replicates = 199)
Computes sample size, power, or other parameters for the exact method of assessing agreement between two measurement methods, as described in Shieh (2019). This method tests whether the central portion of paired differences falls within specified bounds.
power_agreement_exact( n = NULL, delta = NULL, mu = 0, sigma = NULL, p0_star = 0.95, power = NULL, alpha = 0.05, max_iter = 1000, m = 1, icc = 0 )power_agreement_exact( n = NULL, delta = NULL, mu = 0, sigma = NULL, p0_star = 0.95, power = NULL, alpha = 0.05, max_iter = 1000, m = 1, icc = 0 )
n |
Number of subject pairs (sample size), or the number of subjects
when |
delta |
Maximum allowable difference bound (half-width of tolerance interval) |
mu |
Mean of paired differences |
sigma |
Standard deviation of paired differences. With clustered data
( |
p0_star |
The coverage proportion (content) of the tolerance interval. Central proportion under null hypothesis (default = 0.95) |
power |
Target power (probability of rejecting false null) |
alpha |
Significance level (Type I error rate, default = 0.05, Confidence level = 1-alpha) |
max_iter |
Maximum iterations for gamma computation (default = 1000) |
m |
number of paired measurements per subject (default 1, independent pairs). See the "Clustered data" section. |
icc |
intraclass correlation of the paired differences (between-subject
share of the variance of the differences), in [0, 1). Ignored when |
This function implements the exact agreement test procedure of Shieh (2019) for method comparison studies. The test evaluates whether the central proportion of the distribution of paired differences lies within the interval [-delta, delta].
The null hypothesis is: H0: theta_(1-p) <= -delta or delta <= theta_p The alternative is: H1: -delta < theta_(1-p) and theta_p < delta
where p = (1 + p0_star)/2, and theta_p represents the 100p-th percentile of the paired differences.
Specify three of: n, delta, power, and sigma. The fourth will be calculated.
If mu is not specified, it defaults to 0. A sample size can only be found
when the proportion of differences within [-delta, delta] under the
alternative (p1_star) exceeds p0_star; otherwise power cannot exceed
alpha and an error is returned.
Tolerance Interval Interpretation:
The parameter p0_star represents the tolerance coverage proportion,
i.e., the proportion of the population that must fall within the specified
bounds [-delta, delta] under the null hypothesis. This is conceptually
related to tolerance intervals, but formulated as a
hypothesis test rather than an estimation problem.
The power is that of Shieh's test (reject H0 when xbar -/+ gamma \* SE lies
within [-delta, delta]; for independent pairs this test is reported by
agree_test()). It is not the probability that tolerance limits from
tolerance_limit() fall within [-delta, delta]; that decision rule is less
powerful. In simulations at a planned power of 0.80, it was about 0.66 for
independent pairs and about 0.57 to 0.61 for clustered data (m = 3,
icc = 0.1 to 0.6). Allow a larger sample if the analysis will be based on
tolerance limits.
Note: This differs from Bland-Altman's "95% limits of agreement," which are confidence intervals for the 2.5th and 97.5th percentiles, not tolerance intervals.
An object of class "power.htest", a list with components:
n |
Sample size (pairs, or subjects when |
delta |
Maximum allowable difference bound |
mu |
Mean of differences |
sigma |
Standard deviation of differences |
p0_star |
Central proportion (null hypothesis) |
p1_star |
Central proportion (alternative hypothesis) |
alpha |
Significance level |
power |
Power of the test |
critical_value |
Critical value for test statistic |
m, icc
|
The clustering inputs |
n.obs |
Total number of pairs, |
method |
Description of the method |
note |
Additional notes |
With m > 1, the test is Shieh's test with the standard error of the mean
difference taken as S \* sqrt(DEFF / (n \* m)), i.e. with the design effect
treated as known. No function in this package currently carries out that
clustered test, so treat the clustered power as a planning approximation.
When each of n subjects contributes m paired measurements (repeated
measures, or several pairs per subject), the pairs are not independent and
treating n * m pairs as independent overstates precision. Setting m and
icc accounts for this under a balanced one-way random-effects
(compound symmetry) model for the differences:
The mean difference has variance sigma^2 \* DEFF / (n \* m), with design effect DEFF = 1 + (m - 1) \* icc, so the effective sample size for the mean is n \* m / DEFF.
The total variance is estimated as S^2 = MSB / m + (m - 1) / m \* MSW, whose distribution is a weighted sum of two independent chi-square variables with n - 1 and n(m - 1) degrees of freedom. This distribution is integrated numerically in place of the single chi-square of the independent case. It is more skewed than a Satterthwaite approximation suggests, which matters when icc is large.
Under these assumptions the calculation is exact (it agrees with simulation
to Monte Carlo error), and m = 1 gives exactly the published method. The
simple design effect 1 + (m - 1) \* icc applied to the whole sample size
overstates the penalty, because the SD has a much smaller design effect,
1 + (m - 1) \* icc^2. Unbalanced designs, other correlation structures, or
several levels of nesting are not covered; use simulation for those.
icc is the ICC of the differences, which can be estimated from pilot data
with an intercept-only mixed model on the differences (e.g.
lme4::lmer(diff ~ 1 + (1 | id))), and sigma is then the square root of
the sum of the two variance components.
Shieh, G. (2019). Assessing Agreement Between Two Methods of Quantitative Measurements: Exact Test Procedure and Sample Size Calculation. Statistics in Biopharmaceutical Research, 12(3), 352-359. doi:10.1080/19466315.2019.1677495
Bland, J.M. and Altman, D.G. (2007). Agreement between methods of measurement with multiple observations per individual. Journal of Biopharmaceutical Statistics, 17(4), 571-582. doi:10.1080/10543400701329422
# Example 1: Find required sample size power_agreement_exact(delta = 7, mu = 0.5, sigma = 2.5, p0_star = 0.95, power = 0.80, alpha = 0.05) # Example 2: Calculate power for given sample size power_agreement_exact(n = 15, delta = 0.1, mu = 0.011, sigma = 0.044, p0_star = 0.80, alpha = 0.05) # Example 3: Find required delta for given power and sample size power_agreement_exact(n = 50, mu = 0, sigma = 2.5, p0_star = 0.95, power = 0.90, alpha = 0.05) # Example 4: Clustered data, 3 pairs per subject, ICC of the differences 0.15 power_agreement_exact(delta = 7, mu = 0.5, sigma = 2.5, p0_star = 0.95, power = 0.80, m = 3, icc = 0.15)# Example 1: Find required sample size power_agreement_exact(delta = 7, mu = 0.5, sigma = 2.5, p0_star = 0.95, power = 0.80, alpha = 0.05) # Example 2: Calculate power for given sample size power_agreement_exact(n = 15, delta = 0.1, mu = 0.011, sigma = 0.044, p0_star = 0.80, alpha = 0.05) # Example 3: Find required delta for given power and sample size power_agreement_exact(n = 50, mu = 0, sigma = 2.5, p0_star = 0.95, power = 0.90, alpha = 0.05) # Example 4: Clustered data, 3 pairs per subject, ICC of the differences 0.15 power_agreement_exact(delta = 7, mu = 0.5, sigma = 2.5, p0_star = 0.95, power = 0.80, m = 3, icc = 0.15)
Computes the sample size, power, maximal allowable difference, or standard deviation for a method comparison study analyzed with the Bland-Altman limits of agreement, using the method of Lu et al. (2016). Agreement is concluded when the confidence intervals of both limits of agreement fall within [-delta, delta].
power_agreement_limit( n = NULL, delta = NULL, mu = 0, sigma = NULL, power = NULL, conf.level = 0.95, agree.level = 0.95, m = 1, icc = 0 )power_agreement_limit( n = NULL, delta = NULL, mu = 0, sigma = NULL, power = NULL, conf.level = 0.95, agree.level = 0.95, m = 1, icc = 0 )
n |
Number of pairs (sample size), or the number of subjects when
|
delta |
Maximal allowable difference: the methods agree if the confidence limits of the limits of agreement lie within [-delta, delta]. |
mu |
Mean of the paired differences (default 0). |
sigma |
Standard deviation of the paired differences. With clustered
data ( |
power |
Target power. |
conf.level |
Confidence level of the confidence intervals of the limits of agreement (default 0.95). |
agree.level |
Agreement level of the limits of agreement: the proportion of differences the limits should contain (default 0.95). |
m |
number of paired measurements per subject (default 1, independent pairs). See the "Clustered data" section. |
icc |
intraclass correlation of the paired differences (between-subject
share of the variance of the differences), in [0, 1). Ignored when |
Specify three of n, delta, sigma, and power; the fourth is
calculated.
The limits of agreement are mu +/- z \* sigma, with z the
(1 + agree.level) / 2 quantile of the standard normal, and their
confidence intervals are those of Bland and Altman (1999), with standard
error sigma \* sqrt(1/n + z^2 / (2 (n - 1))). This is the interval from
agreement_limit() with loa_calc = "blandaltman". The power is Lu et al.
(2016), eq. 5:
power = 1 - T(t, n - 1, tau1) - T(t, n - 1, tau2),
where T is the CDF of the non-central t distribution, t is the
1 - (1 - conf.level) / 2 quantile of the central t distribution with
n - 1 df, tau1 = (delta - mu - z \* sigma) / SE, and
tau2 = (delta + mu - z \* sigma) / SE. Negative values are set to 0. These
are the values reported by PASS (Chapter 427), which notes that the sample
sizes in Table 1 of Lu et al. (2016), based on their approximate eq. 6, are
sometimes slightly too large.
A sample size can only be found when the true limits of agreement lie within the bounds, i.e. |mu| + z \* sigma < delta; otherwise power goes to 0 as n increases and an error is returned.
With clustered data (m > 1), the standard error of the limits uses the
effective sample size n \* m / (1 + (m - 1) \* icc) for the mean difference
and a Satterthwaite df for the SD, under a balanced compound symmetry model
(see agree_expected_half() for the model). This is an approximation;
m = 1 gives the original results.
For a power curve over a range of sample sizes, see blandPowerCurve().
An object of class "power.htest", a list with components:
n |
Sample size (pairs, or subjects when |
delta |
Maximal allowable difference |
mu |
Mean of differences |
sigma |
Standard deviation of differences |
conf.level |
Confidence level of the CIs of the limits of agreement |
agree.level |
Agreement level of the limits of agreement |
power |
Power of the procedure |
m, icc
|
The clustering inputs |
n.obs |
Total number of pairs, |
method |
Description of the method |
note |
Additional notes |
Lu, M. J., et al. (2016). Sample Size for Assessing Agreement between Two Methods of Measurement by Bland-Altman Method. The International Journal of Biostatistics, 12(2). doi:10.1515/ijb-2015-0039
Bland, J. M., and Altman, D. G. (1999). Measuring agreement in method comparison studies. Statistical Methods in Medical Research, 8(2), 135-160. doi:10.1177/096228029900800204
NCSS (PASS Sample Size Software). Chapter 427: Bland-Altman Method for Assessing Agreement in Method Comparison Studies.
blandPowerCurve(), power_agreement_exact(),
agree_expected_half(), agreement_limit()
# Sample size for 80% power (PASS Example 1: n = 60) power_agreement_limit(delta = 7, mu = 0.5, sigma = 2.5, power = 0.80) # Power at a given sample size (PASS Example 2) power_agreement_limit(n = 201, delta = 2.5, mu = 0.2, sigma = 1) # Largest SD for which 90% power is reached with 100 pairs power_agreement_limit(n = 100, delta = 7, mu = 0.5, power = 0.90) # Clustered data, 3 pairs per subject, ICC of the differences 0.15 power_agreement_limit(delta = 7, mu = 0.5, sigma = 2.5, power = 0.80, m = 3, icc = 0.15)# Sample size for 80% power (PASS Example 1: n = 60) power_agreement_limit(delta = 7, mu = 0.5, sigma = 2.5, power = 0.80) # Power at a given sample size (PASS Example 2) power_agreement_limit(n = 201, delta = 2.5, mu = 0.2, sigma = 1) # Largest SD for which 90% power is reached with 100 pairs power_agreement_limit(n = 100, delta = 7, mu = 0.5, power = 0.90) # Clustered data, 3 pairs per subject, ICC of the differences 0.15 power_agreement_limit(delta = 7, mu = 0.5, sigma = 2.5, power = 0.80, m = 3, icc = 0.15)
Computes the power of the F test of an intraclass correlation
coefficient (ICC) used by icc_test(), or solves for the number of
subjects, the number of measurements per subject, the ICC under the
alternative, or the significance level that gives a target power. Exactly
one of n, k, rho1, power and sig.level must be NULL; that one
is computed from the others, as in stats::power.t.test().
power_icc( n = NULL, k = 2, rho0 = 0, rho1 = NULL, sig.level = 0.05, power = NULL, model = c("oneway", "twoway"), unit = c("single", "average"), missing = 0, alternative = c("two.sided", "greater", "less"), method = c("exact", "approximate") )power_icc( n = NULL, k = 2, rho0 = 0, rho1 = NULL, sig.level = 0.05, power = NULL, model = c("oneway", "twoway"), unit = c("single", "average"), missing = 0, alternative = c("two.sided", "greater", "less"), method = c("exact", "approximate") )
n |
Number of subjects. |
k |
Number of measurements (items: trials, raters or occasions) per subject. When solved for, the smallest whole number of at least 2 that reaches the target is returned. |
rho0 |
The ICC under the null hypothesis (default 0). |
rho1 |
The ICC under the alternative hypothesis. |
sig.level |
Significance level (Type I error rate; default 0.05). |
power |
Power of the test (1 minus the Type II error rate). |
model |
|
unit |
|
missing |
The expected proportion of measurements (subject-by-item
cells) that will be missing, analysed with |
alternative |
|
method |
|
Exact power. For the one-way and the two-way consistency ICC with
complete data, the F ratio of the subject mean square to the error mean
square is a scaled central F variable whatever the true ICC :
with (one-way) or (two-way). The
test of (McGraw & Wong, 1996) divides the F ratio
by the same factor at , so its power is a single F
probability (Donner & Eliasziw, 1987). Non-integer n is allowed, as in
stats::power.t.test(); round up to the next whole number of subjects.
Approximate power uses the closed form of Walter, Eliasziw and Donner (1998),
with in place of for a two-sided test. Walter
et al. (and the PASS software) write n for the number of measurements per
subject and k for the number of subjects; here n is always the number of
subjects and k the number of measurements per subject. The exact method is
preferred; the approximation is provided to reproduce published tables.
Two-sided tests. When solving for rho1 with
alternative = "two.sided", the solution above rho0 is returned; use
alternative = "less" (with the significance level halved) for the one
below.
Average-measure ICCs. The average-measure ICC is the Spearman-Brown
projection of the single-measure ICC to k measurements,
, so a test about ICC(k) or ICC(C,k) is the
same F test as the test about the corresponding single-measure ICC, and
its interval is the projection of the single-measure interval. With
unit = "average" the calculations are therefore exact, not
approximations; k must be given because it defines the quantity being
planned for.
Missing measurements (approximate). With missing = p > 0, the
calculation uses the observed-data quantities that
icc_test(method = "reml") uses for incomplete data:
observations, measurements per
subject in the scale factor of the F ratio and in the interval, and error
degrees of freedom (two-way) or (one-way).
This is an approximation: the F distribution is exact only for complete
data. In simulations of the REML analysis (one-way and two-way ICCs, 15
or 40 subjects, 3 or 5 items, 15% or 30% of measurements missing
completely at random or 30% at random depending on the first
measurement; 1000 replicates per setting), the approximate power was
within 0.035 of the simulated power (Monte Carlo SE 0.016) and the
approximate median interval width within 2.2% of the simulated median.
The REML test itself was slightly liberal with 30% missing (type I error
up to 0.069 missing completely at random and 0.073 at random, for a
nominal 0.05), which the simulated power includes. A warning is given
above 30% missing. The calculation assumes each subject keeps at least
one measurement.
Planning other designs. These functions are exact only for the one-way and two-way consistency ICCs with complete, normally distributed data. They can still guide the planning of other ICC studies, as long as the direction of the error is kept in mind:
Absolute agreement ICC (ICC(A,1), and the CCC for two methods). Not supported exactly: its test has no exact F distribution, and because its denominator degrees of freedom depend on the sample, its power "cannot be determined exactly" (McGraw & Wong, 1996, footnote 3). When the items (trials, raters, methods) differ little systematically, ICC(A,1) is close to ICC(C,1) and the consistency calculation is a good guide. When they differ, the agreement test and interval also depend on the item mean square, which has only k - 1 degrees of freedom: power is lower and intervals wider than the consistency calculation says, most of all with 2 or 3 items. Treat the consistency result as a lower bound on the number of subjects; simulation would be needed for a precise answer.
Missing measurements. Use missing for measurements missing here and
there (see below). For subjects who will be lost entirely, plan for the
subjects who complete the study and recruit for an
expected dropout d.
Non-normal data. The exact calculations assume normality. With
skewed or heavy-tailed data the F-based intervals are wider and cover
less than planned (see icc_test() and reli_err_test() for
simulations), so allow a margin or plan for the bootstrap with a larger
sample.
An object of class "power.htest", printed by
stats::print.power.htest(), with elements n, k, rho0, rho1,
sig.level, power, alternative, method and note.
Donner, A., & Eliasziw, M. (1987). Sample size requirements for reliability studies. Statistics in Medicine, 6(4), 441-448. doi:10.1002/sim.4780060404
McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30-46. doi:10.1037/1082-989X.1.1.30
Walter, S. D., Eliasziw, M., & Donner, A. (1998). Sample size and optimal designs for reliability studies. Statistics in Medicine, 17(1), 101-110. doi:10.1002/(SICI)1097-0258(19980115)17:1<101::AID-SIM727>3.0.CO;2-E
precision_icc() for the sample size for a confidence interval
of a given width; icc_test() for the analysis.
# Subjects needed to show an ICC above 0.6 when it is 0.8, with 2 trials power_icc(rho0 = 0.6, rho1 = 0.8, k = 2, power = 0.8, alternative = "greater") # Power with 30 subjects and 3 trials power_icc(n = 30, k = 3, rho0 = 0.5, rho1 = 0.75, alternative = "greater", model = "twoway") # Trials needed with 20 subjects power_icc(n = 20, k = NULL, rho0 = 0.5, rho1 = 0.8, power = 0.9, alternative = "greater")# Subjects needed to show an ICC above 0.6 when it is 0.8, with 2 trials power_icc(rho0 = 0.6, rho1 = 0.8, k = 2, power = 0.8, alternative = "greater") # Power with 30 subjects and 3 trials power_icc(n = 30, k = 3, rho0 = 0.5, rho1 = 0.75, alternative = "greater", model = "twoway") # Trials needed with 20 subjects power_icc(n = 20, k = NULL, rho0 = 0.5, rho1 = 0.8, power = 0.9, alternative = "greater")
Methods defined for objects returned from the powerCurve function.
find_n(x, power = 0.8) ## S3 method for class 'powerCurve' plot(x, ...)find_n(x, power = 0.8) ## S3 method for class 'powerCurve' plot(x, ...)
x |
object of class |
power |
Level of power (value between 0 and 1) for find_n to find the sample size. |
... |
further arguments passed through, see description of return value
for details.
|
plotReturns a plot of the limits of agreement (type = 1) or concordance plot (type = 2)
find_nFind the smallest sample size at which the desired
power is achieved, for each combination of settings. If the power is not
reached within the sample sizes evaluated, N is NA (with a warning)
and the range of sample sizes must be expanded.
Plans a reliability study so that the confidence interval for an
intraclass correlation coefficient (ICC) from icc_test() is narrow
enough. Computes the assurance (the probability that the interval is no
wider than width), or solves for the number of subjects, the number of
measurements per subject, or the width. Exactly one of n, k, width
and assurance must be NULL.
precision_icc( n = NULL, k = 2, rho = NULL, width = NULL, conf.level = 0.95, assurance = 0.5, model = c("oneway", "twoway"), unit = c("single", "average"), missing = 0, method = c("exact", "approximate") )precision_icc( n = NULL, k = 2, rho = NULL, width = NULL, conf.level = 0.95, assurance = 0.5, model = c("oneway", "twoway"), unit = c("single", "average"), missing = 0, method = c("exact", "approximate") )
n |
Number of subjects. |
k |
Number of measurements (items: trials, raters or occasions) per subject. When solved for, the smallest whole number of at least 2 that reaches the target is returned. |
rho |
The anticipated ICC. |
width |
The full width of the confidence interval (upper minus
lower limit). Zou (2012) uses the half-width |
conf.level |
Confidence level of the interval (default 0.95). |
assurance |
The probability that the realised interval is no wider
than |
model |
|
unit |
|
missing |
The expected proportion of measurements (subject-by-item
cells) that will be missing, analysed with |
method |
|
Exact assurance. The two-sided F-based interval for the one-way or
two-way consistency ICC (Searle's interval; Shrout & Fleiss, 1979; McGraw
& Wong, 1996) is a function of the F ratio only, and the F ratio is a
scaled central F variable for any true ICC (see power_icc()). The
assurance is therefore the probability, under that distribution, of the F
values whose interval is no wider than width. The width is largest at
moderate F and shrinks towards both ends, so this set has two parts; the
lower part (estimates near ) is negligible in realistic
designs. Mondal et al. (2024) recommend this interval for the one-way ICC.
Approximate sample size (Zou, 2012) for a Wald interval with Fisher's
variance, half-width and assurance :
with and .
It requires assurance of at least 0.5. Mondal et al. (2024) note that
Zou used in deriving it and that another expression in his
paper (the variance of the log-F statistic, not used here) is incorrect;
the exact method avoids both issues. For a one-sided lower limit, the
assurance that the lower limit exceeds a value is the power
of the one-sided test: use power_icc(alternative = "greater").
Average-measure ICCs (unit = "average"): the interval for ICC(k) or
ICC(C,k) has limits and (with a and b the
F quantiles), so its width, , decreases with F and the
assurance is a single F probability. width is on the average-measure
scale. The approximate method is for single measures only.
Missing measurements (missing > 0): the interval width uses the
observed-data quantities described in power_icc(), an approximation
whose median widths were within 2.2% of simulated REML intervals with up
to 30% of measurements missing.
The absolute agreement ICC is not supported exactly; see "Planning other
designs" in power_icc() for using these calculations as a guide for it,
for missing data, and for non-normal data.
An object of class "power.htest", printed by
stats::print.power.htest(), with elements n, k, rho, width,
conf.level, assurance, method and note.
McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30-46. doi:10.1037/1082-989X.1.1.30
Mondal, D., Vanbelle, S., Cassese, A., & Candel, M. J. J. M. (2024). Review of sample size determination methods for the intraclass correlation coefficient in the one-way analysis of variance model. Statistical Methods in Medical Research, 33(3), 532-553. doi:10.1177/09622802231224657
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420-428. doi:10.1037/0033-2909.86.2.420
Zou, G. Y. (2012). Sample size formulas for estimating intraclass correlation coefficients with precision and assurance. Statistics in Medicine, 31(29), 3972-3981. doi:10.1002/sim.5466
power_icc(); icc_test() for the analysis.
# Subjects for a median 95% CI width of 0.2 around an ICC of 0.8, 3 trials precision_icc(rho = 0.8, k = 3, width = 0.2) # ... with 90% assurance precision_icc(rho = 0.8, k = 3, width = 0.2, assurance = 0.9) # Zou (2012), Table 1: half-width 0.1, 80% assurance precision_icc(rho = 0.6, k = 2, width = 0.2, assurance = 0.8, method = "approximate")# Subjects for a median 95% CI width of 0.2 around an ICC of 0.8, 3 trials precision_icc(rho = 0.8, k = 3, width = 0.2) # ... with 90% assurance precision_icc(rho = 0.8, k = 3, width = 0.2, assurance = 0.9) # Zou (2012), Table 1: half-width 0.1, 80% assurance precision_icc(rho = 0.6, k = 2, width = 0.2, assurance = 0.8, method = "approximate")
Estimates one measurement error statistic, the standard error of
measurement (SEM), the standard error of estimate (SEE), or the standard
error of prediction (SEP), with a confidence interval and an optional test
against an acceptable error, and returns an object of class
"htest". The defaults give the typical error of Hopkins (2000):
the SEM from a two-way model with systematic differences between items
excluded, .
reli_err_test( x = NULL, data = NULL, measure = NULL, item = NULL, id = NULL, col.names = NULL, stat = c("SEM", "SEE", "SEP"), scale = c("raw", "log", "mean"), model = c("twoway", "oneway"), type = c("consistency", "agreement"), method = c("anova", "reml"), conf.level = 0.95, null.value = NULL, alternative = c("less", "greater", "two.sided"), boot = FALSE, boot_ci = c("auto", "perc", "bca", "stud", "basic"), replicates = 1999, se_method = c("jackknife", "influence", "normal") )reli_err_test( x = NULL, data = NULL, measure = NULL, item = NULL, id = NULL, col.names = NULL, stat = c("SEM", "SEE", "SEP"), scale = c("raw", "log", "mean"), model = c("twoway", "oneway"), type = c("consistency", "agreement"), method = c("anova", "reml"), conf.level = 0.95, null.value = NULL, alternative = c("less", "greater", "two.sided"), boot = FALSE, boot_ci = c("auto", "perc", "bca", "stud", "basic"), replicates = 1999, se_method = c("jackknife", "influence", "normal") )
x |
A formula of the form |
data |
A data frame containing the variables. |
measure, item, id
|
Column names (character) of the measurement, the item (trial, rater, or occasion) and the subject identifier, for data in long format. |
col.names |
Column names (character) of the items, for data in wide format (one row per subject). |
stat |
|
scale |
The scale the statistic is reported on: |
model |
|
type |
|
method |
|
conf.level |
Confidence level of the interval (default 0.95). |
null.value |
An acceptable error to test against, on the scale given
by |
alternative |
|
boot |
If |
boot_ci |
Bootstrap interval type: |
replicates |
Number of bootstrap replicates (default 1999). |
se_method |
How the standard error of the log of the statistic is
estimated: |
Choosing model and type. Choose them as for the ICC (see
"Choosing the form" in icc_test()): type = "agreement" when
systematic differences between items count as error, "consistency"
when they are checked and reported separately. Report the error
statistic that matches the ICC.
Statistics. With the mean squares of the subjects-by-items
analysis of variance (n subjects, k items):
stat |
model |
type |
Estimate | Analytic inference |
| SEM | twoway | consistency | |
exact , df |
| SEM | twoway | agreement | |
Satterthwaite |
| SEM | oneway | agreement | |
exact , df |
| SEE | any | any | |
delta method, log scale |
| SEP | any | any | |
delta method, log scale |
The consistency SEM is the typical error of Hopkins (2000) and the
preferred by Weir (2005); the one-way SEM is the
within-subject SD of Bland and Altman. The agreement SEM adds the
variance between items to the error; with complete data it equals the
one-way SEM, but its degrees of freedom differ.
SEE and SEP (Weir, 2005, eqs. 11 and 15) use the single-measure ICC of
the same model and type (ICC(C,1), ICC(A,1) or ICC(1)). The SEE is the
standard error of a subject's estimated true score; the SEP is used to
judge whether a change from the estimated true score is real. Both depend
on how much subjects differ from each other, so the SEM is generally the
better description of measurement error.
Total SD. SD is the total SD matched to the denominator of the ICC:
for consistency, plus
for agreement. With this SD,
equals the SEM exactly, which is why there is no
separate ICC-based SEM. In mean squares (complete data):
(consistency),
(agreement), and
(one-way). It is not the SD of all the
measurements, , used by Weir (2005) and by
reli_stats(), which estimates none of these variances; so SEE and SEP
differ slightly from Weir's (for his data set A, SD 32.83 vs 31.74, SEE
7.35 vs 7.11, SEP 10.54 vs 10.20). A negative ICC estimate is treated as
zero (SEE = 0, SEP = SD).
Intervals and tests. The SEM interval is the square root of the
interval for the error variance, and the test of
null.value is the test of the variance,
. For the SEE and SEP the interval is a Wald
interval on the log scale, with a delta-method SE from the normal-theory
variances of the mean squares, and the test is the matching z test. With
a one-sided alternative the interval is one-sided. method = "reml" with
missing data uses degrees of freedom based on the observed data, as in
icc_test(), and gives a warning.
Bootstrap. boot = TRUE resamples subjects, as in icc_test(),
including the hybrid step for the two-way agreement statistics (the item
mean square of each replicate is multiplied by an independent
draw). The studentized interval works on the log
scale. The p-value inverts the interval and is approximate.
In simulations (3 trials, 10 or 30 subjects, ICC 0.5 or 0.9, normal or
skewed and heavy-tailed data, complete data), the analytic intervals
covered 0.93 to 0.97 with normal data for the consistency and one-way
statistics, but only 0.79 to 0.90 with non-normal data (the
interval for a variance is sensitive to heavy tails). The
studentized bootstrap was the best method for the consistency SEM, the SEP
and the one-way SEM (0.89 to 0.96 throughout), and BCa for the agreement
SEM and SEP (mean 0.91), hence boot_ci = "auto". The percentile
interval was too narrow with 10 subjects (as low as 0.54). For the SEE the
analytic interval was best with normal data (0.93 to 0.97); no bootstrap
interval was reliable near an ICC of 0.5, where the SEE hardly changes
with the ICC.
Agreement statistics with few items. The agreement SEM and SEP include the variance between items, which is estimated from only k - 1 degrees of freedom. When it is a large part of the error and k is small, every interval under-covers: with 3 trials and an item variance about as large as the error variance (ICC 0.9), the Satterthwaite interval covered 0.81 to 0.91 with normal data and BCa 0.91 to 0.95 (0.80 with 30 subjects and skewed data). Resampling subjects without the hybrid step covered only 0.39 to 0.63 in those settings. More items help more than more subjects.
Coefficient of variation. When the error grows with the size of the measurements, it is better expressed relative to them (Hopkins, 2000; Atkinson and Nevill, 1998; Weir, 2005). Two scales are available:
scale = "log" fits the model to the natural logarithm of the
measurements (all must be positive) and reports
the statistic, in sympercents (Cole, 2000). Because
this is a constant multiple of the log-scale statistic, the SEM keeps its
exact (or Satterthwaite) interval and test. A sympercent
converts to the conventional CV% as
; the two are nearly identical below
about 15% (e.g. 10 sympercents is a CV of 10.03%). The conversion is
monotone, so it can be applied to the interval limits too.
scale = "mean" reports statistic / grand mean (the
"pooled" CV). The interval and test use the delta method on the log
scale, adding the uncertainty in the grand mean, whose normal-theory
variance is with items fixed (consistency, one-way) and
with items random (two-way agreement).
In simulations like those below (population mean 5), the "mean"
interval covered 0.93 to 0.96 with normal data and 30 subjects (except
for the agreement SEM with few items, see below); leaving out the
uncertainty in the grand mean lowered coverage by up to 0.05. With the
bootstrap, the grand mean of each replicate of a two-way agreement
statistic also gets the item mean's normal-theory variation, as in the
hybrid step below.
null.value is on the same scale as the estimate. The log scale is
usually preferable: it treats the error as proportional to each
measurement rather than to the overall mean.
Minimal detectable change. The minimal detectable change (MDC),
smallest real difference, or repeatability coefficient is a scaled SEM:
for 95% (Weir, 2005, eq. 13). mdc_test()
computes it, with its interval and a test, from the data or from an SEM
result of this function. With two trials, agreement_limit() gives the
same quantity as the half width of the limits of agreement, with the bias
reported separately.
A list of class c("reli_htest", "htest") containing:
statistic |
The |
parameter |
The degrees of freedom of the |
p.value |
The p-value, when a test was requested. |
conf.int |
The confidence interval. |
estimate |
The estimate, named by |
stderr |
Standard errors of the estimate (named by |
null.value |
The acceptable error tested against, if any. |
alternative |
The alternative hypothesis. |
method |
A description of the statistic and the inference. |
data.name |
A description of the data. |
fit |
The fitted variance components, mean squares and model. |
data |
The data in long format. |
settings |
The settings used, for the methods. |
boot |
Bootstrap details ( |
Atkinson, G., & Nevill, A. M. (1998). Statistical methods for assessing measurement error (reliability) in variables relevant to sports medicine. Sports Medicine, 26(4), 217-238. doi:10.2165/00007256-199826040-00002
Cole, T. J. (2000). Sympercents: symmetric percentage differences on the 100 log_e scale simplify the presentation of log transformed data. Statistics in Medicine, 19(22), 3109-3125. doi:10.1002/1097-0258(20001130)19:22<3109::AID-SIM558>3.0.CO;2-F
Hopkins, W. G. (2000). Measures of reliability in sports medicine and science. Sports Medicine, 30(1), 1-15. doi:10.2165/00007256-200030010-00001
Weir, J. P. (2005). Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. Journal of Strength and Conditioning Research, 19(1), 231-240. doi:10.1519/15184.1
icc_test(); reli_htest-methods for summary(), plot() and
check().
# Weir (2005) data set A: 8 subjects, 2 trials weir <- data.frame(t1 = c(146, 148, 170, 90, 157, 156, 176, 205), t2 = c(140, 152, 152, 99, 145, 153, 167, 218)) reli_err_test(col.names = c("t1", "t2"), data = weir) # Is the typical error below 10 units? reli_err_test(col.names = c("t1", "t2"), data = weir, null.value = 10) # Typical error as a coefficient of variation (sympercents) reli_err_test(col.names = c("t1", "t2"), data = weir, scale = "log") # Standard error of prediction res <- reli_err_test(col.names = c("t1", "t2"), data = weir, stat = "SEP") summary(res)# Weir (2005) data set A: 8 subjects, 2 trials weir <- data.frame(t1 = c(146, 148, 170, 90, 157, 156, 176, 205), t2 = c(140, 152, 152, 99, 145, 153, 167, 218)) reli_err_test(col.names = c("t1", "t2"), data = weir) # Is the typical error below 10 units? reli_err_test(col.names = c("t1", "t2"), data = weir, null.value = 10) # Typical error as a coefficient of variation (sympercents) reli_err_test(col.names = c("t1", "t2"), data = weir, scale = "log") # Standard error of prediction res <- reli_err_test(col.names = c("t1", "t2"), data = weir, stat = "SEP") summary(res)
This function is experimental and may change; its arguments and name may change too even if its functionality is retained.
Estimates one reliability coefficient or one standard error of
measurement (SEM) when each subject is measured more than once by each
item (rater, device). For example, 3 operators each scanning 20
participants twice. The extra measurements separate error within a
rater from disagreement between raters, which icc_test() and
reli_err_test() cannot do with one measurement per subject and item.
The measurements can be exchangeable repeats within each subject-by-item
cell (the default), or shared occasions (occasion: every rater measured
every subject on day 1 and again on day 2). The model is the
generalizability-theory (G-study) model for that design.
It's G-study capabilities are narrow, which is purposeful at this time.
Returns an
object of class "htest" with a confidence interval and an optional
test.
reli_facet_test( x = NULL, data = NULL, measure = NULL, item = NULL, id = NULL, occasion = NULL, stat = c("ICC", "SEM"), facet = c("intra", "inter", "overall"), type = c("consistency", "agreement"), k_avg = 1, m_avg = 1, r_avg = 1, scale = c("raw", "log", "mean"), method = c("anova", "reml"), conf.level = 0.95, null.value = NULL, alternative = c("two.sided", "greater", "less"), ci_method = c("auto", "mls", "satterthwaite"), boot = FALSE, boot_ci = c("auto", "perc", "bca", "stud", "basic"), replicates = 1999, se_method = c("jackknife", "normal") )reli_facet_test( x = NULL, data = NULL, measure = NULL, item = NULL, id = NULL, occasion = NULL, stat = c("ICC", "SEM"), facet = c("intra", "inter", "overall"), type = c("consistency", "agreement"), k_avg = 1, m_avg = 1, r_avg = 1, scale = c("raw", "log", "mean"), method = c("anova", "reml"), conf.level = 0.95, null.value = NULL, alternative = c("two.sided", "greater", "less"), ci_method = c("auto", "mls", "satterthwaite"), boot = FALSE, boot_ci = c("auto", "perc", "bca", "stud", "basic"), replicates = 1999, se_method = c("jackknife", "normal") )
x |
A formula of the form |
data |
A data frame containing the variables. |
measure, item, id
|
Column names (character) of the measurement, the item (trial, rater, or occasion) and the subject identifier, for data in long format. |
occasion |
|
stat |
|
facet |
Which measurement varies: |
type |
|
k_avg, m_avg, r_avg
|
The planned score (D-study): the mean of |
scale |
For |
method |
|
conf.level |
Confidence level of the interval (default 0.95). |
null.value |
A value to test against, on the scale of the estimate.
The default, |
alternative |
|
ci_method |
The analytic interval when it is not exact: |
boot |
If |
boot_ci |
Bootstrap interval type: |
replicates |
Number of bootstrap replicates (default 1999). |
se_method |
How the standard error on the working scale is
estimated: |
When considered nested versus crossed, ask whether "measurement 2" is the same event for
every item. If each operator simply scanned each participant twice, in no
shared order, the measurements are exchangeable repeats nested in the
subject-by-item cell: leave occasion = NULL. If every operator measured
everyone on day 1 and again on day 2, the days are shared occasions:
give occasion. Treating occasions as repeats counts day-to-day shifts as
repeat error, so the intra-rater error looks larger and cannot be split
into a systematic shift (fixable) and noise. With occasion, the result
(in nested) and summary() also give what the repeat model would have
estimated.
For the model, all effects are random.
Without occasion, with n subjects, k
items and r repeats per cell,
with occasion (m occasions),
where with one measurement per subject, item and occasion the three-way
interaction and the error are not separable (the residual). Further
repeats within each subject, item and occasion are allowed and add a
separate repeat error. The ANOVA estimators are unconstrained: a negative
variance component estimate is kept (as icc_test() keeps a negative
ICC), so a coefficient can be negative. The component table in
components (and summary()) shows each component with an interval and
its share of the variance of a single measurement. The model is the
unrestricted (random facets) one used by lme4 and by McGraw and Wong
(1996); for the consistency forms the items may be regarded as fixed,
which changes the interpretation but not the computation.
Following Brennan (2001), the facet that stays the same is
part of the subject's score, and the facets that vary are error. For a
planned score that is the mean of items (k_avg),
occasions (m_avg) and repeats (r_avg), the error variance
(SEM) is:
facet |
Score | Consistency error | Agreement adds |
| intra, repeats | |
|
(nothing) |
| intra, occasions | |
|
|
| inter, repeats | |
|
|
| inter, occasions | |
|
|
| overall, occasions | |
|
|
(with further repeats within subject, item and occasion, the residual is
split and its error part is also divided by ). The coefficient
(stat = "ICC") is score / (score + error). The intra-rater coefficient
is the correlation between two measurements by the same item, pooled
over items (it assumes the same error for every item; check() shows the
within-cell SD by item). With occasions, the inter-rater coefficient is
for raters on the same occasion, what most inter-rater studies report;
"overall" is for any rater on any occasion, and its coefficients are
the generalizability () and dependability ()
coefficients of G theory. In gauge R&R terms (repeats),
is repeatability and reproducibility.
It is important to know what averaging does for this function. Without occasion and with (the
score averages as many repeats as the study had), the inter-rater
consistency coefficient is the consistency ICC of the cell means,
with , identical to
icc_test() on the cell means; it is not the reliability of a single
measurement unless r = 1. Averaging the repeats before calling
icc_test() is therefore only right when the scores used in practice
are also means of r repeats.
An estimand that is a multiple of one mean
square (an SEM) or a function of the ratio of two (a coefficient) has an
exact interval and test. Without occasion, the intra SEM (
on ) and the inter-rater consistency SEM and coefficient with
. The others are approximate:
"mls": modified large-sample intervals for linear combinations of
mean squares (Graybill & Wang, 1980; Ting et al., 1990; Burdick &
Graybill, 1992). For a coefficient , the interval holds the
values for which the MLS interval of
contains 0. The
p-value inverts the interval, so there is no test statistic.
"satterthwaite": the SEM uses a distribution with
Satterthwaite df; a coefficient uses an F test of
, the positive and
negative parts each with Satterthwaite df (McGraw & Wong's device for
ICC(A,1)), and the interval inverts it.
"auto" uses MLS, chosen in simulations (see "Simulations").
SEMs with scale = "mean" use a Wald interval on the log scale with a
delta-method SE that includes the uncertainty of the grand mean.
When wanting bootstrapped results, boot = TRUE resamples subjects, each with all its
measurements, as in icc_test(). Resampling subjects keeps the same
items and occasions, so for statistics that count systematic differences
between them as error, the mean squares of the effects without subjects
(items, occasions, items by occasions) are also multiplied by independent
draws (the hybrid step). The p-value inverts the
interval and is approximate.
When there is unbalanced data, method = "reml" fits the model with lme4 and keeps
the analytic inference, with degrees of freedom from the observed data
and the harmonic mean number of repeats in place of r, and gives a
warning.
In simulations, repeats (no occasion): with 10 or 30 subjects, 2, 3
or 5 items, 2 or 3 repeats, four mixes of variance components and normal
data (2000 data sets per setting), the exact intervals covered 0.940 to
0.951 and the MLS intervals at least 0.939 for every estimand (0.95 to
0.96 on average; slightly conservative for ), hence
ci_method = "auto" uses MLS. The Satterthwaite intervals were similar
for the intra-rater and consistency statistics but fell to 0.67 to 0.74
for the agreement SEM and coefficient with 2 items when systematic
differences between items were the main error. With REML and unbalanced
data (30 subjects, 20% of repeats missing, or 1 to 3 repeats per cell),
the MLS intervals covered 0.937 to 0.969.
Occasions (2 or 3 items, 2 or 3 occasions, one measurement per cell, four mixes including a large day effect and large rater bias, 2000 data sets per setting): the MLS intervals covered at least 0.934 with 30 subjects and 0.928 with 10, for every facet, type and statistic; the Satterthwaite intervals fell to 0.69 to 0.79 for the agreement statistics.
Warning: there are uneven tails for agreement statistics. The agreement statistics
include a component estimated from very few degrees of freedom (items,
or occasions: k - 1 or m - 1). The MLS bound that this component drives
is wide, and the other bound is close to nominal. With 30 subjects and
95% two-sided intervals, the agreement coefficients fell below their
lower limit 0 to 1% of the time and above their upper limit about 3%
(nominal 2.5% each; two-sided coverage up to 0.97 to 0.99 with
occasions), and the agreement SEMs fell above their upper limit 0 to
0.6% of the time and below their lower limit 3 to 6%. So the usual
one-sided tests are conservative: a coefficient greater than a
threshold (alternative = "greater") and an SEM less than an acceptable
error (alternative = "less"). The opposite one-sided tests can be
somewhat liberal (up to about 6% instead of 2.5% for an agreement SEM
greater than a threshold). With REML and 20% of cells missing,
MLS covered 0.922 to 0.979. Treating the occasions as exchangeable
repeats (leaving out occasion) gave intra-rater SEM intervals that
covered the true agreement SEM only 0.24 to 0.92 of the time (0.04 to
0.87 for the consistency SEM, which it overstated by 30% on average).
With skewed, heavy-tailed data every analytic interval under-covered
(0.78 to 0.90 with repeats). The studentized bootstrap was then the best
interval for every statistic except the agreement SEM, for which BCa was
best, hence boot_ci = "auto": with repeats it covered 0.90 to 0.93 for
the intra-rater and consistency statistics with 30 subjects (2 or 3
items, 2 repeats, 300 data sets of 499 replicates per setting). As in
reli_err_test(), agreement statistics with few items (or occasions)
and large systematic differences between them under-covered with every
method (0.69 to 0.85 with 2 or 3 items); more items help more than more
subjects. Resampling subjects without the hybrid step covered as little
as 0.15 there. With normal data the analytic intervals were better than
any bootstrap interval. With occasions (2 days), the bootstrap gave the
same "auto" choices and helped for the consistency statistics (0.92
to 0.93 with skewed data), but for the agreement statistics the MLS
interval was as good or better even with skewed data, and every interval
under-covered when a day effect on 2 days dominated the error.
Other software: Given the narrow range of application, I want highlight a few R packages.
VCA (variance components for deeper nesting and
laboratory precision designs), rptR (repeatability for non-Gaussian
data), gtheoryr (crossed multi-facet G-studies and D-study design
optimisation), SixSigma (ss.rr(), gauge R&R summaries) and MethComp
(method comparison with linked replicates and method-specific variances)
fit related models. This function adds intervals and tests for the
coefficients and the intra- and inter-rater SEMs, and the link to
mdc_test().
A list of class c("reli_htest", "htest") containing:
statistic, parameter
|
The F (coefficients) or |
p.value |
The p-value, when a test was requested. |
conf.int |
The confidence interval. |
estimate |
The estimate. |
stderr |
Standard errors of the estimate and on the working scale
( |
null.value, alternative, method, data.name
|
As for an |
components |
The variance components with intervals and shares. |
nested |
With |
fit |
The mean squares, degrees of freedom and fitted model. |
data |
The data in long format. |
settings |
The settings used, for the methods. |
boot |
Bootstrap details ( |
Brennan, R. L. (2001). Generalizability Theory. Springer. doi:10.1007/978-1-4757-3456-0
Burdick, R. K., & Graybill, F. A. (1992). Confidence Intervals on Variance Components. Marcel Dekker.
Graybill, F. A., & Wang, C. M. (1980). Confidence intervals on nonnegative linear combinations of variances. Journal of the American Statistical Association, 75(372), 869-873. doi:10.1080/01621459.1980.10477565
McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30-46. doi:10.1037/1082-989X.1.1.30
Ting, N., Burdick, R. K., Graybill, F. A., Jeyaratnam, S., & Lu, T.-F. C. (1990). Confidence intervals on linear combinations of variance components that are unrestricted in sign. Journal of Statistical Computation and Simulation, 35(3-4), 135-143. doi:10.1080/00949659008811240
icc_test() and reli_err_test() for one measurement per
subject and item; mdc_test(); reli_htest-methods for summary(),
plot() and check().
# 12 participants, 3 operators, 2 scans each (exchangeable repeats) set.seed(2) scans <- expand.grid(scan = 1:2, operator = c("A", "B", "C"), id = 1:12) subj <- rnorm(12, 50, 5) bias <- c(A = 0, B = 1.5, C = -1) scans$thickness <- subj[scans$id] + bias[scans$operator] + rnorm(nrow(scans), 0, 1.5) # Intra-rater (repeatability) ICC and SEM reli_facet_test(thickness ~ operator | id, data = scans) reli_facet_test(thickness ~ operator | id, data = scans, stat = "SEM") # Inter-rater agreement for a single scan by any operator res <- reli_facet_test(thickness ~ operator | id, data = scans, facet = "inter", type = "agreement") summary(res) # D-study: the mean of 2 scans by one operator reli_facet_test(thickness ~ operator | id, data = scans, facet = "inter", type = "agreement", r_avg = 2) # If scan 1 and scan 2 were shared days, the same for every operator day1 <- reli_facet_test(thickness ~ operator | id, data = scans, occasion = "scan", type = "agreement") day1 day1$nested # Smallest difference between two operators beyond measurement error mdc_test(reli_facet_test(thickness ~ operator | id, data = scans, stat = "SEM", facet = "inter", type = "agreement"))# 12 participants, 3 operators, 2 scans each (exchangeable repeats) set.seed(2) scans <- expand.grid(scan = 1:2, operator = c("A", "B", "C"), id = 1:12) subj <- rnorm(12, 50, 5) bias <- c(A = 0, B = 1.5, C = -1) scans$thickness <- subj[scans$id] + bias[scans$operator] + rnorm(nrow(scans), 0, 1.5) # Intra-rater (repeatability) ICC and SEM reli_facet_test(thickness ~ operator | id, data = scans) reli_facet_test(thickness ~ operator | id, data = scans, stat = "SEM") # Inter-rater agreement for a single scan by any operator res <- reli_facet_test(thickness ~ operator | id, data = scans, facet = "inter", type = "agreement") summary(res) # D-study: the mean of 2 scans by one operator reli_facet_test(thickness ~ operator | id, data = scans, facet = "inter", type = "agreement", r_avg = 2) # If scan 1 and scan 2 were shared days, the same for every operator day1 <- reli_facet_test(thickness ~ operator | id, data = scans, occasion = "scan", type = "agreement") day1 day1$nested # Smallest difference between two operators beyond measurement error mdc_test(reli_facet_test(thickness ~ operator | id, data = scans, stat = "SEM", facet = "inter", type = "agreement"))
Methods for objects returned by icc_test(), reli_err_test(),
reli_facet_test() and
mdc_test(). They
print as standard
"htest" objects; summary() describes the result in words,
plot() shows the data, and check() gives model diagnostics.
## S3 method for class 'reli_htest' summary(object, style = c("auto", "hypothesis", "estimation"), ...) ## S3 method for class 'summary.reli_htest' print(x, ...) ## S3 method for class 'reli_htest' plot(x, type = c("profile", "subjects", "items", "pairs", "components"), ...) ## S3 method for class 'reli_htest' check(x, ...)## S3 method for class 'reli_htest' summary(object, style = c("auto", "hypothesis", "estimation"), ...) ## S3 method for class 'summary.reli_htest' print(x, ...) ## S3 method for class 'reli_htest' plot(x, type = c("profile", "subjects", "items", "pairs", "components"), ...) ## S3 method for class 'reli_htest' check(x, ...)
style |
How |
... |
Not used. |
x, object
|
An object of class |
type |
For |
summary() describes the estimate and its interval, what was estimated,
the design, and how the inference was done, in one of two styles:
"hypothesis" (Neyman-Pearson; as in the TOSTER package): the test
decision at conf.level, i.e. whether the null
hypothesis can be rejected. Requires a p-value; without one, summary()
falls back to "estimation" with a message.
"estimation" (following Greenland and colleagues): the interval is
described as a compatibility interval, the range of values most
compatible with the data under the model, with no decision and no
. If there is a p-value, it is converted to an S-value
(surprisal), : the bits of information the data
provide against the tested hypothesis, comparable to the surprise of
getting s heads in a row from s tosses of a fair coin.
Qualitative labels such as "good" or "excellent" are never attached to an
ICC; see icc_test() for why.
Diagnostics. The two-way model assumes that subject and item effects
add. A subject-by-item interaction, for example items (raters, devices,
occasions) that differ in scale, is absorbed into the error: it lowers
the consistency ICC and inflates the SEM. check() tests for it with
Mandel's (1961) column-regression test: each item gets its own slope on
the estimated subject effects, and the k - 1 extra slopes are F-tested.
The panel plots the residuals of the additive model against the
estimated subject effect with one line per item, so items whose lines
slope are those that differ in scale. Mandel's test includes the
alternative of Tukey's (1949) one-degree-of-freedom test (scale growing
with the item mean, the multiplicative pattern) and also detects items
that differ in scale but not in location, which Tukey's test misses; for
two items the two tests are identical. The covariates are built from the
fitted additive model, as in Milliken and Graybill (1970), so the F test
remains exact with missing measurements. In simulations (30 subjects, 3
items, 3 missing cells, additive normal data), its type I error was
0.0525 at the 0.05 level. It uses the fixed-effects fit (subjects and
items as factors) even when the ICC came from REML, and it has little
power with few subjects. A rising within-subject SD against the subject
mean means the error grows with the size of the measurement, so a
relative error (reli_err_test() with scale = "log") describes it
better.
For reli_facet_test() results, check() shows the residuals of the full
model (deviations from the cell means), the additivity panel on the
subject-by-item cell means, the within-cell SD by item with Bartlett's
test of equal repeat error across items (the intra-rater coefficient
pools it over items), the within-cell SD against the cell mean, and, with
occasions, each item's mean by occasion (shifts and rater-specific drift).
summary() adds the table of variance components.
summaryAn object of class summary.reli_htest: a
list with the description in text, the style used, the S-value
(s.value, when there is a p-value) and the parts used to build it. It
prints as a paragraph.
plotA ggplot2 plot (see type).
checkDiagnostic plots combined with patchwork: normality and heteroskedasticity of the model residuals; for two-way models, additivity (residuals of the additive subjects + items model against the estimated subject effect, one line per item, with Mandel's test for non-additivity); and the within-subject SD against the subject mean (on the scale the model was fitted). See Details.
Mandel, J. (1961). Non-additivity in two-way analysis of variance. Journal of the American Statistical Association, 56(296), 878-888. doi:10.1080/01621459.1961.10482132
Milliken, G. A., & Graybill, F. A. (1970). Extensions of the general linear hypothesis model. Journal of the American Statistical Association, 65(330), 797-807. doi:10.1080/01621459.1970.10481125
Tukey, J. W. (1949). One degree of freedom for non-additivity. Biometrics, 5(3), 232-242. doi:10.2307/3001938
Greenland, S. (2019). Valid p-values behave exactly as they should: Some misleading criticisms of p-values and their resolution with S-values. The American Statistician, 73(sup1), 106-114. doi:10.1080/00031305.2018.1529625
Rafi, Z., & Greenland, S. (2020). Semantic and cognitive tools to aid statistical science: replace confidence and significance by compatibility and surprise. BMC Medical Research Methodology, 20, 244. doi:10.1186/s12874-020-01105-9
sf <- data.frame(J1 = c(9, 6, 8, 7, 10, 6), J2 = c(2, 1, 4, 1, 5, 2), J3 = c(5, 3, 6, 2, 6, 4), J4 = c(8, 2, 8, 6, 9, 7)) res <- icc_test(col.names = c("J1", "J2", "J3", "J4"), data = sf) summary(res) plot(res) plot(res, type = "subjects") check(res) # A test against a threshold, in both styles res2 <- icc_test(col.names = c("J1", "J2", "J3", "J4"), data = sf, null.value = 0.5, alternative = "greater") summary(res2) summary(res2, style = "estimation")sf <- data.frame(J1 = c(9, 6, 8, 7, 10, 6), J2 = c(2, 1, 4, 1, 5, 2), J3 = c(5, 3, 6, 2, 6, 4), J4 = c(8, 2, 8, 6, 9, 7)) res <- icc_test(col.names = c("J1", "J2", "J3", "J4"), data = sf) summary(res) plot(res) plot(res, type = "subjects") check(res) # A test against a threshold, in both styles res2 <- icc_test(col.names = c("J1", "J2", "J3", "J4"), data = sf, null.value = 0.5, alternative = "greater") summary(res2) summary(res2, style = "estimation")
reli_stats() and reli_aov() return the intraclass correlation
coefficients (ICCs), the coefficient of variation (CV), and the standard
errors of measurement (SEM), estimate (SEE) and prediction (SEP)
described by Weir (2005), all at once.
These functions are superseded: they still work, but new code should use
icc_test() for an ICC (with a test against a meaningful null value and
optional bootstrap inference) and reli_err_test() for the SEM, SEE or
SEP, including the CV via scale = "log" (sympercents) or
scale = "mean". See Details for how the two sets of functions differ.
reli_stats( measure = NULL, item = NULL, id = NULL, data, wide = FALSE, col.names = NULL, se_type = c("MSE", "ICC1", "ICC2", "ICC3", "ICC1k", "ICC2k", "ICC3k"), cv_calc = c("MSE", "residuals", "SEM"), conf.level = 0.95, other_ci = FALSE, type = c("chisq", "perc", "norm", "basic"), replicates = 1999 ) reli_aov( measure = NULL, item = NULL, id = NULL, data, wide = FALSE, col.names = NULL, se_type = c("MSE", "ICC1", "ICC2", "ICC3", "ICC1k", "ICC2k", "ICC3k"), cv_calc = c("MSE", "residuals", "SEM"), conf.level = 0.95, other_ci = FALSE, type = c("chisq", "perc", "norm", "basic"), replicates = 1999 )reli_stats( measure = NULL, item = NULL, id = NULL, data, wide = FALSE, col.names = NULL, se_type = c("MSE", "ICC1", "ICC2", "ICC3", "ICC1k", "ICC2k", "ICC3k"), cv_calc = c("MSE", "residuals", "SEM"), conf.level = 0.95, other_ci = FALSE, type = c("chisq", "perc", "norm", "basic"), replicates = 1999 ) reli_aov( measure = NULL, item = NULL, id = NULL, data, wide = FALSE, col.names = NULL, se_type = c("MSE", "ICC1", "ICC2", "ICC3", "ICC1k", "ICC2k", "ICC3k"), cv_calc = c("MSE", "residuals", "SEM"), conf.level = 0.95, other_ci = FALSE, type = c("chisq", "perc", "norm", "basic"), replicates = 1999 )
measure |
Name of column containing the measurement of interest. |
item |
Name of column containing the items. If this is a test-retest reliability study then this would indicate the time point (e.g., time1, time2, time3, etc.). |
id |
Column with subject identifier. |
data |
Data frame with all data. |
wide |
Logical value (TRUE or FALSE) indicating if data is in a "wide" format. Default is FALSE. |
col.names |
If wide is equal to TRUE then col.names is a list of the column names containing the measurements for reliability analysis. |
se_type |
Type of standard error calculation. The default, |
cv_calc |
Coefficient of variation (CV) calculation: |
conf.level |
The confidence level required. Default is 95%. The ICC intervals are two-sided (see Details). |
other_ci |
Logical value (TRUE or FALSE) indicating whether to calculate confidence intervals for the CV, SEM, SEP, and SEE. |
type |
The type of confidence intervals for the CV, SEM, SEP, and SEE:
|
replicates |
The number of bootstrap replicates. Default is 1999. |
Estimation. Both functions use the engine of icc_test():
reli_stats() fits a linear mixed model by REML (crossed random effects
for subjects and items) and allows missing measurements; reli_aov()
uses the analysis of variance and keeps only subjects measured on every
item (incomplete subjects are dropped with a message). With complete data
the two give the same results (unless a variance component is estimated
at zero). With missing data, reli_stats() uses degrees of freedom based
on the observed data and warns that its intervals are approximate.
ICCs. The six forms of Shrout and Fleiss (1979) are returned, with
the F-based intervals of McGraw and Wong (1996). The intervals are
two-sided conf.level intervals, as in those papers (and in the psych,
irr, SPSS and Stata implementations). Before version 0.4.0 the intervals
used conf.level in each tail, so each limit was a one-sided
conf.level bound and the pair covered only 1 - 2 * (1 - conf.level);
the old output at conf.level = c is reproduced by
conf.level = 1 - 2 * (1 - c) (e.g. old 0.95 = new 0.90). For a one-sided
lower bound on a single ICC, use icc_test(alternative = "greater").
SEM, SEE and SEP. With se_type = "MSE", SEM =
(Weir, 2005, preferred), and SEE = and
SEP = with ICC3. With another se_type, the
named ICC is used for all three, and SEM = . Here SD
is the SD of all the measurements, , as in
Weir (2005). reli_err_test() instead uses the SD matched to the ICC's
denominator, so its SEE and SEP differ slightly. A negative ICC is
treated as zero.
CV. The CV is the SEM (or ) divided by the grand
mean, as a proportion (printed as a percentage).
Intervals for the CV, SEM, SEE and SEP (other_ci = TRUE). With
type = "chisq", the SEM from has the exact
interval; every other statistic has a Wald interval on the
log scale with a delta-method standard error from the normal-theory
variances of the mean squares (and of the grand mean, for the CV). Before
version 0.4.0 these used intervals with the error degrees of
freedom (and Vangel's interval for the CV), which do not apply to these
statistics. The bootstrap types resample subjects (as icc_test() does,
including its hybrid step for the item mean square); before version 0.4.0
reli_stats() used a parametric bootstrap.
Returns single list with the results of the reliability analysis.
icc: Table of ICC results
lmer: The fitted model: a linear mixed model from lme4
(reli_stats()), or a linear model of subjects and items
(reli_aov())
anova: Analysis of Variance table (mean squares reconstructed from
the variance components for reli_stats())
var_comp: Table of Variance Components
n.id: Number of subjects/participants
n.item: Number of items/time points
cv: Coefficient of Variation (a one-row data frame)
SEM: Standard Error of Measurement (a one-row data frame)
SEE: Standard Error of the Estimate (a one-row data frame)
SEP: Standard Error of Prediction (a one-row data frame)
call: the matched call, with the settings used
Weir, J. P. (2005). Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. Journal of Strength and Conditioning Research, 19(1), 231-240. doi:10.1519/15184.1
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420-428. doi:10.1037/0033-2909.86.2.420
McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30-46. doi:10.1037/1082-989X.1.1.30
data('reps') reli_stats(data = reps, wide = TRUE, col.names = c("x","y")) # The recommended replacements (reps has missing measurements, so REML) icc_test(data = reps, col.names = c("x", "y"), method = "reml") reli_err_test(data = reps, col.names = c("x", "y"), method = "reml")data('reps') reli_stats(data = reps, wide = TRUE, col.names = c("x","y")) # The recommended replacements (reps has missing measurements, so REML) icc_test(data = reps, col.names = c("x", "y"), method = "reml") reli_err_test(data = reps, col.names = c("x", "y"), method = "reml")
A fake data set of a agreement study where both measures have replicates.
The data set published in the original Bland & Altman paper on agreement.
reps ba1986reps ba1986
A data frame with 20 rows with 3 variables
Subject identifier
X measurement
Y measurement
A data frame with 17 rows with 5 variables
Subject identifier
PERF measurement #1 using Wright device
PERF measurement #2 using Wright device
PERF measurement #1 using Mini device
PERF measurement #2 using Mini device
Bland, J. M., & Altman, D. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet, 327(8476), 307-310.
Methods defined for objects returned from the agree functions.
## S3 method for class 'simple_agree' print(x, ...) ## S3 method for class 'simple_agree' plot( x, type = 1, x_name = "x", y_name = "y", geom = c("geom_point", "geom_bin2d", "geom_density_2d", "geom_density_2d_filled", "stat_density_2d"), smooth_method = NULL, smooth_se = TRUE, ... ) check(x, ...) ## S3 method for class 'simple_agree' check(x, ...)## S3 method for class 'simple_agree' print(x, ...) ## S3 method for class 'simple_agree' plot( x, type = 1, x_name = "x", y_name = "y", geom = c("geom_point", "geom_bin2d", "geom_density_2d", "geom_density_2d_filled", "stat_density_2d"), smooth_method = NULL, smooth_se = TRUE, ... ) check(x, ...) ## S3 method for class 'simple_agree' check(x, ...)
x |
object of class |
... |
further arguments passed through, see description of return value
for details.
|
type |
Type of plot to output. Default (1) is Bland-Altman plot while type=2 will produce a line-of-identity plot. |
x_name |
Name/label for x values (first measurement) |
y_name |
Name/label for y values (second measurement) |
geom |
String naming the type of geometry to display the data points. Default is "geom_point". Other options include: "geom_bin2d", "geom_density_2d", "geom_density_2d_filled", and "stat_density_2d". |
smooth_method |
Smoothing method (function) to use, accepts either NULL or a character vector, e.g. "lm", "glm", "gam", "loess" or a function. Default is NULL, which will not include a trend line. |
smooth_se |
Display confidence interval around smooth? |
printPrints short summary of the Limits of Agreement
plotReturns a plot of the limits of agreement (type = 1) or concordance plot (type = 2)
checkReturns 2 plots, p_norm and p_het, testing the assumptions of a Bland-Altman analysis. P-values for the normality and heteroskedasticity tests are provided as captions to the plot.
Methods defined for objects returned from error-in-variables models (e.g., dem_reg, pb_reg).
The joint confidence region for the intercept and slope is tested with
joint_test() and plotted with plot_joint().
## S3 method for class 'simple_eiv' print(x, ...) ## S3 method for class 'simple_eiv' formula(x, ...) ## S3 method for class 'simple_eiv' model.frame(formula, data = NULL, na.action = na.pass, ...) ## S3 method for class 'simple_eiv' summary(object, ...) ## S3 method for class 'simple_eiv' confint(object, parm, level = NULL, ...) ## S3 method for class 'simple_eiv' plot( x, x_name = NULL, y_name = NULL, interval = c("none", "confidence"), level = NULL, n_points = 100, data = NULL, ... ) ## S3 method for class 'simple_eiv' check(x, data = NULL, ...) ## S3 method for class 'simple_eiv' vcov(object, ...) ## S3 method for class 'simple_eiv' coef(object, ...) ## S3 method for class 'simple_eiv' fitted(object, type = c("y", "x", "both"), data = NULL, ...) ## S3 method for class 'simple_eiv' residuals(object, type = c("optimized", "x", "y", "raw_y"), data = NULL, ...) ## S3 method for class 'simple_eiv' predict( object, newdata = NULL, interval = c("none", "confidence"), level = NULL, se.fit = FALSE, data = NULL, ... )## S3 method for class 'simple_eiv' print(x, ...) ## S3 method for class 'simple_eiv' formula(x, ...) ## S3 method for class 'simple_eiv' model.frame(formula, data = NULL, na.action = na.pass, ...) ## S3 method for class 'simple_eiv' summary(object, ...) ## S3 method for class 'simple_eiv' confint(object, parm, level = NULL, ...) ## S3 method for class 'simple_eiv' plot( x, x_name = NULL, y_name = NULL, interval = c("none", "confidence"), level = NULL, n_points = 100, data = NULL, ... ) ## S3 method for class 'simple_eiv' check(x, data = NULL, ...) ## S3 method for class 'simple_eiv' vcov(object, ...) ## S3 method for class 'simple_eiv' coef(object, ...) ## S3 method for class 'simple_eiv' fitted(object, type = c("y", "x", "both"), data = NULL, ...) ## S3 method for class 'simple_eiv' residuals(object, type = c("optimized", "x", "y", "raw_y"), data = NULL, ...) ## S3 method for class 'simple_eiv' predict( object, newdata = NULL, interval = c("none", "confidence"), level = NULL, se.fit = FALSE, data = NULL, ... )
... |
further arguments passed through. |
formula |
A simple_eiv object (for model.frame method) |
data |
Optional data frame. Required if model was fitted with |
na.action |
Function for handling NA values (default: na.pass) |
object, x
|
object of class |
parm |
A specification of which parameters are to be given confidence intervals, either a vector of numbers or a vector of names. If missing, all parameters are considered. |
level |
Confidence level for intervals (default uses the model's conf.level). |
x_name |
Name for x-axis label (optional). |
y_name |
Name for y-axis label (optional). |
interval |
Type of interval calculation. Can be "none" (default), or "confidence" |
n_points |
Number of points to use for drawing smooth interval bands (default = 100). |
type |
Type of residuals to return. Options are "optimized" (default), "x", "y", or "raw_y". |
newdata |
An optional data frame containing values of X at which to predict. If omitted, the fitted values are returned. |
se.fit |
Logical. If TRUE, standard errors of predictions are returned. Note: For Passing-Bablok regression without bootstrap, standard errors are not available and this argument is ignored with a warning. |
printPrints short summary of the EIV regression model.
summaryPrints detailed summary.
plotReturns a plot of the regression line and data.
checkReturns plots of residuals.
predictPredicts Y values for new X values.
fittedExtracts fitted values.
residualsExtracts residuals.
coefExtracts model coefficients.
vcovExtracts variance-covariance matrix.
Methods defined for objects returned from reli_stats() and
reli_aov().
## S3 method for class 'simple_reli' print(x, ...) ## S3 method for class 'simple_reli' plot(x, ...) ## S3 method for class 'simple_reli' check(x, ...)## S3 method for class 'simple_reli' print(x, ...) ## S3 method for class 'simple_reli' plot(x, ...) ## S3 method for class 'simple_reli' check(x, ...)
x |
object of class |
... |
further arguments passed through, see description of return value for details. |
printPrints a short summary of the reliability statistics.
plotReturns a plot of the data points used in the reliability analysis.
checkReturns diagnostic plots (normality and heteroskedasticity) of the model residuals.
A dataset from a study on the reliability of human body temperature at different times of day before and after exercise.
temps recpre_longtemps recpre_long
A data frame with 60 rows and 10 variables:
Subject identifier
order in which the experimental trial was completed
Environmental condition and metabolic heat production
Time of Day
Rectal temperature before the beginning of the trial
Rectal temperature at the end of the trial
Change in rectal temperature
Esophageal temperature before the beginning of the trial
Esophageal temperature at the end of the trial
Change in esophageal temperature
An object of class tbl_df (inherits from tbl, data.frame) with 30 rows and 6 columns.
Ravanelli N, Jay O. The Change in Core Temperature and Sweating Response during Exercise Are Unaffected by Time of Day within the Wake Period. Med Sci Sports Exerc. 2020 Dec 1. doi: 10.1249/MSS.0000000000002575. Epub ahead of print. PMID: 33273272.
Methods defined for objects returned from the tolerance_delta function(s).
## S3 method for class 'tolerance_delta' print(x, digits = 4, ...) ## S3 method for class 'tolerance_delta' plot( x, geom = c("geom_point", "geom_bin2d", "geom_density_2d", "geom_density_2d_filled", "stat_density_2d"), delta = NULL, ... ) ## S3 method for class 'tolerance_delta' check(x, ...)## S3 method for class 'tolerance_delta' print(x, digits = 4, ...) ## S3 method for class 'tolerance_delta' plot( x, geom = c("geom_point", "geom_bin2d", "geom_density_2d", "geom_density_2d_filled", "stat_density_2d"), delta = NULL, ... ) ## S3 method for class 'tolerance_delta' check(x, ...)
x |
object of class |
digits |
The number of digits to print. |
... |
further arguments passed through, see description of return value
for details.
|
geom |
String naming the type of geometry to display the data points. Default is "geom_point". Other options include: "geom_bin2d", "geom_density_2d", "geom_density_2d_filled", and "stat_density_2d". |
delta |
The maximal allowable difference. |
printPrints short summary of the tolerance limits, with the SD of the differences (coefficient of variation when log-transformed). If the SD differs between rows of the limits (e.g., by condition), it is shown as a column instead. For clustered models, the between- and within-subject SDs are also printed.
plotReturns a plot of the tolerance limits.
checkReturns plots testing the assumptions of the model. P-values for the normality and heteroskedasticity tests are provided as captions to the plot.
A function for calculating tolerance limits for the difference between two measurements (difference = x-y). This is a procedure that should produce results similar to the Bland-Altman limits of agreement. See vignettes for more details.
tolerance_limit( data, x, y, id = NULL, condition = NULL, time = NULL, pred_level = 0.95, tol_level = 0.95, conf_level = 0.95, tol_method = c("analytic", "boot_cal"), bound_type = c("joint", "iu"), prop_bias = FALSE, log_tf = FALSE, log_tf_display = c("ratio", "sympercent"), model = c("gls", "lme"), cor_type = c("sym", "car1", "ar1", "none"), correlation = NULL, weights = NULL, keep_model = TRUE, replicates = 999 )tolerance_limit( data, x, y, id = NULL, condition = NULL, time = NULL, pred_level = 0.95, tol_level = 0.95, conf_level = 0.95, tol_method = c("analytic", "boot_cal"), bound_type = c("joint", "iu"), prop_bias = FALSE, log_tf = FALSE, log_tf_display = c("ratio", "sympercent"), model = c("gls", "lme"), cor_type = c("sym", "car1", "ar1", "none"), correlation = NULL, weights = NULL, keep_model = TRUE, replicates = 999 )
data |
A data frame containing the variables. |
x |
Name of the column for the first measurement. |
y |
Name of the column for the second measurement. |
id |
Name of the column for the subject ID. With |
condition |
Name of the column indicating different conditions subjects were tested under. This can be left missing if there are no differing conditions to be tested. Supplying a condition adds a separate residual variance for each condition ( |
time |
Name of the column indicating the time points. Only necessary if the data is from time series or repeated measures collection. |
pred_level |
Prediction level for the prediction interval, which is also the content (the proportion of differences to be covered, beta) for the tolerance limits. Default is 95%. |
tol_level |
Confidence level (gamma) for the tolerance limits. Default is 95%. See |
conf_level |
Confidence level for the confidence interval of the bias ( |
tol_method |
Method for calculating the tolerance limits. Options are "analytic" (default) for closed-form limits and "boot_cal" (experimental) for the closed-form limits calibrated by a parametric bootstrap. The analytic limits are recommended; see details. Both target the interval set by |
bound_type |
Which claim the tolerance limits ( |
prop_bias |
Whether to include a proportional bias term in the model. Determines whether proportional bias should be considered for the prediction/tolerance limits calculations. Note that a slope of the differences on the average can appear without any true proportional bias when the two methods have unequal measurement error variances; see "Model assumptions" in details. |
log_tf |
Calculate limits of agreement using log-transformed data. |
log_tf_display |
The type of presentation for log-transformed results. The differences between methods can be displayed as a "ratio" or "sympercent". |
model |
The type of model for the differences. "gls" (default) is a marginal generalized least squares model ( |
cor_type |
The type of correlation structure. "sym" is for Compound Symmetry, "car1" is for continuous autocorrelation structure of order 1, or "ar1" for autocorrelation structure of order 1. The autoregressive options ("ar1" and "car1") have no persistent subject effect; see "Model assumptions" in details. With |
correlation |
an optional corStruct object describing the within-group correlation structure that overrides the default setting. See the documentation of corClasses for a description of the available corStruct classes. If a grouping variable is to be used, it must be specified in the form argument to the corStruct constructor. Defaults to NULL. With |
weights |
an optional varFunc object or one-sided formula describing the within-group heteroskedasticity structure that overrides the default setting. If given as a formula, it is used as the argument to varFixed, corresponding to fixed variance weights. See the documentation on varClasses for a description of the available varFunc classes. Variance covariates must use the internal column names ( |
keep_model |
Logical indicator to retain the fitted model ( |
replicates |
The number of bootstrap replicates. Passed on to the boot function. Default is 999. |
The tolerance limits calculated in this function are based on the papers by Francq & Govaerts (2016), Francq, et al. (2019), and Francq, et al. (2020). The formulas, including how the degrees of freedom are determined for clustered data, are given in vignette("agreement_analysis", package = "SimplyAgree").
The output contains three kinds of interval, which support different claims:
Prediction interval (lower.PL/upper.PL): a beta-expectation tolerance interval. It is expected to contain a single future difference, from a new subject, with probability pred_level, averaged over repeated studies. It is not a confidence statement: any single prediction interval may cover less than pred_level of the differences.
Tolerance limits, bound_type = "joint" (lower.TL/upper.TL): a beta-content, gamma-confidence tolerance interval. With confidence tol_level, at least pred_level of all differences lie within the limits. If the limits lie within a maximal allowable difference of plus or minus delta, one can conclude (at level 1 - tol_level) that at least pred_level of the differences lie within plus or minus delta (Lin, 2000). The tails do not need to be split equally.
Tolerance limits, bound_type = "iu": a one-sided tol_level bound on the lower and on the upper (1 -/+ pred_level)/2 percentile of the differences, i.e., on the limits of agreement. If both bounds lie within plus or minus delta, one can reject, at level 1 - tol_level, that either limit of agreement lies outside plus or minus delta (intersection-union test). Both bounds hold together only about 1 - 2 * (1 - tol_level) of the time, so they should not be reported as a joint tol_level interval. These target the same quantities as the default confidence bounds from agreement_limit() (numerically close but not identical, because the models differ).
The "joint" and "iu" limits test different hypotheses (coverage versus each tail), so they do not nest: the "iu" bounds can be wider than the "joint" limits, and a skewed distribution can pass one test and fail the other.
With tol_method = "analytic":
"joint" uses the approximation of Howe (1969) as described by Francq et al. (2020), with the standard error of prediction (SEP) in place of the sample standard deviation. For independent data this matches Howe's approximation exactly.
"iu" uses the exact noncentral t bound for independent data, and otherwise the MOVER bound of Zou (2013).
Both use a one-sided upper confidence bound for the residual standard deviation (SD). With independent data this is the usual chi-square bound. With compound symmetry (cor_type = "sym"), the between- and within-subject variance components are combined with the MOVER, as in agreement_limit() with data_type = "nest". With other correlation structures, the bound uses an effective degrees of freedom (SD.df) from the approximate covariance of the variance parameters.
The analytic limits are exact for independent data (without a variance function). In simulations (95% target), they were close to nominal with compound symmetry (about 0.95, with 10 to 20 subjects) and with AR(1) correlation (about 0.95). With condition (a separate residual variance per condition) combined with compound symmetry, the variance components for each condition are bounded using that condition's cluster sizes and degrees of freedom; coverage was then about 0.96. Other variance functions (weights) use the cluster sizes of the whole data set and have not been checked by simulation.
With tol_method = "boot_cal", the analytic limits are calibrated by a parametric bootstrap (Loh, 1987; Beran, 1987). New data are simulated from the fitted model (Francq et al., 2019), the model is refit to each replicate, and the analytic limits are computed for each replicate over a range of nominal confidence levels. The level is chosen at which the replicates achieve the target confidence for the fitted model:
"joint": a proportion tol_level of the replicate intervals contain at least pred_level of the fitted distribution of the differences.
"iu": separately for each bound, a proportion tol_level of the replicate bounds lie beyond the fitted limit of agreement.
The limits for the observed data are the analytic limits at the calibrated levels, which are returned as lower.TL.level and upper.TL.level. tol_method = "boot_cal" is experimental. The calibration is judged against the fitted model, so it treats the estimated correlation as known. With few subjects this adds its own error: in simulations with 10 to 15 subjects (95% target), coverage ranged from about 0.93 (compound symmetry, with or without condition) to about 0.96 (AR(1) correlation), while the analytic limits were 0.95 to 0.965 in the same settings. The analytic limits are therefore recommended; the bootstrap may be useful as a check for models whose analytic limits have not been evaluated (e.g., other variance functions or prop_bias = TRUE).
For clustered data (id supplied), all limits refer to a new difference from a new subject: the SD includes both the between- and within-subject variance.
With model = "gls" (default), the model is a marginal (generalized least squares) model fit with nlme::gls(). For the default setup (cor_type = "sym" without a variance function), compound symmetry with a non-negative correlation gives the same likelihood as a random-intercept model, and targets the marginal distribution of a single difference from a randomly chosen subject. With model = "lme", the model is a linear mixed model fit with nlme::lme() with a random intercept for each id; with compound symmetry and no variance function it gives the same fit and limits. With model = "lme", the degrees of freedom for the bias are the containment degrees of freedom (subjects - 1 for the models fit here), because the Satterthwaite approximation for lme models in emmeans can fail. The other options carry assumptions that should be checked:
Autoregressive correlation (cor_type = "ar1" or "car1"): with model = "gls", the correlation between two measurements from the same subject decays towards zero as they get further apart in time. A persistent subject-specific bias (e.g., a subject by method interaction) instead makes all measurements from that subject equally correlated. When such an effect exists, the autoregressive structures miss most of the long-range correlation, so the standard error of the bias is too small and the limits are too narrow; the bootstrap does not correct this, because it simulates from the same model. In one simulation (20 subjects with 19 measurements each, and a random subject effect), the 95% confidence interval for the bias covered the true value 74% of the time with AR(1), versus 96% with compound symmetry. With model = "lme", the serial correlation is added to the residuals on top of a random intercept, which keeps the persistent subject effect. In simulations with a random subject effect and AR(1) residuals (15 subjects, 8 measurements each; 95% target), coverage of the joint tolerance limits was about 0.89 with model = "gls", cor_type = "ar1" and about 0.95 with model = "lme", cor_type = "ar1". Use the autoregressive options with model = "gls" only when no persistent subject effect is expected. The fits can be compared with AIC() on the returned models (a REML comparison is valid because the fixed effects are the same).
Variance functions with compound symmetry (condition, or weights together with cor_type = "sym"): in gls, the correlation applies to the standardized residuals, so the covariance between two measurements from the same subject is rho * sigma_i * sigma_j. The between-subject variance therefore scales with the variance function (e.g., it is larger in a condition with a larger residual SD), rather than being common to all conditions. The marginal variance of each condition, which drives the limits, is not directly affected, but the standard error of each condition's bias and the reported variance components are. With model = "lme", the variance function applies to the residuals only, and the between-subject variance is common to all conditions. In simulations where that was true (12 subjects; 95% target), the joint tolerance limits had coverage of about 0.96 with model = "gls" and about 0.97 with model = "lme".
Clustering levels: model = "gls" has a single grouping factor (id). With model = "lme", id can name two nested levels (e.g., id = c("golfer", "club") for shots with several clubs per golfer), giving random intercepts for the outer level and for the inner level within the outer level. Setting id to the inner level alone (e.g., a golfer-by-club identifier) should be avoided: it drops the correlation across inner levels within the same outer level, which understates the uncertainty. In simulations (15 subjects with 2 to 4 settings each; 95% target), the joint tolerance limits had coverage of about 0.88 to 0.91 with the inner level as id, versus about 0.96 with nested random intercepts. Grouping by the outer level only (id = "golfer") also gave close to nominal coverage (about 0.96), because the inner-level variance is then absorbed into the residual; the nested model additionally reports the inner-level variance component (SD.nested). More than two levels are not supported.
Variance components: the SD of a single difference splits into between-subject (SD.between), inner-level (SD.nested, with two id columns), and within-subject (SD.within) components. With model = "gls" and compound symmetry these are sqrt(rho) * SD and sqrt(1 - rho) * SD; with model = "lme", SD.between is the random-intercept SD and SD.within the residual SD. They can be used, for example, for the point estimate of limits for the mean of m measurements per subject, bias +/- z * sqrt(SD.between^2 + SD.within^2 / m) (without residual correlation), or for the between-subject floor of agreement. They are point estimates only.
Proportional bias (prop_bias = TRUE): a non-zero slope of the differences on the average can appear without any true proportional bias. Whenever the two methods have unequal measurement error variances, cov(difference, average) = (var(x) - var(y)) / 2 is not zero (Bland & Altman, 1999), so the slope should not be read as proportional bias on its own. Errors-in-variables methods such as dem_reg() or pb_reg() are better suited to assessing proportional bias.
Returns single tolerance_delta class object with the results of the agreement analysis with a prediction interval and tolerance limits.
limits: A data frame containing the prediction/tolerance limits. Columns include bias (estimated mean difference), SEM (standard error of the bias), SD (residual standard deviation of a single difference at that row, from the variance function if one is in the model), SEP (standard error of prediction, sqrt(SD^2 + SEM^2)), SD.df (degrees of freedom of the residual variance), SD.upper (one-sided tol_level upper confidence bound for SD), SD.between/SD.nested/SD.within (between-subject, nested (inner level), and within-subject components of SD, under compound symmetry or with model = "lme"; NA otherwise, and SD.nested is NA without nesting), lower.CL/upper.CL (confidence limits for the bias), lower.PL/upper.PL (prediction limits), lower.TL/upper.TL (tolerance limits), and, for tol_method = "boot_cal", lower.TL.level/upper.TL.level (the calibrated nominal levels at which the analytic limits were computed).
model: The fitted gls or lme model; NULL if keep_model set to FALSE. The data (with the internal column names x, y, delta, avg, id (the row number if not supplied), and, if supplied, condition and time), correlation structure, and variance function are stored with the model, so update() and nlme::getData() can be used on it directly.
call: The matched call.
Francq, B. G., & Govaerts, B. (2016). How to regress and predict in a Bland–Altman plot? Review and contribution based on tolerance intervals and correlated‐errors‐in‐variables models. Statistics in mMdicine, 35(14), 2328-2358.
Francq, B. G., Lin, D., & Hoyer, W. (2019). Confidence, prediction, and tolerance in linear mixed models. Statistics in Medicine, 38(30), 5603-5622.
Francq, B. G., Berger, M., & Boachie, C. (2020). To tolerate or to agree: A tutorial on tolerance intervals in method comparison studies with BivRegBLS R Package. Statistics in Medicine, 39(28), 4334-4349.
Howe, W. G. (1969). Two-sided tolerance limits for normal populations—some improvements. Journal of the American Statistical Association, 64(326), 610-620.
Lin, L. I. (2000). Total deviation index for measuring individual agreement with applications in laboratory performance and bioequivalence. Statistics in Medicine, 19(2), 255-270.
Loh, W. Y. (1987). Calibrating confidence coefficients. Journal of the American Statistical Association, 82(397), 155-162.
Beran, R. (1987). Prepivoting to reduce level error of confidence sets. Biometrika, 74(3), 457-468.
Zou, G. Y. (2013). Confidence interval estimation for the Bland–Altman limits of agreement with multiple observations per individual. Statistical Methods in Medical Research, 22(6), 630-642.
data('reps') # Simple tolerance_limit(x = "x", y ="y", data = reps) # Nested tolerance_limit(x = "x", y ="y", data = reps, id = "id") # Nested, random intercept (mixed) model tolerance_limit(x = "x", y ="y", data = reps, id = "id", model = "lme") # Nested random intercepts (e.g., measurements within settings within subjects) reps2 = reps reps2$setting = rep(1:2, length.out = nrow(reps2)) tolerance_limit(x = "x", y = "y", data = reps2, id = c("id", "setting"), model = "lme")data('reps') # Simple tolerance_limit(x = "x", y ="y", data = reps) # Nested tolerance_limit(x = "x", y ="y", data = reps, id = "id") # Nested, random intercept (mixed) model tolerance_limit(x = "x", y ="y", data = reps, id = "id", model = "lme") # Nested random intercepts (e.g., measurements within settings within subjects) reps2 = reps reps2$setting = rep(1:2, length.out = nrow(reps2)) tolerance_limit(x = "x", y = "y", data = reps2, id = c("id", "setting"), model = "lme")