Criterion-dependent diagnostics
Criterion dependent diagnostics usually rely on some arbitrary decision criterion (specified a priori) to decide whether a given patient belongs to one classification state or another (e.g., knowledgeable vs unknowledgeable). However, as we have seen in the present report, individuals with scores near a diagnostic threshold may not present consistently above or below that threshold upon re-sampling or re-testing. Therefore, we suggest that researchers take at least one of the following easily derived and intuitive methods into account when conducting criterion-dependent analyses: 1) the 95% confidence interval around the mean, or 2) the probability of each classification state. We note that neither of these options are mutually exclusive and can be used in conjunction to make more informed decisions.
The 95% confidence interval around the BSITER score can easily be calculated for a subject using the rBS method and deriving +/- 2 standard deviations11Note, as discussed by Luck et al., (2021) and Efron & Tibshirani (1994), the standard deviation of the bootstrap mean estimates the standard error, which is why +/- 2 standard deviations produces the 95% CI in this example, and not +/- 2 standard error . from the mean of that subject’s BSITER scores. We propose an intuitive rule for determining patient diagnoses using the 95% CI: if the patient’s 95% CI overlaps with the diagnostic threshold, then that patient is labeled as “indeterminate”. One can only make a diagnosis with acceptable statistical certainty if the participant’s 95% CI does not overlap with the diagnostic cutpoint. Table 1 showcases several illustrative examples of participant results using this method.