Criterion-dependent diagnostics
Criterion dependent diagnostics usually rely on some arbitrary decision
criterion (specified a priori) to decide whether a given patient belongs
to one classification state or another (e.g., knowledgeable vs
unknowledgeable). However, as we have seen in the present report,
individuals with scores near a diagnostic threshold may not present
consistently above or below that threshold upon re-sampling or
re-testing. Therefore, we suggest that researchers take at least one of
the following easily derived and intuitive methods into account when
conducting criterion-dependent analyses: 1) the 95% confidence interval
around the mean, or 2) the probability of each classification state. We
note that neither of these options are mutually exclusive and can be
used in conjunction to make more informed decisions.
The 95% confidence interval around the BSITER score can easily be
calculated for a subject using the rBS method and deriving +/- 2
standard deviations11Note, as discussed by Luck et al., (2021)
and Efron & Tibshirani (1994), the standard deviation of the
bootstrap mean estimates the standard error, which is why +/- 2
standard deviations produces the 95% CI in this example, and
not +/- 2 standard error . from the mean of that subject’s
BSITER scores. We propose an intuitive rule for determining patient
diagnoses using the 95% CI: if the patient’s 95% CI overlaps with the
diagnostic threshold, then that patient is labeled as “indeterminate”.
One can only make a diagnosis with acceptable statistical certainty if
the participant’s 95% CI does not overlap with the diagnostic cutpoint.
Table 1 showcases several illustrative examples of participant results
using this method.