Limitations & Future directions
We do acknowledge that the analyses presented in this article have some
limitations. First, we used data from the Rosenfeld Lab only, so these
results may not apply to other labs using other equipment. Further
analyses performed on data from different laboratories could not only
verify the utility of the presented approach but additionally may or may
not also show a difference in BSITER score variability between
laboratories, and ultimately could allow for better quality control and
understanding of diagnostic metrics in the field of concealed
information detection.
Second, we analyzed only “simple guilty” participants - knowledgeable
subjects who did not use countermeasure techniques. Therefore,
determining the BSITER score variability among other groups, such as
innocent subjects or countermeasure users is not possible at this time.
Performing the rBS procedure with data from innocent or countermeasure
participants is required to provide information about the reliability
and repeatability of the classification results in these groups.
Third, the sample size used in these analyses was limited to 81
participants. Although we acknowledge that our results may not
generalize from the current sample, we believe that our sample size is
not small in the case of an ERP study. Nevertheless, larger and more
diverse samples are needed to further support arguments on the utility
of the rBS approach in diagnostic psychophysiology.
As we mentioned earlier, in traditional concealed information detection
studies, an individual’s diagnosis was based on a single-point result
(i.e., one BSITER score) without noting its variability. The rBS method
could be a new standard in each future P300-CIT study because it
provides information about the stability of the obtained result and the
reliability of the classification.
Future studies and analyses should evaluate whether BSITER score
variability could differ between protocols (e.g., standard 3SP vs. CTP
or single probe vs. multiple probe protocols) or between experimental
conditions (e.g., motivated vs. unmotivated participants). The rBS
method could also provide new indicators for individual diagnosis. Apart
from calculating the mean BSITER score and its confidence interval, rBS
allows the analysis of the distribution of eg. 100 BSITER scores for
every individual, from which many statistics can be derived. For
instance, we can hypothesize that the distribution of BSITER scores
should be normally distributed in the case of innocent individuals -
since they are unknowledgeable, their mean BSITER score should be around
50% (for innocents, the probe is just another irrelevant stimulus, see
eg. Meixner and Rosenfeld, 2011). However, for guilty individuals, since
their expected mean BSITER score is close to the boundary (close to or
above 85-90%), we can also expect that the distribution of their single
BSITER scores will be leptokurtic and left skewed. Therefore, the
normality test could be another metric employed in diagnosis. This is
one example of an empirical question that could be addressed in future
studies and analyses.