Limitations & Future directions
We do acknowledge that the analyses presented in this article have some limitations. First, we used data from the Rosenfeld Lab only, so these results may not apply to other labs using other equipment. Further analyses performed on data from different laboratories could not only verify the utility of the presented approach but additionally may or may not also show a difference in BSITER score variability between laboratories, and ultimately could allow for better quality control and understanding of diagnostic metrics in the field of concealed information detection.
Second, we analyzed only “simple guilty” participants - knowledgeable subjects who did not use countermeasure techniques. Therefore, determining the BSITER score variability among other groups, such as innocent subjects or countermeasure users is not possible at this time. Performing the rBS procedure with data from innocent or countermeasure participants is required to provide information about the reliability and repeatability of the classification results in these groups.
Third, the sample size used in these analyses was limited to 81 participants. Although we acknowledge that our results may not generalize from the current sample, we believe that our sample size is not small in the case of an ERP study. Nevertheless, larger and more diverse samples are needed to further support arguments on the utility of the rBS approach in diagnostic psychophysiology.
As we mentioned earlier, in traditional concealed information detection studies, an individual’s diagnosis was based on a single-point result (i.e., one BSITER score) without noting its variability. The rBS method could be a new standard in each future P300-CIT study because it provides information about the stability of the obtained result and the reliability of the classification.
Future studies and analyses should evaluate whether BSITER score variability could differ between protocols (e.g., standard 3SP vs. CTP or single probe vs. multiple probe protocols) or between experimental conditions (e.g., motivated vs. unmotivated participants). The rBS method could also provide new indicators for individual diagnosis. Apart from calculating the mean BSITER score and its confidence interval, rBS allows the analysis of the distribution of eg. 100 BSITER scores for every individual, from which many statistics can be derived. For instance, we can hypothesize that the distribution of BSITER scores should be normally distributed in the case of innocent individuals - since they are unknowledgeable, their mean BSITER score should be around 50% (for innocents, the probe is just another irrelevant stimulus, see eg. Meixner and Rosenfeld, 2011). However, for guilty individuals, since their expected mean BSITER score is close to the boundary (close to or above 85-90%), we can also expect that the distribution of their single BSITER scores will be leptokurtic and left skewed. Therefore, the normality test could be another metric employed in diagnosis. This is one example of an empirical question that could be addressed in future studies and analyses.