Selecting optimal models using sequential approaches
Out of the 11 threshold dependent omission rates Maxent produces in its
output we used “10th percentile training presence
test omission” (hereafter ‘percentile OR’) and “balance training
omission, predicted area and threshold values test omission” (hereafter
‘balance OR’) for the sequential model selection approaches. We chose
percentile OR (Radosavljevic and Anderson, 2014; Galante et al. 2018)
over the “minimum training presence test omission” (Shcheglovitova and
Anderson, 2013; Radosavljevic and Anderson, 2014) since the latter is
more sensitive to extreme localities and over predicts when calibration
localities are many (Radosavljevic and Anderson, 2014). We used balance
OR to assess utility of a new thresholding rule and its OR in selecting
optimal model. Through different sequential combinations of the two ORs,
AUCTEST and AUCDIFF we formulated four
sequential approaches. They were: (i) sequential combination of
percentile OR followed by AUCTEST (hereafter
ORTEST_PER), (ii) sequential
combination of balance OR followed by AUCTEST (hereafter
ORTEST_BAL), (iii) sequential combination of percentile
OR followed by AUCDIFF and then by
AUCTEST (hereafter AUCDIFF_PER), and
(iv) sequential combination of balance OR followed by
AUCDIFF and then by AUCTEST (hereafter
AUCDIFF_BAL) approaches.
We used composite models instead of the jackknife iterations (Galante et
al., 2018) for each RM-FC combination to select the optimal model.
However, Maxent averages all the jackknife iterations to produce the
composite model irrespective of whether some individual jackknife models
have good model discrimination
(AUCTEST>.5), marginal discrimination
(AUCTEST<.5) or no discrimination at all
(AUCTEST=.5) (Figure S1). When the composite models are
comprised of jackknife models with no discrimination they would have
lower average ORs since Maxent assigns zero OR to the models with no
discriminatory power, and thereby favours these as optimal models.
Therefore, we first sorted composite models into four hierarchical
groups beginning with (i) the composite models with all jackknife
iterations with AUCTEST>.5, (ii) followed
by ones with some jackknife iteration models with
AUCTEST<.5, (iii) then with some jackknife
iteration models with AUCTEST=.5 and (iv) ended with
composite models with all their jackknife iteration models having
AUCTEST=.5.
Following the above hierarchical groups, we then ranked the ORs,
AUCDIFF and AUCTEST of the composite
models. We accorded the highest rank to the models with the lowest OR
since ORs higher than the theoretically expected value indicate
overfitting (Radosavljevic and Anderson 2014). Similarly, we accorded
the highest rank to the models with the lowest AUCDIFFsince less overfitting models are expected to have lower
AUCDIFF (Warren and Seifert 2011, Radosavljevic and
Anderson 2014). Here, we also considered negative
AUCDIFF as equal to zero, the lowest
AUCDIFF for model selection (Muscarella et al. 2014),
though we used raw values for general analysis. For the
AUCTEST we accorded the highest rank to the models with
the highest AUCTEST since higher AUCTESTmeans better model performance or discriminatory ability (Radosavljevic
and Anderson, 2014).
Once thus ranked, we followed the steps outlined in Figure 1. We chose
the composite model or subset of composite models with the highest OR
rank (corresponding to Step 1 of Figure 1). Since we used OR as the
first criteria to select the optimal models if only a single composite
model had the highest OR rank (i.e., the lowest ORs among the models) we
considered it the optimal model for both ORTEST and
AUCDIFF approaches (Figure 1). If Step 1 resulted in a
subset of composite models we chose either a composite model or subset
of models with the highest AUCTEST rank for the two
ORTEST approaches (corresponding to Step 2b, Figure 1)
(Shcheglovitova and Anderson 2013, Galante et al. 2018). Whereas, for
AUCDIFF approaches we chose the model or models with the
best ranked AUCDIFF (corresponding to Step 2a, Figure 1)
followed by Step 2b (Radosavljevic and Anderson, 2014) depending on the
outcome of Step 2a (Figure 1). After Step 2b, depending on the outcome,
we followed Steps 3 to 5 for both the ORTEST and
AUCDIFF approaches (Figure 1). In Step 3 we chose the
models with the lowest average number of parameters since models with
lower numbers of parameters are considered less complex and better
models (Galante et al. 2018). We derived the average number of
parameters for each candidate composite optimal model by dividing the
sum of the number of parameters with non-zero lambda coefficients for
each individual model extracted from the LAMBDA text file (Galante et
al. 2018) by the number of iterations used for building SDM since Maxent
does not provide directly the average number of parameters in the result
for the composite models unlike it does for the threshold values and
ORs. Further, when multiple optimal models had equal average number of
parameters, we then chose models with the lower average lambda
coefficients obtained by dividing the sum of the absolute value of
lambda coefficients by the total number of parameters. However, for some
species multiple optimal models with same RM values but different FCs
had equal average numbers of parameters as well as the average absolute
lambda coefficients. In such cases we used the composite models with
simpler FC as the final optimal model since lower FCs are considered
better for species with smaller occurrence in Maxent (Shcheglovitova and
Anderson 2013).