Material and Methods

The model

To analyse change within a plot we use the partitioning approach from Godsoe et al. (2021). Within a given plot the absolute abundance of each species is \(n_{i}\) and the relative abundance of each species is\(p_{i}=n_{i}/\sum_{i}n_{i}\). For many diversity indices, the contribution that each individual makes to diversity can be defined as zi, the rarity of the species to which it belongs (Equation 1). For Shannon entropy, the measure of rarity is zi=–log(pi). Because we are interested in change between one period and another we can define rarity scores for Gini-Simpson’s diversity as zi = -pi(Godsoe et al. 2021). Diversity is an average measurement of rarity across all individuals in a plot (Patil and Taillie 1982); For Shannon entropy, the arithmetic mean of rarity scores are weighted by the relative abundance of each species (Jost 2006, 2007, Cover and Thomas 2012).
\(D=\sum_{i}{p_{i}z_{i}}\).
Total change in diversity over time is the difference between a measurement of diversity in the present and diversity in the past (Equation 2). Here and elsewhere, describes change over time and the prime superscript denotes present measurements (Frank 2012).
\(D=D^{{}^{\prime}}-D\) .
To understand the effects of immigrants on diversity change, we further divide the present community is into two components:
\(D^{{}^{\prime}}=\text{\ ω}\sum_{i}{p_{i}^{\prime}z_{i}^{\prime}}+\mu\sum_{j}{a_{j}z_{j}^{*}}\).
The first term represents what we will call “descendants”, individuals that are not immigrants. They may represent individuals that were in the community in the first survey, or their offspring. Here \(\omega\) is the proportion of individuals in the present community that descended from the past community. Among descendants the proportion belonging to species i is pi\({}^{\prime}\), and the rarity score of species i is zi \({}^{\prime}\) (note this is rarity as a proportion of total individuals in both time steps). The second term describes the contribution of immigrants, where \(\mu\) is the proportion of individuals in the present community which immigrated to the community since the initial measurement. Among immigrants, the proportion belonging to species i is aj\({}^{\prime}\), and the rarity score of species j is zj* (note this is rarity as a proportion of all individuals, not just immigrants).
Overall diversity change in equation 2 can then be partitioned using extensions of the Price equation (Price 1970, Kerr and Godfrey‐Smith 2009). This leads to explicit definitions for the effects of selection, immigration and rarity shifts on diversity (Equation 4).
(4) \(D=\underset{\par \begin{matrix}\text{Selection}\\ (+\ drift)\\ \end{matrix}}{}+\underset{\text{immigration}}{}+\underset{\par \begin{matrix}\text{rarity\ }\\ \text{shifts}\\ \end{matrix}}{}\)
In Equation 4, the first term describes the effect of selection, where the tendency of species i to leave more descendants increases its frequency relative to other descendants (\(p_{i}=p_{i}^{{}^{\prime}}-p_{i})\). This term implicitly includes effects of drift because both mechanisms change species’ relative abundances (Rice 2004). In this framework, selection on species identity emerges when one species tends to leave more descendants than another either by leaving more offspring or having higher survival (Vellend 2016). Drift emerges when one species increases in relative abundance due to stochastic sampling. The second term describes immigration, which changes diversity when diversity among immigrants \(\sum_{j}{a_{j}z_{j}^{*}}\) is different from the diversity among resident ancestors. The final term denotes rarity shifts where the rarity score for descendants is different from the rarity scores of ancestors \(z_{i}\), as in (Figure 1 C).
We have presented diversity indices that are familiar and easy to partition on a linear scale. More complexities emerge when analysing so called “Hill numbers”, a re-scaled version of diversity indices expressed in common units. Both Shannon entropy and Simpsons diversity indices can be converted into common units the equivalent number of uniformly distributed species needed to produce the observed diversity index (Jost 2006, 2007). While not used in this paper, the numbers equivalent Shannon entropy can be partitioned by exponentiation Equation 4 (Godsoe et al. 2022). This conversion changes the scale of measurement from additive to multiplicative but preserves many of the qualitative patterns. At present, techniques exist to other Hill numbers into selection and transmission terms, in the absence of immigration (Frank and Godsoe 2020), but this condition is unrealistic for most observations in our study.

The Data

We analysed the causes of biodiversity change from plots in 15 studies compiled in the BioTIME database (Dornelas et al. 2018). As of December 2021, the BioTIME database contained 361 studies, 201 of which were terrestrial surveys. We selected vegetation studies with multiple plots of area greater than 1 m2, including observations from 3 or more time periods, 5 or more species, and counts of individuals (as opposed to biomass, presence or vegetation cover). This resulted in a database of 3341 observations of diversity change, with each observation representing the difference in diversity between one sample and a previous sample.
To compare the effect of rarity shifts with other mechanisms, we calculated the change in Shannon entropy and Gini-Simpson’s diversity within each plot across each time step using Equations 1 and 2. This change in diversity was partitioned using Equation 4. We used histograms to compare the prevalence of rarity shifts relative to other sources of diversity change.
To determine how the strength of rarity shifts changes with other mechanisms, we used a generalized additive model (GAM). This was done using the mgcv package (Wood and Wood 2015) where the strength of rarity shifts is modelled as a smoothed non-linear function of selection and immigration, with study and plot treated as random effects. Increasing effective degrees of freedom (edf) indicate greater model complexity, where edf = 1 indicates a straight line. The model was fit using Restricted Maximum Likelihood (REML). Rarity shifts had an extremely long tail and so to improve the interpretability of Figure 4, ~3% of the data with the lowest rarity shifts were removed from the analysis of Shannon entropy. We also removed one unusually large estimate of selection.