Over the last ten years, satellite and geographically constrained in situ observations largely focused on the northern hemisphere have suggested that annual phytoplankton biomass cycles in bloom-forming ocean regions cannot be fully understood from environmental properties controlling phytoplankton division rates (e.g., nutrients and light). Here, we use multi-year observations from a very large array of robotic drifting buoys in the Southern Ocean to determine key factors governing phytoplankton biomass dynamics over the annual cycle. Our analysis reveals phytoplankton blooming events occurring during periods of declining division rates, an observation that clearly highlights the importance of changing loss processes in dictating the evolution of the bloom. Bloom magnitude is found to be greatest in areas with high dissolved iron concentrations, consistent with iron being a well-established primary limiting nutrient in the Southern Ocean. Projections for expected future seasonal variations in nutrient and light availability indicate a 10% change in phytoplankton division rate may be associated with a 50% reduction in mean bloom magnitude and annual primary productivity in the Southern Ocean. Our results highlight the importance of quantifying and accounting for both changing phytoplankton division and loss processes when modeling future changes in phytoplankton bloom cycles.