Accurate state estimation of the high-dimensional, chaotic Earth atmosphere marks a Sisyphean task, yet is indispensable for initiating weather forecast and gauging climate variability. While much effort is devoted to assimilating observations and forecasts to infer weather state, the inherent low-dimensional statistical structure in atmospheric circulation, shaped by geophysical laws and geographic boundaries, is underutilized as informative prior for state inference, or as reference for assessing representative of existing observations and planning new ones. We realize these potential by learning climatological distribution from climate reanalysis/simulation, using deep generative model. For a case study of estimating 2 m temperature spatial patterns, the learned distribution faithfully reproduces climatology statistics. A combination of the learned climatological prior with few station observations yields strong posterior of spatial pattern estimates, which are spatially coherent, faithful and adaptive to observation constraints, and uncertainty-aware. This allows us to evaluate each observation’s value in reducing state estimation uncertainty, and guide optimal observation network design by pinpointing the most informative sites. Our study showcases how generative models can extract and utilize information produced in the chaotic evolution of climate system.

Precipitation nowcasting is a crucial element in current weather service systems. Data-driven methods have proven highly advantageous, due to their flexibility in utilizing detailed initial hydrometeor observations, and their capability to approximate meteorological dynamics effectively given sufficient training data. However, current data-driven methods often encounter severe approximation/optimization errors, rendering their predictions and associated uncertainty estimates unreliable. Here we develop a probabilistic diffusion model-based precipitation nowcasting methodology, overcoming the notorious blurriness and mode collapse issues in existing practices. Our approach results in a 3.7% improvement in continuous ranked probability score compared to state-of-the-art generative adversarial model-based method. Critically, we significantly enhance the reliability of forecast uncertainty estimates, evidenced in a 68% gain of spread-skill ratio skill. As a result, our approach provides more reliable probabilistic precipitation nowcasting, showing the potential to better support weather-related decision makings.

Numerical weather prediction models often struggle to accurately simulate high-resolution precipitation processes, due to resolution limits and difficulties in representing convection and cloud microphysics. Data-driven methods often offer more accurate approximation of these unresolved processes by learning from high-fidelity referential data. Yet, existing approaches fail to yield fine-resolution, spatially coherent, reliable ensemble simulations/predictions, particularly for high-impact extreme events. To address these limitations, we develop a latent diffusion modeling (LDM) framework for quantitative precipitation estimation and forecast. The LDM leverages low-resolution (25 km) circulation and topographic information to estimate precipitation at a 4 km resolution. The latent diffusion model (LDM) learns the probability distribution of precipitation patterns by first compressing high-resolution spatial data into a compact, Quasi-Gaussian latent space. It then gradually refines estimates through a deep neural network parameterized reverse diffusion process, effectively capturing complex precipitation dynamics and delivering superior ensemble predictions. This approach enables LDMs to outperform traditional deep learning models like convolutional neural nets and generative adversarial neural nets, particularly for extreme events, while avoiding issues such as mode collapse, blurring artifacts, or underestimation of extremes. Compared to the dynamical method (WRF and GFS), the LDM offers significant performance gains, particularly for extreme precipitation events, improving over 30% in root mean squared error and over 40% in critical success index. For the extreme precipitation (> 300 mm/d) in California on October 25, 2021, LDM can provide effective forecasting up to 7 days in advance, forced by circulation prediction from a data-driven weather forecasting model.