Performance across ecological forecasting and uncertainty
communication learning objectives
Student performance improved from pre- to post-module in correctly
answering questions on both foundational ecological forecasting concepts
(LO1) and uncertainty communication (LO2; Figure 5). The increase in
performance was stronger for students who scored lower on the pre-survey
(Figure 5b). Students were more likely to correctly answer questions on
ecological forecasting concepts than questions on uncertainty
communication concepts on the pre-module assessment (Figure 5a). Both
LOs showed strong growth after module completion, with many students who
answered zero questions correctly on the pre-survey answering all
questions correctly on the post-survey in both categories (Figure 5b).
More students scored 100% (all LO-specific questions answered
correctly) on foundational ecological forecasting concepts than on
uncertainty communication both before and after module completion,
though we note that the number of questions in the two categories
differed (Figure 1).
Discussion
Our results indicate that completion of a 3-hour module can
significantly improve undergraduate ecology students’ understanding of
uncertainty communication and ecological forecasting. While the
percentage of correct answers increased for all assessment questions
after module completion, students were more likely to perform higher on
foundational ecological forecasting concepts than uncertainty
communication concepts prior to module completion. Higher initial
performance on ecological forecasting questions may be because students
were more familiar with ecological forecasting concepts relative to
uncertainty communication concepts before completing the module.
However, students showed more growth in describing multiple ways to
communicate uncertainty than in identifying benefits of ecological
forecasting (Figure 4c, 4d). Below, we explore the implications of our
results for undergraduate education in uncertainty communication,
ecological forecasting, and ecology
broadly.Improved uncertainty communication
skills and implications for visualization literacy
Students identified significantly
more ways to communicate uncertainty following module completion,
indicating that the module introduced students to a toolbox of
approaches for developing and understanding uncertainty in ecological
visualizations. Before completing our module, the majority (85%) of
students were unable to describe any ways to communicate uncertainty,
while after module completion 72% were able to describe one or more
ways to communicate uncertainty (Figure 4d). Being able to identify and
describe multiple methods for uncertainty communication is an important
skill, as the method used to visualize uncertainty can have a
substantial effect on user comprehension and decision-making
(Nadav-Greenberg et al. 2008, Ramos et al. 2013, Cheong et al. 2016,
McKenzie et al. 2016, Kinkeldey et al. 2017). For example, using summary
visualizations (e.g., boxplots) can decrease users’ cognitive load and
increase the speed of decision-making, but are more likely to lead to
misinterpretation (Ruginski et al. 2016, Correll et al. 2018). In
contrast, ensemble-based visualizations (i.e., forecast visualizations
that show all possible model outputs) may provide users with more
information about the whole spread of uncertainty, but viewers may
overweight certain ensemble members, leading to inconsistent
decision-making (Padilla et al. 2017a). Given that there is no single
“best” visualization method for uncertainty communication due to
differences in decision-making needs (Spiegelhalter et al. 2011), the
ability to create a variety of visualization options and adapt
visualizations based on forecast user feedback is critical for
developing effective uncertainty visualizations.
All of the methods for uncertainty communication included in the
students’ post-module responses are aligned with current
state-of-the-art practices for uncertainty communication in
visualization science. “Visual” and “numeric,” the two uncertainty
communication methods most commonly reported by students in post-module
responses (Figure 4b), mirror the two key uncertainty representation
techniques (“visualization” and “quantification”) identified in a
recent review (Kamal et al. 2021). Probabilistic methods, which were
also significantly more common in student responses following module
completion, can decrease cognitive load and increase use and
understanding of visualizations for decision support (Kox et al. 2018).
Some students (n = 8) also identified “text” as a useful method for
uncertainty communication (Figure 4b), but “text” was almost always
(7/8 students) reported in addition to another form of communication
(e.g., “visual,” “numeric,” or “probability”), following
literature which shows that text is most useful for explaining and
providing context for visualizations (Carr et al. 2018). Additionally,
common keywords throughout student responses included “color” and
“icon.” Thoughtful use of color palettes (e.g., by using discrete
rather than continuous color palettes) has been shown to be a powerful
tool in representing ranges of uncertainty (Padilla et al. 2017b,
Correll et al. 2018). Similarly, the use of icons or symbols has been
shown to improve user understanding and usability of decision support
tools in diverse settings (Galesic et al. 2009, Garcia-Retamero et al.
2010, Zikmund-Fisher et al. 2014, Kamal et al. 2021), potentially by
decreasing the cognitive load required to interpret the communication.
While neither text, icons, or numbers alone are typically most effective
in scientific communication of complex ideas (Larkin and Simon 1987,
Tait et al. 2010), the student post-module responses are reflective of a
common theme in the visualization literature that using multiple
communication forms increases user comprehension and confidence in
decision-making (Fagerlin et al. 2005, Spiegelhalter et al. 2011).
Ultimately, the module increased students’ ability to communicate
uncertainty using multiple approaches, a key skill for developing
decision support tools for forecast users.
Overall, the module shows promise for increasing visualization literacy
and introducing much needed skills in uncertainty communication to
undergraduate ecology students. Most students who completed this module
had little to no prior experience with uncertainty communication, but
showed substantial improvement in performance after module completion,
indicating that a 3-hour module can help build these critical skills
(Figure 5). Students’ lack of previous exposure to uncertainty
communication is likely because undergraduate ecology classes do not
currently include uncertainty communication and visualization literacy
topics as often as, e.g., ecological modeling and prediction (Willson et
al. 2022). Because the communication of uncertainty is just as
important as the quantification of uncertainty in forecasts for
ensuring that forecast visualizations guide end users’ decision-making,
it is critical that science communication and visualization science,
including incorporation of end user decision needs in visualization
development, are included in ecological forecasting training (Robinson
et al. 2012, Schwartz et al. 2017, Eisenhauer et al. 2021).
Overall, given the importance of uncertainty communication not only in
ecology, but across scientific disciplines broadly (e.g., medicine,
meteorology, economics; Tait et al. 2010, Ferstl et al. 2017, Wesslen et
al. 2022), improving students’ ability to interpret and produce
uncertainty visualizations may help enable student participation in a
variety of scientific disciplines. Moreover, providing students with
improved visualization literacy and uncertainty communication skills
will yield a more data-literate population, regardless of students’
future careers. This module provides an important first step for
incorporating visualization literacy coursework across undergraduate
curricula broadly and initiating training in critical visualization
interpretation and communication skills.Increase
in student understanding of foundational ecological forecasting concepts
In addition to expanding students’ uncertainty communication skills,
completion of the 3-hour module improved students’ understanding of
foundational ecological forecasting concepts. Following module
completion, students were significantly more likely to correctly define
an ecological forecast as a future prediction of environmental
conditions with uncertainty (Figure 3a; Appendix S1: Table S1). Overall,
developing a common definition of “forecast” is important for
furthering the field of ecological forecasting, as having common
definitions enables meaningful discourse on topics within and across
disciplines, providing a scaffold for interdisciplinary work to address
complex socioecological problems (Lélé and Norgaard 2005, Robinson et
al. 2012). Given that ecological forecasting is an emerging field
(Woelmer et al. 2021), codifying definitions in training materials
enables undergraduate ecology students to more effectively discuss and
learn forecasting topics.
Student understanding of the benefits of ecological forecasting also
significantly increased after module completion. Specifically, we found
a significant increase in the number of students who identified
‘decision-making’ and ‘prediction’ as benefits, but saw only a minimal
increase for ‘policy,’ and decreases for ‘management’ and
‘understanding.’ Since ‘decision-making’ and ‘prediction’ were
emphasized throughout the module, it is unsurprising that these two
benefits of ecological forecasting were most commonly provided in
student responses. However, the small decrease in responses related to
management and understanding is surprising, and may indicate that the
module did not sufficiently focus on the benefits of forecasts for
management or ecological understanding. For example, while Activity B
used a management-centered role-playing example, the activity was
primarily focused on the effect of visualization type on
decision-making, rather than how forecasts could be integrated into
management workflows. Similarly, the module did not emphasize how
ecological forecasts can advance understanding of ecosystems and testing
of ecological theory or be integrated into policy-making decisions,
leaving an opportunity to bolster these forecast applications in future
iterations of the module. Alternatively, it is possible that students
who identified “decision-making” as a benefit of forecasts may have
had policy or management decisions in mind, but not specifically stated
this.
We note that students’ ability to correctly identify that uncertainty
should increase the further into the future a forecast is made (Q2)
showed only marginal growth (Figure 3b), leaving room for improvement in
teaching this foundational concept of ecological forecasting
(sensu Dietze et al. 2018). To complement this module and provide
additional training in foundational ecological forecasting concepts, we
suggest pairing this module with other Macrosystems EDDIE modules
(Module 5: Introduction to Ecological Forecasting, Moore et al. 2022b;
Module 6: Understanding Uncertainty in Ecological Forecasts, Moore et
al. 2021; or Module 7: Using Data to Improve Ecological Forecasts,
Lofton et al. 2022).
Integration of decision support concepts into ecology curricula Integrating applied decision-making concepts into ecological
forecasting and uncertainty communication lessons heeds a widespread
call to make ecological research more societally relevant (e.g.,
Belovsky et al. 2004, Ruhl et al. 2022). Training that incorporates
components of translational ecology (e.g., science communication, end
user engagement, structured decision-making, multidisciplinary training)
has long been recommended for ecologists at all career stages (Robinson
et al. 2012, Schwartz et al. 2017, Eisenhauer et al. 2021), but
resources targeted at the undergraduate level have been lacking
(Bakermans and Pfeifer 2018). Our module aims to close this gap by
incorporating complex water management decision-making scenarios with
multiple end users to engage students in solving real-world problems,
while also developing visualization literacy skills. While we did not
quantitatively assess the engagement of students in the module, we
received open-ended positive feedback from many students. Students
reported that the module was enjoyable and important (“I really enjoyed
looking at decision analysis in an ecology class;” “I think it’s very
important to talk about this in science classes”). Additionally,
student responses suggest that they found that the module was
interesting (“This was informative and a really interesting way for me
to realize the actual impacts of ecological forecasting”) and novel
(“It was helpful because I came in with no information”).
Ultimately, introducing students to applications of ecological
forecasting for real-world decision-making may help recruit students to
work in this subdiscipline, as well as highlight the importance of using
ecology to produce actionable tools to address societal problems (e.g.,
Enquist et al. 2017). Many ecological forecasters have already begun to
integrate decision support and uncertainty communication components into
their forecasts by making forecasts which are actionable, useful, and
targeted towards forecast users (e.g., Gerst et al. 2019, Turner et al.
2020, Jackson-Blake et al. 2022). Our experience with this module
indicates that even a short (3-hour) exposure to decision support and
uncertainty communication concepts can increase students’ understanding
of potential applications and benefits of ecological forecasting.