\documentclass[10pt]{article}

\usepackage{fullpage}
\usepackage{setspace}
\usepackage{parskip}
\usepackage{titlesec}
\usepackage[section]{placeins}
\usepackage{xcolor}
\usepackage{breakcites}
\usepackage{lineno}
\usepackage{hyphenat}





\PassOptionsToPackage{hyphens}{url}
\usepackage[colorlinks = true,
            linkcolor = blue,
            urlcolor  = blue,
            citecolor = blue,
            anchorcolor = blue]{hyperref}
\usepackage{etoolbox}
\makeatletter
\patchcmd\@combinedblfloats{\box\@outputbox}{\unvbox\@outputbox}{}{%
  \errmessage{\noexpand\@combinedblfloats could not be patched}%
}%
\makeatother


\usepackage{natbib}




\renewenvironment{abstract}
  {{\bfseries\noindent{\abstractname}\par\nobreak}\footnotesize}
  {\bigskip}

\titlespacing{\section}{0pt}{*3}{*1}
\titlespacing{\subsection}{0pt}{*2}{*0.5}
\titlespacing{\subsubsection}{0pt}{*1.5}{0pt}


\usepackage{authblk}


\usepackage{graphicx}
\usepackage[space]{grffile}
\usepackage{latexsym}
\usepackage{textcomp}
\usepackage{longtable}
\usepackage{tabulary}
\usepackage{booktabs,array,multirow}
\usepackage{amsfonts,amsmath,amssymb}
\providecommand\citet{\cite}
\providecommand\citep{\cite}
\providecommand\citealt{\cite}
% You can conditionalize code for latexml or normal latex using this.
\newif\iflatexml\latexmlfalse
\providecommand{\tightlist}{\setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}%

\AtBeginDocument{\DeclareGraphicsExtensions{.pdf,.PDF,.eps,.EPS,.png,.PNG,.tif,.TIF,.jpg,.JPG,.jpeg,.JPEG}}

\usepackage[utf8]{inputenc}
\usepackage[T2A]{fontenc}
\usepackage[ngerman,polish,english]{babel}



\usepackage{float}








\begin{document}

\title{A modular curriculum to teach undergraduates~ ecological forecasting
improves student and instructor confidence in their data~ science skills}



\author[1]{Mary E. Lofton}%
\author[1]{Tadhg N. Moore}%
\author[1]{Whitney M. Woelmer}%
\author[1]{R. Quinn Thomas}%
\author[1]{Cayelan C. Carey}%
\affil[1]{Affiliation not available}%


\vspace{-1em}



  
  \date{May 22, 2024}


\begingroup
\let\center\flushleft
\let\endcenter\endflushleft
\maketitle
\endgroup








\sloppy


\par\null

Mary Lofton\textsuperscript{1,2a}, Tadhg Moore\textsuperscript{1b},
Whitney Woelmer\textsuperscript{1c}, Quinn
Thomas\textsuperscript{1,2,3d}, Cayelan Carey\textsuperscript{1,2e}

\textsuperscript{1}Department of Biological Sciences, Virginia Tech. 926
West Campus Drive, Blacksburg, Virginia, 24061, United States

\textsuperscript{2}Center for Ecosystem Forecasting, Virginia Tech. 1015
Life Science Circle, Blacksburg, VA 24061, United States

\textsuperscript{3}Department of Forest Resources and Environmental
Conservation, Virginia Tech. 310 West Campus Drive, Blacksburg,
Virginia, 24061, United States

\textsuperscript{a} Postdoctoral Research Associate; melofton@vt.edu

\textsuperscript{b} Postdoctoral Research Associate;
tadhg.moore6@gmail.com

\textsuperscript{c} Ph.D. Candidate; wwoelmer@vt.edu

\textsuperscript{d} Associate Professor; rqthomas@vt.edu

\textsuperscript{e} Professor; cayelan@vt.edu

\par\null

\section*{Abstract}

{\label{abstract}}

Data science skills (e.g., analyzing, modeling, and visualizing large
datasets) are increasingly needed by undergraduates in environmental
science. However, a lack of both student and instructor confidence in
data science skills presents a barrier to their inclusion in
undergraduate curricula. To reduce this barrier, we developed four
teaching modules in the Macrosystems EDDIE (Environmental Data-Driven
Inquiry \& Exploration) program to introduce undergraduate students and
instructors to ecological forecasting, an emerging subdiscipline which
integrates multiple data science skills. Ecological forecasting aims to
improve natural resource management by providing future predictions of
ecosystems with uncertainty. We assessed the efficacy of the modules
with 596 students and 26 instructors over three years and found that
module completion increased students' confidence in their understanding
of ecological forecasting and instructors' likelihood to work with
long-term, high-frequency sensor network data. Our modules constitute
one of the first formalized data science curricula on ecological
forecasting for undergraduates.

\textbf{Keywords:} active learning, ecosystem modeling, National
Ecological Observatory Network (NEON), training program, undergraduate
education

\section*{}

{\label{section}}

\section*{Introduction}

{\label{section}}

Data science skills, such as visualizing, analyzing, and modeling large
datasets, are increasingly needed by ecology and environmental science
undergraduate students (Farrell and Carey 2018, Auker and Barthelmess
2020, Feng et al. 2020, Cooke et al. 2021). Recent advancements in
environmental monitoring technology (e.g., Mcloughlin et al. 2019,
Nathan et al. 2022, Dauphin et al. 2023) and the rise of environmental
observatory networks (Keller et al. 2008, Weathers et al. 2013, Cleverly
et al. 2019) have resulted in a deluge of ``big data'' in ecology
(Hampton et al. 2013, LaDeau et al. 2017, Farley et al. 2018). As a
result, analysis of large datasets is now required across a variety of
environmental science and ecology careers, necessitating new approaches
to training researchers, instructors, and students in data science
skills (Hampton et al. 2017, National Academies of Sciences 2018, Feng
et al. 2020, Emery et al. 2021).

Currently, a lack of both student and instructor familiarity with data
science concepts, methods, and tools presents a major barrier to
incorporation of data science into undergraduate life science curricula
(Williams et al. 2019, Emery et al. 2021, Naithani et al. 2022,
Cuddington et al. 2023). This gap often exists because instructors
themselves have not received training in data science skills (Williams
et al. 2019, Emery et al. 2021), and students do not have the requisite
background skills and confidence to effectively engage in data science
training (Williams et al. 2019, Cuddington et al. 2023). Consequently,
development of educational materials approachable to both instructors
and students is needed to lower the barrier to data science education in
ecology and environmental science.

Ecological forecasting is an ideal topic for engaging instructors and
students in data science training (Willson et al. 2023). First,
ecological forecasting has the potential to guide environmental
management decisions (Johnson et al. 2018, Liu et al. 2020, Bodner et
al. 2021, Heilman et al. 2022), thereby engaging students in real-world
problem-solving. Ecological forecasts, which provide predictions of the
future state of ecosystems with uncertainty (Luo et al. 2011, Petchey et
al. 2015), are critically needed to help manage natural resources
increasingly threatened by climate and land use change (Bradford et al.
2020). Examples of societally important forecasts exist for many
ecological systems, including river temperature forecasts to guide
reservoir water release decisions and protect fish species
(Ouellet-Proulx et al. 2017), temperature-based spring onset forecasts
to inform agricultural decision-making (Carrillo et al. 2018), and
forecasts of endangered ocean species to avoid bycatch (Hazen et al.
2018).

Second, generating ecological forecasts requires students to step
through the scientific method (Moore et al. 2022, Lewis et al. 2023),
providing critical skills in developing and testing hypotheses, which
are transferable across scientific disciplines. In the iterative
forecast cycle, similar to the scientific method, researchers develop
hypotheses about how ecosystems function; instantiate hypotheses into a
predictive model; use the model to generate forecasts into the future;
evaluate forecasts with observations once the future arrives and new
data are available; and use evaluation results to iteratively update and
improve hypotheses, models, and predictions (Dietze et al. 2018).

Third, ecological forecasting problems are particularly well-suited for
active learning, in which students learn by doing rather than passively
listening or watching (Bonwell and Eison 1991). Active learning has been
shown to enhance student outcomes (Freeman et al. 2014), especially for
underrepresented groups (Theobald et al. 2020). Key components of active
learning that can be easily embedded within ecological forecasting
curricula include: authentic assessments that engage students in
real-world, relevant problems similar to what they will encounter in
their future careers (Villarroel et al. 2018); scaffolding to help
students progressively build more complex skills and problem-solve
(Belland 2014); and formative assessments that provide students with
specific, actionable guidance on their progress, with opportunities to
apply that guidance moving forward (Wiliam 2011).

To effectively use ecological forecasting as a platform for teaching
data science in undergraduate classrooms, instructors must have both
pedagogical knowledge of active learning \emph{and} disciplinary
knowledge of data science and ecological forecasting (Auerbach and
Andrews 2018, Andrews et al. 2019). However, research has demonstrated
substantial gaps in instructor knowledge in both active learning
(Auerbach and Andrews 2018, Andrews et al. 2019) and data science
(Williams et al. 2019, Emery et al. 2021). Given that ecological
forecasting is an emerging field (Lewis et al. 2022), and educational
resources in ecological forecasting remain rare (Willson et al. 2023),
it is unlikely that many instructors have training in this area.

To address gaps in instructor knowledge in the life sciences, multiple
models of instructor professional development have been trialed,
including short, intensive trainings for teaching assistants (Hughes and
Ellefson 2013, Schussler et al. 2015), department-wide training programs
for faculty (Owens et al. 2018), and multi-year, multi-institutional
programs for postdoctoral researchers (Ebert-May et al. 2011, D'Avanzo
et al. 2012, Derting et al. 2016). Outcomes of these professional
development activities frequently rely solely on instructor feedback
(Ebert-May et al. 2011). However, instructor and student perceptions of
the effectiveness of teaching practices in the classroom can differ from
each other (Heim and Holt 2018). Consequently, the effectiveness of
instructor professional development should be evaluated using multiple
methods (e.g., reflection and feedback, observing teaching practices,
student assessments; Ebert-May et al. 2011, Heim and Holt 2018) and
incorporate input from both students and faculty. Moreover, as lack of
time is often cited as a barrier to instructor professional development
(Williams et al. 2019), instructional materials associated with active
learning activities should include short, accessible definitions and
examples of key concepts to provide ``just-in-time'' (\emph{sensu} Novak
et al. 1999) pedagogical, data science, and ecological forecasting
training for instructors as well as students.

To lower the barrier of entry to data science for both undergraduate
students and instructors in ecology and environmental science, we
developed and assessed a modular curriculum within the Macrosystems
EDDIE (Environmental Data-Driven Inquiry and Exploration) program (Carey
et al. 2020, Hounshell et al. 2021, Moore et al. 2022, Woelmer et al.
2023a) that uses active learning techniques to teach data science skills
in the context of ecological forecasting. While previous educational
materials on ecological forecasting have been developed for advanced
students, primarily at the graduate level (e.g., Dietze 2017a, Ernest et
al. 2023), our curriculum is one of the first that is specifically
targeted to undergraduates (Willson et al. 2023). In addition, all
materials are designed to be approachable to both instructors and
students, as coding experience is not a necessary prerequisite and each
module is accompanied by substantial introductory and supporting
materials for instructors. Moreover, students work with data from the
U.S.'s National Ecological Observatory Network (NEON) to address
relevant societal challenges such as predicting freshwater quality
impairment.

Here, we present an overview of the Macrosystems EDDIE ecological
forecasting curriculum and examples of how it has been implemented in
various course contexts. We also analyze student and instructor
assessment data to address the following questions: (1) How does
completion of Macrosystems EDDIE ecological forecasting modules affect
student confidence and understanding of data science and ecological
forecasting skills? (2) What are instructor perceptions of module ease
of use and efficacy in teaching data science and ecological forecasting
concepts? We were specifically focused on student confidence and
instructor perceptions, as previous work has shown that two major
barriers to integrating data science active learning activities into
existing curricula are a lack of instructor training (Williams et al.
2019, Emery et al. 2021) and student confidence (Williams et al. 2019,
Cuddington et al. 2023).

\section*{}

{\label{overview-of-the-macrosystems-eddie-curriculum}}

\section*{Overview of the Macrosystems EDDIE
curriculum}

{\label{overview-of-the-macrosystems-eddie-curriculum}}

The Macrosystems EDDIE ecological forecasting curriculum for
undergraduates comprises four standalone modules: Introduction to
Ecological Forecasting, Understanding Uncertainty in Ecological
Forecasts, Using Data to Improve Ecological Forecasts, and Using
Ecological Forecasts to Guide Decision-Making (Fig. 1). Like all EDDIE
modules, Macrosystems EDDIE ecological forecasting modules are designed
using the 5E (Engagement, Exploration, Explanation, Expansion,
Evaluation) instructional model (Bybee et al. 2006), which is
implemented through a scaffolded A-B-C structure (O'Connell et al.
2024). In all modules, Activity A Engages students and asks them to
Explore the module's focal topic, Activity B further Explains and asks
students to Expand on that topic, and Activity C Evaluates students'
understanding of the topic (Carey et al. 2015, O'Reilly et al. 2017).
The three-part scaffolded structure also maximizes the adaptability of
Macrosystems EDDIE modules to various classroom contexts, as instructors
can choose whether to complete just Activity A, Activities A and B, or
all three activities in one to three-hour course periods. Each module
can be taught individually or instructors may choose to implement
multiple modules throughout their curriculum; example use cases are
detailed in the \emph{Course implementation} section below.

The modules in the Macrosystems EDDIE ecological forecasting curriculum
are designed to both 1) introduce ecological forecasting concepts and 2)
develop data science skills (Fig. 1). To accomplish the first goal, each
module covers a foundational concept in ecological forecasting, and
students then apply the forecasting concept to a NEON lake site of their
choice. To develop data science skills, students use environmental data
collected by NEON (Keller et al. 2008, Goodman et al. 2015) as the basis
for their forecasting analyses. Working with NEON datasets requires
students to evaluate the quality of the data (e.g., gaps, outliers,
biases) and confront how inherent variability and error in environmental
datasets may affect their analyses. In addition, each module asks
students to interpret data visualized using various methods, ranging
from time series and scatterplots to probabilistic forecasts and
histograms. Finally, each module focuses on one or more foundational
quantitative skills in ecological forecasting, including building and
calibrating ecological models, generating forecasts, quantifying the
uncertainty associated with predictions, using new observations to
update forecast models, and designing forecast visualizations to
effectively communicate forecast output.

Macrosystems EDDIE modules include a comprehensive set of instruction
materials and are suitable for implementation in a variety of class
contexts (Fig. 1). All modules are delivered through an R Shiny
interface, where R code is used to render a website that students can
access in their internet browser (Chang et al. 2023). This permits a
user-friendly, point-and-click interface for introductory students and
aims to lower the intimidation barrier to ecological forecasting, as
students do not need to have any coding skills to generate a forecast.
For classrooms where gaining R coding skills is a learning objective,
two of the modules (Understanding Uncertainty in Ecological Forecasts
and Using Data to Improve Ecological Forecasts) have Rmarkdown
activities in addition to R Shiny materials. The Rmarkdown activities
enable students to access and modify the code underlying the R Shiny app
and complete module activities in the R programming environment (Xie et
al. 2018).

All Macrosystems EDDIE ecological forecasting materials are designed to
provide instructors with ``just-in-time'' training (\emph{sensu} Novak
et al. 1999) on data science skills as they prepare to teach the modules
in their classrooms. In addition to the R Shiny application (and
RMarkdown file if applicable), each module includes an introductory
(\textasciitilde{}30 minute) Microsoft PowerPoint presentation with
slide notes; a Microsoft Word student handout with pre-class readings,
activities, and questions associated with the module; a comprehensive
instructor manual with learning objectives; detailed guidelines for
module implementation and answer keys; and a ``quick start'' guide to
the R Shiny applications. Notably, instructor manuals include strategies
for teaching and recommendations for implementing the modules across a
variety of course schedules (e.g., three, one-hour class sessions vs.
one, three-hour lab period) and modalities (e.g., virtual, face-to-face,
hybrid).

All module teaching materials are licensed under the CC BY-NC-SA 3.0
license allowing modification for classroom use and are published in the
Environmental Data Initiative repository (Moore et al. 2023a, 2024b,
Woelmer et al. 2023b, Lofton et al. 2024c), and all module code is
published in the Zenodo repository (Moore et al. 2023b, 2023c, 2024a,
Woelmer et al. 2022, Lofton et al. 2024a, 2024b). In addition, all
module code is maintained and updated at the Macrosystems EDDIE GitHub
organization (https://github.com/MacrosystemsEDDIE). We encourage and
welcome instructors and students to adapt and modify these materials for
their classrooms, projects, and research.

\section*{}

{\label{section-1}}

\section*{Module descriptions}

{\label{section-1}}

\emph{Introduction to Ecological Forecasting (Intro to Forecasting)}

This module introduces students to the ecological forecasting cycle,
which includes the following steps: create a hypothesis, build a model,
quantify model uncertainty, generate a forecast, communicate the
forecast, assess the forecast, and update the model as new data become
available (Dietze et al. 2018). Students complete each step in the cycle
as they generate water quality forecasts for various NEON lake sites.
See \emph{http://module5.macrosystemseddie.org} for a detailed
description of the module; module code and instructor materials are also
published with DOIs in Moore et al. (2022a) and Moore et al. (2022b),
respectively.

\emph{Understanding Uncertainty in Ecological Forecasts (Forecasts \&
Uncertainty)}

This module introduces students to concepts and methods for quantifying
forecast uncertainty, which entails identifying the range of possible
future model outcomes (Dietze 2017b). Students build simple linear
models to forecast water temperature at a NEON lake site of their choice
and calculate the uncertainty associated with the forecasts.
See\emph{http://module6.macrosystemseddie.org} for a detailed
description of the module; module code for the R Shiny application and
RMarkdown as well as instructor materials are also published with DOIs
in Moore et al. (2023b), Moore et al. (2023c), and Moore et al. (2023a),
respectively.

\emph{Using Data to Improve Ecological Forecasts (Forecasts \& Data)}

This module introduces students to concepts and methods for data
assimilation, or the process of updating forecast models to incorporate
new data as they become available (Niu et al. 2014). Students fit an
autoregressive time series model to predict chlorophyll-a at a NEON lake
site of their choice and examine the effect of updating the initial
(starting) conditions of the model with chlorophyll-a data at different
temporal frequencies (e.g., updating the model once a week vs. once a
day) and with low vs. high observation uncertainty.
See\emph{http://module7.macrosystemseddie.org} for a detailed
description of all module materials; module code for the R Shiny
application and RMarkdown as well as instructor materials are also
published with DOIs in Lofton et al. (2024a), Lofton et al. (2024b), and
Lofton et al. (2024c), respectively.

\emph{Using Ecological Forecasts to Guide Decision Making (Forecasts \&
Decisions)}

This module explores how different methods of visualizing and
communicating forecasts can affect decision-making. Students are asked
to critically evaluate, interpret, and design different ecological
forecast visualizations for water quality management.
See\emph{http://module8.macrosystemseddie.org} for a detailed
description of the module; module code and instructor materials are also
published with DOIs in Woelmer et al. (2022) and Woelmer et al. (2023b),
respectively.

\section*{}

{\label{course-implementation}}

\section*{Course implementation}

{\label{course-implementation}}

To date, Macrosystems EDDIE modules have been implemented and assessed
in life science courses at a wide range of higher education
institutions, ranging from small, primarily undergraduate institutions
to large, research-focused universities (Table S1; Carey et al. 2020,
Hounshell et al. 2021). Notably, as all materials are publicly
available, instructors can integrate modules into their curricula
independently of module developers. Below, we provide three examples of
courses in which Macrosystems EDDIE ecological forecasting modules have
been implemented (following Fig. 1). These examples were selected to
illustrate both the breadth of courses across which the modules have
been applied, as well as the various ways in which instructors choose to
adapt Macrosystems EDDIE ecological forecasting materials for their
classes. Institutional designations are provided following the Carnegie
Classification of Institutions of Higher Education
(https://carnegieclassifications.acenet.edu/).

\emph{Ecology: Forecasts \& Decisions in R Shiny}

\emph{Ecology} is a third-year lecture/laboratory undergraduate course
of \textasciitilde{}250 students at a public R1 state university. Key
learning outcomes of the laboratory curriculum include communicating
scientific knowledge in writing, designing and implementing ecological
studies and data analyses, and conducting collaborative team science,
with an emphasis on inquiry-based learning. The instructor
taught\emph{Using Ecological Forecasts to Guide Decision Making} in the
R Shiny interface to introduce students to the emerging field of
ecological forecasting, as well as encourage them to consider
connections between sociological and ecological systems, such as how
communication of forecasts can affect resource management and therefore
water quality. For this course, the module was taught across 11 lab
sections of \textasciitilde{}24 students each by a team of teaching
assistants in a single, 3-hour laboratory period.

\emph{Freshman Ecology and Evolution Seminar: Forecasts \& Uncertainty
in R Shiny}

\emph{Freshman Seminar: Ecology and Evolution} is a first-year course
designed for \textasciitilde{}20 students to introduce them to the
Biology major at a public Master's 2 state institution. Key learning
outcomes of the course include explaining patterns of energy and matter
flow through ecosystems, understanding ecological relationships among
organisms and their environment, and explaining how humans interact with
the environment via ecosystem functions and services. The instructor
taught \emph{Understanding Uncertainty in Ecological Forecasts} in R
Shiny over three, 90-minute class periods to introduce students to
ecological forecasting and explore contributions to uncertainty in
models.

\emph{Environmental Data Science: two modules in R Shiny and RMarkdown}

\emph{Environmental Data Science} is a third-year undergraduate course
of \textasciitilde{}20 students within the Environmental Data Science
major at a public R1 state university. Key skills developed in this
course include advanced R coding, environmental data wrangling,
visualization, and interpretation, and data-driven modeling. Students
are expected to have basic to intermediate R coding skills upon
enrollment in the course. The instructor designed a two-week unit (four
75-minute class periods) using Macrosystems EDDIE ecological forecasting
materials. The dual goals of the unit were to introduce students to the
emerging field of ecological forecasting as well as to better understand
model uncertainty and how to calculate it. During the first week,
students completed \emph{Introduction to Ecological Forecasting} using
the R Shiny app. During the second week, students
completed\emph{Understanding Uncertainty in Ecological Forecasts} using
RMarkdown. This format permitted students to be introduced to a new
concept (ecological forecasting) in a user-friendly interface (R Shiny),
and then subsequently apply this new knowledge to a more in-depth task
(uncertainty quantification) while reinforcing and developing coding
skills (in RMarkdown).

\section*{}

{\label{curriculum-assessment-methods}}

\section*{Curriculum assessment
methods}

{\label{curriculum-assessment-methods}}

Independent assessment of the effectiveness of each Macrosystems EDDIE
ecological forecasting module was administered by the Science Education
Resource Center (SERC) at Carleton College. Through SERC's secure,
online portal, we delivered pre- and post-assessments of students who
completed one or more modules, and also collected instructor feedback
after module completion. Each module's student assessment included
Likert scale ranking and multiple choice questions that were consistent
across all modules as well as multiple choice and short answer questions
that were specific to the individual module. Students who completed
multiple modules, regardless of which modules they completed, were given
the pre- and post-assessments for the \emph{Introduction to Ecological
Forecasting} module to avoid survey fatigue. Instructor feedback surveys
included Likert scale ranking questions about module ease of use and
efficacy in meeting learning objectives, as well as multiple choice and
short answer questions about the delivery of the module, how likely the
instructor was to use high-frequency/long-term data and data from sensor
networks after teaching a module, and any other feedback the instructor
wished to provide.

Modules were assessed from January 2021 through May 2023 in 32 courses
across 22 institutions (Table S1), with a total of 596 students
completing one or more questions for both the pre- and post-assessment
and 26 instructors completing a feedback survey. We compared pre- and
post-module responses with paired Wilcoxon signed rank tests to evaluate
student growth. Due to widely varying numbers of students across
experience levels and courses (Table S1), we aggregated all students'
responses together for statistical analysis. All module assessment and
instructor survey questions, as well as details regarding the analysis
of assessment responses, can be found in the supplemental material (Text
S1, Tables S2-S7). All assessment was conducted following approved
Institutional Review Board (IRB) protocols (Virginia Tech IRB 19-669 and
Carleton College IRB 19-20 065). Anonymized, aggregated student
assessment and instructor feedback data and code to reproduce figures
and statistics are published in Lofton et al. (2024d).

\subsection*{}

{\label{curriculum-assessment-results}}

\section*{Curriculum assessment
results}

{\label{975798}}

\emph{Q1: How does completion of Macrosystems EDDIE ecological
forecasting modules affect student perceptions and knowledge of data
science and ecological forecasting skills?}

Completing one or more Macrosystems EDDIE ecological forecasting modules
improved student confidence in both their ecological forecasting and
data science skills (Fig. 2a, Fig. S1). Students gained the most
confidence from pre-module to post-module in their ecological
forecasting skills, such as generating a forecast (with 76\% of students
reporting an increase in confidence), quantifying uncertainty (74\%),
and communicating a forecast (71\%). Among data science skills, the
largest percentage of students gained confidence in modeling data
(65\%), followed by analyzing (52\%) and graphing data (43\%). Of the
students who did not exhibit a gain in confidence in their ecological
forecasting and data science skills, most (71-81\%) reported no change
in confidence rather than a decrease in confidence (Fig. 2a).

Module completion similarly improved student understanding of ecological
forecasting concepts, regardless of which module was completed (Fig.
2b). We observed a significant increase in students' ability to
correctly define an ecological forecast after completing any one of the
modules or multiple modules. Across modules, 31\% of students could
define an ecological forecast before module completion, while 76\% of
students could correctly complete this task after module completion.
Students also improved in their ability to define and describe other
ecological forecasting concepts, such as data assimilation and
uncertainty propagation; however, gains in student understanding in
these areas were uneven (Fig. S2, S3).

\emph{Q2: What are instructor perceptions of module ease of use and
efficacy in teaching data science and ecological forecasting concepts?}

Instructors reported that modules were usable and effective in teaching
data science skills and ecological forecasting (Fig. 3a). Importantly,
this increase in skills translated into gains outside of the classroom,
as most instructors reported that they were more likely to use
high-frequency, long-term data as well as data from established
ecological observatory sensor networks (e.g., NEON, GLEON) in their
research after teaching a module (Fig. 3b). Instructor responses to
module feedback surveys indicated that modules were `very effective' in
teaching data science skills and modeling, and `very effective' to
`extremely effective' in teaching ecological forecasting. In addition,
instructors reported that modules were `very easy' for students to use
across both the R Shiny materials and other materials (e.g., instructor
manual and introductory presentation), and `very easy' to teach (Fig.
3a).

Most instructors reported that they would use the modules again (Fig.
3b). Seven of the 26 instructors who filled out the feedback survey were
teaching assistants who were not responsible for course curriculum
design and might not teach the class in future years, and the rest were
faculty instructors of record for the course with the ability to make
future decisions regarding course curriculum. For a summary of
instructor qualitative feedback on the modules, see Text S2.

\section*{}

{\label{discussion}}

\section*{Discussion}

{\label{discussion}}

Through formal assessment of the Macrosystems EDDIE ecological
forecasting curriculum for undergraduates, we found that modules were
successful in increasing student confidence and knowledge of ecological
forecasting and data science (Fig. 2) and lowered the barrier of entry
to these fields for instructors (Fig. 3). In an era when data science
and ecological forecasting skills are increasingly needed to tackle
pressing biological and environmental science problems (Hampton et al.
2017, National Academies of Sciences 2018, Feng et al. 2020, Emery et
al. 2021), the Macrosystems EDDIE curriculum provides one pathway to
introducing these skills to both students and instructors.

Our results indicate that flexible, short, and easy-to-use modules
increase student confidence in data science and ecological forecasting
skills. In particular, students showed the greatest gains in confidence
in ecological forecasting skills (Fig. 2a), likely because they had
lower initial confidence in ecological forecasting skills (e.g.,
generating forecasts, for which students reported a median pre-module
Likert score of 2, or `slightly confident'). In comparison, student
confidence in their data science skills was relatively higher prior to
completing the module (e.g., graphing data, with a median pre-module
score of 4, or `very confident'; Fig. S1). The Dunning-Kruger effect
(Kruger and Dunning 1999) may explain the few students that exhibited
decreases in confidence (ranging from n = 24 students for the skill of
generating a forecast to n = 77 students for the skill of graphing
data), in which novice students overestimate their abilities, and as
they progress, are much better able to estimate their abilities, which
are less than they previously thought (Fig. S1). Ultimately, increased
student confidence and knowledge of data science and forecasting are
relevant beyond the life sciences, as workers with data science and
predictive modeling skills are sought across multiple sectors (Stanton
and Stanton 2019).

Instructor feedback after teaching a module indicates that the
Macrosystems EDDIE approach of ``just-in-time'' background skills
training (\emph{sensu} Novak et al. 1999) and robust instructional
supporting material may be successful strategies for instructor
professional development in data science. We received positive feedback
regarding the effect of Macrosystems EDDIE modules on both the growth of
instructor pedagogical (e.g., active learning) and disciplinary (e.g.,
data science and ecological forecasting) knowledge (Auerbach and Andrews
2018, Andrews et al. 2019). Most instructors said that Macrosystems
EDDIE modules were easy to use and very to extremely effective in
teaching ecological forecasting and data science concepts (Fig. 3).
Qualitative responses to our instructor survey indicated that a
comprehensive introduction to the structure, development, and
interpretation of the forecasting models used in each module (e.g.,
reviewing the structure of a simple ecosystem primary productivity model
in the \emph{Intro to Forecasting} module) was helpful to both students
and instructors (Text S2). In addition, instructors reported that the
accompanying instructor manual with detailed talking points for each
slide in the introductory presentation and suggested timing for each
activity within the module were helpful for classroom implementation.
Finally, most instructors reported that they were better equipped to use
long-term and high-frequency data and more likely to use sensor network
data after teaching a module (Fig. 3b), indicating that the modules
build skills and data science familiarity with instructors as well as
students. Overall, an important achievement of this adaptable,
accessible curriculum is ``training the trainers,'' in which an
instructor gains skills and knowledge in a new area, which are then
transferred to students (Beyer et al. 2009, Emery et al. 2021).

Modules were iteratively revised in response to student and faculty
feedback. For example, we revised early versions of the modules to
provide a more in-depth introduction in Activity A to the modeling
approaches used for forecasting as a method of ``just-in-time'' training
for both students and instructors. In addition, RMarkdown versions of
the \emph{Forecasts \& Uncertainty} and \emph{Forecasts \& Data} modules
were developed based on requests from instructors. The RMarkdown files
provide scaffolding for both students and instructors, who can start by
working through materials in the point-and-click R Shiny interface and
then move to the code ``under the hood'' of the Shiny application if
they wish. Importantly, this scaffolding may enable students and
instructors to transfer skills learned from teaching the module to their
own research projects, as they can modify the code for their own
datasets and research questions.

Macrosystems EDDIE ecological forecasting modules may facilitate the use
and analysis of large datasets, including NEON data, by instructors who
have not had extensive data science training. While interdisciplinary
collaborations with, e.g., computer scientists can facilitate analyses
with large computational demands, ecologists must still possess basic
data science skills, such as coding and data wrangling, modeling, and
visualization, to make these collaborations a success (Cheruvelil et al.
2014, Cheruvelil and Soranno 2018, Carey et al. 2019). In sum, we found
that the development of comprehensive supporting materials aimed to
provide background skills and pedagogical training for instructors is
critical for the effective implementation of new data science material
into existing undergraduate curricula and may also facilitate new
research efforts for instructors. Up-to-date versions of the modules are
available on GitHub (https://github.com/MacrosystemsEDDIE) and feedback
on module content and ease of use is welcome and can be submitted at
MacrosystemsEDDIE.org.

To train ecological and environmental scientists in data science and
ecological forecasting concepts and skills, these topics need to be
presented in a relevant, approachable way for both students and
instructors. Our data indicate that the Macrosystems EDDIE approach is
effective in engaging both instructors and students in data science and
ecological forecasting, and our observed increases in student confidence
may foster greater student ``science identity'' and retention in STEM
(Stets et al. 2017, Vincent-Ruz and Schunn 2018, O'Brien et al. 2020,
Bowser and Cid 2021). Ultimately, increased data science confidence and
proficiency by both undergraduate students and instructors unleashes
tremendous potential to leverage large datasets for addressing
environmental challenges.

\par\null

\section*{Acknowledgements}

{\label{acknowledgements}}

We thank the students and instructors who tested Macrosystems EDDIE
ecological forecasting modules, especially Kait Farrell, Matt Hipsey,
Leah Johnson, Nick Record, and Kiyoko Yokota. We also thank the Virginia
Tech Reservoir Group and our undergraduate focus groups for their
feedback, especially Caroline Bryant, Arpita Das, George Haynie, Ryan
Keverline, Michael Kricheldorf, Rose Thai, Evelyn Tipper, and Jacob
Wynne. We thank Monica Bruckner, Ashley Carlson, Kristin O'Connell, and
Cailin Huyck Orr at the Science Education Resource Center at Carleton
College for administrative assistance with module testing. All module
testing and assessment was conducted following approved Institutional
Review Board (IRB) protocols (Virginia Tech IRB 19-669 and Carleton
College IRB 19-20 065). This work was funded by NSF DEB-1926050,
DBI-1933016, and EF-2318861.

\par\null

\section*{Data Availability Statement}

{\label{236536}}

Anonymized, aggregated student assessment and instructor feedback data
as well as code to recreate figures and statistics associated with the
manuscript are published with a DOI in the Zenodo repository in Lofton
et al. (2024d); access at \url{https://doi.org/10.5281/zenodo.10932209}.
All students and faculty consented to participate in the study per our
Institutional Review Board (IRB) protocols (Virginia Tech IRB 19-669 and
Carleton College IRB 19-20 065). The published data have been
pre-processed to remove any sensitive or personally identifying
information.

\par\null

\section*{References}

{\label{references}}

Andrews TC, Auerbach AJJ, Grant EF. 2019. Exploring the relationship
between teacher knowledge and active-learning implementation in large
college biology courses. CBE---Life Sciences Education 18: ar48.

Auerbach AJJ, Andrews TC. 2018. Pedagogical knowledge for
active-learning instruction in large undergraduate biology courses: a
large-scale qualitative investigation of instructor thinking.
International Journal of STEM Education 5: 19.

Auker LA, Barthelmess EL. 2020. Teaching R in the undergraduate ecology
classroom: approaches, lessons learned, and recommendations. Ecosphere
11: e03060.

Belland BR. 2014. Scaffolding: Definition, Current Debates, and Future
Directions. Pages 505--518 in Spector JM, Merrill MD, Elen J, and Bishop
MJ, eds. Handbook of Research on Educational Communications and
Technology. Springer New York.

Beyer CJ, Delgado C, Davis EA, Krajcik J. 2009. Investigating teacher
learning supports in high school biology curricular programs to inform
the design of educative curriculum materials. Journal of Research in
Science Teaching 46: 977--998.

Bodner K, Rauen Firkowski C, Bennett JR, Brookson C, Dietze M, Green S,
Hughes J, Kerr J, Kunegel-Lion M, Leroux SJ, McIntire E, Moln\selectlanguage{ngerman}ár PK,
Simpkins C, Tekwa E, Watts A, Fortin M. 2021. Bridging the divide
between ecological forecasts and environmental decision making.
Ecosphere 12: e03869.

Bonwell CC, Eison JA. 1991. Active learning: Creating excitement in the
classroom. 1991 Association for the Study of Higher Education-Education
Resources Information Center (ASHE-ERIC) higher education report. ERIC.

Bowser G, Cid CR. 2021. Developing the ecological scientist mindset
among underrepresented students in ecology fields. Ecological
Applications 31: e02348.

Bradford JB, Weltzin J, McCormick ML, Baron J, Bowen Z, Bristol S,
Carlisle D, Crimmins T, Cross PC, DeVivo J, Dietze M, Freeman M,
Goldberg J, Hooten M, Hsu L, Jenni K, Keisman JL, Kennen J, Lee K,
Lesmes DP, Loftin K, Miller BW, Murdoch PS, Newman J, Prentice KL,
Rangwala I, Read J, Sieracki J, Sofaer H, Thur S, Toevs G, Werner F,
White CL, White T, Wiltermuth MT. 2020. Ecological forecasting---21st
century science for 21st century management. U.S. Geological Survey.
Open-File Report 2020--1073.

Bybee RW, Taylor JA, Gardner A, Van Scotter P, Powell JC, Westbrook A,
Landes N. 2006. The BSCS 5E instructional model: Origins and
effectiveness. Colorado Springs, Co: BSCS Science Learning.

Carey CC, Darner Gougis R, Klug JL, O'Reilly CM, Richardson DC. 2015. A
model for using environmental data\selectlanguage{english}-driven Inquiry and exploration to
teach limnology to undergraduates. Limnology and Oceanography Bulletin
24: 32--35.

Carey CC, Farrell KJ, Hounshell AG, O'Connell K. 2020. Macrosystems
EDDIE teaching modules significantly increase ecology students'
proficiency and confidence working with ecosystem models and use of
systems thinking. Ecology and Evolution 10: 12515--12527.

Carey CC, Ward NK, Farrell KJ, Lofton ME, Krinos AI, McClure RP,
Subratie KC, Figueiredo RJ, Doubek JP, Hanson PC, Papadopoulos P,
Arzberger P. 2019. Enhancing collaboration between ecologists and
computer scientists: lessons learned and recommendations forward.
Ecosphere 10: e02753.

Carrillo CM, Ault TR, Wilks DS. 2018. Spring onset predictability in the
North American multimodel ensemble. Journal of Geophysical Research:
Atmospheres 123: 5913--5926.

Chang W, Cheng J, Allaire JJ, Sievert C, Schloerke B, Xie Y, Allen J,
McPherson J, Dipert A, Borges B. 2023. Shiny: Web Application Framework
for R.

Cheruvelil KS, Soranno PA. 2018. Data-intensive ecological research is
catalyzed by open science and team science. BioScience 68: 813--822.

Cheruvelil KS, Soranno PA, Weathers KC, Hanson PC, Goring SJ, Filstrup
CT, Read EK. 2014. Creating and maintaining high\selectlanguage{english}-performing
collaborative research teams: the importance of diversity and
interpersonal skills. Frontiers in Ecology and the Environment 12:
31--38.

Cleverly J, Eamus D, Edwards W, Grant M, Grundy MJ, Held A, Karan M,
Lowe AJ, Prober SM, Sparrow B, Morris B. 2019. TERN, Australia's land
observatory: addressing the global challenge of forecasting ecosystem
responses to climate variability and change. Environmental Research
Letters 14: 095004.

Cooke J, Araya Y, Bacon KL, Bagniewska JM, Batty LC, Bishop TR, Burns M,
Charalambous M, Daversa DR, Dougherty LR, Dyson M, Fisher AM, Forman D,
Garcia C, Harney E, Hesselberg T, John EA, Knell RJ, Maseyk K, Mauchline
AL, Peacock J, Pernetta AP, Pritchard J, Sutherland WJ, Thomas RL, Tigar
B, Wheeler P, White RL, Worsfold NT, Lewis Z. 2021. Teaching and
learning in ecology: a horizon scan of emerging challenges and
solutions. Oikos 130: 15--28.

Cuddington K, Abbott KC, Adler FR, Aydeniz M, Dale R, Gross LJ, Hastings
A, Hobson EA, Karatayev VA, Killion A, Madamanchi A, Marraffini ML,
McCombs AL, Samyono W, Shiu S-H, Watanabe KH, White ER. 2023. Challenges
and opportunities to build quantitative self-confidence in biologists.
BioScience 73: 364--375.

Dauphin B, Rellstab C, Wuest RO, Karger DN, Holderegger R, Gugerli F,
Manel S. 2023. Re-thinking the environment in landscape genomics. Trends
in Ecology \& Evolution 38: 261--274.

D'Avanzo C, Anderson CW, Hartley LM, Pelaez N. 2012. A
faculty-development model for transforming introductory biology and
ecology courses. BioScience 62: 416--427.

Derting TL, Ebert-May D, Henkel TP, Maher JM, Arnold B, Passmore HA.
2016. Assessing faculty professional development in STEM higher
education: Sustainability of outcomes. Science Advances 2: e1501422.

Dietze M. 2017a. Ecological Forecasting. Princeton University Press.

Dietze MC. 2017b. Prediction in ecology: a first\selectlanguage{english}-principles framework.
Ecological Applications 27: 2048--2060.

Dietze MC, Fox A, Beck-Johnson LM, Betancourt JL, Hooten MB, Jarnevich
CS, Keitt TH, Kenney MA, Laney CM, Larsen LG, Loescher HW, Lunch CK,
Pijanowski BC, Randerson JT, Read EK, Tredennick AT, Vargas R, Weathers
KC, White EP. 2018. Iterative near-term ecological forecasting: Needs,
opportunities, and challenges. Proceedings of the National Academy of
Sciences 115: 1424--1432.

Ebert-May D, Derting TL, Hodder J, Momsen JL, Long TM, Jardeleza SE.
2011. What we say is not what we do: Effective evaluation of faculty
professional development programs. BioScience 61: 550--558.

Emery NC, Crispo E, Supp SR, Farrell KJ, Kerkhoff AJ, Bledsoe EK,
O'Donnell KL, McCall AC, Aiello-Lammens ME. 2021. Data science in
undergraduate life science education: A need for instructor skills
training. BioScience 71: 1274--1287.

Ernest SKM, Ye H, White EP. 2023. Ecological Forecasting and Dynamics: A
graduate course on the fundamentals of time series and forecasting in
ecology. Journal of Open Source Education 6: 198.

Farley SS, Dawson A, Goring SJ, Williams JW. 2018. Situating ecology as
a big-data science: Current advances, challenges, and solutions.
BioScience 68: 563--576.

Farrell KJ, Carey CC. 2018. Power, pitfalls, and potential for
integrating computational literacy into undergraduate ecology courses.
Ecology and Evolution 8: 7744--7751.

Feng X, Qiao H, Enquist BJ. 2020. Doubling demands in programming skills
call for ecoinformatics education. Frontiers in Ecology and the
Environment 18: 123--124.

Freeman S, Eddy SL, McDonough M, Smith MK, Okoroafor N, Jordt H,
Wenderoth MP. 2014. Active learning increases student performance in
science, engineering, and mathematics. Proceedings of the National
Academy of Sciences 111: 8410--8415.

Goodman KJ, Parker SM, Edmonds JW, Zeglin LH. 2015. Expanding the scale
of aquatic sciences: the role of the National Ecological Observatory
Network (NEON). Freshwater Science 34: 377--385.

Hampton SE, Jones MB, Wasser LA, Schildhauer MP, Supp SR, Brun J,
Hernandez RR, Boettiger C, Collins SL, Gross LJ, Fernandez DS, Budden A,
White EP, Teal TK, Labou SG, Aukema JE. 2017. Skills and knowledge for
data-intensive environmental research. BioScience 67: 546--557.

Hampton SE, Strasser CA, Tewksbury JJ, Gram WK, Budden AE, Batcheller
AL, Duke CS, Porter JH. 2013. Big data and the future of ecology.
Frontiers in Ecology and the Environment 11: 156--162.

Hazen EL, Scales KL, Maxwell SM, Briscoe DK, Welch H, Bograd SJ, Bailey
H, Benson SR, Eguchi T, Dewar H, Kohin S, Costa DP, Crowder LB, Lewison
RL. 2018. A dynamic ocean management tool to reduce bycatch and support
sustainable fisheries. Science Advances 4: eaar3001.

Heilman KA, Dietze MC, Arizpe AA, Aragon J, Gray A, Shaw JD, Finley AO,
Klesse S, DeRose RJ, Evans MEK. 2022. Ecological forecasting of tree
growth: Regional fusion of tree\selectlanguage{english}-ring and forest inventory data to
quantify drivers and characterize uncertainty. Global Change Biology 28:
2442--2460.

Heim AB, Holt EA. 2018. Comparing student, instructor, and expert
perceptions of learner-centeredness in post-secondary biology
classrooms. PLOS ONE 13: e0200524.

Hounshell AG, Farrell KJ, Carey CC. 2021. Macrosystems EDDIE teaching
modules increase students' ability to define, interpret, and apply
concepts in macrosystems ecology. Education Sciences 11: 382.

Hughes PW, Ellefson MR. 2013. Inquiry-based training improves teaching
effectiveness of biology teaching assistants. PLoS ONE 8: e78540.

Johnson LR, Gramacy RB, Cohen J, Mordecai E, Murdock C, Rohr J, Ryan SJ,
Stewart-Ibarra AM, Weikel D. 2018. Phenomenological forecasting of
disease incidence using heteroskedastic Gaussian processes: A dengue
case study. The Annals of Applied Statistics 12: 27--66.

Keller M, Schimel DS, Hargrove WW, Hoffman FM. 2008. A continental
strategy for the National Ecological Observatory Network. The Ecological
Society of America: 282-284.

Kruger J, Dunning D. 1999. Unskilled and unaware of it: How difficulties
in recognizing one's own incompetence lead to inflated self-assessments.
Journal of Personality and Social Psychology 77: 1121--1134.

LaDeau SL, Han BA, Rosi-Marshall EJ, Weathers KC. 2017. The next decade
of big data in ecosystem science. Ecosystems 20: 274--283.

Lewis ASL, Rollinson CR, Allyn AJ, Ashander J, Brodie S, Brookson CB,
Collins E, Dietze MC, Gallinat AS, Juvigny\selectlanguage{english}-Khenafou N, Koren G, McGlinn
DJ, Moustahfid H, Peters JA, Record NR, Robbins CJ, Tonkin J, Wardle GM.
2023. The power of forecasts to advance ecological theory. Methods in
Ecology and Evolution 14: 746--756.

Lewis ASL, Woelmer WM, Wander HL, Howard DW, Smith JW, McClure RP,
Lofton ME, Hammond NW, Corrigan RS, Thomas RQ, Carey CC. 2022. Increased
adoption of best practices in ecological forecasting enables comparisons
of forecastability. Ecological Applications 32: e02500.

Liu Q, Rowe MD, Anderson EJ, Stow CA, Stumpf RP, Johengen TH. 2020.
Probabilistic forecast of microcystin toxin using satellite remote
sensing, in situ observations and numerical modeling. Environmental
Modelling \& Software 128: 104705.

Lofton ME, Moore TN, Thomas RQ, Carey CC. 2024a. Macrosystems EDDIE
Module 7: Using Data to Improve Ecological Forecasts, R Shiny version 1.
\url{https://doi.org/10.5281/zenodo.10903839}~

Lofton ME, Moore TN, Thomas RQ, Carey CC. 2024b. Macrosystems EDDIE
Module 7: Using Data to Improve Ecological Forecasts, RMarkdown version
1. \url{https://doi.org/10.5281/zenodo.10909589}~

Lofton ME, Moore TN, Thomas RQ, Carey CC. 2024c. Macrosystems EDDIE
Module 7: Using Data to Improve Ecological Forecasts (Instructor
Materials).
\url{https://doi.org/10.6073/pasta/6c8478d9aa04eeab55646ffa8e62b278}~

Lofton ME, Moore TN, Woelmer WM, Thomas RQ, Carey CC. 2024d. A modular
curriculum to teach undergraduates ecological forecasting improves
student and instructor confidence in their data science skills (Code
Repository). \url{https://doi.org/10.5281/zenodo.10932209}~

Luo Y, Ogle K, Tucker C, Fei S, Gao C, LaDeau S, Clark JS, Schimel DS.
2011. Ecological forecasting and data assimilation in a data-rich era.
Ecological Applications 21: 1429--1442.

Mcloughlin MP, Stewart R, McElligott AG. 2019. Automated bioacoustics:
methods in ecology and conservation and their potential for animal
welfare monitoring. Journal of The Royal Society Interface 16: 20190225.

Moore TN, Lofton ME, Carey CC, Thomas RQ. 2023a. Macrosystems EDDIE
Module 6: Understanding Uncertainty in Ecological Forecasts (Instructor
Materials).
\url{https://doi.org/10.6073/pasta/1ce758925388a9273083c24ed0ee0c05}~

Moore TN, Lofton ME, Carey CC, Thomas RQ. 2023b. Macrosystems EDDIE
Module 6: Understanding Uncertainty in Ecological Forecasts, R Shiny
version 2. \url{https://doi.org/10.5281/zenodo.10380760}~

Moore TN, Lofton ME, Carey CC, Thomas RQ. 2023c. Macrosystems EDDIE
Module 6: Understanding Uncertainty in Ecological Forecasts, RMarkdown
version 1. \url{https://doi.org/10.5281/zenodo.10380340}~

Moore TN, Lofton, ME, Carey CC, Thomas RQ. 2024a. Macrosystems EDDIE
Module 5: Introduction to Ecological Forecasting, R Shiny version 2.
\url{https://doi.org/10.5281/zenodo.10733117}~

Moore TN, Lofton, ME, Carey CC, Thomas RQ. 2024b. Macrosystems EDDIE
Module 5 version 2: Introduction to Ecological Forecasting (Instructor
Materials).
\url{https://doi.org/10.6073/pasta/b3953d81b3b4158e0ec5375c04774bd2}~

Moore TN, Thomas RQ, Woelmer WM, Carey CC. 2022. Integrating ecological
forecasting into undergraduate ecology curricula with an R Shiny
application-based teaching module. Forecasting 4: 604--633.

Naithani K, Jones M, Grayson KL. 2022. Building communities of teaching
practice and data\selectlanguage{english}-driven open education resources with NEON faculty
mentoring networks. Ecosphere 13: e4210.

Nathan R, Monk CT, Arlinghaus R, Adam T, Alos J, Assaf M, Baktoft H,
Beardsworth CE, Bertram MG, Bijleveld AI, Brodin T, Brooks JL,
Campos-Candela A, Cooke SJ, Gjelland KO, Gupte PR, Harel R, Hellstrom G,
Jeltsch F, Killen SS, Klefoth T, Langrock R, Lennox RJ, Lourie E, Madden
JR, Orchan Y, Pauwels IS, \selectlanguage{polish}Ř\selectlanguage{ngerman}íha M, Roeleke M, Schlägel UE, Shohami D,
Signer J, Toledo S, Vilk O, Westrelin S, Whiteside MA, Jari\selectlanguage{polish}ć \selectlanguage{english}I. 2022.
Big-data approaches lead to an increased understanding of the ecology of
animal movement. Science 375: eabg1780.

National Academies of Sciences E. 2018. Data science for undergraduates:
Opportunities and options. National Academies Press.

Niu S, Luo Y, Dietze MC, Keenan TF, Shi Z, Li J, Iii FSC. 2014. The role
of data assimilation in predictive ecology. Ecosphere 5: 1--16.

Novak GM, Patterson ET, Gavrin AD, Christian W. 1999. Just in time
teaching. American Association of Physics Teachers.

O'Brien LT, Bart HL, Garcia DM. 2020. Why are there so few ethnic
minorities in ecology and evolutionary biology? Challenges to inclusion
and the role of sense of belonging. Social Psychology of Education 23:
449--477.

O'Connell K, Altermatt E, Darner R, Iverson E, Meixner T, O'Reilly C,
Orr CH, Soule D. 2024. Project EDDIE Module Development Rubric.

O'Reilly CM, Gougis RD, Klug JL, Carey CC, Richardson DC, Bader NE,
Soule DC, Castendyk D, Meixner T, Stomberg J, Weathers KC, Hunter W.
2017. Using Large Data Sets for Open-Ended Inquiry in Undergraduate
Science Classrooms. BioScience 67: 1052--1061.

Ouellet-Proulx S, St-Hilaire A, Boucher M-A. 2017. Water Temperature
Ensemble Forecasts: Implementation Using the CEQUEAU Model on Two
Contrasted River Systems. Water 9: 457.

Owens MT, Trujillo G, Seidel SB, Harrison CD, Farrar KM, Benton HP,
Blair JR, Boyer KE, Breckler JL, Burrus LW, Byrd DT, Caporale N,
Carpenter EJ, Chan Y-HM, Chen JC, Chen L, Chen LH, Chu DS, Cochlan WP,
Crook RJ, Crow KD, De La Torre JR, Denetclaw WF, Dowdy LM, Franklin D,
Fuse M, Goldman MA, Govindan B, Green M, Harris HE, He Z-H, Ingalls SB,
Ingmire P, Johnson ARB, Knight JD, LeBuhn G, Light TL, Low C, Lund L,
M\selectlanguage{ngerman}árquez-Magaña LM, Miller-Sims VC, Moffatt CA, Murdock H, Nusse GL,
Parker VT, Pasion SG, Patterson R, Pennings PS, Ramirez JC, Ramirez RM,
Riggs B, Rohlfs RV, Romeo JM, Rothman BS, Roy SW, Russo-Tait T, Sehgal
RNM, Simonin KA, Spicer GS, Stillman JH, Swei A, Tempe LC, Vredenburg
VT, Weinstein SL, Zink AG, Kelley LA, Domingo CR, Tanner KD. 2018.
Collectively improving our teaching: Attempting biology department--wide
professional development in scientific teaching. CBE---Life Sciences
Education 17: ar2.

Petchey OL, Pontarp M, Massie TM, Kéfi S, Ozgul A, Weilenmann M,
Palamara GM, Altermatt F, Matthews B, Levine JM, Childs DZ, McGill BJ,
Schaepman ME, Schmid B, Spaak P, Beckerman AP, Pennekamp F, Pearse IS.
2015. The ecological forecast horizon, and examples of its uses and
determinants. Ecology Letters 18: 597--611.

Schussler EE, Read Q, Marbach-Ad G, Miller K, Ferzli M. 2015. Preparing
biology braduate teaching assistants for their roles as instructors: An
assessment of institutional approaches. CBE---Life Sciences Education
14: ar31.

Stanton AD, Stanton WW. 2019. Closing the skills gap: Finding skilled
analytics professionals for a dynamically changing data-driven
environment. Applied Marketing Analytics 5: 170--184.

Stets JE, Brenner PS, Burke PJ, Serpe RT. 2017. The science identity and
entering a science occupation. Social Science Research 64: 1--14.

Theobald EJ, Hill MJ, Tran E, Agrawal S, Arroyo EN, Behling S, Chambwe
N, Cintrón DL, Cooper JD, Dunster G, Grummer JA, Hennessey K, Hsiao J,
Iranon N, Jones L, Jordt H, Keller M, Lacey ME, Littlefield CE, Lowe A,
Newman S, Okolo V, Olroyd S, Peecook BR, Pickett SB, Slager DL,
Caviedes-Solis IW, Stanchak KE, Sundaravardan V, Valdebenito C, Williams
CR, Zinsli K, Freeman S. 2020. Active learning narrows achievement gaps
for underrepresented students in undergraduate science, technology,
engineering, and math. Proceedings of the National Academy of Sciences
117: 6476--6483.

Villarroel V, Bloxham S, Bruna D, Bruna C, Herrera-Seda C. 2018.
Authentic assessment: creating a blueprint for course design. Assessment
\& Evaluation in Higher Education 43: 840--854.

Vincent-Ruz P, Schunn CD. 2018. The nature of science identity and its
role as the driver of student choices. International Journal of STEM
Education 5: 48.

Weathers KC, Hanson PC, Arzberger P, Brentrup J, Brookes J, Carey CC,
Gaiser E, Gaiser E, Hamilton DP, Hong GS, Ibelings B, Istvánovics V,
Jennings E, Kim B, Kratz T, Lin F, Muraoka K, O'Reilly C, Rose KC, Ryder
E, Zhu G. 2013. The Global Lake Ecological Observatory Network (GLEON):
The evolution of grassroots network science. Limnology and Oceanography
Bulletin 22: 71--73.

Wiliam D. 2011. What is assessment for learning? Studies in Educational
Evaluation 37: 3--14.

Williams JJ, Drew JC, Galindo-Gonzalez S, Robic S, Dinsdale E, Morgan
WR, Triplett EW, Burnette JM, Donovan SS, Fowlks ER, Goodman AL,
Grandgenett NF, Goller CC, Hauser C, Jungck JR, Newman JD, Pearson WR,
Ryder EF, Sierk M, Smith TM, Tosado-Acevedo R, Tapprich W, Tobin TC,
Toro-Martínez A, Welch LR, Wilson MA, Ebenbach D, McWilliams M,
Rosenwald AG, Pauley MA. 2019. Barriers to integration of bioinformatics
into undergraduate life sciences education: A national study of US life
sciences faculty uncover significant barriers to integrating
bioinformatics into undergraduate instruction. PLOS ONE 14: e0224288.

Willson AM, Gallo H, Peters JA, Abeyta A, Bueno~Watts N, Carey CC, Moore
TN, Smies G, Thomas RQ, Woelmer WM, McLachlan JS. 2023. Assessing
opportunities and inequities in undergraduate ecological forecasting
education. Ecology and Evolution 13: e10001.

Woelmer W, Moore T, Thomas Q, Carey. 2022. Macrosystems EDDIE Module 8:
Using Ecological Forecasts to Guide Decision-Making (R Shiny
application). \url{https://doi.org/10.5281/zenodo.7074674}~

Woelmer WM, Moore TN, Lofton ME, Thomas RQ, Carey CC. 2023a. Embedding
communication concepts in forecasting training increases students'
understanding of ecological uncertainty. Ecosphere 14: e4628.

Woelmer WM, Thomas RQ, Moore TN, Carey CC. 2023b. Macrosystems EDDIE
Module 8: Using Ecological Forecasts to Guide Decision-Making
(Instructor Materials).
\url{https://doi.org/10.6073/pasta/8bf4a076433f0e9f74f1d764d5bd4c3f}~

Xie Y, Allaire JJ, Grolemund G. 2018. R markdown: The definitive guide.
CRC Press.

\par\null

\section*{Biographical information}

{\label{494354}}

\textbf{Mary Lofton} is a postdoctoral research associate in the
Virginia Tech Center for Ecosystem Forecasting. Her research interests
are modeling and forecasting of freshwater ecosystems and development of
educational resources to help students gain data science and systems
thinking skills.

\textbf{Tadhg Moore} is currently a lake scientist at Limnotrack in
Hamilton, New Zealand. He develops and delivers online platforms for
water quality monitoring and modelling data for water resource managers.

\textbf{Whitney Woelmer} is currently a postdoctoral researcher at the
University of Waikato in Hamilton, New Zealand. She is a quantitative
limnologist interested in understanding and predicting freshwater
ecosystem dynamics.

\textbf{R. Quinn Thomas} is an associate professor and co-director of
the Virginia Tech Center for Ecosystem Forecasting. His research
interests are modeling and forecasting of forest and freshwater
ecosystem dynamics.

\textbf{Cayelan Carey} is a professor and co-director of the Virginia
Tech Center for Ecosystem Forecasting. Her research integrates field
studies, modeling, and forecasting to understand freshwater ecosystem
dynamics.

\par\null

\section*{Figures}

{\label{358820}}\par\null\selectlanguage{english}
\begin{figure}[H]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/Fig1/Fig1}
\caption{{Conceptual diagram of Macrosystems EDDIE ecological forecasting
curriculum content and workflow to guide instructors on potential ways
the modules could be implemented into their courses, depending on
learning objectives and student experience level.
{\label{426132}}%
}}
\end{center}
\end{figure}\selectlanguage{english}
\begin{figure}[H]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/image2/image2}
\caption{{Effect of module completion on (a) student confidence in data science
(graph data, analyze data, model data) and forecasting (generate
forecast, quantify uncertainty, communicate forecast) skills and (b)
knowledge of ecological forecasting. (a) Student confidence was assessed
via Likert scores, where 1 was ``not confident at all'' and 5 was
``extremely confident''. Changes in student confidence were calculated
by subtracting each student's pre-module score from the post-module
score. Numbers above each bar indicate the number of student responses
obtained for each assessment question. (b) Differences in students'
ability to identify the definition of an ecological forecast before and
after module completion were assessed via paired, two-sided Wilcoxon
signed rank tests; * p \textless{} 0.05; ** p \textless{} 0.01; *** p
\textless{} 0.001.%
}}
\end{center}
\end{figure}\selectlanguage{english}
\begin{figure}[H]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/image3/image3}
\caption{{Instructor perceptions of module ease of use and efficacy in teaching
data science and ecological forecasting skills (a) and the number of
instructors who reported they would use the module again, were better
equipped to use high-frequency/long-term data, and were more likely to
use sensor network data (b). (a) Numbers above each violin plot indicate
the number of instructor responses received for each question. (b) NEON
= National Ecological Observatory Network; GLEON = Global Lake
Ecological Observatory Network.%
}}
\end{center}
\end{figure}

\section*{}

{\label{573258}}

\section*{Supplemental Information}

{\label{573258}}

\textbf{Hosted file}

\verb`supplement.docx`  available at \url{https://authorea.com/users/540012/articles/741831-a-modular-curriculum-to-teach-undergraduates-ecological-forecasting-improves-student-and-instructor-confidence-in-their-data-science-skills}

\selectlanguage{english}
\FloatBarrier
\end{document}

