\documentclass[10pt]{article}

\usepackage{fullpage}
\usepackage{setspace}
\usepackage{parskip}
\usepackage{titlesec}
\usepackage[section]{placeins}
\usepackage{xcolor}
\usepackage{breakcites}
\usepackage{lineno}
\usepackage{hyphenat}





\PassOptionsToPackage{hyphens}{url}
\usepackage[colorlinks = true,
            linkcolor = blue,
            urlcolor  = blue,
            citecolor = blue,
            anchorcolor = blue]{hyperref}
\usepackage{etoolbox}
\makeatletter
\patchcmd\@combinedblfloats{\box\@outputbox}{\unvbox\@outputbox}{}{%
  \errmessage{\noexpand\@combinedblfloats could not be patched}%
}%
\makeatother


\usepackage[round]{natbib}
\let\cite\citep




\renewenvironment{abstract}
  {{\bfseries\noindent{\abstractname}\par\nobreak}\footnotesize}
  {\bigskip}

\titlespacing{\section}{0pt}{*3}{*1}
\titlespacing{\subsection}{0pt}{*2}{*0.5}
\titlespacing{\subsubsection}{0pt}{*1.5}{0pt}


\usepackage{authblk}


\usepackage{graphicx}
\usepackage[space]{grffile}
\usepackage{latexsym}
\usepackage{textcomp}
\usepackage{longtable}
\usepackage{tabulary}
\usepackage{booktabs,array,multirow}
\usepackage{amsfonts,amsmath,amssymb}
\providecommand\citet{\cite}
\providecommand\citep{\cite}
\providecommand\citealt{\cite}
% You can conditionalize code for latexml or normal latex using this.
\newif\iflatexml\latexmlfalse
\providecommand{\tightlist}{\setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}%

\AtBeginDocument{\DeclareGraphicsExtensions{.pdf,.PDF,.eps,.EPS,.png,.PNG,.tif,.TIF,.jpg,.JPG,.jpeg,.JPEG}}

\usepackage[utf8]{inputenc}
\usepackage[ngerman,english]{babel}








% Use this header.tex file for:
% 1. frontmatter/preamble LaTeX definitions
%  - Example: \usepackage{xspace}
% 2. global macros available in all document blocks
%  - Example: \def\example{This is an example macro.}
%
% You should ONLY add such definitions in this header.tex space,
%  and treat the main article content as the body/mainmatter of your document
% Preamble-only macros such as \documentclass and \usepackage are
% NOT allowed in the main document, and definitions will be local to the current block.


\begin{document}

\title{Review: ``CoRR --- The Cloud of Reproducible Records''}



\author[1]{Fernando Chirigati}%
\affil[1]{Computing in Science and Engineering}%


\vspace{-1em}



  
  \date{May 27, 2024}


\begingroup
\let\center\flushleft
\let\endcenter\endflushleft
\maketitle
\endgroup








\sloppy


This is a review of manuscript CiSESI-2018-02-0016 submitted to
Computing in Science \& Engineering: ``CoRR --- The Cloud of
Reproducible Records'' (Congo, Traor\selectlanguage{ngerman}é, Hill and Wheeler, 2018).

\section*{}

{\label{355712}}

\section*{Overview}

{\label{355712}}\par\null

The paper presents CoRR (Cloud of Reproducible Records), a Web platform
for storing and managing records from different tools that create
snapshots of computational environments for reproducibility purposes.
The authors refer to these tools as CVC (Computation Version Control)
tools. In a nutshell, these tools capture the state of the environment
in which computational environments are run (e.g.: OS and hardware
information, library dependencies, system variables, etc.), in addition
to the code and the data. Examples of CVC tools include Sumatra,
ReproZip, and CDE. The authors argue that CVC tools are facing major
issues in adoption, and that one of the main reasons is related to the
lack of a Web interface for sharing and managing CVC records (similar to
what GitHub or BitBucket do for SVC tools). CoRR was designed to fill
this gap and to facilitate the integration among these tools by
providing a common management platform.

\par\null

A common platform for these tools is indeed interesting and useful for
reproducibility. However, the contributions of the manuscript are still
not clear. More details are provided in the next section, but here is a
summary of the main issues:

\par\null

\begin{enumerate}
\tightlist
\item
  The differences between CoRR and existing data repositories need to be
  made more clear.~
\item
  The name CVC is misleading.
\item
  The way that the metadata is stored in the platform is not clear.
\item
  Diffs in the platform are manual rather than automatic, and there is
  no discussion on the challenges related to these diffs.
\item
  Related work about provenance,~workflows, and repositories are
  missing.
\end{enumerate}

\par\null

My recommendation is "\emph{Author Should Prepare A Major Revision For A
Second Review}".

\par\null

\section*{Detailed Review}

{\label{765048}}\par\null

I should note that I tried to get access to the platform, but I wasn't
able to (no confirmation email was sent as of yet). I also tried to use
the search feature in the main Website, but it keeps loading after
pressing the return key and no results are returned.

\par\null

\subsection*{1. The differences between CoRR and existing data
repositories need to be made more
clear.}

{\label{574946}}

\subsection*{}

{\label{838234}}

CoRR is a repository of computation records. But what makes it different
from other data repositories? This is still not clear to me.

\par\null

It seems that one of the main benefits of CoRR is the ability of
exposing the metadata that the CVC tools capture, and allowing these to
be queryable. For instance, in a regular data repository, if I want to
search for projects that used scikit-learn, I would only be able to find
such information if it were present in the description of the artifacts.
On the other hand, in CoRR, one could make this information
automatically available for querying, since tools like Sumatra or
ReproZip capture such dependencies.

\par\null

Are these metadata indeed queryable in CoRR? If yes, this is a major
benefit and should be made more explicit in the paper. In general, the
paper would benefit from a section where authors explicitly discuss the
main differences between CoRR and existing data repositories when it
comes to CVC tools, i.e., why would someone choose to use CoRR and not
any of the existing repositories?

\par\null

\subsection*{2. The name CVC is
misleading.}

{\label{677206}}\par\null

CVC stands for Computation Version Control, but neither ReproZip nor CDE
do version control: they do create a snapshot of the computation, but
they do not have a mechanism for version control. The authors seem to be
referring to tools that capture provenance related to the computational
environment, but not necessarily that provide version control, so the
nomenclature should be changed.

\par\null

\subsection*{3. The way that the metadata is stored in the platform is
not
clear.}

{\label{664433}}\par\null

The section ``Adaptive and Open Database Model'' was not clear enough to
me. How are different metadata (from different tools) stored in a single
data store? The authors do present the MongoDB's models, but there are
no details on how different metadata are integrated into a single model.
And why not use some representation like
PROV~(\url{https://www.w3.org/TR/prov-primer/}) for integrating the
models?

\par\null

\subsection*{4. Diffs in the platform are manual rather than automatic,
and there is no discussion on the challenges related to these
diffs.}

{\label{894256}}\par\null

At the end of the paper, the authors discuss the concept of diff as a
way to tell whether a computation X is a replicate, a repeat, or a
reproduction of a computation Y. This is a really cool feature, but it
looks like users in CoRR need to define such diffs manually, which is
certainly not scalable when dealing with hundreds of computations.
Automatically figuring out if two computations are similar in terms of
reproducibility is challenging, in particular if they were captured by
different tools. But this is certainly a very useful feature for a
repository such as CoRR. I was expecting at least a more detailed
discussion about this.

\par\null

\subsection*{5. Related work about provenance, workflows, and
repositories are
missing.}

{\label{215986}}\par\null

Capturing provenance from computations is certainly not a novel topic,
and some references are missing. Authors should acknowledge scientific
workflow management systems (e.g.: Taverna~\citep{Missier_2010},
Kepler~\citep{Ludascher:2006:SWM:1148437.1148454}, and VisTrails \citep{vo2011}), since they
are known for capturing provenance from experiments \citep{Davidson_2008}
\citep{Freire_2008}. In terms of representing provenance information, the
authors should take a look at PROV
(\url{https://www.w3.org/TR/prov-primer/}).~

\par\null

There are also other tools that capture provenance from computational
environments: although some might not be widely adopted, it is worth
mentioning them. These are: PTU~\citep{pham},
CARE~\citep{Janin_2014}, Arnold~\citep{186206}, and noWorkflow
\citep{Murta_2015}.

\par\null

I~ also recommend taking a look at the related work section of these
papers to see if there are additional relevant references as well.

\par\null

Finally, since CoRR is a repository, it is important to acknowledge
existing collaborative data repositories, e.g.: Dataverse, figshare,
OSF, etc. Again, as I mentioned before, it is important to provide a
detailed comparison against these repositories.

\par\null

\section*{Additional Comments}

{\label{920682}}\par\null

\begin{itemize}
\tightlist
\item
  The authors mention in the Introduction that the lack of a platform
  for storing and managing computation records is probably one of the
  main reasons for the slow adoption of CVC tools. However, there is no
  evidence for that. Is there any reference or further study that the
  authors can provide to back this up? The motivation is not clear.
\item
  There are other tools (including noWorkflow) supported by CoRR (Figure
  3). Why aren't these tools mentioned in the paper?
\item
  Using forked repositories from the existing tools might not be ideal.
  It might be hard to keep updating these repositories as the original
  ones keep changing. For instance, a user might need the latest
  features of Sumatra, but the CoRR-related Sumatra repository might be
  a few commits behind. Are there any thoughts on that? Why not just
  have a standalone CoRR software that reads project information from
  Sumatra / ReproZip / CDE and uploads data to the platform?
\end{itemize}

\par\null

\section*{Disclaimer}

{\label{574966}}

For the sake of transparency, it should be noted that I am one of the
developers for ReproZip, one of the software tools reviewed in the
manuscript and used in the case study.

\selectlanguage{english}
\FloatBarrier
\bibliographystyle{plainnat}
\bibliography{bibliography/converted_to_latex.bib%
}

\end{document}

