Use Cases

Describe some real world examples of Jupyter usage at NERSC
We noticed some common patterns in our engagements with scientific users that use Jupyter for their computational workflows on NERSC systems. At the highest level there is a need for combining exploration of very large datasets with some computational and analytical capabilities. Crucially the scale of data or compute (or both) required to enable these workflows typically exceeds the capacity of the users own machines and the users need a user-friendly way to drive these large-scale workflows interactively.
We often see a two phased approach, where the user performs some local notebook development and then runs these on machines like NERSC on their production data and compute pipelines. It is very important to be able to seamlessly go between these modes and our approach is grounded in trying to make sure that a user can easily take a notebook and its associated environment over to our systems, with minimal effort and making sure that they have a consistent user experience.

Conclusion

[ROUGH] Jupyter in HPC is now commonplace. We have been able to give hundreds of HPC users a rich user interface to HPC through Jupyter. In the supercomputing context, we look at Jupyter as a tool that will help make it easier for our users to take advantage of supercomputing hardware and software. Some of that will come from us at supercomputing centers. Jupyter as a project needs to not make design decisions that break things for us, or lock us into one way of doing things. Each HPC center is different and that means that for Jupyter to remain useful to HPC centers and supercomputing it needs to maintain its high level of abstraction. We should make this into a bulleted list of demands :)

Acknowledgments

This work was supported by Lawrence Berkeley National Laboratory, through the U.S. Department of Energy under Contract No. DE-AC02-05CH11231. This work used resources of the National Energy Research Scientific Computing Center (NERSC), a U.S. Department of Energy Office of Science User Facility located at Lawrence Berkeley National Laboratory.