Cross Facility Workflows

Documentation and best practices on how to run workflows across ASCR facilities

A number of users and science teams would like to be able to run their workflows at multiple HPC facilities. This is particularly true for users analyzing data generated at experimental facilities or observational instruments (such as light sources, genome sequencers, accelerators, telescopes, and electron microscopes) which are increasingly requiring High Performance Computing (HPC) scale capabilities for data analysis and workflow processing. The reasons given for needing cross facility workflows are varied. Often large data analysis pipelines from experimental facilities require near real-time computing and need to have an alternative analysis site if their primary site is down or undergoing maintenance. In other instances, teams have acquired allocations at multiple facilities and need to be able to use the allocations efficiently.

Historically, running workflows across facilities has been challenging. In this documentation we highlight tools, techniques, and best practies for running workflows across facilities.

Jump to our GitHub project page

Technologies

• Containers
• Persistent Services Platforms
• APIs for Facilities
• Workflow Tools
• Identity Management
• Data Transfer
• Data Management

Use cases

• Using CCTBX in a Superfacility Workflow
• Cross-facility Analysis with XPCS

Research efforts

• Team
• Publications
• Projects