Storage

Credit: Hard Drives in Abstract Computer Storage by ken123.

Overview

To support diverse data management and computational needs, the ND-HNC has partnered with the Center for Research Computing (CRC) to provide researchers with a shared, multi-tiered data storage infrastructure.

Storage Tiers

Storage resources are organized into three distinct tiers based on the data’s activity level and backup requirements:

Storage Tier Total Capacity Mount Location Primary Purpose Backup Policy
Active 50 TB /groups/hnc Intermediate storage of publication-critical files. Nightly
Cold 250 TB /goldfs/hnc Long-term storage srcdata, rawdata, publication files. Nightly
Scratch 500 TB /leadfs/hnc Temporary workspace for intermediate generation. None

Notes

  1. Scratch is intended only for transient data, do not store critical or irreplaceable files here.
  2. Scratch storage is accessible in two ways:
    1. After connecting to the login node epycfe.crc.nd.edu, or
    2. When working in a scheduled session on the regular login nodes ($ qrsh -q long -pe smp 1).
  3. Compute nodes do not have access to files stored in Cold storage.

Access

Usage

Using storage tiers for their intended use is critical when multiple users are sharing common resources. Instead of dividing available resources across all ND-HNC affiliated labs, we currently allow all users to access these resources. Such an approach:

  • ensures there is sufficient capacity to support large projects (e.g. HCP data),
  • supports multiple projects at various stages of completion, and
  • requires active data management and coordination among the ND-HNC group.

While these storage resources provide ample space for well-managed projects, they are nevertheless finite. To ensure these resources remain available to the entire group, coordinate with Nathan Muncy directly if your lab intends to use more than 5TB Active, 25TB Cold, and/or 50TB Scratch storage. As projects progress from data collection to publication, reference the graphic below for data migration and appropriate storage tier usage:

NDHNC_Data_Flow cluster_tier2 Cold Storage (250 TB, Nightly Backups) cluster_tier3 Scratch Workspace (500 TB, NO BACKUPS) cluster_local Local Storage cluster_tier1 Active Storage (50 TB, Nightly Backups) tier2_raw Original Data: - srcdata DICOM - rawdata NIfTI - Assessment metrics - Task responses tier3_proc Intermediate Data: - Preprocessing - Subject models - Group models tier2_raw->tier3_proc Transfer for analysis tier2_archive Long-Term Archive: - Project code - Final dataframes - Statistical output - Figures tier3_trash Purge / Delete tier3_proc->tier3_trash Removed when unused local_store Complicated Data: - Difficult generation - Long runtime tier3_proc->local_store Manage large data footprints tier1_active Final Data: - Statistical Maps - Figures tier3_proc->tier1_active Results (pre-publication) tier1_active->tier2_archive Paper accepted: Archive final data

Dataflow Description

  1. Cold storage (blue) provides a secure location for organized (e.g. BIDS) original data.
  2. Data is transferred to Scratch (red) for preprocessing and modelling.
    • Unused intermediates are purged.
    • Long-term storage of complicated intermediates are managed locally (gray).
  3. Final results are transferred to Active storage (green) for manuscript generation.
  4. Upon publication, files in Active storage are transferred to Cold Storage.

Organization

Labs should organize their data within a PI directory at each storage location. For instance, Active storage would have the following structure:

groups
└── hnc
    ├── barbey
    ├── forest
    ├── jdu
    └── muncy
        ├── proj_a
        ├── proj_b
        └── proj_c

Additionally, it will be useful to set a umask to ensure that data is accessible only by the intended individuals.