Storage

Credit: Hard Drives in Abstract Computer Storage by ken123.

Overview

To support diverse data management and computational needs, the ND-HNC has partnered with the Center for Research Computing (CRC) to provide researchers with a shared, multi-tiered data storage infrastructure.

These resources are categorized into three distinct tiers based on the data’s activity level and backup requirements:

Storage Tier Total Capacity Mount Location Primary Purpose Backup Policy
Active Storage 50 TB /groups/hnc Intermediate storage of publication-critical files. Nightly
Cold Storage 250 TB /goldfs/hnc Long-term storage srcdata, rawdata, publication files. Nightly
Scratch 500 TB TBD Temporary workspace for intermediate generation. None

Access

Usage

Using storage tiers for their intended use is critical when multiple users are sharing common resources. Instead of dividing available resources across all ND-HNC affiliated labs, we currently allow all users to have full access to all resources. Such an approach:

  • Ensures there is sufficient capacity to support large projects (e.g. HCP data).
  • Supports multiple projects at various stages of completion.
  • Requires active data management and coordination among the ND-HNC group.

As projects progress from data collection to publication, reference the graphic below for data migration and appropriate use-case:

NDHNC_Data_Flow cluster_tier2 Cold Storage (250 TB, Nightly Backups) cluster_tier3 Scratch Workspace (500 TB, NO BACKUPS) cluster_local Local Storage cluster_tier1 Active Storage (50 TB, Nightly Backups) tier2_raw Original Data: - srcdata DICOM - rawdata NIfTI - Assessment metrics - Task responses tier3_proc Intermediate Data: - Preprocessing - Subject models - Group models tier2_raw->tier3_proc Transfer for analysis tier2_archive Long-Term Archive: - Project code - Final dataframes - Statistical output - Figures tier3_trash Purge / Delete tier3_proc->tier3_trash Removed when unused local_store Complicated Data: - Difficult generation - Long runtime tier3_proc->local_store Manage large data footprints tier1_active Final Data: - Statistical Maps - Figures tier3_proc->tier1_active Results (pre-publication) tier1_active->tier2_archive Paper accepted: Archive final data

  1. Cold Storage provides a secure location for organized (e.g. BIDS) original data.
  2. Data is transferred to Scratch for preprocessing and modelling.
    • Unused intermediates are purged.
    • Long-term storage of complicated intermediates are managed locally.
  3. Final results are transferred to Active Storage for manuscript generation.
  4. Upon publication, files in Active Storage are transferred to Cold Storage.

Guidelines

  • Build the dataflow into your workflow/codebase.
  • As storage is shared across the ND-HNC community, please:
    • Manage your allocations responsibly,
    • Migrate data to the appropriate tier as your project progresses,
    • Maintain only the requisite intermediates for subsequent pipeline steps.

Notes

  • CRC compute nodes do not have access to files stored in Cold storage.
  • Scratch is intended only for transient data, do not store critical or irreplaceable files here.