Long Running Processes
Abel users have a disk capacity of 200 GB; but you might have more data than this. This page summarises some good practices on using Abel for large jobs. With a specific example for WRF.
Use job scripts
From the Abel user's guide:
"When you log in to Abel, you are logged in to a login node. The login nodes are meant for logging in, copying files, editing, compiling, running short tests (no more than a couple of minutes), submitting jobs, checking job status, etc.
To run a job on the cluster, you submit a job script into a job queue, and the job is started when a suitable compute node (or group of nodes) is available. The job queue is managed by a queue system (scheduler and resource manager) called SLURM (SLURM's documentation page).
Note that it is not allowed to run jobs directly on the login nodes of Abel. If you fail to comply with this rule, your access to Abel might be suspended."
In short, all simulations should be submitted as jobs. Read more on these links:
Work directory $SCRATCH
When submitting a job, it is good practice to copy all relevant files to a work directory (/work/...). "The name of the directory is stored in the environment variable $SCRATCH. As a general rule, all jobs should use the scratch directory ($SCRATCH) as its work directory.
There are several reasons for using $SCRATCH:
$SCRATCH is on a faster file system than user home directories. There is less risk of interfering with running jobs by accidentally modifying or deleting the jobs' input or output files. Temporary files are automatically cleaned up, because the scratch directory is removed when the job finishes. It avoids taking unneeded backups of temporary and partial files, because $SCRATCH is not backed up
All Abel users have a folder called nobackup. "Directories with name "nobackup" (and all the sub-tree) will be excluded from the daily backup (to be used as much as possible to help reducing the task of the daily backup !)"
Copy to vann (or other storage)
When a simulation is finished, the output should be copied to a storage (for instance vann). For large datasets, or many small files, the command rsync is preferred over scp. The following scripts show automatic copying from Abel to vann. Note that ssh keys between vann (sverdrup) and Abel must be set up to make this automatic, otherwise it asks for a password every time.
Job script hrldas.job:
#!/bin/bash # Job name: #SBATCH --job-name=spinup # # Project: #SBATCH --account=geofag # # Wall clock limit: #SBATCH --time=20:0:0 # # Max memory usage per task: #SBATCH --mem-per-cpu=2000M # # Number of tasks (cores): #SBATCH --ntasks-per-node=1 # # Number of nodes: #SBATCH --nodes=1 # ###SBATCH --constraint=amd # Set up job environment source /cluster/bin/jobsetup ulimit -s unlimited export LANG=en_US.UTF-8 export LC_ALL=en_US # cp ../Noah_hrldas_beta $SCRATCH # do not need this when using absolute file paths # cp kopi.sh $SCRATCH chkfile "RESTART*" "199*1200*LDASOUT*" # copy result files from $SCRATCH to $SUBMITDIR when job is finished cp kopi.job $SCRATCH cp namelist.input $SCRATCH cp namelist.hrldas $SCRATCH cp RESTART.1992123100_DOMAIN1 $SCRATCH # cd $SCRATCH # Need this to do the actual simulation in the work directory export GRIB_ROOT~/hrldas/hrldas-v3.6/HRLDAS_COLLECT_DATA/GRIB_TABLES ~/hrldas/hrldas-v3.6/HRLDAS_COLLECT_DATA/consolidate_grib.exe # ~/hrldas/hrldas-v3.6/Run/Noah_hrldas_beta # IF COPY TO SCRATCH, ./Noah_hrldas_beta sbatch kopi.job # start copying when the job is done (if relatively short job) # scancel <jobid to kopi.job> # remember to cancel the copy job.
Copy script kopi.job
#!/bin/bash # Job name: #SBATCH --job-name=kopi # # Project: #SBATCH --account=geofag # # Wall clock limit: #SBATCH --time=05:0:0 # # Max memory usage per task: #SBATCH --mem-per-cpu=2000M # # Number of tasks (cores): #SBATCH --ntasks-per-node=1 # # Number of nodes: #SBATCH --nodes=1 # ###SBATCH --constraint=amd # Set up job environment source /cluster/bin/jobsetup ulimit -s unlimited export LANG=en_US.UTF-8 export LC_ALL=en_US mpirun ./kopi_hrldas_sverdrup.sh
-bash-4.1$ more wrf_kopi_hjem.sh
#!/bin/bash #Script to copy too home folder (must mkdir to copy to each time) # 1) create a directory on Sverdrup/vann for the outfiles, ex mkdir full1992and1993 cd "/usit/abel/u1/irenebn/nobackup/1992and1993/simulated_LDASOUT" # Output files 199*LDASOUT* are created here rsync -F -tv namelist.hrldas firstname.lastname@example.org:/mn/vann/metos-d2/irenebn/HRLDAS/Resultater/noah-mp_1993 rsync -F -tv *.TBL email@example.com:/mn/vann/metos-d2/irenebn/HRLDAS/Resultater/noah_mp_1993 while true; do date # note that this loop never ends! ls -l 199*LDASOUT*>>~/kopi-status.txt time rsync -F -tv ./199*LDASOUT* firstname.lastname@example.org:/mn/vann/metos-d2/irenebn/HRLDAS/Resultater/noah_mp_1993 >>~/kopi-status.txt 2>&1 rsync -F -tv ./RESTART* email@example.com:/mn/vann/metos-d2/irenebn/HRLDAS/Resultater/noah_mp_1993 >>~/kopi-status.txt 2>&1 sleep 600 # the script is activated once each 10 minutes (600 s) done
Remember to run scancel to stop the copy script, because the loop is infinite.
scancel jobid # where jobid is the number before your job when writing squeue -U geofag