Running NOAH
This page contains the steps needed to run NOAH (using the offline driver code HRLDAS). It requires that NOAH is set up; that the data are downloaded and preprocessed. If you haven't done that, please refer to the NOAH page.
You probably have lots of data stored on vann (/mn/vann/metos-d2/irenebn/; exchange irenebn with your user name) or any of the other storage disks at UiO. Abel users have a disk capacity of 200 GB; make sure not to exceed that. Please read this page on how to handle big jobs on Abel.
Contents
Preprocessing
Please refer to points 6), 7) and 8) in the NOAH page to do the preprocessing. NOTE! Before running extract_var_after_split.sh, make sure that _only_ gribfiles on the hourly timescale are present in the folder (otherwise, incorrect files called "T2_19930131.g" or "T2_199301.mar" will be produced instead of " T2_1993013100".
cd ~/FILESTORE mkdir <name of new simulation> # for instance fraBjorg1993jan cp -r ../TestInvertlatTilSlutt/GRBHRS-empty/ <name of new simulation>/GRIBHRS cd <name of new simulation> cp ../split_time.sh . cp ../TdPa2qq.sh . cp ../TestInvertlatTilSlutt/rawfiles1992/extract_var_after_split.sh . cp ../TestInvertlatTilSlutt/rawfiles1992/split_rule . cp ../TestInvertlatTilSlutt/rawfiles1992/split_rule_169_175_228 . cp ../TestInvertlatTilSlutt/rawfiles1992/split_rule_134_167_168 . emacs split_time.sh &
Split_time takes in .mars files on the format yyyymm and outputs gribfiles on the format yyyymmdd. Make sure the file paths are correct, and that the numbers {file:50:8} are set so that they only extract the date (on the format yyyymmdd). Running split_time.sh will extract and save the following variables in its respective folder under <name of new simulation>/GRIBHRS:
source split_time.sh 169 SW 175 LW 228 PCP
Check that the variables are found under ./GRIBHRS/
Do as much preprocessing as possible on vann
First, log on to sverdrup to access the data at vann and do all preprocessing there. When everything is done, transfer a small portion of the data (the relevant files for a run) to Abel. Note that your storage limit on Abel is 200 G.
ssh -YC irenebn@sverdrup.uio.no cd /mn/vann/metos-d2/irenebn/HRLDAS/data
If the data are given as separate files (fc_sfc_<startdate>_<enddate>.grb, an_sfc_<startdate>_<enddate>.grb), concatenate them into one .mars file first (note that February must be added, depending on its number of days).
cat an_sfc_19960101_19960131.grb fc_sfc_19960101_19960131.grb > ma199601.mars cat an_sfc_19960301_19960331.grb fc_sfc_19960301_19960331.grb > ma199603.mars cat an_sfc_19960501_19960531.grb fc_sfc_19960501_19960531.grb > ma199605.mars cat an_sfc_19960701_19960731.grb fc_sfc_19960701_19960731.grb > ma199607.mars cat an_sfc_19960801_19960831.grb fc_sfc_19960801_19960831.grb > ma199608.mars cat an_sfc_19961001_19961031.grb fc_sfc_19961001_19961031.grb > ma199610.mars cat an_sfc_19961201_19961231.grb fc_sfc_19961201_19961231.grb > ma199612.mars # cat an_sfc_19960401_19960430.grb fc_sfc_19960401_19960430.grb > ma199604.mars cat an_sfc_19960601_19960630.grb fc_sfc_19960601_19960630.grb > ma199606.mars cat an_sfc_19960901_19960930.grb fc_sfc_19960901_19960930.grb > ma199609.mars cat an_sfc_19961101_19961130.grb fc_sfc_19961101_19961130.grb > ma199611.mars # cat an_sfc_19960201_19960229.grb fc_sfc_19960201_19960229.grb > ma199602.mars
For the initial time step, also the pressure layer field (an_pl*.grb) must be included, but this is not required after the initial time step. Also remember NOT to delete any raw data! Vann should have plenty of storage, and if in doubt, ask the METOS IT.
cat an_sfc_19930101_19930131.grb an_pl_19930101_19930131.grb fc_sfc_19930101_19930131.grb > ma199301.mars
To look at the data (especially important for the first time step), first clip the monthly file into days using split_time.sh. Check the data for one day. Are all variables there? Are all pressure layers available for the first time step? There should be 38 pressure layers, and all variables should be available.
Note that the clipping must be done for surface variables and for atmospheric (pressure) levels differently. Therefore use grib_filter instead of CDO for this step.
# On Vann: module load cdo
# On Vann: module load ncview
# On Abel: module load wrf
# On Abel: module load hrldas
# On Abel: module load grib_api
# these module load commands should be added to your ~/.bashrc file on Vann and ~/.bash_login file on Abel.
cdo splitday ma199301.mars ma199301
# for f in ma1993*.mars; do echo cdo splitday $f ~/sunspot/HRLDAS/create_GRBHRS/${f%.mars}; done
#
grib_ls ma*01.grb >& infoJan-ls-slett.txt
g1print ma*01.grb >& infoJan-g1-slett.txt
#
less infoJan-g1-slett.txt
# less infoJan-ls-slett.txt
In the terminal, you may search for words by typing / (shift+7) and the word. To quit the text reader less (or more), type q.
If the data are insufficient, order new. Otherwise, navigate to WPS and create a new directory.
Create GRIBHRS files on Vann
Next, create GRIBHRS files - input to HRLDAS. Navigate to ~/sunspot/HRLDAS/create_GRBHRS/ and run this script:
#!/bin/bash
##########################################################
# Script extracting 4-hourly values from a set of grib files (.mars).
# Save this script in the same directory as the files.
# Run the script by typing split_time.sh in the terminal.
# First, extract LW, SW and precip from the mars files.
# This must be done before splithour
##########################################################
module load grib_api
module load cdo
#______________________________Write paths here______________________________
# ABEL directories: fildir='/usit/abel/u1/irenebn/nobackup/79test'
# ABEL directories: savedir='/usit/abel/u1/irenebn/nobackup/79test/GRIBHRS'
fildir='/uio/kant/geo-mhgeo-u1/irenebn/sunspot/HRLDAS/create_GRBHRS'
savedir='/uio/kant/geo-mhgeo-u1/irenebn/sunspot/HRLDAS/create_GRBHRS/GRIBHRS'
for file in /uio/kant/geo-mhgeo-u1/irenebn/sunspot/HRLDAS/data/ma1996*.mars; do
echo $file
grib_filter $fildir/split_rule_169_175_228 $file # her m<E5> det inn en -mars-fil, ikke oppdelte gribfiler. ## <- use this
cdo splitday $file ${file%.mars}
done
for file in $fildir/ma199*.grb; do # use an existing set of gribfiles to get the dates right
echo ${file:62:8}
# Forecast fields (LW, SW, PCP) must be divided by 21600 (6*60*60) to get values per second.
cdo invertlat $fildir/tempLW_${file:62:8} $fildir/tempLW_invert${file:62:8}
cdo divc,21600.0 $fildir/tempLW_invert${file:62:8} $fildir/tempLW_invertDiv${file:62:8}
cdo splithour $fildir/tempLW_invertDiv${file:62:8} $savedir/LW/LW_${file:62:8}
#
cdo invertlat $fildir/tempSW_${file:62:8} $fildir/tempSW_invert${file:62:8}
cdo divc,21600.0 $fildir/tempSW_invert${file:62:8} $fildir/tempSW_invertDiv${file:62:8}
cdo splithour $fildir/tempSW_invertDiv${file:62:8} $savedir/SW/SW_${file:62:8}
#
cdo invertlat $fildir/tempPCP_${file:62:8} $fildir/tempPCP_invert${file:62:8}
cdo divc,21600.0 $fildir/tempPCP_invert${file:62:8} $fildir/tempPCP_invertDiv${file:62:8}
cdo splithour $fildir/tempPCP_invertDiv${file:62:8} $savedir/PCP/PCP_${file:62:8}
done
for f in $savedir/PCP/PCP_199*01*.grb; do mv $f ${f%.grb}; done
for f in $savedir/LW/LW_199*01*.grb; do mv $f ${f%.grb}; done
for f in $savedir/SW/SW_199*01*.grb; do mv $f ${f%.grb}; done
rm $savedir/temp*
rm $fildir/temp*
rm $fildir/*.mars*.grb
# mv $file $file.mars
##
## Then, split the files into whichever timestep is wanted.
## (splithour is preferred in my case)
##
# for file in ma*; do
# echo $file
# cdo splityear $file $file
# rm $file
# for file in *.grb; do mv $file ${file%.grb};done
# done
for file in $fildir/ma199*.mars; do
echo $file
cdo splitday $file ${file%.mars}
done
[irenebn@sverdrup SW]$ less ../../split_rule_169_175_228
##########################################################
if (indicatorOfParameter == 169) {
write "tempSW_[dataDate]";
}
if (indicatorOfParameter == 175) {
write "tempLW_[dataDate]";
}
if (indicatorOfParameter == 228) {
write "tempPCP_[dataDate]";
}
Then navigate to the folders GRIBHRS/LW GRIBHRS/SW and GRIBHRS/PCP to check the output and remove .grb from the filename:
for file in *.grb; do echo mv $file ${file%.grb};done
Now it is time to calulate the specific humidity, QQ.
#!/bin/bash #convert Era-Interim surface dew point temp (167) to specific humidity using surface pressure (134) and formulas in EC-documentation: # Look at equations 7.4 and 7.5 (pages 91-92) in part IV http://www.ecmwf.int/research/ifsdocs/CY36r1/PHYSICS/IFSPart4.pdf savedir='/uio/kant/geo-mhgeo-u1/irenebn/sunspot/HRLDAS/create_GRBHRS/GRIBHRS' fildir='/uio/kant/geo-mhgeo-u1/irenebn/sunspot/HRLDAS/create_GRBHRS' #savedir='/uio/kant/geo-mhgeo-u1/irenebn/sunspot/HRLDAS/prosesserteFiler/1992and1993/GRIBHRS' # fildir='/uio/kant/geo-mhgeo-u1/irenebn/sunspot/HRLDAS/prosesserteFiler/1992and1993' #storedir='/uio/kant/geo-mhgeo-u1/irenebn/sunspot/HRLDAS/data' ##cdo showname gives var134 (pressure) and var168 (Td) module load cdo module load grib_api ##cdo -expr,'es=611.21*exp(var168-273.16)/(var168-32.19);' maQQPa ES for file in ~/sunspot/HRLDAS/data/ma1999*.mars; do echo $file echo ${file:53:6} grib_filter $fildir/split-needMARSfile_134_167_168 $file # her maa det inn en .mars-fil, ikke oppdelte gribfiler. Det lages en filter.out med alle tre variablene. ## mv filter.out filter${file:46:6}.out ## echo cdo -gec,273.16 -selname,var167 filter.out $fildir/aboveZero_${file:46:6}.grb # monthly time scale ## echo cdo -ltc,273.16 -selname,var167 filter.out $fildir/belowZero_${file:46:6}.grb # mv filter.out filter_${file:53:6}.out # filter.out is saved in $filedir, along with all temp files. done for file in ~/sunspot/HRLDAS/data/ma1998*.mars; do echo $file grib_filter $fildir/split-needMARSfile_134_167_168 $file # her maa det inn en .mars-fil, ikke oppdelte gribfiler. Det lages en filter.out med alle tre variablene. mv filter.out filter_${file:53:6}.out # filter.out is saved in $filedir, along with all temp files. done for file in $fildir/ma1996*.grb; do # bruk grb for <E5> f<E5> med datoene, ikke bare m<E5>neden echo $file cdo splitday filter_${file:62:6}.out temp_QQdata_${file:62:6} # days (not hours yet) cdo -gec,273.16 -selname,var167 temp_QQdata_${file:62:8}.grb check_aboveZero_${file:62:8}.grb # writes 0 for days where T2 < 273.16 Kelvin. Daily time scale cdo -expr,'qsat=0.62188*(611.21*exp(17.502*(var168-273.16)/(var168-32.19)))/(var134-(1-0.62188)*(611.21*exp(17.502*((var168-273.16)/(var168-32.19)))));' temp_QQdata_${file:62:8}.grb temp_QQ2m_aboveZero_${file:62:8} cdo -expr,'qsat=0.62188*(611.21*exp(22.587*(var168-273.16)/(var168-(-20.7))))/(var134-(1-0.62188)*(611.21*exp(22.587*((var168-273.16)/(var168-(-20.7))))));' temp_QQdata_${file:62:8}.grb temp_QQ2m_belowZero_${file:62:8} # To select all field elements of ifile2 if the corresponding field element of ifile1 is greater than 0 and from ifile3 otherwise use: # cdo ifthenelse ifile1 ifile2 ifile3 ofile cdo ifthenelse check_aboveZero_${file:62:8}.grb temp_QQ2m_aboveZero_${file:62:8} temp_QQ2m_belowZero_${file:62:8} temp_QQ2m_${file:62:8} cdo invertlat $fildir/temp_QQ2m_${file:62:8} $fildir/temp_QQinvert_${file:62:8} cdo chcode,1,133 $fildir/temp_QQinvert_${file:62:8} $fildir/temp_QQsurf_${file:62:8} cdo chparam,133,133.128 $fildir/temp_QQsurf_${file:62:8} $fildir/temp_QQsCode_${file:62:8} cdo splithour $fildir/temp_QQsCode_${file:62:8} $savedir/QQ/QQ_${file:62:8} done # for file in $fildir/ma1997*.grb; do # bruk grb for <E5> f<E5> med datoene, ikke bare m<E5>neden echo $file cdo splitday filter_${file:62:6}.out temp_QQdata_${file:62:6} # days (not hours yet) cdo -gec,273.16 -selname,var167 temp_QQdata_${file:62:8}.grb check_aboveZero_${file:62:8}.grb # writes 0 for days where T2 < 273.16 Kelvin. Daily time scale cdo -expr,'qsat=0.62188*(611.21*exp(17.502*(var168-273.16)/(var168-32.19)))/(var134-(1-0.62188)*(611.21*exp(17.502*((var168-273.16)/(var168-32.19)))));' temp_QQdata_${file:62:8}.grb temp_QQ2m_aboveZero_${file:62:8} cdo -expr,'qsat=0.62188*(611.21*exp(22.587*(var168-273.16)/(var168-(-20.7))))/(var134-(1-0.62188)*(611.21*exp(22.587*((var168-273.16)/(var168-(-20.7))))));' temp_QQdata_${file:62:8}.grb temp_QQ2m_belowZero_${file:62:8} cdo ifthenelse check_aboveZero_${file:62:8}.grb temp_QQ2m_aboveZero_${file:62:8} temp_QQ2m_belowZero_${file:62:8} temp_QQ2m_${file:62:8} cdo invertlat $fildir/temp_QQ2m_${file:62:8} $fildir/temp_QQinvert_${file:62:8} cdo chcode,1,133 $fildir/temp_QQinvert_${file:62:8} $fildir/temp_QQsurf_${file:62:8} cdo chparam,133,133.128 $fildir/temp_QQsurf_${file:62:8} $fildir/temp_QQsCode_${file:62:8} cdo splithour $fildir/temp_QQsCode_${file:62:8} $savedir/QQ/QQ_${file:62:8} done
NOTE! For long time series, this processing will take time. However, there's no need to do this serially. Once split_time.sh or TdPa2QQ.sh is started, extract_var_after_split.sh might be started on Abel (sbatch extract.job). If all initial files are created, consolidate.job can be started, and after that, run.job can be started. It works because it takes longer time to make LDASIN files (consolidate) than to extract the variables; and even longer time to run the model than to consolidate.
Transfer data to Abel
The last piece of processing must be performed on Abel, because the script uses gribextract (a part of HRLDAS). It would be possible to rewrite the script to use grib_filter, but it has not been done yet. Therefore; LW, SW, PCP and QQ fields are copied to Abel, where the other fields will be extracted. To check which years are saved in the folder, type
ll QQ/QQ_199*010100
Then transfer the relevant files
scp QQ/QQ_1997* irenebn@abel.uio.no:~/nobackup/1992and1993/GRIBHRS/QQ
And run extract_var_after_split.sh
#!/bin/bash
#Extract grib-parameters from ERA-Interim files (GRIB1, format maYYYMMDDHH 00,06,12,18)
#_________________________________________Paths_________________________________________
fildir='/usit/abel/u1/irenebn/nobackup/79test'
savedir='/usit/abel/u1/irenebn/nobackup/79test/GRIBHRS'
extractdir='/usit/abel/u1/irenebn/hrldas/hrldas-v3.4.1/Utility_programs' #MAA SETTES
for file in $fildir/ma197901*.grb; do ## this takes a file on the format yyyymmdd.grb
echo ${file:40:8} # hvis filbanen er fildir='/usit/abel/u1/irenebn/FILESTORE/1992and1993
#
$extractdir/gribextract -c 165 -l 1,0,0 $file $savedir/tempUU ## ${file:46:10}
cdo invertlat $savedir/tempUU $savedir/tempUU_invert
cdo mulc,1.0 $savedir/tempUU_invert $savedir/tempUU_daily
cdo splithour $savedir/tempUU_daily $savedir/UU/UU_${file:40:8}
#
$extractdir/gribextract -c 166 -l 1,0,0 $file $savedir/tempVV ##${file:46:10}
cdo invertlat $savedir/tempVV $savedir/tempVV_invert
cdo mulc,1.0 $savedir/tempVV_invert $savedir/tempVV_daily
cdo splithour $savedir/tempVV_daily $savedir/VV/VV_${file:40:8}
#
$extractdir/gribextract -c 167 -l 1,0,0 $file $savedir/tempT2 ##${file:46:10}
cdo invertlat $savedir/tempT2 $savedir/tempT2_invert
cdo mulc,1.0 $savedir/tempT2_invert $savedir/tempT2_daily
cdo splithour $savedir/tempT2_daily $savedir/T2/T2_${file:40:8}
#
$extractdir/gribextract -c 134 -l 1,0,0 $file $savedir/tempPa ##${file:46:10}
cdo invertlat $savedir/tempPa $savedir/tempPa_invert
cdo mulc,1.0 $savedir/tempPa_invert $savedir/tempPa_daily
cdo splithour $savedir/tempPa_daily $savedir/Pa/Pa_${file:40:8}
#
for f in $savedir/VV/VV_19790101*.grb; do mv $f ${f%.grb}; done
for f in $savedir/UU/UU_19790101*.grb; do mv $f ${f%.grb}; done
for f in $savedir/T2/T2_19790101*.grb; do mv $f ${f%.grb}; done
for f in $savedir/Pa/Pa_19790101*.grb; do mv $f ${f%.grb}; done
rm $savedir/temp*
done
Remember that on Abel, we should not run jobs on the login nodes. The following script submits extract_var_after_split as a job, and liberates resources on the login node. Therefore, run sbatch extract.job.
#!/bin/bash # Job name: #SBATCH --job-name=extract # # Project: #SBATCH --account=geofag # # Wall clock limit: #SBATCH --time=05:0:0 # # Max memory usage per task: #SBATCH --mem-per-cpu=2000M # # Number of tasks (cores): #SBATCH --ntasks-per-node=1 # # Number of nodes: #SBATCH --nodes=1 # ###SBATCH --constraint=amd # Set up job environment source /cluster/bin/jobsetup ulimit -s unlimited export LANG=en_US.UTF-8 export LC_ALL=en_US ./extract_var_after_split.sh
Now, navigate to LDASIN and change the namelist.input. Then run consolidate.job. Navigate to the run folder (for instance 1996to99) and change the date of the latest restart file. Then change the date in namelist.hrldas and run run.job.
WPS
Before running WPS, make sure you have a yyyymm.mars file containing all relevant fields, stored under ~/FILESTORE/<name of new simulation>. (fraBjoerg)
cd ~/WRF/WPSv3.6/WPS/ mkdir <name of new simulation> # for instance fraBjorg1993jan cp namelist.wps <name of new simulation> cp runningWPS.sh <name of new simulation> cd <name of new simulation> emacs namelist.wps &
Check that the start date in the namelist corresponds to the input data, but ends 48 hours after that (for instance, WPS and real.exe can be run from 1992-01-01_00:00 to 1992-01-03_00:00). The reason is that when NOAH is run in offline mode, which is what HRLDAS is used for, only static fields from WRF are used; that is, geogrid and wrfinput. Start all simulations at midnight (00:00) (because some of the consolidate_grib modules only look for initialization values at midnight). Also check the gridsize, dx and dy. Lastly, check that under &ungrib, it says
prefix = 'SURF', !// 'ATM' # that is, it is looking for the surf Vtable.
Run WPS
Then open runningWPS.sh and check the Vtables and the input data.
emacs runningWPS.sh &
ln -sf ../ungrib/Vtable.ECSURF Vtable ../link_grib.csh ~/FILESTORE/fraBjorg1993jan/ma199301.mars rm SURF:* rm ATM:* echo geogrid.exe >& geogrid.log geogrid.exe >& geogrid.log tail geogrid.log echo ungrib.exe >& ungrib_SURF.log ungrib.exe >& ungrib_SURF.log tail ungrib_SURF.log ln -sf ../ungrib/Vtable.ECATM Vtable rm ATM:* emacs -nw namelist.wps &
To run WPS, simply type ./runningWPS.sh and things should go automatically.
Next, change 'SURF' into 'ATM' in namelist.wps and run the following
ln -sf ../ungrib/Vtable.ECATM Vtable ungrib.exe >& ungrib_ATM.log tail ungrib_ATM.log metgrid.exe >& metgrid.log # note: this should take a while. tail metgrid.log
Check that geo_em_d01.nc is created. To prepare for the next run, change the namelist.wps back to have the prefix SURF.
Check WPS
Now, we need to check the output. With ncdump, you can check if all the variables stated in Vtable.ECSURF and Vtable.ECATM are present. Note that the variables have different names from the variable names in Era-Interim here. For instance, variable 144 - WEASD - is called SNOW because that is the name it has been given by the Vtable. The Vtables used are standard tables for Era-Interim data. Note! The Vtables are different for different versions of WRF. The correct tables for your version should be available in the folder (change irenebn to your username)
/cluster/home/irenebn/WRF/WPS/ungrib/Variable_Tables/Vtable...
rd_intermediate.exe makes it possible to look into the contents of the intermediate files SURF: and ATM:.
ncdump -h met_em-<date> >& info-met_em.txt rd_intermediate.exe ATM\:* >& info-atm.txt rd_intermediate.exe SURF\:* >& info-surf.txt
Again, there should be 38 atmospheric levels, four soil levels and all variables stated in the Vtables should be available.
WRF
Then navigate to the WRFV3/run folder and create a directory named the same as that in WPS.
cd ~/WRF/WRFv3.6/WRFV3/run mkdir <name of new simulation> # for instance fraBjorg1993jan cp namelist.input <name of new simulation> cd <name of new simulation> emacs -nw namelist.input &
Check that the start and end dates in the namelist corresponds to the input data and with namelist.wps. Also check the gridsize, dx and dy.
Running real.exe
Check what wrf.job does (there should be no reason to change this). Then run real.exe in parallel using sbatch.
less ../wrf.job ln -sf ~/WRF/WPSv3.6/WPS/<name of new simulation>/met_em* . # link to input files sbatch ../wrf.job # submit the job. Should take a while to run. squeue -u irenebn # check how your job is doing squeue -U geofag # check how many jobs your group is running
Running real.exe for the first two days takes less than five minutes. When your job disappears from the list, the job is done. In order not to use more geofag time than necessary, type
scancel #jobid
Check that wrfinput_d01 is created, and check this result file by typing
tail rsl.error.0000
Hopefully, the model has either run perfectly and tells you it is a success, or it gives a useful error message.
Check that the wrfinput and wrfbdy files exist, (use ncdump -h *.nc). The wrfbdy file is not used by HRLDAS (in offline mode), only wrfinput (and geogrid).
HRLDAS
HRLDAS needs the geogrid and wrfinput files. When real.exe is ready, NOAH (HRLDAS) can be run. Still, the preprocessing and consolidate_grib.exe can be done while real.exe is producing wrfinput.
Consolidate_grib.exe
First, create a new directory with the same name as the previous ones.
cd ~/hrldas/hrldas-v3.4.1/HRLDAS_COLLECT_DATA/ mkdir <name of new simulation> # for instance fraBjorg1993jan cp jan92/namelist.input <name of new simulation> emacs namelist.input &
Check that the start and end dates in the namelist corresponds to the input data. It does not have to end at the same time as the date in namelist.wps (it should preferably be much longer). Also, check all filepaths. Next, link to the appropriate grib tables and run consolidate_grib.exe. Note that all programs on Abel must be submitted as a job. The difference is that jobs are not run on the login node. Make one job script per program, so for consolidate_grib.exe, make the script consolidate.job:
#!/bin/bash # Job name: #SBATCH --job-name=consolidate # # Project: #SBATCH --account=geofag # # Wall clock limit: #SBATCH --time=20:0:0 # # Max memory usage per task: #SBATCH --mem-per-cpu=4000M # # Number of tasks (cores): #SBATCH --ntasks-per-node=1 # # Number of nodes: #SBATCH --nodes=1 # ###SBATCH --constraint=amd # Set up job environment source /cluster/bin/jobsetup ulimit -s unlimited export LANG=en_US.UTF-8 export LC_ALL=en_US export GRIB_ROOT=~/hrldas/hrldas-v3.4.1/HRLDAS_COLLECT_DATA/GRIB_TABLES ../consolidate_grib.exe
Then, run consolidate.job in the terminal and check the status of the job with squeue.
sbatch consolidate.job squeue -U geofag
Check that the LDASIN files are produced and contain the correct variables.
Noah_hrldas_beta
Navigate to the folder ~/hrldas/hrldas-v3.4.1/Run (not ../run, but ../../Run). Again, create a new directory with the same name as the previous ones.
cd ~/hrldas/hrldas-v3.4.1/Run mkdir <name of new simulation> # for instance fraBjorg1993jan cp ./namelist.hrldas <name of new simulation> cp ./*.TBL <name of new simulation> emacs namelist.hrldas &
Check that the start and end dates in the namelist corresponds to the input data. Also, check all filepaths.
Before running Noah, read README.namelist.
Next, run Noah_hrldas_beta by submitting a job script (do NOT write ../Noah_hrldas_beta in the terminal, see http://www.uio.no/english/services/it/research/hpc/abel/help/user-guide/job-scripts.html)
#!/bin/bash # Job name: #SBATCH --job-name=hrldasoffline # # Project: #SBATCH --account=geofag # # Wall clock limit: #SBATCH --time=20:0:0 # # Max memory usage per task: #SBATCH --mem-per-cpu=2000M # # Number of tasks (cores): #SBATCH --ntasks-per-node=1 # # Number of nodes: #SBATCH --nodes=1 # ###SBATCH --constraint=amd # Set up job environment source /cluster/bin/jobsetup ulimit -s unlimited export LANG=en_US.UTF-8 export LC_ALL=en_US # cp ../Noah_hrldas_beta $SCRATCH # cp kopi.sh $SCRATCH # cd $SCRATCH #sbatch kopi_hrldas_sverdrup.sh ../Noah_hrldas_beta # IF COPY TO SCRATCH, ./Noah_hrldas_beta
sbatch hrldas.job
Check that the LDASOUT files are produced and contain the correct variables. Now, ncview is a good tool to use. First, you need to load a module (note that this module may interfere with real.exe or Noah_hrldas_beta, so don't load it until all model parts are run).
module load ncview ncview LDASOUT # any of the time steps
For the last time, create a directory with the name of your simulation on vann. Transfer the LDASOUT files to vann from Abel by the first of the following commands, or from vann by the second one.
scp LDASO* irenebn@sverdrup.uio.no:/mn/vann/metos-d2/irenebn/HRLDAS/Resultater/<name of new simulation> scp irenebn@abel.uio.no:~/hrldas/hrldas-v3.4.1/Run/<name of new simulation>/*LDASOUT* . (or scp irenebn@abel.uio.no:/usit/abel/u1/irenebn/hrldas/hrldas-v3.4.1/Run/<name of new simulation>/*LDASOUT* .)
Then, remove the LDAS* files from Abel. Use du -h to check how much storage is used.
du -h rm 199*LDASOUT* cd ../../HRLDAS_COLLECT_DATA/fraBjorg1993jan/ rm 199*LDASIN*
Advanced: spin up in loop and copy between disks
Spin up
The longer spinup time, the less dependent are the results on initial conditions. Helene looped one year nine times as spinup. First, consolidate_grib is run to produce LDASIN files, which are needed to force Noah. Let's say you have found that 1993 is a normal year[1] (which it is, in terms of annual precipitation) and would like to run this in loop. In the namelist file, specify how often you'd like to produce a restart file. [1] You chould choose a normal year to spin up in order to avoid reinforced anomalies. It is good practice to plot soil temperature and soil moisture in the bottom layer to check that it stabilises.
RESTART_FREQUENCY_HOURS = 1460 # With this frequency, you get a restart file every second month.
When the first year is run, you get a restart file starting at the end of 1993. The next run is resumed from that restart time. However, because you only have one year of input data, the date of this file must be changed, so that the next run starts off with the previous restart file. That means: your restart file should be edited so that it starts at the beginning of 1993 instead of end.
Helene has made a clever script called 'changedateRESTART.ncl' to take care of this. Remember to module load ncl first.
&NOAHLSM_OFFLINE # HRLDAS_CONSTANTS_FILE ="/usit/abel/u1/helenbe/hrldas-v3.4.1/Run/constants/test/wrfinput_d02" INDIR = "/usit/abel/u1/helenbe/hrldas-v3.4.1/Run/forcingd02/" OUTDIR = "/usit/abel/u1/helenbe/hrldas-v3.4.1/Run/OUTPUT/HBE_VEGSENSEXP_d02/CTRLRAINSNOW/" # START_YEAR = 2008 START_MONTH = 10 START_DAY = 01 START_HOUR = 00 START_MIN = 00 # RESTART_FILENAME_REQUESTED = "./RESTART.2009100100_DOMAIN2" # Last day of the run # KDAY = 366 # FORCING_TIMESTEP = 3600 NOAH_TIMESTEP = 3600 OUTPUT_TIMESTEP = 21600 # RESTART_FREQUENCY_HOURS = 1460 ! Split output after split_output_count output times. SPLIT_OUTPUT_COUNT = 4 # ! SUBWINDOW_XSTART = 32 ! SUBWINDOW_XEND = 32 ! SUBWINDOW_YSTART = 60 ! SUBWINDOW_YEND = 60 # NSOIL=4 ZSOIL(1) = -0.10 ZSOIL(2) = -0.40 ZSOIL(3) = -1.00 ZSOIL(4) = -2.00 # ZLVL = 2.0 ZLVL_WIND = 10.0 # IZ0TLND = 0 SFCDIF_OPTION = 0 UPDATE_SNOW_FROM_FORCING = .FALSE. /
Spin up/kjøre-script -bash-4.1$ more runscript.sh
#!/bin/bash
#Spins-up i years. First must be spun up in advance, unless you want to change sscript to change namelist (i.e. if i eq 1 use namelist version w/o spin up, else copy to namelist with restart.. easy)
#Check if set up for correct domain number here, ncl-script and namelist
OUTDIR="/usit/abel/u1/helenbe/hrldas-v3.4.1/Run/OUTPUT/NEWVEG_CONTROL/d02/n"
OUTDIRspin="/usit/abel/u1/helenbe/hrldas-v3.4.1/Run/OUTPUT/NEWVEG_CONTROL/d02"
=1 --mem-per-cpu=2000M
for i in {3..9}
do
echo "Initializing spin-up starting on last, $i"
cp $OUTDIR/RESTART.2009100100_DOMAIN2 . # Every run ends with a restart file named 2009*.
mv $OUTDIR/* $OUTDIRspin/$i # Because all outfiles have the same date; use different directories
ncl changedateRESTART.ncl # This script tells Noah which files to look for; here: 2008*LDASIN
mpirun -np 8 ./Noah_hrldas_beta # Run Noah in parallel.
done
mv $OUTDIR/* $OUTDIRspin/10
Har du ncl installert med hslure fil, eller hva det heter? for om man looper spinup år må man endre dato på filen:
-bash-4.1$ more changedateRESTART.ncl
load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl" # These only work from the script, not the terminal.
load "$NCARG_ROOT/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl" # also, on my system, $NCARG_ROOT is projects/metos/annfou/ncl/3.6
begin
a = addfile("./RESTART.2009100100_DOMAIN2"+".nc","w") # This is the restart file produced at the end of a run
times=(a->Times)
printVarSummary(times)
print(times)
eight=tocharacter("8") # here, we want Noah to look for years 2008 instead of 2009.
ait=eight(0)
times(0,3)=ait
printVarSummary(times)
print(times)
a->Times=times
end
This is conceptually how the program should loop:
ncl changedateRESTART.ncl # change the file so the restart file can be used from the beginning of the year
mv ../RESTART.1993123* RESTART.19931231-3 # save the restart that started this run, in folder 4
# Restart1993*-3 er altså på begynnelsen av kjøring 4; Restart1993*00_Domain er på slutten.
scp RESTART.19931231*-* irenebn@sverdrup.uio.no:/mn/vann/metos-d2/irenebn/HRLDAS/Resultater/full1992and1993/spinup/4
cd 4 # move to another folder
cp ../namelist.hrldas . # move namelist..
cp ../RESTART.1993123100* . # the modified restart file...
cp ../*.TBL . # and all table files necessary to run hrldas
sbatch ../spinup-hrldas.job # and run the job
scp ../RESTART.1993123100* irenebn@sverdrup.uio.no:/mn/vann/metos-d2/irenebn/HRLDAS/Resultater/full1992and1993/spinup/4
cp RESTART.1993123100* .. # copy out the result to next iteration.
cd .. scp ../RESTART.1993123100* irenebn@sverdrup.uio.no:/mn/vann/metos-d2/irenebn/HRLDAS/Resultater/full1992and1993/spinup/5 ncl changedateRESTART.ncl cd 5 cp ../namelist.hrldas . cp ../RESTART1993123* . cp ../*.TBL . sbatch ../spinup-hrldas.job mv ../RESTART.1993123* RESTART.19931231-4 # save the restart that started this run, in folder 5 scp RESTART.19931231*-* irenebn@sverdrup.uio.no:/mn/vann/metos-d2/irenebn/HRLDAS/Resultater/full1992and1993/spinup/5 cp RESTART1993123100* ..
cd .. scp ../RESTART.1993123100* irenebn@sverdrup.uio.no:/mn/vann/metos-d2/irenebn/HRLDAS/Resultater/full1992and1993/spinup/6 ncl changedateRESTART.ncl cd 6 cp ../namelist.hrldas . cp ../RESTART1993123* . cp ../*.TBL . sbatch ../spinup-hrldas.job # så langt har jeg kommi # emacs ../kopi_hrldas_sverdrup.sh # change the numbers to next folder; in this case both 5's to 6. # sjekk at mv ../RESTART.1993123* RESTART.19931231-5 # save the restart that started this run, in folder 6 scp RESTART.19931231*-* irenebn@sverdrup.uio.no:/mn/vann/metos-d2/irenebn/HRLDAS/Resultater/full1992and1993/spinup/6 cp RESTART1993123100* ..
cd .. scp ../RESTART.1993123100* irenebn@sverdrup.uio.no:/mn/vann/metos-d2/irenebn/HRLDAS/Resultater/full1992and1993/spinup/7 ncl changedateRESTART.ncl cd 7 cp ../namelist.hrldas . cp ../RESTART1993123* . cp ../*.TBL . sbatch ../spinup-hrldas.job mv ../RESTART.1993123* RESTART.19931231-6 # save the restart that started this run, in folder 7 scp RESTART.19931231*-* irenebn@sverdrup.uio.no:/mn/vann/metos-d2/irenebn/HRLDAS/Resultater/full1992and1993/spinup/7 cp RESTART1993123100* ..
cd .. scp ../RESTART.1993123100* irenebn@sverdrup.uio.no:/mn/vann/metos-d2/irenebn/HRLDAS/Resultater/full1992and1993/spinup/8 ncl changedateRESTART.ncl cd 8 cp ../namelist.hrldas . cp ../RESTART1993123* . cp ../*.TBL . sbatch ../spinup-hrldas.job mv ../RESTART.1993123* RESTART.19931231-7 # save the restart that started this run, in folder 8 scp RESTART.19931231*-* irenebn@sverdrup.uio.no:/mn/vann/metos-d2/irenebn/HRLDAS/Resultater/full1992and1993/spinup/8 cp RESTART1993123100* ..
cd .. scp ../RESTART.1993123100* irenebn@sverdrup.uio.no:/mn/vann/metos-d2/irenebn/HRLDAS/Resultater/full1992and1993/spinup/9 ncl changedateRESTART.ncl cd 9 cp ../namelist.hrldas . cp ../RESTART1993123* . cp ../*.TBL . sbatch ../spinup-hrldas.job mv ../RESTART.1993123* RESTART.19931231-8 # save the restart that started this run, in folder 9 scp RESTART.19931231*-* irenebn@sverdrup.uio.no:/mn/vann/metos-d2/irenebn/HRLDAS/Resultater/full1992and1993/spinup/9 cp RESTART1993123100* ..
Copy between disks
Neste steg er å synkronisere filene mellom Abel og vann. Foreløpig har jeg laget et script for å kopiere resultatene fra Abel til vann (se under). Såvidt jeg skjønner må dette settes på før modellen kjører. Fil Kopi.job:
#!/bin/bash # Job name: #SBATCH --job-name=kopi # # Project: #SBATCH --account=geofag # # Wall clock limit: #SBATCH --time=20:0:0 # # Max memory usage per task: #SBATCH --mem-per-cpu=2000M # # Number of tasks (cores): #SBATCH --ntasks-per-node=1 # # Number of nodes: #SBATCH --nodes=1 # ###SBATCH --constraint=amd # Set up job environment source /cluster/bin/jobsetup ulimit -s unlimited export LANG=en_US.UTF-8 export LC_ALL=en_US ./kopi_hrldas_sverdrup.sh
Fil kopi_hrldas_sverdrup.sh:
#!/bin/bash #Script too cpy to home folder on Sverdrup # 1) create a directory on Sverdrup/vann for the outfiles, fex mkdir full1992and1993 cd "/usit/abel/u1/irenebn/hrldas/hrldas-v3.4.1/Run/1992and1993" # Output files 199*LDASOUT* are created here while true; do date ls -l 199*LDASOUT*>>~/kopi-status.txt time rsync -F -tv ./199*LDASOUT* irenebn@sverdrup.uio.no:/mn/vann/metos-d2/irenebn/HRLDAS/Resultater/full1992and1993 >>~/kopi-status.txt 2>&1 sleep 600 # copies once every 10 minutes done #kopi_hrldas_sverdrup.sh (END)
Scriptet kopi_hrldas_sverdrup.sh slutter aldri å kjøre ("while true;"), så etter at simuleringene er ferdige må jobben stoppes (scancel #jobid). Finnes det en smart måte å ekstrahere #jobid på, eller må jeg stoppe jobben manuelt etter hver simulering?
Look at results
ncview 1/1992*0112.LDA* 2/1993*0112.LDASO* 3/1993*0112.LDASO* 4/1993*0112.LDASO* 5/1993*0112.LDASO* 6/1993*0112.LDA* 7/1993*0112.LDA* 8/1993*0112.LDA* 9/1993*0112.LDA*
