Difference between revisions of "5. Retrieve larger data sets on MARS"
m |
|||
Line 1: | Line 1: | ||
+ | Back to ECMWF overview[http://muspelheim.nilu.no/w/index.php/ECMWF] | ||
+ | |||
+ | |||
+ | == BATCH JOBS == | ||
+ | |||
+ | Log in to ECgate. | ||
+ | |||
+ | The retrieval of data is done through a submission of a shell-script. | ||
+ | |||
+ | When running a program on the two systems use the batch system (not interactive mode). That means you submit the job with explicit commands so that the job is run unattended under Unix. | ||
+ | |||
+ | The batch system currently available on ECgate and HPCF is called '''LoadLeveler''' and jobs are submitted with the command ''llsubmit''. | ||
+ | |||
+ | The default shell is either Korn (ksh) or C-shell (csh). | ||
+ | |||
+ | |||
+ | |||
+ | == CREATING A SCRIPT == | ||
+ | |||
+ | |||
To retrieve large numbers of files from MARS create a '''ksh script''' on your home directory on ecgate. | To retrieve large numbers of files from MARS create a '''ksh script''' on your home directory on ecgate. | ||
Line 57: | Line 77: | ||
-remove \ | -remove \ | ||
-verbose | -verbose | ||
+ | |||
+ | |||
+ | == SUBMIT YOUR JOB == | ||
+ | |||
+ | |||
To submit your script: | To submit your script: | ||
Line 78: | Line 103: | ||
ecd “cd” equivalent | ecd “cd” equivalent | ||
erm “rm” equivalent | erm “rm” equivalent | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− |
Revision as of 14:23, 12 March 2013
Back to ECMWF overview[1]
BATCH JOBS
Log in to ECgate.
The retrieval of data is done through a submission of a shell-script.
When running a program on the two systems use the batch system (not interactive mode). That means you submit the job with explicit commands so that the job is run unattended under Unix.
The batch system currently available on ECgate and HPCF is called LoadLeveler and jobs are submitted with the command llsubmit.
The default shell is either Korn (ksh) or C-shell (csh).
CREATING A SCRIPT
To retrieve large numbers of files from MARS create a ksh script on your home directory on ecgate.
For a script example see the script from Sam-Erik:
/nilu/home/sec/ecmwf/ecmwf_starg_all.ksh
In the beginning of the script, set the Batch System keywords:
#@ shell = /usr/bin/ksh #@ job_type = serial #@ job_name = request_to_MARS #@ output = /scratch/ms/no/$user/MARS_out #@ error = /scratch/ms/no/$user/MARS_err #@ initialdir = /scratch/ms/no/$user/ #@ class = normal #@ notification = complete #@ account_no = spnoflex #@ queue
If retrieving fields for different times:
in a loop over dates:
- create a file “ct1ed” containing the request information for data for a given date (need to specify keywords)
- execute this script on mars: mars ct1ed
- this script generates the specified file
- retrieve the file using ectrans command
The request information looks like this:
retrieve, class = od, stream = enfo, expver = 1, date = $cdate, time = 00:00, step = 0/to/72/by/6, type = cf, levtype = pl, levelist = 100/150/200/250/300/400/500/700/850/925/1000, param = 129.128/130.128/131.128/132.128/133.128, grid = 0.5/0.5, area = 65/0/55/20, target = "ecmwf_ctl3d_$cdate.grib"
A summary of MARS keywords can be found here:
http://www.ecmwf.int/publications/manuals/mars/guide/MARS_keywords.html
To transfer files to zardoz add the following to your ksh script:
ectrans -remote my_ms_association@genericFtp \ -source ecmwf_ctl3d_$cdate.grib \ -mailto user@nilu.no \ -onfailure \ -remove \ -verbose
SUBMIT YOUR JOB
To submit your script:
llsubmit myscript
To cancel your script:
llcancel myscript
To view where your job is in the queue:
llq
To append file2 to file1:
cat file2 >> file1
To concatenate 2 files to make a new file:
cat file1 file2 > file3
Working with files on the archive
els “ls” equivalent ectrans “mv” equivalent ecd “cd” equivalent erm “rm” equivalent