Difference between revisions of "5. Retrieve larger data sets on MARS"
m |
m |
||
Line 35: | Line 35: | ||
#@ account_no = spnoflex | #@ account_no = spnoflex | ||
#@ queue | #@ queue | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
Line 68: | Line 56: | ||
http://www.ecmwf.int/publications/manuals/mars/guide/MARS_keywords.html | http://www.ecmwf.int/publications/manuals/mars/guide/MARS_keywords.html | ||
+ | |||
+ | If retrieving fields for different times: | ||
+ | |||
+ | in a loop over dates: | ||
+ | |||
+ | - create a file “ct1ed” containing the request information for data for a given date (need to specify keywords) | ||
+ | |||
+ | - execute this script on mars: mars ct1ed | ||
+ | |||
+ | - this script generates the specified file | ||
+ | |||
+ | - retrieve the file using ectrans command | ||
Revision as of 15:02, 12 March 2013
Back to ECMWF overview[1]
BATCH JOBS
The retrieval of data is done through a submission of a shell-script.
When running a program on the UNIX system use the batch system (not interactive mode). That means you submit the job with explicit commands so that the job is run unattended under Unix.
The batch system currently available on ECgate and HPCF is called LoadLeveler and jobs are submitted with the command llsubmit.
The default shell is either Korn (ksh) or C-shell (csh).
CREATING A SCRIPT
Log in to ECgate.
To retrieve large numbers of files from MARS create a ksh script on your home directory on ecgate.
For a script example see the script from Sam-Erik:
/nilu/home/sec/ecmwf/ecmwf_starg_all.ksh
In the beginning of the script, set the Batch System keywords:
#@ shell = /usr/bin/ksh #@ job_type = serial #@ job_name = request_to_MARS #@ output = /scratch/ms/no/$user/MARS_out #@ error = /scratch/ms/no/$user/MARS_err #@ initialdir = /scratch/ms/no/$user/ #@ class = normal #@ notification = complete #@ account_no = spnoflex #@ queue
The request information looks like this:
retrieve, class = od, stream = enfo, expver = 1, date = $cdate, time = 00:00, step = 0/to/72/by/6, type = cf, levtype = pl, levelist = 100/150/200/250/300/400/500/700/850/925/1000, param = 129.128/130.128/131.128/132.128/133.128, grid = 0.5/0.5, area = 65/0/55/20, target = "ecmwf_ctl3d_$cdate.grib"
A summary of MARS keywords can be found here:
http://www.ecmwf.int/publications/manuals/mars/guide/MARS_keywords.html
If retrieving fields for different times:
in a loop over dates:
- create a file “ct1ed” containing the request information for data for a given date (need to specify keywords)
- execute this script on mars: mars ct1ed
- this script generates the specified file
- retrieve the file using ectrans command
To transfer files to zardoz add the following to your ksh script:
ectrans -remote my_ms_association@genericFtp \ -source ecmwf_ctl3d_$cdate.grib \ -mailto user@nilu.no \ -onfailure \ -remove \ -verbose
SUBMIT YOUR JOB
To submit your script:
llsubmit myscript
To cancel your script:
llcancel myscript
To view where your job is in the queue:
llq
To append file2 to file1:
cat file2 >> file1
To concatenate 2 files to make a new file:
cat file1 file2 > file3
Working with files on the archive
els “ls” equivalent ectrans “mv” equivalent ecd “cd” equivalent erm “rm” equivalent