5. Retrieve larger data sets on MARS
To retrieve large numbers of files from MARS create a ksh script on your home directory on ecgate.
For a script example see the script from Sam-Erik:
/nilu/home/sec/ecmwf/ecmwf_starg_all.ksh
In the beginning of the script, set the Batch System keywords:
#@ shell = /usr/bin/ksh #@ job_type = serial #@ job_name = request_to_MARS #@ output = /scratch/ms/no/$user/MARS_out #@ error = /scratch/ms/no/$user/MARS_err #@ initialdir = /scratch/ms/no/$user/ #@ class = normal #@ notification = complete #@ account_no = spnoflex #@ queue
If retrieving fields for different times:
in a loop over dates:
- create a file “ct1ed” containing the request information for data for a given date (need to specify keywords)
- execute this script on mars: mars ct1ed
- this script generates the specified file
- retrieve the file using ectrans command
The request information looks like this:
retrieve, class = od, stream = enfo, expver = 1, date = $cdate, time = 00:00, step = 0/to/72/by/6, type = cf, levtype = pl, levelist = 100/150/200/250/300/400/500/700/850/925/1000, param = 129.128/130.128/131.128/132.128/133.128, grid = 0.5/0.5, area = 65/0/55/20, target = "ecmwf_ctl3d_$cdate.grib"
A summary of MARS keywords can be found here:
http://www.ecmwf.int/publications/manuals/mars/guide/MARS_keywords.html
To transfer files to zardoz add the following to your ksh script:
ectrans -remote my_ms_association@genericFtp \ -source ecmwf_ctl3d_$cdate.grib \ -mailto user@nilu.no \ -onfailure \ -remove \ -verbose
To submit your script:
llsubmit myscript
To cancel your script:
llcancel myscript
To view where your job is in the queue:
llq
To append file2 to file1:
cat file2 >> file1
To concatenate 2 files to make a new file:
cat file1 file2 > file3
Working with files on the archive
els “ls” equivalent ectrans “mv” equivalent ecd “cd” equivalent erm “rm” equivalent
Back to ECMWF overview[1]