Difference between revisions of "5. Retrieve larger data sets on MARS"

Revision as of 13:09, 29 October 2012

To retrieve large numbers of files from MARS create a ksh script on your home directory on ecgate.

For a script example see the script from Sam-Erik:

/nilu/home/sec/ecmwf/ecmwf_starg_all.ksh

In the beginning of the script, set the Batch System keywords:

#@ shell        = /usr/bin/ksh
#@ job_type     = serial
#@ job_name     = request_to_MARS
#@ output       = /scratch/ms/no/$user/MARS_out
#@ error        = /scratch/ms/no/$user/MARS_err
#@ initialdir   = /scratch/ms/no/$user/
#@ class        = normal
#@ notification = complete
#@ account_no   = spnoflex
#@ queue

If retrieving fields for different times:

in a loop over dates:

- create a file “ct1ed” containing the request information for data for a given date (need to specify keywords)

- execute this script on mars: mars ct1ed

- this script generates the specified file

- retrieve the file using ectrans command

The request information looks like this:

retrieve,
 class    = od,
 stream   = enfo,
 expver   = 1,
 date     = $cdate,
 time     = 00:00,
 step     = 0/to/72/by/6,
 type     = cf,
 levtype  = pl,
 levelist = 100/150/200/250/300/400/500/700/850/925/1000,
 param    = 129.128/130.128/131.128/132.128/133.128,
 grid     = 0.5/0.5,
 area     = 65/0/55/20,
 target   = "ecmwf_ctl3d_$cdate.grib"

A summary of MARS keywords can be found here:

http://www.ecmwf.int/publications/manuals/mars/guide/MARS_keywords.html

To transfer files to zardoz add the following to your ksh script:

ectrans -remote my_ms_association@genericFtp \
-source ecmwf_ctl3d_$cdate.grib \
-mailto user@nilu.no \
-onfailure \
-remove \
-verbose

To submit your script:

llsubmit myscript

To cancel your script:

llcancel myscript

To view where your job is in the queue:

llq

To append file2 to file1:

cat file2 >> file1

To concatenate 2 files to make a new file:

cat file1 file2 > file3 

Working with files on the archive

els		“ls” equivalent
ectrans	“mv” equivalent
ecd		“cd” equivalent
erm		“rm” equivalent

Back to ECMWF overview[1]