5. Retrieve larger data sets on MARS

From mn/geo/geoit
Revision as of 13:23, 12 March 2013 by Nik (talk | contribs)

Jump to: navigation, search

Back to ECMWF overview[1]


BATCH JOBS

The retrieval of data is done through a submission of a shell-script.

When running a program on the UNIX system use the batch system (not interactive mode). That means you submit the job with explicit commands so that the job is run unattended under Unix.

The batch system currently available on ECgate and HPCF is called LoadLeveler and jobs are submitted with the command llsubmit.

The default shell is either Korn (ksh) or C-shell (csh).


CREATING A SCRIPT

Log in to ECgate.

To retrieve large numbers of files from MARS create a ksh script on your home directory on ecgate.

For a script example see the script from Sam-Erik:

/nilu/home/sec/ecmwf/ecmwf_starg_all.ksh

In the beginning of the script, set the Batch System keywords:

#@ shell        = /usr/bin/ksh
#@ job_type     = serial
#@ job_name     = request_to_MARS
#@ output       = /scratch/ms/no/$user/MARS_out
#@ error        = /scratch/ms/no/$user/MARS_err
#@ initialdir   = /scratch/ms/no/$user/
#@ class        = normal
#@ notification = complete
#@ account_no   = spnoflex
#@ queue

If retrieving fields for different times:

in a loop over dates:

- create a file “ct1ed” containing the request information for data for a given date (need to specify keywords)

- execute this script on mars: mars ct1ed

- this script generates the specified file

- retrieve the file using ectrans command


The request information looks like this:

retrieve,
 class    = od,
 stream   = enfo,
 expver   = 1,
 date     = $cdate,
 time     = 00:00,
 step     = 0/to/72/by/6,
 type     = cf,
 levtype  = pl,
 levelist = 100/150/200/250/300/400/500/700/850/925/1000,
 param    = 129.128/130.128/131.128/132.128/133.128,
 grid     = 0.5/0.5,
 area     = 65/0/55/20,
 target   = "ecmwf_ctl3d_$cdate.grib"

A summary of MARS keywords can be found here:

http://www.ecmwf.int/publications/manuals/mars/guide/MARS_keywords.html


To transfer files to zardoz add the following to your ksh script:

ectrans -remote my_ms_association@genericFtp \
-source ecmwf_ctl3d_$cdate.grib \
-mailto user@nilu.no \
-onfailure \
-remove \
-verbose


SUBMIT YOUR JOB

To submit your script:

llsubmit myscript

To cancel your script:

llcancel myscript

To view where your job is in the queue:

llq

To append file2 to file1:

cat file2 >> file1

To concatenate 2 files to make a new file:

cat file1 file2 > file3 


Working with files on the archive

els		“ls” equivalent
ectrans 	“mv” equivalent
ecd		“cd” equivalent
erm		“rm” equivalent