Difference between revisions of "5. Retrieve larger data sets on MARS"

From mn/geo/geoit
Jump to: navigation, search
(Created page with " Back to ECMWF overview[http://muspelheim.nilu.no/w/index.php/ECMWF]")
 
Line 1: Line 1:
 +
To retrieve large numbers of files from MARS create a ksh script on your home directory on ecgate.
  
 +
For a script example see the script from Sam-Erik:
 +
/nilu/home/sec/ecmwf/ecmwf_starg_all.ksh
  
 +
In the beginning of the script, set the Batch System keywords:
 +
#@ shell        = /usr/bin/ksh
 +
#@ job_type    = serial
 +
#@ job_name    = request_to_MARS
 +
#@ output      = /scratch/ms/no/$user/MARS_out
 +
#@ error        = /scratch/ms/no/$user/MARS_err
 +
#@ initialdir  = /scratch/ms/no/$user/
 +
#@ class        = normal
 +
#@ notification = complete
 +
#@ account_no  = spnoflex
 +
#@ queue
 +
 +
If retrieving fields for different times:
 +
 +
in a loop over dates:
 +
 +
 create a file “ct1ed” containing the request information for data for a  given date (need to specify keywords)
 +
 +
 execute this script on mars: mars ct1ed
 +
 +
 this script generates the specified file
 +
 +
 retrieve the file using ectrans command
 +
 +
The request information looks like this:
 +
retrieve,
 +
  class    = od,
 +
  stream  = enfo,
 +
  expver  = 1,
 +
  date    = $cdate,
 +
  time    = 00:00,
 +
  step    = 0/to/72/by/6,
 +
  type    = cf,
 +
  levtype  = pl,
 +
  levelist = 100/150/200/250/300/400/500/700/850/925/1000,
 +
  param    = 129.128/130.128/131.128/132.128/133.128,
 +
  grid    = 0.5/0.5,
 +
  area    = 65/0/55/20,
 +
  target  = "ecmwf_ctl3d_$cdate.grib"
 +
 +
A summary of MARS keywords can be found here:
 +
 +
http://www.ecmwf.int/publications/manuals/mars/guide/MARS_keywords.html
 +
 +
To transfer files to zardoz add the following to your ksh script:
 +
ectrans -remote my_ms_association@genericFtp \
 +
-source ecmwf_ctl3d_$cdate.grib \
 +
-mailto user@nilu.no \
 +
-onfailure \
 +
-remove \
 +
-verbose
 +
 +
To submit your script:
 +
llsubmit myscript
 +
 +
To cancel your script:
 +
llcancel myscript
 +
 +
To view where your job is in the queue:
 +
llq
 +
 +
To append file2 to file1:
 +
cat file2 >> file1
 +
To concatenate 2 files to make a new file:
 +
cat file1 file2 > file3
 +
 +
 +
Working with files on the archive
 +
els “ls” equivalent
 +
ectrans “mv” equivalent
 +
ecd “cd” equivalent
 +
erm “rm” equivalent
  
  

Revision as of 13:07, 29 October 2012

To retrieve large numbers of files from MARS create a ksh script on your home directory on ecgate.

For a script example see the script from Sam-Erik: 

/nilu/home/sec/ecmwf/ecmwf_starg_all.ksh

In the beginning of the script, set the Batch System keywords: 

#@ shell        = /usr/bin/ksh
#@ job_type     = serial
#@ job_name     = request_to_MARS
#@ output       = /scratch/ms/no/$user/MARS_out
#@ error        = /scratch/ms/no/$user/MARS_err
#@ initialdir   = /scratch/ms/no/$user/
#@ class        = normal
#@ notification = complete
#@ account_no   = spnoflex
#@ queue

If retrieving fields for different times:

in a loop over dates:

 create a file “ct1ed” containing the request information for data for a given date (need to specify keywords)

 execute this script on mars: mars ct1ed

 this script generates the specified file

 retrieve the file using ectrans command

The request information looks like this: 

retrieve,
 class    = od,
 stream   = enfo,
 expver   = 1,
 date     = $cdate,
 time     = 00:00,
 step     = 0/to/72/by/6,
 type     = cf,
 levtype  = pl,
 levelist = 100/150/200/250/300/400/500/700/850/925/1000,
 param    = 129.128/130.128/131.128/132.128/133.128,
 grid     = 0.5/0.5,
 area     = 65/0/55/20,
 target   = "ecmwf_ctl3d_$cdate.grib"

A summary of MARS keywords can be found here:

http://www.ecmwf.int/publications/manuals/mars/guide/MARS_keywords.html

To transfer files to zardoz add the following to your ksh script: 

ectrans -remote my_ms_association@genericFtp \
-source ecmwf_ctl3d_$cdate.grib \
-mailto user@nilu.no \
-onfailure \
-remove \
-verbose

To submit your script: 

llsubmit myscript

To cancel your script: 

llcancel myscript

To view where your job is in the queue: 

llq

To append file2 to file1: 

cat file2 >> file1

To concatenate 2 files to make a new file: 

cat file1 file2 > file3


Working with files on the archive 

els		“ls” equivalent
ectrans	“mv” equivalent
ecd		“cd” equivalent
erm		“rm” equivalent










Back to ECMWF overview[1]

Retrieved from "https://wiki.uio.no/mn/geo/geoit/index.php?title=5._Retrieve_larger_data_sets_on_MARS&oldid=461"