5. Retrieve larger data sets on MARS
Back to ECMWF overview[1]
BATCH JOBS
The retrieval of data is done through a submission of a shell-script.
When running a program on the UNIX system use the batch system (not interactive mode). That means you submit the job with explicit commands so that the job is run unattended under Unix. nohup is similarly submission of jobs to run unattended on a Unix system, but there exists more sophisticated batch systems for handling jobs.
The batch system currently available on ECgate and HPCF is called LoadLeveler and jobs are submitted with the command llsubmit.
The default shell is either Korn (ksh) or C-shell (csh).
CREATING A SCRIPT
Log in to ECgate.
To retrieve large numbers of files from MARS create a ksh script on your home directory on ecgate.
For a script example see the script from Sam-Erik:
/nilu/home/sec/ecmwf/ecmwf_starg_all.ksh
In the beginning of the script, set the Batch System keywords:
#@ shell = /usr/bin/ksh #@ job_type = serial #@ job_name = request_to_MARS #@ output = /scratch/ms/no/$user/MARS_out #@ error = /scratch/ms/no/$user/MARS_err #@ initialdir = /scratch/ms/no/$user/ #@ class = normal #@ notification = complete #@ account_no = spnoflex #@ queue
Do not use environmental variables like $SCRATCH, $USER to define LL keywords.
Then add your request information which might look like this:
retrieve, class = od, ("Operational archive" stream = oper, ("operational Atmospheric model") expver = 1, ("Experiment version", always use 1) date = 1, ("Specifies the Analysis/Forecast base date", n is the number of days before today) time = 00:00, ("Specifies the Analysis/Forecast base time" step = 0/to/72/by/6, type = cf, levtype = pl, levelist = 100/150/200/250/300/400/500/700/850/925/1000, param = 129.128/130.128/131.128/132.128/133.128, grid = 0.5/0.5, area = 65/0/55/20, target = "ecmwf_data.grib" (Output file containing the data)
A summary of MARS keywords can be found here:
http://www.ecmwf.int/publications/manuals/mars/guide/MARS_keywords.html
To transfer files to zardoz add the following to your ksh script:
ectrans -remote my_ms_association@genericFtp \ -source ecmwf_data.grib \ -mailto user@nilu.no \ -onfailure \ -remove \ -verbose
Instead of the "retrieve" keyword in the above script you can use other keywords to "read", "compute" or "write" data.
As a rule - you should have as few retrieval routines as possible, but rather have each retrieval routine
SUBMIT YOUR JOB
Submit your job as a batch job to LoadLeveler.
To submit your script:
llsubmit myscript
To cancel your script:
llcancel myscript
MONITOR YOUR JOB
llq -u <UserId> To view where your job is in the queue llq -l <jobId> To get a detailed description for a job
llq -s <jobId> To determine why the job is not running
See man llq for more options