Difference between revisions of "7. Retrieve ECMWF fields for FLEXPART"

From mn/geo/geoit
Jump to: navigation, search
m
 
Line 22: Line 22:
  
  
'''LIST OF PROCEDURES TO FOLLOW TO RETRIEVE DATA:'''
+
 
 +
== LIST OF PROCEDURES TO FOLLOW TO RETRIEVE DATA ==
  
 
If retrieving a small number of files this can be done on ECGATE, but for larger data sets it is recommended to use the supercomputer facilities (HPCF-High Performance Computing Facility) when logging in (c2a)
 
If retrieving a small number of files this can be done on ECGATE, but for larger data sets it is recommended to use the supercomputer facilities (HPCF-High Performance Computing Facility) when logging in (c2a)
Line 136: Line 137:
  
  
'''OVERVIEW OVER PARAMETERS THAT MAY BE CHANGED BY THE USER (default values in brackets):'''
+
 
 +
== OVERVIEW OVER PARAMETERS THAT MAY BE CHANGED BY THE USER (default values in brackets) ==
 +
 
  
 
  FORCE_E40  - (0)  -  Retrieve data from class E4 (ERA-40)
 
  FORCE_E40  - (0)  -  Retrieve data from class E4 (ERA-40)

Latest revision as of 15:38, 12 March 2013

Back to ECMWF overview[1]


Updated by nik: 12 March 2013


The retrieval procedure uses a number of fortran scripts which are run either ECgate or the HPCF and retrieves data from MARS. In short:

- Retrieves the meteorological model data output from the MARS (see bottom of page for a list of which data are retrieved)

- De-accumulates atmospheric flux data (FLXACC2 fortran code under source.tar)

- Calculates the vertical velocity from the continuity equation (CONVERT2 fortran code under source.tar)

- Prepares the met.files (ENyyyymmddhh) that can be used for FLEXPART.


More details are given in the documents on the webpage:

http://transport.nilu.no/flexpart
under Routines for retrieval of FLEXTRA/FLEXPART input data from ECMWF



LIST OF PROCEDURES TO FOLLOW TO RETRIEVE DATA

If retrieving a small number of files this can be done on ECGATE, but for larger data sets it is recommended to use the supercomputer facilities (HPCF-High Performance Computing Facility) when logging in (c2a)


1. Need to copy the following directory and files to your home directory on ecgate:

cp -r /home/ms/sbc/flex_extract_ecgate_V4/ YourDestination

List of files needed:

flex_ecmwf_91_global_1.0_ecgate
extract_ecmwf_ftpbase
seq_control.V4
seq_extract.V4
submit
source.tar (contains Fortran90 source files and makefiles for the data processing of vertical fluxes etc)

2. Edit the times in the file seq_control.V4:

./seq_extract.V4 1990 1990 06 06 29 29 1
This extract the files from 1990 06 29 to 1990 06 29 - for 1 day!

3. Check that the file flex_ecmwf_91_global_1.0_ecgate contains the following:

(for an overview over the parameters and what they mean - see bottom of page)

-----------------------------------
BEGIN: modification of config file
-----------------------------------
cat <<EOF >CONTROL_ERA
DAY1 YYYYMMDD1
DAY2 YYYYMMDD2
DTIME 3
FORCE_EPS 0
FORCE_EI 0
FORCE_ERA 0
M_GRID 100  
M_LEFT -17800
M_LOWER -9000
M_UPPER 9000
M_RIGHT 18100
M_LEVEL 91
M_RESOL 255
M_GAUSS 1
M_ACCURACY 24
M_OMEGA 0
M_OMEGADIFF 0
M_SMOOTH 179
M_ADDPAR /27/28/173/186/187/188/235/139/39
M_FORMAT GRIB2
GATEWAY ecaccess.ecmwf.int
DESTINATION flexpart                                (or change to another of your MS associations)
ECSTORAGE 1
ECTRANS 1
ECFSDIR ectmp:/${USER}/oper_YYYY_MM/
MAILOPS ${USER}
MAILFAIL ${USER}
EXEDIR .
SOURCECODE /home/ms/no/sbc/flex_extract_ecgate_V4     (CHANGE TO YOUR USER!!)
EOF   
-----------------------------------
- END: modification of config file
-----------------------------------


It is important to check that the three parameters M_GAUSS=1, M_RESOL=255, M_SMOOTH=179 have the set values as this is optimal for the calculation of vertical velocities for a 1.0 degree grid.


4. In the scripts, reference to the right ECMWF userID must be made. Use the command

grep $USER *

to list all lines in the files containing a reference to the ECMWF userId and that should be set to your own user.

You will need to change in: submit, seq_control.V4, seq_extract.V4 and flex_ecmwf_91_global_1.0_ecgate


5. Copy the template file flex_ecmwf_91_global_1.0_ecgate to extract_ecmwf_ftpbase

cp flex_ecmwf_91_global_1.0_ecgate extract_ecmwf_ftpbase


6. Run the job:

./submit


7. Check your job:

llq            to see all jobs in the queue 
llq -u $USER   to see only jobs for your userID
eoj <jobid> 	provides details on the performance of a running job. (also automatically provided at end of job)

You can also follow your execution on the Web access facilities on ECMWF

You will receive an email notifying the start of the script and the successful completion of the script.

You will also get a file finished.* in your directory indicating that the script successfully completed.

The scripts can take from a few minutes up to a few hours to complete, depending on the load of the MARS system.


8. Possible error messages:

LoadL_starter: The program, extract_ecmwf_ftp01.5046746, exited normally and returned an exit code of 1.

In the file *.out search for "exit" to find the line where exit code 1 is given. Here might be hints to find the error message.


9. Get the files:

The retrieved files are automatically transferred to NILU via ftp using the MS association that you specified under "DESTINATION" and that exist under your account (check this under 2. Logging in and account settings).

If you cannot find your files in the defined NILU-directory, check the directory you specified in flex_ecmwf_91_global_1.0_ecgate * (ECFSDIR ectmp:/${USER}/oper_YYYY_MM/) for where it is temporarily stored on the ECWMF system. Then try ftp it over again using the ftp commands specified under 6. File transfer via ftp



OVERVIEW OVER PARAMETERS THAT MAY BE CHANGED BY THE USER (default values in brackets)

FORCE_E40  - (0)  -  Retrieve data from class E4 (ERA-40)
FORCE_EI   - (0)  -  Retrieve data from class EI (ERA-Interim)
FORCE_EPS  - (0)  -  Retrieve data from ensemble prediction system
M_STREAM   -(OPER)-  Allows to retrieve data from IFS experiment
M_EXPVER   - (1)  -  Experiment number, necessary for ERA-Interim
M_NUMBER   - (1)  -  Forecast leg, necessary for EPS. 1=T399 forecasts, 2=T255
M_GRID     -(100) -  Horizontal resolution of Lat/Lon grid in 1/100 degree
M_RESOL    -(159) -  Horizontal resolution of spectral fields
M_LEVEL    - (91) -  Number of vertical levels
M_LEVELIST -(ALL) -  List of vertical levels for MARS request
M_ADDPAR          -  Additional surface parameters  (/27/28/173/186/187/188/235/139/39)
M_UPPER           -  Latitude of upper right corner of grid area to be retrieved in Hundredth of a degree (e.g. 3750 for 37.5N)
M _LOWER          -  Latitude of lower left corner of grid area in 1/100 degree
M_LEFT            -  Longitude of lower left corner of grid area in 1/100 degree
M_RIGHT           -  Longitude of upper right corner of grid area in1/100 degree.
M_ACCURACY - (24) -  Accuracy of GRIB data in bits
M_GAUSS    -  (1) -  Calculation of etadot on Lat/Lon grid (0) or on Gaussian grid (1)
M_OMEGA    -  (0) -  Retrieve Omega from MARS and put it to file OMEGAyymmddhh
M_OMEGADIFF-  (0) -  Calculate Omega and Dps/Dt from continuity equation for diagnostic purposes and include it in file OMEGAyymmddhh
M_SMOOTH   -  (0) -  Spectral truncation of etadot after calculation on Gaussian grid
GATEWAY           -  Name of ECaccess gateway server
DESTINATION       -  Name of destination at ECaccess gateway server
ECSTORAGE  -  (1) -  Store EN-files in ECFS file system 0/1 (1)
ECTRANS    -  (1) -  Transfer EN-files to gateway server 0/1 (1)  
ECFSDIR           -  Destination directory on ECFS file system
MAILOPS           -  List of email addresses for operational log-files
MAILFAIL          -  List of email addresses for failure log-files
EXEDIR     - (./) -  Location of FORTRAN executables
SOURCECODE        -  Location of file source.tar containing source code and makefiles ($HOME/flex_extract_ecgate_V2)



LIST OF METEOROLOGICAL PARAMETERS RETRIEVED FOR FLEXPART

FLEXPART-variable         Description             paramid           MARS shortName           Unit
tth                       Temperature                130                       t                 K
uuh                       U-velocity                 131                       u                 ms-1
vvh                       V-velocity                 132                       v                 ms-1
qvh                       Specific humidity          133                       q                 kgkg-1
sp                        Surface Pressure           134                       sp                Pa
wwh                       W-velocity                 135                       w                 Pa s-1
sd                        Snow depth                 141                       sd                m of water equivalent
msl                       Sea level pressure         151                       msl               Pa
tcc                       Total cloud cover          164                       tcc               (0-1)
u10                       10m U velocity             165                       10u               ms-1
v10                       10m V velocity             166                       10v               ms-1
tt2                       2m temperature             167                       2t                K
td2                       2m dew point               168                       2d                K
lsprec                    Large scale precipitation  142                       lsp               m
convprec                  Convective precipitation   143                       cp                m
sshf                      Sensible heat flux         146                       sshf              Jm-2
ssr                       Solar radiation            176                       ssr               Jm-2
ewss                      EW surface stress          180                       ewss              Nm-2s
nsss                      NS surface stress          181                       nsss              Nm-2s
oro                       Orography                  129                       z                 m2s-2
excessoro                 Standard error orography   160                       sdor              ~
lsm                       Land-Sea mask              172                       lsm               (0-1)

Total of 22 parameters are retrieved.


The paramid (parameter ID) is following the table nr 128.

http://www.ecmwf.int/publications/manuals/d/gribapi/param/filter=grib1/order=paramId/order_type=asc/p=1/table=128/

The paramid is edition independent, meaning it is the same id for both GRIB1 and GRIB2 files.