Difference between revisions of "11. File systems and data structure at ECMWF - where to store and get data!"

From mn/geo/geoit
Jump to: navigation, search
Line 42: Line 42:
 
Commands are same as for unix but starting with "e":
 
Commands are same as for unix but starting with "e":
  
els, ecd, ecat, ecp, epwd, emv, echmod,
+
els, ecd, ecat, ecp, epwd, emv, echmod...
  
 
For a manual of the commands type "man els".
 
For a manual of the commands type "man els".

Revision as of 17:27, 11 March 2013

ECgate

is a Unix server (soon to be changed to a Linux server) where each user has directories:

- HOME: Small permanent files, daily backup/snapshot. Quota: 3.6 GB. To check quota: ecquota
- SCRATCH: Temporary data, removed regularly. Quota: 100 GB
- SCRATCHDIR: Files are autodeleted after end of job. Part of SCRATCH and quota.


HPCF

(High performance computing facility) is the supercomputer (c2a) where each user has directories:

- HOME: Small permanent files, daily backup/snapshot. Quota: 320 MB. To check quota: quota
- SCRATCH: Temporary data, removed regularly. Quota: 25 TB.
- SCRATCHDIR: Autodeled after end of job. 
- PERM: Large permanent files, no backup. Quota: 26 GB. Accessible from ECgate as /c2a/perm

The two systems use the same name convension - but different file systems.



On both ECgate and HPCF you can access 2 different archives.


1) MARS

which contains the meteorological forecast/analysis/re-analysis data in GRIB format, as well as observation data in BUFR format.


2) ECFS (ECMWF File storage system):

This is a User Archive where you can store any long-term data on tape.

On ECFS you have two domains:

- ec: Permanent domain where files are stored indefinitively.
- ectmp: temporary domain where files are stored for 90 days after which they are automatically deleted (and cannot be recovered).

To store/access files in ECFS you must issue explicit commands which are set-up to use a UNIX like interface.

Commands are same as for unix but starting with "e":

els, ecd, ecat, ecp, epwd, emv, echmod...

For a manual of the commands type "man els".

els lists your data under ec:
ls lists your data under $HOME (on ECgate or HPCF dependent on where you are logged in)