Difference between revisions of "File systems and data structure at ECMWF - where to store and get data!"
From mn/geo/geoit
(Created page with "Back to ECMWF overview[http://muspelheim.nilu.no/w/index.php/ECMWF] ''Updated by nik: 12 March 2013'' '''ECgate''' is a Unix server (soon to be changed to a Linux server) wh...") |
Basal@uio.no (talk | contribs) |
||
| (5 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
| − | Back to | + | Back to [[ECMWF]] overview |
| + | ''Updated by nik: 12 March 2013'' | ||
| − | + | <br/>A good overview of the Computing services is given here: | |
| + | [http://www.ecmwf.int/services/computing/ http://www.ecmwf.int/services/computing/] | ||
| − | '''ECgate''' | + | <br/>'''ECgate''' |
is a Unix server (soon to be changed to a Linux server) where each user has directories: | is a Unix server (soon to be changed to a Linux server) where each user has directories: | ||
| Line 13: | Line 15: | ||
- SCRATCHDIR: Files are autodeleted after end of job. Part of SCRATCH and quota. | - SCRATCHDIR: Files are autodeleted after end of job. Part of SCRATCH and quota. | ||
| + | <br/>'''HPCF''' | ||
| − | + | (High performance computing facility) is the supercomputer (c2a) where each user has directories: | |
| − | |||
| − | |||
- HOME: Small permanent files, daily backup/snapshot. Quota: 320 MB. To check quota: quota | - HOME: Small permanent files, daily backup/snapshot. Quota: 320 MB. To check quota: quota | ||
- SCRATCH: Temporary data, removed regularly. Quota: 25 TB. | - SCRATCH: Temporary data, removed regularly. Quota: 25 TB. | ||
| Line 24: | Line 25: | ||
The two systems use the same name convension - but different file systems. | The two systems use the same name convension - but different file systems. | ||
| + | |||
| + | <br/>'''Accsessing ECGATE from HPCF''' | ||
| + | |||
| + | If you need to access data on ECGATE from the HPCF you can find them in /ws/$HOME | ||
| + | |||
| Line 30: | Line 36: | ||
'''On both ECgate and HPCF you can access 2 different archives.''' | '''On both ECgate and HPCF you can access 2 different archives.''' | ||
| − | + | <br/>'''1) MARS (Meteorological Archival and Retrieval System)''' | |
| − | '''1) MARS (Meteorological Archival and Retrieval System)''' | ||
which contains the meteorological forecast/analysis/re-analysis data in GRIB format, as well as observation data in BUFR format. | which contains the meteorological forecast/analysis/re-analysis data in GRIB format, as well as observation data in BUFR format. | ||
| + | <br/>'''2) ECFS (ECMWF File storage system):''' | ||
| − | |||
| − | |||
This is a User Archive where you can store any long-term data on tape. | This is a User Archive where you can store any long-term data on tape. | ||
On ECFS you have two domains: | On ECFS you have two domains: | ||
| + | |||
- ec: Permanent domain where files are stored indefinitively. | - ec: Permanent domain where files are stored indefinitively. | ||
- ectmp: temporary domain where files are stored for 90 days after which they are automatically deleted (and cannot be recovered). | - ectmp: temporary domain where files are stored for 90 days after which they are automatically deleted (and cannot be recovered). | ||
| − | To store/access files in ECFS you must issue explicit commands which are set-up to use a UNIX like interface. | + | To store/access files in ECFS you must issue explicit commands which are set-up to use a UNIX like interface. |
Commands are same as for unix but starting with "e": | Commands are same as for unix but starting with "e": | ||
| Line 54: | Line 59: | ||
''els'' lists your data under ec: | ''els'' lists your data under ec: | ||
''ls'' lists your data under $HOME (on ECgate or HPCF dependent on where you are logged in) | ''ls'' lists your data under $HOME (on ECgate or HPCF dependent on where you are logged in) | ||
| + | |||
| + | [[Category:Tools]][[Category:Models]][[Category:ECMWF]] | ||
Latest revision as of 17:21, 30 January 2015
Back to ECMWF overview
Updated by nik: 12 March 2013
A good overview of the Computing services is given here:
http://www.ecmwf.int/services/computing/
ECgate
is a Unix server (soon to be changed to a Linux server) where each user has directories:
- HOME: Small permanent files, daily backup/snapshot. Quota: 3.6 GB. To check quota: ecquota - SCRATCH: Temporary data, removed regularly. Quota: 100 GB - SCRATCHDIR: Files are autodeleted after end of job. Part of SCRATCH and quota.
HPCF
(High performance computing facility) is the supercomputer (c2a) where each user has directories:
- HOME: Small permanent files, daily backup/snapshot. Quota: 320 MB. To check quota: quota - SCRATCH: Temporary data, removed regularly. Quota: 25 TB. - SCRATCHDIR: Autodeled after end of job. - PERM: Large permanent files, no backup. Quota: 26 GB. Accessible from ECgate as /c2a/perm
The two systems use the same name convension - but different file systems.
Accsessing ECGATE from HPCF
If you need to access data on ECGATE from the HPCF you can find them in /ws/$HOME
On both ECgate and HPCF you can access 2 different archives.
1) MARS (Meteorological Archival and Retrieval System)
which contains the meteorological forecast/analysis/re-analysis data in GRIB format, as well as observation data in BUFR format.
2) ECFS (ECMWF File storage system):
This is a User Archive where you can store any long-term data on tape.
On ECFS you have two domains:
- ec: Permanent domain where files are stored indefinitively. - ectmp: temporary domain where files are stored for 90 days after which they are automatically deleted (and cannot be recovered).
To store/access files in ECFS you must issue explicit commands which are set-up to use a UNIX like interface.
Commands are same as for unix but starting with "e":
els, ecd, ecat, ecp, epwd, emv, echmod...
For a manual of the commands type "man els".
els lists your data under ec: ls lists your data under $HOME (on ECgate or HPCF dependent on where you are logged in)
