Difference between revisions of "11. File systems and data structure at ECMWF - where to store and get data!"

From mn/geo/geoit
Jump to: navigation, search
m
 
(18 intermediate revisions by the same user not shown)
Line 1: Line 1:
−
'''ECgate is a Unix server (soon to be changed to a Linux server) where you have home directories:'''
+
Back to ECMWF overview[http://muspelheim.nilu.no/w/index.php/ECMWF]
 +
 
 +
 
 +
''Updated by nik: 12 March 2013''
 +
 
 +
 
 +
'''ECgate'''
 +
 
 +
is a Unix server (soon to be changed to a Linux server) where each user has directories:
  
 
  - HOME: Small permanent files, daily backup/snapshot. Quota: 3.6 GB. To check quota: ecquota
 
  - HOME: Small permanent files, daily backup/snapshot. Quota: 3.6 GB. To check quota: ecquota
Line 5: Line 13:
 
  - SCRATCHDIR: Files are autodeleted after end of job. Part of SCRATCH and quota.
 
  - SCRATCHDIR: Files are autodeleted after end of job. Part of SCRATCH and quota.
  
−
'''HPCF (High performance computing facility) is the supercomputer (c2a) where you have home directories:'''
+
 
 +
'''HPCF'''
 +
 
 +
(High performance computing facility) is the supercomputer (c2a) where each user has directories:
 
   
 
   
−
- HOME: Small permanent files, daily backup/snapshot. Quota: 320 MB. To check quota: quota
+
- HOME: Small permanent files, daily backup/snapshot. Quota: 320 MB. To check quota: quota
 
  - SCRATCH: Temporary data, removed regularly. Quota: 25 TB.
 
  - SCRATCH: Temporary data, removed regularly. Quota: 25 TB.
 
  - SCRATCHDIR: Autodeled after end of job.  
 
  - SCRATCHDIR: Autodeled after end of job.  
−
  - PERM: Large paermanent files, no backup. Quota: 26 GB. Accessible from ECgate as /c2a/perm
+
  - PERM: Large permanent files, no backup. Quota: 26 GB. Accessible from ECgate as /c2a/perm
  
 
The two systems use the same name convension - but different file systems.
 
The two systems use the same name convension - but different file systems.
 +
 +
  
  
 
'''On both ECgate and HPCF you can access 2 different archives.'''
 
'''On both ECgate and HPCF you can access 2 different archives.'''
  
−
''1) MARS which contains the meteorological data.''
 
  
−
''2) ECFS (ECMWF File storage system): This is a User Archive where you can store long-term data on tape. ''
+
'''1) MARS (Meteorological Archival and Retrieval System)'''
−
  On ECFS you have two domains:
+
 
−
  - ec: Permanent domain where files are stored indefinitively.
+
which contains the meteorological forecast/analysis/re-analysis data in GRIB format, as well as observation data in BUFR format.
−
  - ectmp: temporary domain where files are stored for 90 days after which they are automatically deleted (and cannot be recovered).
+
 
 +
 
 +
'''2) ECFS (ECMWF File storage system):'''
 +
 
 +
This is a User Archive where you can store any long-term data on tape.
 +
 
 +
On ECFS you have two domains:
 +
- ec: Permanent domain where files are stored indefinitively.
 +
- ectmp: temporary domain where files are stored for 90 days after which they are automatically deleted (and cannot be recovered).
 +
 
 +
To store/access files in ECFS you must issue explicit commands which are set-up to use a UNIX like interface.
 +
 
 +
Commands are same as for unix but starting with "e":
  
−
  To store/access fiels in ECFS you must issue explicit commands which is set-up to use a UNIX like interface. Commands are same as for unix but starting with "e":
+
els, ecd, ecat, ecp, epwd, emv, echmod...
−
  els, ecd, ecat..  
 
−
  For a manual of the commands type "man els".
 
  
−
  els lists your data under ec:
+
For a manual of the commands type "man els".
−
  ls lists your data under $HOME
 
  
−
  ECFS can be accessed both from ECgate and HPCF.
+
''els'' lists your data under ec:
 +
''ls'' lists your data under $HOME (on ECgate or HPCF dependent on where you are logged in)

Latest revision as of 13:43, 12 March 2013

Back to ECMWF overview[1]


Updated by nik: 12 March 2013


ECgate

is a Unix server (soon to be changed to a Linux server) where each user has directories: 

- HOME: Small permanent files, daily backup/snapshot. Quota: 3.6 GB. To check quota: ecquota
- SCRATCH: Temporary data, removed regularly. Quota: 100 GB
- SCRATCHDIR: Files are autodeleted after end of job. Part of SCRATCH and quota.


HPCF

(High performance computing facility) is the supercomputer (c2a) where each user has directories: 

- HOME: Small permanent files, daily backup/snapshot. Quota: 320 MB. To check quota: quota
- SCRATCH: Temporary data, removed regularly. Quota: 25 TB.
- SCRATCHDIR: Autodeled after end of job. 
- PERM: Large permanent files, no backup. Quota: 26 GB. Accessible from ECgate as /c2a/perm

The two systems use the same name convension - but different file systems.



On both ECgate and HPCF you can access 2 different archives.


1) MARS (Meteorological Archival and Retrieval System)

which contains the meteorological forecast/analysis/re-analysis data in GRIB format, as well as observation data in BUFR format.


2) ECFS (ECMWF File storage system):

This is a User Archive where you can store any long-term data on tape.

On ECFS you have two domains: 

- ec: Permanent domain where files are stored indefinitively.
- ectmp: temporary domain where files are stored for 90 days after which they are automatically deleted (and cannot be recovered).

To store/access files in ECFS you must issue explicit commands which are set-up to use a UNIX like interface.

Commands are same as for unix but starting with "e": 

els, ecd, ecat, ecp, epwd, emv, echmod...

For a manual of the commands type "man els". 

els lists your data under ec:
ls lists your data under $HOME (on ECgate or HPCF dependent on where you are logged in)
Retrieved from "https://wiki.uio.no/mn/geo/geoit/index.php?title=11._File_systems_and_data_structure_at_ECMWF_-_where_to_store_and_get_data!&oldid=645"