Data storage on GEO servers (lagringshotell)

To get information on how to access data storage areas (lagringshotell), from a variety of machines and locations, refer to the end user documentation for the lagringshotell for Department of Geosciences. It can be found [1](here).

Storage for researchers

Researchers at University of Oslo may be granted up to 1TB of additional storage space to accommodate any extra requirements they might have when performing their research activities. For more information follow this link.

NorStore: the Norwegian Storage Infrastructure

The resources that are available for researchers in the national high performance computing facilities are usually accompanied with limited resources and services for storing larger data collections, sharing and staging of data collections and long-term preservation of data.

NorStore offers a scalable, network accessible storage infrastructure. This is accomplished by providing resources and services for general data storage that allow users to store and access seamlessly data using a hierarchy of storage media. NorStore provides independent storage resources and services for data collections that are not easily managed or hosted on other existing systems or by research groups themselves. Storage space is available as a separate independent data allocation in the national e-Infrastructure and is intended for short- and long-term storage, and staging and sharing of data collections in databases or other specific services on disk or tape.

For more information, visit norStore website.

How to access my project area from my local machine?

From a Windows laptop or stationary Windows machine:

You can install sshfs yourself (or ask for help if needed). You can get sshfs for windows at  https://github.com/dimov-cz/win-sshfs/releases

A nice documentation how to set up win-sshfs can be found at [1] (instructions for windows 7 too). Create a new entry for your NorStore data storage (host: login.norstore.uio.no) and login with your NorStore credentials. You can configure win-sshfs to automically mount your NorStore disk when you logon to your Windows pc. Your data is now available like any other disk you have access to from Windows and can be used in any Windows application. The I/O speed from and to the NorStore disk is good although not comparable with your local C-drive, but sufficient for most purposes. The advantage is large amount of secure storing with backup.

From a Mac or Linux laptop or stationary Mac/linux machine:

You can use sshfs. Please contact us (drift@geo.uio.no)  if it does not work or if sshfs is not avaliable on your machine.

First you need to create a directory you will use to mount your NorStore data to, e.g.

[heereman@sverdrup ~]$ mkdir /var/sshfs/heereman

(Do use a mountpoint on the local filesystem, ie. not on your NFS home directory! The directory /var/sshfs is available on sverdrup for this purpose. On other machines, creating a directory under /var/tmp/ is suggested.)

You can now mount your NorStore project using the following command (use your own username!!):

[heereman@sverdrup ~]$ sshfs heereman@login.norstore.uio.no:/norstore_osl/ /var/sshfs/heereman

Fill in your NorStore password.

Now cd to /var/sshfs/heereman

An ls -l command gives the following result:

[heereman@sverdrup heereman]$ ls -l
total 76
drwxr-xr-x. 1 root root  2048 Oct 13  2014 diverse
drwxr-xr-x. 1 root root 67584 Oct 29 13:29 home
dr-xr-xr-x. 1 root root     0 Oct 27 08:05 projects
lrwxrwxrwx. 1 root root    18 Apr 25  2013 software -> /diverse/software/

The directory 'home' stores your NorStore home-directory, e.g.:

[heereman@sverdrup heereman]$ cd home/heereman
[heereman@sverdrup heereman]$ ls -l
total 28637268
-rw-------. 1 200611 norstore 28276491960 Jun 26 15:23 HEEREMANS.9390559889.SEGY.00001.ST09M03-FULL-OFFSET.3D.MIG_FIN
-rw-r--r--. 1 200611 norstore    82259088 Jun  3  2014 NBR10-327239-F_Amplitudes.sgy
-rw-r--r--. 1 200611 norstore    21998832 Jun 29 13:53 NBR10-327239-F.zgy.segy
-rw-r--r--. 1 200611 norstore   797966336 Jun 29 13:19 ST09M03-FULL-OFFSET.zgy
-rw-r--r--. 1 200611 norstore     6831288 Apr 20  2012 ST8624-105.SEGY
-rw-r--r--. 1 200611 norstore     9568656 Apr 20  2012 ST8624-107.SEGY
....
....
etc.

The projects directory contains the project-area you have been assigned, e.g.:

[heereman@sverdrup norstore]$ cd projects/NS1000K
[heereman@sverdrup NS1000K]$ ls -l
total 32724
drwxrwsr-x. 1 root     211000     2048 Apr 10  2015 ARCHIVE_STAGEDIR
drwxr-sr-x. 1 200447   211000     6144 Oct  5 13:43 ECMWF
drwxr-xr-x. 1 200447   211000     8192 Sep 23  2014 geog
drwxr-sr-x. 1 200447   211000     2048 Aug 31 13:23 ICESat
....
....
etc

Since your disk is now mounted, you can access the data directly in any kind of script using the correct path, in my case for my NorStore home-directory:

/var/sshfs/heereman/norstore/home/heereman

but this depend ofcourse, at what mountpoint you have mounted the disk.

When finished, disconnect the sshfs drive:

fusermount -u /var/sshfs/heereman

or

umount /var/sshfs/heereman

Can I process data stored in my project area directly from norStore?

It is sometimes very convenient (and fast!) to process your data directly on norStore. For this you can login on a norStore machine called:

cruncher.norstore.uio.no

To get the list of available software on cruncher:

module avail

or you can get the up to date list from [1]

You can contact us (drift@geo.uio.no) or directly norstore support (support@norstore.no) if you wish to have a new software/library installed on cruncher.