Storage Analysis

Procedures for storing, sharing and analysing data

All instruments store data to the local disk of the computer running the instrument.
After each experiment the data needs to be transferred to a secure storage space. Our secure storage space is on fys-server1.uio.no, more specifically on

/mn/fys-server1/kondfys and /mn/fys-server1/lab1 These disks are backuped to tape, that is, your data is safe.

On these disks you may make your own directories for storing all your primary experimental data.
When you want to analyse the data you start by transferring data from fys-server1 to the computer where you want to do the calculations:
- Your own laptop/desktop
- Our common computation machine located in the room of Ali and Thomas (fys-lab-flow, more information below)
- The Avizo server
- Some supercomputer resource

Analysis software that you write should be put in a github repository, either github.uio.no or github.com
Analysis will normally reduce the data by a factor 10³-10⁹ so the results may be stored on the github repository

This machine is located in Ali's and Thomas' office and can be used for jobs that need moderate/high CPU RAM resources. Franzi and Thomas are administrating this machine.
New users should ask the administrators for having an account.
By default, new users do not get administrator rights. You should ask an administrator to install new whole system packages. Note that, this does not concern you if you want to install python packages via pip as they are installed on your local account.
It can be accessed physically or via ssh:
- if you are on the 802.1X network, you can connect directly to it using ssh username@IP, where the username is your username on this machine. IP is not displayed on this website, you can find it on the machine or ask an administrator.
- if you are using an UiO managed computer or are connecting from another network, you need install a reverse ssh bridge. Ask Thomas for that but before make sure you have an ssh key set between fys-lab-flow and login and one set between your computer and login.
- Do NOT turn off this machine without asking the administrators first

You cannot login on this server.
If you want to login to the server to check out files, change directory structure… you have to go through login.uio.no.
- On linux and terminal on mac this is: ssh login.uio.no
- use your UiO user name and password.

You may want to set up a ssh key (google or ask Thomas or here):
There are other file management tools you may use like Cyberduck (mac)
Cyberduck: Set up new connection> Choose SFTP > server fys-server1.uio.no … Connect. Navigate to our storage space /mn/fys-server1/kondfys or /mn/fys-server1/lab1
For Windows: use Winscp: https://www.mn.uio.no/ifi/tjenester/it/hjelp/startpakke/Programmer/Win/SSH/winscp-gml.html
linux or mac terminal you may use scp and rsync directly to transfer. scp copies everything, rsync only copies what does not exist or older at the destination. Here are two ways how I could transfer all data from all the directories under 2018 to the current directory:
- rsync -auv dagkd@login.uio.no:/mn/fys-server1/kondfys/Dag/Bildedisk/2018/ .
- scp -r dagkd@login.uio.no:/mn/fys-server1/kondfys/Dag/Bildedisk/2018/ .
- Here I included my username because my UiO user name is different from the user name on my computer. (this is not necessary if you have a ssh key)

Some of us also have access to server hotels. Here is an example of copying all data from the Ludi directory on the ITOM storage hotel:
- scp -r login.uio.no:/net/hypatia/uio/fs01/lh-med-imb-itom/Ludi/* .
- to navigate there on login.uio.no:
  - cd /net/hypatia/uio/fs01/lh-med-imb-itom/Ludi/

sshfs allows you to mount a remote drive than you can access from ssh on your local machine, a bit like what Dropbox is doing. The good thing is that it allows to performs computation on remote data without actively copying them. Of course, for codes that spend a lot of time accessing data this is not practical as it slows the transfers dramatically.