Difference between revisions of "Storage Analysis"

Revision as of 14:22, 15 November 2023

Procedures for storing, sharing and analysing data

• All instruments store data to the local disk of the computer running the instrument.
• After each experiment the data needs to be transferred to a secure storage space. Our secure storage space is on fys-server1.uio.no, more specifically on

/mn/fys-server1/kondfys and /mn/fys-server1/lab1 These disks are backuped to tape, that is, your data is safe.

• On these disks you may make your own directories for storing all your primary experimental data.
• When you want to analyse the data you start by transferring data from fys-server1 to the computer where you want to do the calculations:
  • Your own laptop/desktop
  • Our common computation machine located in the room of Ali and Thomas (fys-lab-flow, more information below)
  • The Avizo server
  • Some supercomputer resource

• Analysis software that you write should be put in a github repository, either github.uio.no or github.com
• Analysis will normally reduce the data by a factor 10³-10⁹ so the results may be stored on the github repository

More about the fys-lab-flow machine

• This machine is located in Ali's and Thomas' office and can be used for jobs that need moderate/high CPU RAM resources. Franzi and Thomas are administrating this machine.
• New users should ask the administrators for having an account.
• It can be accessed physically or via ssh:
  • if you are on the 802.1X network, you can connect directly to it using ssh username@IP, where the username is your username on ```this machine```.

More about transferring files to and from fys-server1.uio.no

• You cannot login on this server.
• If you want to login to the server to check out files, change directory structure… you have to go through login.uio.no.
  • On linux and terminal on mac this is: ssh login.uio.no
  • use your UiO user name and password.

• You may want to set up a ssh key (google or ask Thomas)
• There are other file management tools you may use like Cyberduck (mac)
• Cyberduck: Set up new connection> Choose SFTP > server fys-server1.uio.no … Connect. Navigate to our storage space /mn/fys-server1/kondfys or /mn/fys-server1/lab1
• For Windows: use Winscp: https://www.mn.uio.no/ifi/tjenester/it/hjelp/startpakke/Programmer/Win/SSH/winscp-gml.html
• linux or mac terminal you may use scp and rsync directly to transfer. scp copies everything, rsync only copies what does not exist or older at the destination. Here are two ways how I could transfer all data from all the directories under 2018 to the current directory:
  • rsync -auv dagkd@login.uio.no:/mn/fys-server1/kondfys/Dag/Bildedisk/2018/ .
  • scp -r dagkd@login.uio.no:/mn/fys-server1/kondfys/Dag/Bildedisk/2018/ .
  • Here I included my username because my UiO user name is different from the user name on my computer. (this is not necessary if you have a ssh key)

• Some of us also have access to server hotels. Here is an example of copying all data from the Ludi directory on the ITOM storage hotel:
  • scp -r login.uio.no:/net/hypatia/uio/fs01/lh-med-imb-itom/Ludi/* .
  • to navigate there on login.uio.no:
    • cd /net/hypatia/uio/fs01/lh-med-imb-itom/Ludi/

• sshfs allows you to mount a remote drive than you can access from ssh on your local machine, a bit like what Dropbox is doing. The good thing is that it allows to performs computation on remote data without actively copying them. Of course, for codes that spend a lot of time accessing data this is not practical as it slows the transfers dramatically.