Difference between revisions of "Storage Analysis"
From mn.fysikk.laglivlab
Dagkd@uio.no (talk | contribs) (Created page with "Procedures for storing, sharing and analysing data * All instruments store data to the local disk of the computer running the instrument. * After each experiment the data nee...") |
Dagkd@uio.no (talk | contribs) (First complete procedure) |
||
| Line 1: | Line 1: | ||
| − | Procedures for storing, sharing and analysing data | + | == Procedures for storing, sharing and analysing data == |
| − | |||
* All instruments store data to the local disk of the computer running the instrument. | * All instruments store data to the local disk of the computer running the instrument. | ||
| − | * After each experiment the data needs to be transferred to a secure storage space. Our secure storage space is on fys-server1.uio.no, more specifically on | + | * After each experiment the data needs to be transferred to a secure storage space. Our secure storage space is on '''fys-server1.uio.no''', more specifically on |
| − | /mn/fys-server1/kondfys and /mn/fys-server1/lab1 These disks are backuped to tape, that is, your data is safe. | + | '''/mn/fys-server1/kondfys''' and '''/mn/fys-server1/lab1''' These disks are backuped to tape, that is, your data is safe. |
* On these disks you may make your own directories for storing all your primary experimental data. | * On these disks you may make your own directories for storing all your primary experimental data. | ||
* When you want to analyse the data you start by transferring data from fys-server1 to the computer where you want to do the calculations: | * When you want to analyse the data you start by transferring data from fys-server1 to the computer where you want to do the calculations: | ||
| − | + | ** Your own laptop/desktop | |
| − | + | ** Our common computation machine located in the room of Ali and Thomas | |
| − | + | ** The Avizo server | |
| − | + | ** Some supercomputer resource | |
| + | |||
* Analysis software that you write should be put in a github repository, either github.uio.no or github.com | * Analysis software that you write should be put in a github repository, either github.uio.no or github.com | ||
| − | * Analysis will normally reduce the data by a factor 10 | + | * Analysis will normally reduce the data by a factor 10<sup>3</sup>-10<sup>9</sup> so the results may be stored on the github repository |
| − | More about transferring files to and from fys-server1.uio.no | + | == More about transferring files to and from fys-server1.uio.no == |
* You cannot login on this server. | * You cannot login on this server. | ||
| − | * If you want to login to the server to check out files, change directory structure… you have to go through login.uio.no. On linux and terminal on mac this is: | + | * If you want to login to the server to check out files, change directory structure… you have to go through login.uio.no. |
| − | ssh login.uio.no | + | ** On linux and terminal on mac this is: ssh login.uio.no |
| − | use your UiO user name and password. | + | ** use your UiO user name and password. |
| + | |||
* You may want to set up a ssh key (google or ask Thomas) | * You may want to set up a ssh key (google or ask Thomas) | ||
| − | * There are other file management tools you may use like Cyberduck (mac) | + | * There are other file management tools you may use like '''Cyberduck''' (mac) |
* Cyberduck: Set up new connection> Choose SFTP > server fys-server1.uio.no … Connect. Navigate to our storage space /mn/fys-server1/kondfys or /mn/fys-server1/lab1 | * Cyberduck: Set up new connection> Choose SFTP > server fys-server1.uio.no … Connect. Navigate to our storage space /mn/fys-server1/kondfys or /mn/fys-server1/lab1 | ||
| − | * For Windows: use Winscp: https://www.mn.uio.no/ifi/tjenester/it/hjelp/startpakke/Programmer/Win/SSH/winscp-gml.html | + | * For Windows: use '''Winscp''': https://www.mn.uio.no/ifi/tjenester/it/hjelp/startpakke/Programmer/Win/SSH/winscp-gml.html |
| − | * linux or mac terminal you may use scp and rsync directly to transfer. scp copies everything, rsync only | + | * linux or mac terminal you may use '''scp''' and '''rsync''' directly to transfer. scp copies everything, rsync only copies what does not exist or older at the destination. Here are two ways how I could transfer all data from all the directories under 2018 to the current directory: |
| − | + | ** rsync -auv dagkd@login.uio.no:/mn/fys-server1/kondfys/Dag/Bildedisk/2018/ . | |
| − | + | ** scp -r dagkd@login.uio.no:/mn/fys-server1/kondfys/Dag/Bildedisk/2018/ . | |
| − | + | ** Here I included my username because my UiO user name is different from the user name on my computer. (this is not necessary if you have a ssh key) | |
| − | Some of us also have access to server hotels. Here is an example of copying all data from the Ludi directory on the ITOM storage hotel: | + | * Some of us also have access to server hotels. Here is an example of copying all data from the Ludi directory on the ITOM storage hotel: |
| − | scp -r login.uio.no:/net/hypatia/uio/fs01/lh-med-imb-itom/Ludi/* . | + | ** scp -r login.uio.no:/net/hypatia/uio/fs01/lh-med-imb-itom/Ludi/* . |
| − | to navigate there on login.uio.no: cd /net/hypatia/uio/fs01/lh-med-imb-itom/Ludi/ | + | ** to navigate there on login.uio.no: |
| + | *** cd /net/hypatia/uio/fs01/lh-med-imb-itom/Ludi/ | ||
Revision as of 16:00, 29 September 2023
Procedures for storing, sharing and analysing data
- All instruments store data to the local disk of the computer running the instrument.
- After each experiment the data needs to be transferred to a secure storage space. Our secure storage space is on fys-server1.uio.no, more specifically on
/mn/fys-server1/kondfys and /mn/fys-server1/lab1 These disks are backuped to tape, that is, your data is safe.
- On these disks you may make your own directories for storing all your primary experimental data.
- When you want to analyse the data you start by transferring data from fys-server1 to the computer where you want to do the calculations:
- Your own laptop/desktop
- Our common computation machine located in the room of Ali and Thomas
- The Avizo server
- Some supercomputer resource
- Analysis software that you write should be put in a github repository, either github.uio.no or github.com
- Analysis will normally reduce the data by a factor 103-109 so the results may be stored on the github repository
More about transferring files to and from fys-server1.uio.no
- You cannot login on this server.
- If you want to login to the server to check out files, change directory structure… you have to go through login.uio.no.
- On linux and terminal on mac this is: ssh login.uio.no
- use your UiO user name and password.
- You may want to set up a ssh key (google or ask Thomas)
- There are other file management tools you may use like Cyberduck (mac)
- Cyberduck: Set up new connection> Choose SFTP > server fys-server1.uio.no … Connect. Navigate to our storage space /mn/fys-server1/kondfys or /mn/fys-server1/lab1
- For Windows: use Winscp: https://www.mn.uio.no/ifi/tjenester/it/hjelp/startpakke/Programmer/Win/SSH/winscp-gml.html
- linux or mac terminal you may use scp and rsync directly to transfer. scp copies everything, rsync only copies what does not exist or older at the destination. Here are two ways how I could transfer all data from all the directories under 2018 to the current directory:
- rsync -auv dagkd@login.uio.no:/mn/fys-server1/kondfys/Dag/Bildedisk/2018/ .
- scp -r dagkd@login.uio.no:/mn/fys-server1/kondfys/Dag/Bildedisk/2018/ .
- Here I included my username because my UiO user name is different from the user name on my computer. (this is not necessary if you have a ssh key)
- Some of us also have access to server hotels. Here is an example of copying all data from the Ludi directory on the ITOM storage hotel:
- scp -r login.uio.no:/net/hypatia/uio/fs01/lh-med-imb-itom/Ludi/* .
- to navigate there on login.uio.no:
- cd /net/hypatia/uio/fs01/lh-med-imb-itom/Ludi/
