Difference between revisions of "Prometheus.uio.no"
(→Preparing the host) |
(→How to force an upgrade of cryosparc) |
||
(18 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
− | prometheus.uio.no is the front-end node for the NCMM cryoSPARC platform. | + | <i>prometheus.uio.no</i> is the front-end node for the NCMM cryoSPARC platform. [https://cryosparc.com/ cryoSPARC] is a platform used for obtaining 3D structural information from single-particle cryoEM data |
+ | This wiki page describes how to configure the host and install the cryosparc software. | ||
− | + | NOTE: After completing the above, navigate your browser to http://<workstation_hostname>:39000 to access the cryoSPARC user interface. | |
+ | |||
+ | == Apply for licenses == | ||
+ | |||
+ | Proceed to https://cryosparc.com/download/ and apply for a license or two. You should get a reply to your request within 24hrs. Please mind you that Structura Biotechnology Inc. is in Toronto, Canada, which means EST timezone ( GMT - 5), so adjust your expectations for a reply, accordingly. | ||
== Preparing the host == | == Preparing the host == | ||
Line 88: | Line 93: | ||
mount -a | mount -a | ||
df -h | df -h | ||
+ | </pre> | ||
+ | |||
+ | === Misc settings for comfortable editing === | ||
+ | |||
+ | <pre> | ||
+ | vi ~/.vimrc | ||
+ | |||
+ | 1 syntax enable | ||
+ | 2 set tabstop=4 | ||
+ | 3 set softtabstop=4 | ||
+ | 4 set expandtab | ||
+ | 5 set number | ||
+ | 6 set showcmd " show command in bottom bar | ||
+ | 7 set cursorline " highlight current line | ||
+ | 8 filetype indent on " load filetype-specific indent files | ||
+ | 9 set wildmenu | ||
+ | 10 set showmatch | ||
+ | 11 set incsearch " search as characters are entered | ||
+ | 12 set hlsearch " highlight matches | ||
+ | 13 nnoremap <leader><space> :nohlsearch<CR> | ||
+ | 14 " set foldenable " enable folding | ||
+ | 15 " set foldlevelstart=10 " open most folds by default | ||
+ | 16 " set foldnestmax=10 " 10 nested fold max | ||
+ | 17 " space open/closes folds | ||
+ | 18 nnoremap <space> za | ||
+ | 19 " set foldmethod=indent " fold based on indent level | ||
+ | |||
+ | vi ~/.bashrc | ||
+ | alias ls='LC_ALL=C ls --color=auto --literal --human-readable --group-directories-first --classify' | ||
+ | </pre> | ||
+ | |||
+ | === Configuring collectd === | ||
+ | |||
+ | ==== General configuration ==== | ||
+ | |||
+ | <pre> | ||
+ | systemctl stop collectd | ||
+ | systemctl enable collectd | ||
+ | ssh prometheus.uio.no cp /etc/collectd.conf /etc/collectd.conf.orig | ||
+ | scp ~georgmar/ncmm/files/etc/collectd.conf prometheus:/etc/collectd.conf | ||
+ | vi /etc/collectd.conf | ||
+ | :s/TEMPLATEHOST\.uio\.no/prometheus.uio.no/g | ||
+ | :s/TEMPLATEINTHOST\.internal\.biotek/intprometheus.internal.biotek/g | ||
+ | :wq | ||
+ | /usr/sbin/collectd -T | ||
+ | systemctl start collectd | ||
+ | </pre> | ||
+ | |||
+ | ==== Set up firewall on panoptis.uio.no ==== | ||
+ | |||
+ | <pre> | ||
+ | ssh root@panoptis -- "firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.144/32" port port=2003 protocol="tcp" accept' && firewall-cmd --reload" | ||
+ | ssh root@panoptis -- "firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.144/32" port port=2004 protocol="tcp" accept' && firewall-cmd --reload" | ||
+ | </pre> | ||
+ | |||
+ | |||
+ | === Setup firewall on prometheus.uio.no === | ||
+ | |||
+ | After looking at the cryosparc logs, it looks like cryosparc is using more ports than documented to communicate. so, enable more firewall ports. | ||
+ | |||
+ | On the public interface we specify dias.uio.no ( 129.240.235.130 ) for the source address, as intaristotle.internal.biotek uses dias for the NAT: | ||
+ | |||
+ | <pre> | ||
+ | #public interface | ||
+ | firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="129.240.235.130/32" port port=3900 protocol="tcp" accept' && firewall-cmd --reload | ||
+ | firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="129.240.235.130/32" port port=3901 protocol="tcp" accept' && firewall-cmd --reload | ||
+ | firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="129.240.235.130/32" port port=3902 protocol="tcp" accept' && firewall-cmd --reload | ||
+ | firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="129.240.235.130/32" port port=3903 protocol="tcp" accept' && firewall-cmd --reload | ||
+ | firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="129.240.235.130/32" port port=3904 protocol="tcp" accept' && firewall-cmd --reload | ||
+ | </pre> | ||
+ | |||
+ | On the private interface we specify the address of intaristotle.internal.biotek ( 192.168.8.109 ): | ||
+ | <pre> | ||
+ | # internal interface | ||
+ | firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.109/32" port port=3900 protocol="tcp" accept' && firewall-cmd --reload | ||
+ | firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.109/32" port port=3901 protocol="tcp" accept' && firewall-cmd --reload | ||
+ | firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.109/32" port port=3902 protocol="tcp" accept' && firewall-cmd --reload | ||
+ | firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.109/32" port port=3903 protocol="tcp" accept' && firewall-cmd --reload | ||
+ | firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.109/32" port port=3904 protocol="tcp" accept' && firewall-cmd --reload | ||
+ | </pre> | ||
+ | |||
+ | The cryosparc installer, as we will see, defines only the base port for the web application. [http://supervisord.org/ supervisord] and [https://www.mongodb.com/ mongodb] run at base port +1 and +2, respectively. | ||
+ | |||
+ | We specified that the base port is 3900, so we have to open ports 3900, 3901, 3902, 3903,for the web app, mongodb, supervisord and wsgi, respectively. | ||
+ | |||
+ | |||
+ | === Setting up NFS on the master node === | ||
+ | |||
+ | <pre> | ||
+ | [root@pythagoras ~]# vi /etc/exports.d/intprometheus.internal.biotek.exports | ||
+ | /div/pythagoras/u6 192.168.8.144(rw,sync) | ||
+ | /div/pythagoras/u4 192.168.8.144(rw,sync) | ||
+ | /div/pythagoras/u1 192.168.8.144(rw,sync) | ||
+ | /div/pythagoras/u2 192.168.8.144(rw,sync) | ||
+ | |||
+ | [root@pythagoras ~]# exportfs -a | ||
+ | |||
+ | [root@perun ~]# vi /etc/exports | ||
+ | /storage/hudelarea 192.168.15.2(rw,sync) 129.240.189.230(rw,sync) 129.240.237.151(rw,sync) 192.168.8.38(rw,sync) 192.168.15.4(rw,sync,no_root_squash) 192.168.8.144(rw,sync) | ||
+ | /storage/hudelarea2 192.168.15.2(rw,sync) 129.240.189.230(rw,sync) 192.168.8.38(rw,sync) 129.240.237.151(rw,sync) 192.168.8.144(rw,sync) | ||
+ | |||
+ | [root@pythagoras ~]# exportfs -a | ||
+ | |||
+ | </pre> | ||
+ | |||
+ | ==== Automatically mounting all the partitions needed ==== | ||
+ | |||
+ | <pre> | ||
+ | [root@prometheus ~]# vi /etc/fstab | ||
+ | |||
+ | # | ||
+ | # /etc/fstab | ||
+ | # Created by anaconda on Mon Jul 8 11:28:47 2019 | ||
+ | # | ||
+ | # Accessible filesystems, by reference, are maintained under '/dev/disk' | ||
+ | # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info | ||
+ | # | ||
+ | /dev/mapper/internvg-root / xfs defaults 0 0 | ||
+ | UUID=c12a7820-0b28-408e-8cea-0f3ebaeb864a /boot xfs defaults 0 0 | ||
+ | /dev/mapper/internvg-opt /opt xfs defaults 0 0 | ||
+ | /dev/mapper/internvg-tmp /tmp xfs defaults 0 0 | ||
+ | /dev/mapper/internvg-usr /usr xfs defaults 0 0 | ||
+ | /dev/mapper/internvg-var /var xfs defaults 0 0 | ||
+ | /dev/mapper/internvg-swap swap swap defaults 0 0 | ||
+ | |||
+ | #RT 3485324 | ||
+ | UUID=450b41db-e9a4-49df-beb5-d4ccef7461a3 /lsc xfs defaults 0 0 | ||
+ | 192.168.8.132:/div/pythagoras/u6 /div/pythagoras/u6 nfs rw,nodev,nosuid,intr,nfsvers=3 | ||
+ | 192.168.8.132:/div/pythagoras/u4 /div/pythagoras/u4 nfs rw,nodev,nosuid,intr,nfsvers=3 | ||
+ | 192.168.8.132:/div/pythagoras/u1 /div/pythagoras/u1 nfs rw,nodev,nosuid,intr,nfsvers=3 | ||
+ | 192.168.8.132:/div/pythagoras/u2 /div/pythagoras/u2 nfs rw,nodev,nosuid,intr,nfsvers=3 | ||
+ | |||
+ | 192.168.8.22:/storage/hudelarea /storage/hudelarea nfs rw,nodev,nosuid,intr,nfsvers=3 | ||
+ | 192.168.8.22:/storage/hudelarea2 /storage/hudelarea2 nfs rw,nodev,nosuid,intr,nfsvers=3 | ||
+ | |||
+ | [root@prometheus ~]# mount -a | ||
+ | |||
+ | </pre> | ||
+ | |||
+ | == Installing cryoSPARC == | ||
+ | |||
+ | === Set up cryoSPARC master node === | ||
+ | |||
+ | Full instructions here: https://cryosparc.com/docs/reference/install/ | ||
+ | |||
+ | |||
+ | <pre> | ||
+ | [root@intaristotle ~]# /usr/sbin/useradd --home-dir /storage/hudelarea2 -g ncmm-luecke-fg --no-create-home --shell /bin/bash --uid 1010 cryosparc_user | ||
+ | [root@prometheus ~]# /usr/sbin/useradd --home-dir /storage/hudelarea2 -g ncmm-luecke-fg --no-create-home --shell /bin/bash --uid 1010 cryosparc_user | ||
+ | [root@prometheus ~]# mkdir /storage/hudelarea2/.ssh | ||
+ | [root@prometheus ~]# chmod 700 /storage/hudelarea2/.ssh | ||
+ | [root@prometheus ~]# chown cryosparc_user:ncmm-luecke-fg /storage/hudelarea2/.ssh/ | ||
+ | [root@prometheus ~]# su - cryosparc_user | ||
+ | </pre> | ||
+ | |||
+ | === Setting up the password-less ssh auth === | ||
+ | Make sure that there is no password on the ssh key, cryoSPARC does not interact with ssh to ask you, the user, for a password. | ||
+ | <pre> | ||
+ | [cryosparc_user@prometheus ~ ]$ ssh-keygen -t rsa | ||
+ | [cryosparc_user@prometheus ~ ]$ mkdir .ssh/sockets | ||
+ | [cryosparc_user@prometheus ~ ]$ chmod 700 .ssh/sockets | ||
+ | [cryosparc_user@prometheus ~ ]$ cat .ssh/id_rsa.pub > .ssh/authorized_keys && chmod 600 .ssh/authorized_keys | ||
+ | </pre> | ||
+ | |||
+ | === Setting up the master node === | ||
+ | <pre> | ||
+ | [root@prometheus ~]# /usr/bin/mkdir -p /lsc/sources/cryosparc/2.8/1048c1bc-99d6-11e9-9456-bb2320f16355/ | ||
+ | [root@prometheus ~]# /usr/bin/cd /lsc/sources/cryosparc/2.8/1048c1bc-99d6-11e9-9456-bb2320f16355/ | ||
+ | [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# cat > $PWD/env.sources | ||
+ | export LICENSE_ID="1048c1bc-99d6-11e9-9456-bb2320f16355" | ||
+ | [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# source env.sources | ||
+ | [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# curl -L https://get.cryosparc.com/download/master-latest/$LICENSE_ID [Open URL] > cryosparc2_master.tar.gz | ||
+ | [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# for file in *.tar.gz; do tar zxvf $file; done | ||
+ | [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# cd cryosparc2_master/ | ||
+ | [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_MASTER_HOSTNAME=$(/usr/bin/hostname -f) | ||
+ | [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_USER=cryosparc_user | ||
+ | [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_VERSION=2.9 | ||
+ | [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_USER_HOME=/lsc/cryosparc/$CRYOSPARC_VERSION/$LICENSE_ID | ||
+ | [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_INSTALL_PATH=$CRYOSPARC_USER_HOME | ||
+ | [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_DB_PATH=$CRYOSPARC_INSTALL_PATH/cryosparc2_database | ||
+ | [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_MASTER_PORT_NUMBER=39000 | ||
+ | [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_USER_EMAIL=george.marselis@ncmm.uio.no | ||
+ | [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_USER_PASSWORD=###REDACTED### | ||
+ | [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# cd /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master | ||
+ | [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# ./install.sh --license $LICENSE_ID --hostname $CRYOSPARC_MASTER_HOSTNAME --dbpath $CRYOSPARC_DB_PATH --port $CRYOSPARC_MASTER_PORT_NUMBER | ||
+ | [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# cat > ~/.bashrc | ||
+ | export CRYOSPARC_MASTER_HOSTNAME=$(/usr/bin/hostname -f) | ||
+ | export CRYOSPARC_USER=cryosparc_user | ||
+ | export CRYOSPARC_VERSION=2.9 | ||
+ | export CRYOSPARC_USER_HOME=/lsc/cryosparc/$CRYOSPARC_VERSION/$LICENSE_ID | ||
+ | export CRYOSPARC_INSTALL_PATH=$CRYOSPARC_USER_HOME | ||
+ | export CRYOSPARC_DB_PATH=$CRYOSPARC_INSTALL_PATH/cryosparc2_database | ||
+ | export CRYOSPARC_MASTER_PORT_NUMBER=80 | ||
+ | [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# source ~/.bashrc | ||
+ | [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# mkdir run && touch run/command_core.log && touch run/database.log && touch run/webapp.log | ||
+ | </pre> | ||
+ | |||
+ | === Setup systemd service for cryosparc master === | ||
+ | |||
+ | We create a hand-made systemd service unit file, since cryosparc does not provide one. | ||
+ | |||
+ | <pre> | ||
+ | cat > /etc/systemd/system/cryosparc.service | ||
+ | |||
+ | [Unit] | ||
+ | Description = cryoSPARC | ||
+ | After = network.target | ||
+ | |||
+ | [Service] | ||
+ | Type = forking | ||
+ | RestartSec = 10 | ||
+ | User = cryosparc_user | ||
+ | ExecStart = /bin/bash /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/bin/cryosparcm start | ||
+ | ExecStop = /bin/bash /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/bin/cryosparcm stop | ||
+ | PIDFile = /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/run/supervisord.pid | ||
+ | #StartLimitIntervalSec = 10 | ||
+ | |||
+ | [Install] | ||
+ | WantedBy=multi-user.target | ||
+ | |||
+ | </pre> | ||
+ | |||
+ | == Set up cryosparc worker == | ||
+ | |||
+ | === Prepare the host === | ||
+ | <pre> | ||
+ | [root@intaristotle ~]# /usr/bin/mkdir -p /lsc/sources/cryosparc/2.8/1048c1bc-99d6-11e9-9456-bb2320f16355/ | ||
+ | [root@intaristotle ~]# /usr/bin/cd /lsc/sources/cryosparc/2.8/1048c1bc-99d6-11e9-9456-bb2320f16355/ | ||
+ | [root@intaristotle 1048c1bc-99d6-11e9-9456-bb2320f16355]# cat > $PWD/env.sources | ||
+ | export LICENSE_ID="1048c1bc-99d6-11e9-9456-bb2320f16355" | ||
+ | [root@intaristotle 1048c1bc-99d6-11e9-9456-bb2320f16355]# source env.sources | ||
+ | [root@intaristotle 1048c1bc-99d6-11e9-9456-bb2320f16355]# curl -L https://get.cryosparc.com/download/worker-latest/$LICENSE_ID [Open URL] > cryosparc2_worker.tar.gz | ||
+ | [root@intaristotle 1048c1bc-99d6-11e9-9456-bb2320f16355]# su - cryosparc_user | ||
+ | [cryosparc_user@intaristotle cryosparc2_worker]$ export MASTER_HOSTNAME=prometheus.uio.no | ||
+ | [cryosparc_user@intaristotle cryosparc2_worker]$ export PORT_NUM=3900 | ||
+ | [cryosparc_user@intaristotle cryosparc2_worker]$ export WORKER_HOSTNAME=intaristotle.internal.biotek | ||
+ | [cryosparc_user@intaristotle cryosparc2_worker]$ export CUDA_PATH=/lsc/nvidia/cuda/8.0-GA1/ | ||
+ | </pre> | ||
+ | |||
+ | |||
+ | === Connect the cryosparc worker software === | ||
+ | <pre> | ||
+ | |||
+ | [root@intaristotle cryosparc2_worker]# usermod -a -G vglusers cryosparc_user # allow cryosparc_user to run /usr/bin/nvidia-smi to enumerate the graphics cards | ||
+ | [root@intaristotle cryosparc2_worker]# bin/cryosparcw connect --worker intaristotle.internal.biotek --master intprometheus.internal.biotek --port 3900 --ssdpath /storage/scratch --sshstr cryosparc_user@intaristotle.internal.biotek | ||
+ | --------------------------------------------------------------- | ||
+ | CRYOSPARC CONNECT -------------------------------------------- | ||
+ | --------------------------------------------------------------- | ||
+ | Attempting to register worker intaristotle.internal.biotek to command intprometheus.internal.biotek:3902 | ||
+ | Connecting as unix user root | ||
+ | Will register using ssh string: cryosparc_user@intaristotle.internal.biotek | ||
+ | If this is incorrect, you should re-run this command with the flag --sshstr <ssh string> | ||
+ | --------------------------------------------------------------- | ||
+ | Connected to master. | ||
+ | --------------------------------------------------------------- | ||
+ | Current connected workers: | ||
+ | --------------------------------------------------------------- | ||
+ | Autodetecting available GPUs... | ||
+ | Detected 4 CUDA devices. | ||
+ | |||
+ | id pci-bus name | ||
+ | --------------------------------------------------------------- | ||
+ | 0 0000:02:00.0 GeForce GTX 1080 Ti | ||
+ | 1 0000:03:00.0 GeForce GTX 1080 Ti | ||
+ | 2 0000:83:00.0 GeForce GTX 1080 Ti | ||
+ | 3 0000:84:00.0 GeForce GTX 1080 Ti | ||
+ | --------------------------------------------------------------- | ||
+ | All devices will be enabled now. | ||
+ | This can be changed later using --update | ||
+ | --------------------------------------------------------------- | ||
+ | Worker will be registered with SSD cache location /storage/scratch | ||
+ | --------------------------------------------------------------- | ||
+ | Autodetecting the amount of RAM available... | ||
+ | This machine has 128.64GB RAM . | ||
+ | --------------------------------------------------------------- | ||
+ | --------------------------------------------------------------- | ||
+ | Registering worker... | ||
+ | Done. | ||
+ | |||
+ | You can now launch jobs on the master node and they will be scheduled | ||
+ | on to this worker node if resource requirements are met. | ||
+ | --------------------------------------------------------------- | ||
+ | Final configuration for intaristotle.internal.biotek | ||
+ | lane : default | ||
+ | name : intaristotle.internal.biotek | ||
+ | title : Worker node intaristotle.internal.biotek | ||
+ | resource_slots : {u'GPU': [0, 1, 2, 3], u'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], u'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]} | ||
+ | hostname : intaristotle.internal.biotek | ||
+ | worker_bin_path : /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_worker/bin/cryosparcw | ||
+ | cache_path : /storage/scratch | ||
+ | cache_quota_mb : None | ||
+ | resource_fixed : {u'SSD': True} | ||
+ | cache_reserve_mb : 10000 | ||
+ | type : node | ||
+ | ssh_str : cryosparc_user@intaristotle.internal.biotek | ||
+ | desc : None | ||
+ | --------------------------------------------------------------- | ||
+ | </pre> | ||
+ | |||
+ | Done | ||
+ | |||
+ | == Additional Info == | ||
+ | |||
+ | === How to delete a worker node === | ||
+ | |||
+ | Connect to the master node (prometheus.uio.no) and | ||
+ | |||
+ | <pre>[cryosparc_user@prometheus ~] /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/bin/cryosparcm cluster remove intaristotle.internal.biotek</pre> | ||
+ | |||
+ | |||
+ | === How to check for new updates for cryosparc === | ||
+ | |||
+ | Connect to the master node (prometheus.uio.no) and | ||
+ | |||
+ | <pre> | ||
+ | [root@prometheus ~] su - cryosparc_user | ||
+ | [cryosparc_user@prometheus ~] source /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/env.sources | ||
+ | [cryosparc_user@prometheus ~] /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/bin/cryosparcm update --check | ||
+ | </pre> | ||
+ | |||
+ | === How to list all versions of cryosparc available === | ||
+ | |||
+ | <pre> | ||
+ | [root@prometheus ~] su - cryosparc_user | ||
+ | [cryosparc_user@prometheus ~] source /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/env.sources | ||
+ | [cryosparc_user@prometheus ~] /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/bin/cryosparcm update --list | ||
+ | </pre> | ||
+ | To get a full list of available versions (including old versions in case of a downgrade). | ||
+ | |||
+ | === How to upgrade cryosparc === | ||
+ | |||
+ | <pre> | ||
+ | [root@prometheus ~] su - cryosparc_user | ||
+ | [cryosparc_user@prometheus ~] source /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/env.sources | ||
+ | [cryosparc_user@prometheus ~] /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/bin/cryosparcm update | ||
+ | </pre> | ||
+ | |||
+ | === How to downgrade cryosparc === | ||
+ | |||
+ | <pre> | ||
+ | [root@prometheus ~] su - cryosparc_user | ||
+ | [cryosparc_user@prometheus ~] source /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/env.sources | ||
+ | [cryosparc_user@prometheus ~] /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/bin/cryosparcm update --version=vX.Y.Z | ||
+ | </pre> | ||
+ | |||
+ | version vX.Y.Z is the version you want to downgrade to, which you can find from bin/cryosparcm update --list | ||
+ | |||
+ | === How to force an upgrade of cryosparc === | ||
+ | |||
+ | If, for any reason, the upgrade does not succeed, you can force the upgrade. This means that bin/cryosparcm will re-download the latest software and re-install all of it from scratch. | ||
+ | |||
+ | Specifically, the bin/cryosparcm program will stop the cryoSPARC service on the master node, will download the latest software and deploy it as needed. The data in the current database will be preserved. | ||
+ | |||
+ | The way to force the upgrade is | ||
+ | |||
+ | <pre> | ||
+ | [root@prometheus ~] su - cryosparc_user | ||
+ | [cryosparc_user@prometheus ~] source /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/env.sources | ||
+ | [cryosparc_user@prometheus ~] /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/bin/cryosparcm update --force | ||
+ | </pre> | ||
+ | |||
+ | After updating the master node, bin/cryosparcm will try to connect to each worker node, in this case intaristotle, to copy the new cryosparc2_worker.tar.gz ; If for any reason copying fails, the software will notify you to copy /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/cryosparc2_worker.tar.gz to the worker node under /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_worker/cryosparc2_worker.tar.gz | ||
+ | |||
+ | After you manually copy the tarball you can run manually | ||
+ | |||
+ | <pre> | ||
+ | [root@intaristotle ~] su - cryosparc_user | ||
+ | [cryosparc_user@intaristotle ~] source /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/env.sources | ||
+ | [cryosparc_user@intaristotle ~] /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_worker/bin/cryosparcm update --force | ||
+ | </pre> | ||
+ | |||
+ | For more details, check the [https://cryosparc.com/docs/reference/install#updating cryoSPARC manual on the update section] | ||
+ | |||
+ | === Changing the path of the software, both in the master and the worker(s) will disable the software === | ||
+ | |||
+ | As [the manual says] under "Management and Monitoring" -> "Advanced": | ||
+ | |||
+ | <pre> | ||
+ | master and worker installation directories cannot be moved to a different absolute path once installed. | ||
+ | To move the installation, dump the database, install cryoSPARC again in the new location, and restore the database. | ||
</pre> | </pre> |
Latest revision as of 15:00, 6 May 2020
prometheus.uio.no is the front-end node for the NCMM cryoSPARC platform. cryoSPARC is a platform used for obtaining 3D structural information from single-particle cryoEM data
This wiki page describes how to configure the host and install the cryosparc software.
NOTE: After completing the above, navigate your browser to http://<workstation_hostname>:39000 to access the cryoSPARC user interface.
Contents
- 1 Apply for licenses
- 2 Preparing the host
- 2.1 Configuring the virsh serial port for the guest OS
- 2.2 Installing base-line software and prerequisites
- 2.3 Allocating extra space for the /lsc partition
- 2.4 Misc settings for comfortable editing
- 2.5 Configuring collectd
- 2.6 Setup firewall on prometheus.uio.no
- 2.7 Setting up NFS on the master node
- 3 Installing cryoSPARC
- 4 Set up cryosparc worker
- 5 Additional Info
- 5.1 How to delete a worker node
- 5.2 How to check for new updates for cryosparc
- 5.3 How to list all versions of cryosparc available
- 5.4 How to upgrade cryosparc
- 5.5 How to downgrade cryosparc
- 5.6 How to force an upgrade of cryosparc
- 5.7 Changing the path of the software, both in the master and the worker(s) will disable the software
Apply for licenses
Proceed to https://cryosparc.com/download/ and apply for a license or two. You should get a reply to your request within 24hrs. Please mind you that Structura Biotechnology Inc. is in Toronto, Canada, which means EST timezone ( GMT - 5), so adjust your expectations for a reply, accordingly.
Preparing the host
from jump-biotek, after pxebooting and installing the OS:
ssh prometheus.uio.no
Configuring the virsh serial port for the guest OS
cp /etc/default/grub /etc/default/grub.orig vi /etc/default/grub insert the following lines for serial console: GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)" GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT="console serial" GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=internvg/root rd.lvm.lv=internvg/swap rd.lvm.lv=internvg/usr console=tty1 console=ttyS0,115200" GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1" GRUB_DISABLE_RECOVERY="true" cp /boot/grub2/grub.cfg /boot/grub2/grub.cfg.orig grub2-mkconfig -o /boot/grub2/grub.cfg systemctl reboot
Installing base-line software and prerequisites
yum install -y fail2ban* vim* etckeeper* git* collectd* lm_* ncdu bzip2 lsof --skip-broken
Allocating extra space for the /lsc partition
parted /dev/vdb (parted) mklabel gtp New disk label type? gpt mkpart primary xfs 1 -1 (parted) p Model: Virtio Block Device (virtblk) Disk /dev/vdb: 42.9GB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1049kB 42.9GB 42.9GB primary (parted) quit Information: You may need to update /etc/fstab. quit
Format the partition
[root@prometheus ~]# mkfs.xfs /dev/vdb1 meta-data=/dev/vdb1 isize=512 agcount=4, agsize=2621312 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=0, sparse=0 data = bsize=4096 blocks=10485248, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=5119, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 [root@prometheus ~]# blkid /dev/vdb1 /dev/vdb1: UUID="450b41db-e9a4-49df-beb5-d4ccef7461a3" TYPE="xfs" PARTLABEL="primary" PARTUUID="d189b7ce-c419-4457-9281-13b9f988caa6"
Test the partition and mount it
#add an auxiliary space under /lsc vi /etc/fstab #RT 3485324 UUID=450b41db-e9a4-49df-beb5-d4ccef7461a3 /lsc xfs defaults 0 0 mount /dev/vdb1 /lsc df -h umount /lsc mount -a df -h
Misc settings for comfortable editing
vi ~/.vimrc 1 syntax enable 2 set tabstop=4 3 set softtabstop=4 4 set expandtab 5 set number 6 set showcmd " show command in bottom bar 7 set cursorline " highlight current line 8 filetype indent on " load filetype-specific indent files 9 set wildmenu 10 set showmatch 11 set incsearch " search as characters are entered 12 set hlsearch " highlight matches 13 nnoremap <leader><space> :nohlsearch<CR> 14 " set foldenable " enable folding 15 " set foldlevelstart=10 " open most folds by default 16 " set foldnestmax=10 " 10 nested fold max 17 " space open/closes folds 18 nnoremap <space> za 19 " set foldmethod=indent " fold based on indent level vi ~/.bashrc alias ls='LC_ALL=C ls --color=auto --literal --human-readable --group-directories-first --classify'
Configuring collectd
General configuration
systemctl stop collectd systemctl enable collectd ssh prometheus.uio.no cp /etc/collectd.conf /etc/collectd.conf.orig scp ~georgmar/ncmm/files/etc/collectd.conf prometheus:/etc/collectd.conf vi /etc/collectd.conf :s/TEMPLATEHOST\.uio\.no/prometheus.uio.no/g :s/TEMPLATEINTHOST\.internal\.biotek/intprometheus.internal.biotek/g :wq /usr/sbin/collectd -T systemctl start collectd
Set up firewall on panoptis.uio.no
ssh root@panoptis -- "firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.144/32" port port=2003 protocol="tcp" accept' && firewall-cmd --reload" ssh root@panoptis -- "firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.144/32" port port=2004 protocol="tcp" accept' && firewall-cmd --reload"
Setup firewall on prometheus.uio.no
After looking at the cryosparc logs, it looks like cryosparc is using more ports than documented to communicate. so, enable more firewall ports.
On the public interface we specify dias.uio.no ( 129.240.235.130 ) for the source address, as intaristotle.internal.biotek uses dias for the NAT:
#public interface firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="129.240.235.130/32" port port=3900 protocol="tcp" accept' && firewall-cmd --reload firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="129.240.235.130/32" port port=3901 protocol="tcp" accept' && firewall-cmd --reload firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="129.240.235.130/32" port port=3902 protocol="tcp" accept' && firewall-cmd --reload firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="129.240.235.130/32" port port=3903 protocol="tcp" accept' && firewall-cmd --reload firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="129.240.235.130/32" port port=3904 protocol="tcp" accept' && firewall-cmd --reload
On the private interface we specify the address of intaristotle.internal.biotek ( 192.168.8.109 ):
# internal interface firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.109/32" port port=3900 protocol="tcp" accept' && firewall-cmd --reload firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.109/32" port port=3901 protocol="tcp" accept' && firewall-cmd --reload firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.109/32" port port=3902 protocol="tcp" accept' && firewall-cmd --reload firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.109/32" port port=3903 protocol="tcp" accept' && firewall-cmd --reload firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.109/32" port port=3904 protocol="tcp" accept' && firewall-cmd --reload
The cryosparc installer, as we will see, defines only the base port for the web application. supervisord and mongodb run at base port +1 and +2, respectively.
We specified that the base port is 3900, so we have to open ports 3900, 3901, 3902, 3903,for the web app, mongodb, supervisord and wsgi, respectively.
Setting up NFS on the master node
[root@pythagoras ~]# vi /etc/exports.d/intprometheus.internal.biotek.exports /div/pythagoras/u6 192.168.8.144(rw,sync) /div/pythagoras/u4 192.168.8.144(rw,sync) /div/pythagoras/u1 192.168.8.144(rw,sync) /div/pythagoras/u2 192.168.8.144(rw,sync) [root@pythagoras ~]# exportfs -a [root@perun ~]# vi /etc/exports /storage/hudelarea 192.168.15.2(rw,sync) 129.240.189.230(rw,sync) 129.240.237.151(rw,sync) 192.168.8.38(rw,sync) 192.168.15.4(rw,sync,no_root_squash) 192.168.8.144(rw,sync) /storage/hudelarea2 192.168.15.2(rw,sync) 129.240.189.230(rw,sync) 192.168.8.38(rw,sync) 129.240.237.151(rw,sync) 192.168.8.144(rw,sync) [root@pythagoras ~]# exportfs -a
Automatically mounting all the partitions needed
[root@prometheus ~]# vi /etc/fstab # # /etc/fstab # Created by anaconda on Mon Jul 8 11:28:47 2019 # # Accessible filesystems, by reference, are maintained under '/dev/disk' # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info # /dev/mapper/internvg-root / xfs defaults 0 0 UUID=c12a7820-0b28-408e-8cea-0f3ebaeb864a /boot xfs defaults 0 0 /dev/mapper/internvg-opt /opt xfs defaults 0 0 /dev/mapper/internvg-tmp /tmp xfs defaults 0 0 /dev/mapper/internvg-usr /usr xfs defaults 0 0 /dev/mapper/internvg-var /var xfs defaults 0 0 /dev/mapper/internvg-swap swap swap defaults 0 0 #RT 3485324 UUID=450b41db-e9a4-49df-beb5-d4ccef7461a3 /lsc xfs defaults 0 0 192.168.8.132:/div/pythagoras/u6 /div/pythagoras/u6 nfs rw,nodev,nosuid,intr,nfsvers=3 192.168.8.132:/div/pythagoras/u4 /div/pythagoras/u4 nfs rw,nodev,nosuid,intr,nfsvers=3 192.168.8.132:/div/pythagoras/u1 /div/pythagoras/u1 nfs rw,nodev,nosuid,intr,nfsvers=3 192.168.8.132:/div/pythagoras/u2 /div/pythagoras/u2 nfs rw,nodev,nosuid,intr,nfsvers=3 192.168.8.22:/storage/hudelarea /storage/hudelarea nfs rw,nodev,nosuid,intr,nfsvers=3 192.168.8.22:/storage/hudelarea2 /storage/hudelarea2 nfs rw,nodev,nosuid,intr,nfsvers=3 [root@prometheus ~]# mount -a
Installing cryoSPARC
Set up cryoSPARC master node
Full instructions here: https://cryosparc.com/docs/reference/install/
[root@intaristotle ~]# /usr/sbin/useradd --home-dir /storage/hudelarea2 -g ncmm-luecke-fg --no-create-home --shell /bin/bash --uid 1010 cryosparc_user [root@prometheus ~]# /usr/sbin/useradd --home-dir /storage/hudelarea2 -g ncmm-luecke-fg --no-create-home --shell /bin/bash --uid 1010 cryosparc_user [root@prometheus ~]# mkdir /storage/hudelarea2/.ssh [root@prometheus ~]# chmod 700 /storage/hudelarea2/.ssh [root@prometheus ~]# chown cryosparc_user:ncmm-luecke-fg /storage/hudelarea2/.ssh/ [root@prometheus ~]# su - cryosparc_user
Setting up the password-less ssh auth
Make sure that there is no password on the ssh key, cryoSPARC does not interact with ssh to ask you, the user, for a password.
[cryosparc_user@prometheus ~ ]$ ssh-keygen -t rsa [cryosparc_user@prometheus ~ ]$ mkdir .ssh/sockets [cryosparc_user@prometheus ~ ]$ chmod 700 .ssh/sockets [cryosparc_user@prometheus ~ ]$ cat .ssh/id_rsa.pub > .ssh/authorized_keys && chmod 600 .ssh/authorized_keys
Setting up the master node
[root@prometheus ~]# /usr/bin/mkdir -p /lsc/sources/cryosparc/2.8/1048c1bc-99d6-11e9-9456-bb2320f16355/ [root@prometheus ~]# /usr/bin/cd /lsc/sources/cryosparc/2.8/1048c1bc-99d6-11e9-9456-bb2320f16355/ [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# cat > $PWD/env.sources export LICENSE_ID="1048c1bc-99d6-11e9-9456-bb2320f16355" [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# source env.sources [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# curl -L https://get.cryosparc.com/download/master-latest/$LICENSE_ID [Open URL] > cryosparc2_master.tar.gz [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# for file in *.tar.gz; do tar zxvf $file; done [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# cd cryosparc2_master/ [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_MASTER_HOSTNAME=$(/usr/bin/hostname -f) [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_USER=cryosparc_user [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_VERSION=2.9 [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_USER_HOME=/lsc/cryosparc/$CRYOSPARC_VERSION/$LICENSE_ID [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_INSTALL_PATH=$CRYOSPARC_USER_HOME [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_DB_PATH=$CRYOSPARC_INSTALL_PATH/cryosparc2_database [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_MASTER_PORT_NUMBER=39000 [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_USER_EMAIL=george.marselis@ncmm.uio.no [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_USER_PASSWORD=###REDACTED### [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# cd /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# ./install.sh --license $LICENSE_ID --hostname $CRYOSPARC_MASTER_HOSTNAME --dbpath $CRYOSPARC_DB_PATH --port $CRYOSPARC_MASTER_PORT_NUMBER [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# cat > ~/.bashrc export CRYOSPARC_MASTER_HOSTNAME=$(/usr/bin/hostname -f) export CRYOSPARC_USER=cryosparc_user export CRYOSPARC_VERSION=2.9 export CRYOSPARC_USER_HOME=/lsc/cryosparc/$CRYOSPARC_VERSION/$LICENSE_ID export CRYOSPARC_INSTALL_PATH=$CRYOSPARC_USER_HOME export CRYOSPARC_DB_PATH=$CRYOSPARC_INSTALL_PATH/cryosparc2_database export CRYOSPARC_MASTER_PORT_NUMBER=80 [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# source ~/.bashrc [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# mkdir run && touch run/command_core.log && touch run/database.log && touch run/webapp.log
Setup systemd service for cryosparc master
We create a hand-made systemd service unit file, since cryosparc does not provide one.
cat > /etc/systemd/system/cryosparc.service [Unit] Description = cryoSPARC After = network.target [Service] Type = forking RestartSec = 10 User = cryosparc_user ExecStart = /bin/bash /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/bin/cryosparcm start ExecStop = /bin/bash /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/bin/cryosparcm stop PIDFile = /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/run/supervisord.pid #StartLimitIntervalSec = 10 [Install] WantedBy=multi-user.target
Set up cryosparc worker
Prepare the host
[root@intaristotle ~]# /usr/bin/mkdir -p /lsc/sources/cryosparc/2.8/1048c1bc-99d6-11e9-9456-bb2320f16355/ [root@intaristotle ~]# /usr/bin/cd /lsc/sources/cryosparc/2.8/1048c1bc-99d6-11e9-9456-bb2320f16355/ [root@intaristotle 1048c1bc-99d6-11e9-9456-bb2320f16355]# cat > $PWD/env.sources export LICENSE_ID="1048c1bc-99d6-11e9-9456-bb2320f16355" [root@intaristotle 1048c1bc-99d6-11e9-9456-bb2320f16355]# source env.sources [root@intaristotle 1048c1bc-99d6-11e9-9456-bb2320f16355]# curl -L https://get.cryosparc.com/download/worker-latest/$LICENSE_ID [Open URL] > cryosparc2_worker.tar.gz [root@intaristotle 1048c1bc-99d6-11e9-9456-bb2320f16355]# su - cryosparc_user [cryosparc_user@intaristotle cryosparc2_worker]$ export MASTER_HOSTNAME=prometheus.uio.no [cryosparc_user@intaristotle cryosparc2_worker]$ export PORT_NUM=3900 [cryosparc_user@intaristotle cryosparc2_worker]$ export WORKER_HOSTNAME=intaristotle.internal.biotek [cryosparc_user@intaristotle cryosparc2_worker]$ export CUDA_PATH=/lsc/nvidia/cuda/8.0-GA1/
Connect the cryosparc worker software
[root@intaristotle cryosparc2_worker]# usermod -a -G vglusers cryosparc_user # allow cryosparc_user to run /usr/bin/nvidia-smi to enumerate the graphics cards [root@intaristotle cryosparc2_worker]# bin/cryosparcw connect --worker intaristotle.internal.biotek --master intprometheus.internal.biotek --port 3900 --ssdpath /storage/scratch --sshstr cryosparc_user@intaristotle.internal.biotek --------------------------------------------------------------- CRYOSPARC CONNECT -------------------------------------------- --------------------------------------------------------------- Attempting to register worker intaristotle.internal.biotek to command intprometheus.internal.biotek:3902 Connecting as unix user root Will register using ssh string: cryosparc_user@intaristotle.internal.biotek If this is incorrect, you should re-run this command with the flag --sshstr <ssh string> --------------------------------------------------------------- Connected to master. --------------------------------------------------------------- Current connected workers: --------------------------------------------------------------- Autodetecting available GPUs... Detected 4 CUDA devices. id pci-bus name --------------------------------------------------------------- 0 0000:02:00.0 GeForce GTX 1080 Ti 1 0000:03:00.0 GeForce GTX 1080 Ti 2 0000:83:00.0 GeForce GTX 1080 Ti 3 0000:84:00.0 GeForce GTX 1080 Ti --------------------------------------------------------------- All devices will be enabled now. This can be changed later using --update --------------------------------------------------------------- Worker will be registered with SSD cache location /storage/scratch --------------------------------------------------------------- Autodetecting the amount of RAM available... This machine has 128.64GB RAM . --------------------------------------------------------------- --------------------------------------------------------------- Registering worker... Done. You can now launch jobs on the master node and they will be scheduled on to this worker node if resource requirements are met. --------------------------------------------------------------- Final configuration for intaristotle.internal.biotek lane : default name : intaristotle.internal.biotek title : Worker node intaristotle.internal.biotek resource_slots : {u'GPU': [0, 1, 2, 3], u'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], u'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]} hostname : intaristotle.internal.biotek worker_bin_path : /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_worker/bin/cryosparcw cache_path : /storage/scratch cache_quota_mb : None resource_fixed : {u'SSD': True} cache_reserve_mb : 10000 type : node ssh_str : cryosparc_user@intaristotle.internal.biotek desc : None ---------------------------------------------------------------
Done
Additional Info
How to delete a worker node
Connect to the master node (prometheus.uio.no) and
[cryosparc_user@prometheus ~] /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/bin/cryosparcm cluster remove intaristotle.internal.biotek
How to check for new updates for cryosparc
Connect to the master node (prometheus.uio.no) and
[root@prometheus ~] su - cryosparc_user [cryosparc_user@prometheus ~] source /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/env.sources [cryosparc_user@prometheus ~] /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/bin/cryosparcm update --check
How to list all versions of cryosparc available
[root@prometheus ~] su - cryosparc_user [cryosparc_user@prometheus ~] source /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/env.sources [cryosparc_user@prometheus ~] /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/bin/cryosparcm update --list
To get a full list of available versions (including old versions in case of a downgrade).
How to upgrade cryosparc
[root@prometheus ~] su - cryosparc_user [cryosparc_user@prometheus ~] source /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/env.sources [cryosparc_user@prometheus ~] /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/bin/cryosparcm update
How to downgrade cryosparc
[root@prometheus ~] su - cryosparc_user [cryosparc_user@prometheus ~] source /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/env.sources [cryosparc_user@prometheus ~] /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/bin/cryosparcm update --version=vX.Y.Z
version vX.Y.Z is the version you want to downgrade to, which you can find from bin/cryosparcm update --list
How to force an upgrade of cryosparc
If, for any reason, the upgrade does not succeed, you can force the upgrade. This means that bin/cryosparcm will re-download the latest software and re-install all of it from scratch.
Specifically, the bin/cryosparcm program will stop the cryoSPARC service on the master node, will download the latest software and deploy it as needed. The data in the current database will be preserved.
The way to force the upgrade is
[root@prometheus ~] su - cryosparc_user [cryosparc_user@prometheus ~] source /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/env.sources [cryosparc_user@prometheus ~] /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/bin/cryosparcm update --force
After updating the master node, bin/cryosparcm will try to connect to each worker node, in this case intaristotle, to copy the new cryosparc2_worker.tar.gz ; If for any reason copying fails, the software will notify you to copy /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/cryosparc2_worker.tar.gz to the worker node under /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_worker/cryosparc2_worker.tar.gz
After you manually copy the tarball you can run manually
[root@intaristotle ~] su - cryosparc_user [cryosparc_user@intaristotle ~] source /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/env.sources [cryosparc_user@intaristotle ~] /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_worker/bin/cryosparcm update --force
For more details, check the cryoSPARC manual on the update section
Changing the path of the software, both in the master and the worker(s) will disable the software
As [the manual says] under "Management and Monitoring" -> "Advanced":
master and worker installation directories cannot be moved to a different absolute path once installed. To move the installation, dump the database, install cryoSPARC again in the new location, and restore the database.