Difference between revisions of "Prometheus.uio.no"
(→Connect the cryosparc worker software) |
|||
Line 334: | Line 334: | ||
<pre> | <pre> | ||
+ | [root@intaristotle cryosparc2_worker]# usermod -a -G vglusers cryosparc_user # allow cryosparc_user to run /usr/bin/nvidia-smi to enumerate the graphics cards | ||
[root@intaristotle cryosparc2_worker]# bin/cryosparcw connect --worker intaristotle.internal.biotek --master intprometheus.internal.biotek --port 3900 --ssdpath /storage/scratch --sshstr cryosparc_user@intaristotle.internal.biotek | [root@intaristotle cryosparc2_worker]# bin/cryosparcw connect --worker intaristotle.internal.biotek --master intprometheus.internal.biotek --port 3900 --ssdpath /storage/scratch --sshstr cryosparc_user@intaristotle.internal.biotek | ||
--------------------------------------------------------------- | --------------------------------------------------------------- |
Revision as of 17:51, 17 July 2019
prometheus.uio.no is the front-end node for the NCMM cryoSPARC platform. cryoSPARC is a platform used for obtaining 3D structural information from single-particle cryoEM data
This wiki page describes how to configure the host and install the cryosparc software.
Contents
- 1 Apply for licenses
- 2 Preparing the host
- 2.1 Configuring the virsh serial port for the guest OS
- 2.2 Installing base-line software and prerequisites
- 2.3 Allocating extra space for the /lsc partition
- 2.4 Misc settings for comfortable editing
- 2.5 Configuring collectd
- 2.6 Setup firewall on prometheus.uio.no
- 2.7 Setting up NFS on the master node
- 3 Installing cryoSPARC
- 4 Set up cryosparc worker
- 5 Additional Info
Apply for licenses
Proceed to https://cryosparc.com/download/ and apply for a license or two. You should get a reply to your request within 24hrs. Please mind you that Structura Biotechnology Inc. is in Toronto, Canada, which means EST timezone ( GMT - 5), so adjust your expectations for a reply, accordingly.
Preparing the host
from jump-biotek, after pxebooting and installing the OS:
ssh prometheus.uio.no
Configuring the virsh serial port for the guest OS
cp /etc/default/grub /etc/default/grub.orig vi /etc/default/grub insert the following lines for serial console: GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)" GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT="console serial" GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=internvg/root rd.lvm.lv=internvg/swap rd.lvm.lv=internvg/usr console=tty1 console=ttyS0,115200" GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1" GRUB_DISABLE_RECOVERY="true" cp /boot/grub2/grub.cfg /boot/grub2/grub.cfg.orig grub2-mkconfig -o /boot/grub2/grub.cfg systemctl reboot
Installing base-line software and prerequisites
yum install -y fail2ban* vim* etckeeper* git* collectd* lm_* ncdu bzip2 lsof --skip-broken
Allocating extra space for the /lsc partition
parted /dev/vdb (parted) mklabel gtp New disk label type? gpt mkpart primary xfs 1 -1 (parted) p Model: Virtio Block Device (virtblk) Disk /dev/vdb: 42.9GB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1049kB 42.9GB 42.9GB primary (parted) quit Information: You may need to update /etc/fstab. quit
Format the partition
[root@prometheus ~]# mkfs.xfs /dev/vdb1 meta-data=/dev/vdb1 isize=512 agcount=4, agsize=2621312 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=0, sparse=0 data = bsize=4096 blocks=10485248, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=5119, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 [root@prometheus ~]# blkid /dev/vdb1 /dev/vdb1: UUID="450b41db-e9a4-49df-beb5-d4ccef7461a3" TYPE="xfs" PARTLABEL="primary" PARTUUID="d189b7ce-c419-4457-9281-13b9f988caa6"
Test the partition and mount it
#add an auxiliary space under /lsc vi /etc/fstab #RT 3485324 UUID=450b41db-e9a4-49df-beb5-d4ccef7461a3 /lsc xfs defaults 0 0 mount /dev/vdb1 /lsc df -h umount /lsc mount -a df -h
Misc settings for comfortable editing
vi ~/.vimrc 1 syntax enable 2 set tabstop=4 3 set softtabstop=4 4 set expandtab 5 set number 6 set showcmd " show command in bottom bar 7 set cursorline " highlight current line 8 filetype indent on " load filetype-specific indent files 9 set wildmenu 10 set showmatch 11 set incsearch " search as characters are entered 12 set hlsearch " highlight matches 13 nnoremap <leader><space> :nohlsearch<CR> 14 " set foldenable " enable folding 15 " set foldlevelstart=10 " open most folds by default 16 " set foldnestmax=10 " 10 nested fold max 17 " space open/closes folds 18 nnoremap <space> za 19 " set foldmethod=indent " fold based on indent level vi ~/.bashrc alias ls='LC_ALL=C ls --color=auto --literal --human-readable --group-directories-first --classify'
Configuring collectd
General configuration
systemctl stop collectd systemctl enable collectd ssh prometheus.uio.no cp /etc/collectd.conf /etc/collectd.conf.orig scp ~georgmar/ncmm/files/etc/collectd.conf prometheus:/etc/collectd.conf vi /etc/collectd.conf :s/TEMPLATEHOST\.uio\.no/prometheus.uio.no/g :s/TEMPLATEINTHOST\.internal\.biotek/intprometheus.internal.biotek/g :wq /usr/sbin/collectd -T systemctl start collectd
Set up firewall on panoptis.uio.no
ssh root@panoptis -- "firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.144/32" port port=2003 protocol="tcp" accept' && firewall-cmd --reload" ssh root@panoptis -- "firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.144/32" port port=2004 protocol="tcp" accept' && firewall-cmd --reload"
Setup firewall on prometheus.uio.no
After looking at the cryosparc logs, it looks like cryosparc is using more ports than documented to communicate. so, enable more firewall ports.
On the public interface we specify dias.uio.no ( 129.240.235.130 ) for the source address, as intaristotle.internal.biotek uses dias for the NAT:
#public interface firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="129.240.235.130/32" port port=3900 protocol="tcp" accept' && firewall-cmd --reload firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="129.240.235.130/32" port port=3901 protocol="tcp" accept' && firewall-cmd --reload firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="129.240.235.130/32" port port=3902 protocol="tcp" accept' && firewall-cmd --reload firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="129.240.235.130/32" port port=3903 protocol="tcp" accept' && firewall-cmd --reload firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="129.240.235.130/32" port port=3904 protocol="tcp" accept' && firewall-cmd --reload
On the private interface we specify the address of intaristotle.internal.biotek ( 192.168.8.109 ):
# internal interface firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.109/32" port port=3900 protocol="tcp" accept' && firewall-cmd --reload firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.109/32" port port=3901 protocol="tcp" accept' && firewall-cmd --reload firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.109/32" port port=3902 protocol="tcp" accept' && firewall-cmd --reload firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.109/32" port port=3903 protocol="tcp" accept' && firewall-cmd --reload firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.109/32" port port=3904 protocol="tcp" accept' && firewall-cmd --reload
The cryosparc installer, as we will see, defines only the base port for the web application. supervisord and mongodb run at base port +1 and +2, respectively.
We specified that the base port is 3900, so we have to open ports 3900, 3901, 3902, 3903,for the web app, mongodb, supervisord and wsgi, respectively.
Setting up NFS on the master node
[root@pythagoras ~]# vi /etc/exports.d/intprometheus.internal.biotek.exports /div/pythagoras/u6 192.168.8.144(rw,sync) /div/pythagoras/u4 192.168.8.144(rw,sync) /div/pythagoras/u1 192.168.8.144(rw,sync) /div/pythagoras/u2 192.168.8.144(rw,sync) [root@pythagoras ~]# exportfs -a [root@perun ~]# vi /etc/exports /storage/hudelarea 192.168.15.2(rw,sync) 129.240.189.230(rw,sync) 129.240.237.151(rw,sync) 192.168.8.38(rw,sync) 192.168.15.4(rw,sync,no_root_squash) 192.168.8.144(rw,sync) /storage/hudelarea2 192.168.15.2(rw,sync) 129.240.189.230(rw,sync) 192.168.8.38(rw,sync) 129.240.237.151(rw,sync) 192.168.8.144(rw,sync) [root@pythagoras ~]# exportfs -a
Automatically mounting all the partitions needed
[root@prometheus ~]# vi /etc/fstab # # /etc/fstab # Created by anaconda on Mon Jul 8 11:28:47 2019 # # Accessible filesystems, by reference, are maintained under '/dev/disk' # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info # /dev/mapper/internvg-root / xfs defaults 0 0 UUID=c12a7820-0b28-408e-8cea-0f3ebaeb864a /boot xfs defaults 0 0 /dev/mapper/internvg-opt /opt xfs defaults 0 0 /dev/mapper/internvg-tmp /tmp xfs defaults 0 0 /dev/mapper/internvg-usr /usr xfs defaults 0 0 /dev/mapper/internvg-var /var xfs defaults 0 0 /dev/mapper/internvg-swap swap swap defaults 0 0 #RT 3485324 UUID=450b41db-e9a4-49df-beb5-d4ccef7461a3 /lsc xfs defaults 0 0 192.168.8.132:/div/pythagoras/u6 /div/pythagoras/u6 nfs rw,nodev,nosuid,intr,nfsvers=3 192.168.8.132:/div/pythagoras/u4 /div/pythagoras/u4 nfs rw,nodev,nosuid,intr,nfsvers=3 192.168.8.132:/div/pythagoras/u1 /div/pythagoras/u1 nfs rw,nodev,nosuid,intr,nfsvers=3 192.168.8.132:/div/pythagoras/u2 /div/pythagoras/u2 nfs rw,nodev,nosuid,intr,nfsvers=3 192.168.8.22:/storage/hudelarea /storage/hudelarea nfs rw,nodev,nosuid,intr,nfsvers=3 192.168.8.22:/storage/hudelarea2 /storage/hudelarea2 nfs rw,nodev,nosuid,intr,nfsvers=3 [root@prometheus ~]# mount -a
Installing cryoSPARC
Set up cryoSPARC master node
Full instructions here: https://cryosparc.com/docs/reference/install/
[root@intaristotle ~]# /usr/sbin/useradd --home-dir /storage/hudelarea2 -g ncmm-luecke-fg --no-create-home --shell /bin/bash --uid 1010 cryosparc_user [root@prometheus ~]# /usr/sbin/useradd --home-dir /storage/hudelarea2 -g ncmm-luecke-fg --no-create-home --shell /bin/bash --uid 1010 cryosparc_user [root@prometheus ~]# mkdir /storage/hudelarea2/.ssh [root@prometheus ~]# chmod 700 /storage/hudelarea2/.ssh [root@prometheus ~]# chown cryosparc_user:ncmm-luecke-fg /storage/hudelarea2/.ssh/ [root@prometheus ~]# su - cryosparc_user
Setting up the password-less ssh auth
Make sure that there is no password on the ssh key, cryoSPARC does not interact with ssh to ask you, the user, for a password.
[cryosparc_user@prometheus ~ ]$ ssh-keygen -t rsa [cryosparc_user@prometheus ~ ]$ mkdir .ssh/sockets [cryosparc_user@prometheus ~ ]$ chmod 700 .ssh/sockets [cryosparc_user@prometheus ~ ]$ cat .ssh/id_rsa.pub > .ssh/authorized_keys && chmod 600 .ssh/authorized_keys
Setting up the master node
[root@prometheus ~]# /usr/bin/mkdir -p /lsc/sources/cryosparc/2.8/1048c1bc-99d6-11e9-9456-bb2320f16355/ [root@prometheus ~]# /usr/bin/cd /lsc/sources/cryosparc/2.8/1048c1bc-99d6-11e9-9456-bb2320f16355/ [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# cat > $PWD/env.sources export LICENSE_ID="1048c1bc-99d6-11e9-9456-bb2320f16355" [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# source env.sources [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# curl -L https://get.cryosparc.com/download/master-latest/$LICENSE_ID [Open URL] > cryosparc2_master.tar.gz [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# for file in *.tar.gz; do tar zxvf $file; done [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# cd cryosparc2_master/ [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_MASTER_HOSTNAME=$(/usr/bin/hostname -f) [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_USER=cryosparc_user [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_VERSION=2.9 [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_USER_HOME=/lsc/cryosparc/$CRYOSPARC_VERSION/$LICENSE_ID [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_INSTALL_PATH=$CRYOSPARC_USER_HOME [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_DB_PATH=$CRYOSPARC_INSTALL_PATH/cryosparc2_database [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_MASTER_PORT_NUMBER=39000 [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_USER_EMAIL=george.marselis@ncmm.uio.no [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_USER_PASSWORD=###REDACTED### [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# cd /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# ./install.sh --license $LICENSE_ID --hostname $CRYOSPARC_MASTER_HOSTNAME --dbpath $CRYOSPARC_DB_PATH --port $CRYOSPARC_MASTER_PORT_NUMBER [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# cat > ~/.bashrc export CRYOSPARC_MASTER_HOSTNAME=$(/usr/bin/hostname -f) export CRYOSPARC_USER=cryosparc_user export CRYOSPARC_VERSION=2.9 export CRYOSPARC_USER_HOME=/lsc/cryosparc/$CRYOSPARC_VERSION/$LICENSE_ID export CRYOSPARC_INSTALL_PATH=$CRYOSPARC_USER_HOME export CRYOSPARC_DB_PATH=$CRYOSPARC_INSTALL_PATH/cryosparc2_database export CRYOSPARC_MASTER_PORT_NUMBER=80 [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# source ~/.bashrc [root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# mkdir run && touch run/command_core.log && touch run/database.log && touch run/webapp.log
Setup systemd service for cryosparc master
We create a hand-made systemd service unit file, since cryosparc does not provide one.
cat > /etc/systemd/system/cryosparc.service [Unit] Description = cryoSPARC After = network.target [Service] Type = forking RestartSec = 10 User = cryosparc_user ExecStart = /bin/bash /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/bin/cryosparcm start ExecStop = /bin/bash /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/bin/cryosparcm stop PIDFile = /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/run/supervisord.pid #StartLimitIntervalSec = 10 [Install] WantedBy=multi-user.target
Set up cryosparc worker
Prepare the host
[root@intaristotle ~]# /usr/bin/mkdir -p /lsc/sources/cryosparc/2.8/1048c1bc-99d6-11e9-9456-bb2320f16355/ [root@intaristotle ~]# /usr/bin/cd /lsc/sources/cryosparc/2.8/1048c1bc-99d6-11e9-9456-bb2320f16355/ [root@intaristotle 1048c1bc-99d6-11e9-9456-bb2320f16355]# cat > $PWD/env.sources export LICENSE_ID="1048c1bc-99d6-11e9-9456-bb2320f16355" [root@intaristotle 1048c1bc-99d6-11e9-9456-bb2320f16355]# source env.sources [root@intaristotle 1048c1bc-99d6-11e9-9456-bb2320f16355]# curl -L https://get.cryosparc.com/download/worker-latest/$LICENSE_ID [Open URL] > cryosparc2_worker.tar.gz [root@intaristotle 1048c1bc-99d6-11e9-9456-bb2320f16355]# su - cryosparc_user [cryosparc_user@intaristotle cryosparc2_worker]$ export MASTER_HOSTNAME=prometheus.uio.no [cryosparc_user@intaristotle cryosparc2_worker]$ export PORT_NUM=3900 [cryosparc_user@intaristotle cryosparc2_worker]$ export WORKER_HOSTNAME=intaristotle.internal.biotek [cryosparc_user@intaristotle cryosparc2_worker]$ export CUDA_PATH=/lsc/nvidia/cuda/8.0-GA1/
Connect the cryosparc worker software
[root@intaristotle cryosparc2_worker]# usermod -a -G vglusers cryosparc_user # allow cryosparc_user to run /usr/bin/nvidia-smi to enumerate the graphics cards [root@intaristotle cryosparc2_worker]# bin/cryosparcw connect --worker intaristotle.internal.biotek --master intprometheus.internal.biotek --port 3900 --ssdpath /storage/scratch --sshstr cryosparc_user@intaristotle.internal.biotek --------------------------------------------------------------- CRYOSPARC CONNECT -------------------------------------------- --------------------------------------------------------------- Attempting to register worker intaristotle.internal.biotek to command intprometheus.internal.biotek:3902 Connecting as unix user root Will register using ssh string: cryosparc_user@intaristotle.internal.biotek If this is incorrect, you should re-run this command with the flag --sshstr <ssh string> --------------------------------------------------------------- Connected to master. --------------------------------------------------------------- Current connected workers: --------------------------------------------------------------- Autodetecting available GPUs... Detected 4 CUDA devices. id pci-bus name --------------------------------------------------------------- 0 0000:02:00.0 GeForce GTX 1080 Ti 1 0000:03:00.0 GeForce GTX 1080 Ti 2 0000:83:00.0 GeForce GTX 1080 Ti 3 0000:84:00.0 GeForce GTX 1080 Ti --------------------------------------------------------------- All devices will be enabled now. This can be changed later using --update --------------------------------------------------------------- Worker will be registered with SSD cache location /storage/scratch --------------------------------------------------------------- Autodetecting the amount of RAM available... This machine has 128.64GB RAM . --------------------------------------------------------------- --------------------------------------------------------------- Registering worker... Done. You can now launch jobs on the master node and they will be scheduled on to this worker node if resource requirements are met. --------------------------------------------------------------- Final configuration for intaristotle.internal.biotek lane : default name : intaristotle.internal.biotek title : Worker node intaristotle.internal.biotek resource_slots : {u'GPU': [0, 1, 2, 3], u'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], u'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]} hostname : intaristotle.internal.biotek worker_bin_path : /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_worker/bin/cryosparcw cache_path : /storage/scratch cache_quota_mb : None resource_fixed : {u'SSD': True} cache_reserve_mb : 10000 type : node ssh_str : cryosparc_user@intaristotle.internal.biotek desc : None ---------------------------------------------------------------
Done
Additional Info
How to delete a worker node
Connect to the master node (prometheus.uio.no) and
[cryosparc_user@prometheus ~] /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/bin/cryosparcm cluster remove intaristotle.internal.biotek