Prometheus.uio.no

From medicin.ncmm.IT
Jump to: navigation, search

prometheus.uio.no is the front-end node for the NCMM cryoSPARC platform. cryoSPARC is a platform used for obtaining 3D structural information from single-particle cryoEM data

This wiki page describes how to configure the host and install the cryosparc software.

Apply for licenses

Proceed to https://cryosparc.com/download/ and apply for a license or two. You should get a reply to your request within 24hrs. Please mind you that Structura Biotechnology Inc. is in Toronto, Canada, which means EST timezone ( GMT - 5), so adjust your expectations for a reply, accordingly.

Preparing the host

from jump-biotek, after pxebooting and installing the OS:

ssh prometheus.uio.no

Configuring the virsh serial port for the guest OS

cp /etc/default/grub /etc/default/grub.orig
vi /etc/default/grub 
insert the following lines for serial console:
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console serial"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=internvg/root rd.lvm.lv=internvg/swap rd.lvm.lv=internvg/usr console=tty1 console=ttyS0,115200"
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"
GRUB_DISABLE_RECOVERY="true"

cp /boot/grub2/grub.cfg /boot/grub2/grub.cfg.orig
grub2-mkconfig -o /boot/grub2/grub.cfg
systemctl reboot

Installing base-line software and prerequisites

yum install -y fail2ban* vim* etckeeper* git* collectd* lm_* ncdu bzip2 lsof --skip-broken

Allocating extra space for the /lsc partition

parted /dev/vdb
(parted) mklabel gtp
New disk label type? gpt
mkpart primary xfs 1 -1
(parted) p
Model: Virtio Block Device (virtblk)
Disk /dev/vdb: 42.9GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number Start End Size File system Name Flags
1 1049kB 42.9GB 42.9GB primary

(parted) quit 
Information: You may need to update /etc/fstab.
quit


Format the partition

[root@prometheus ~]# mkfs.xfs /dev/vdb1 
meta-data=/dev/vdb1 isize=512 agcount=4, agsize=2621312 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=10485248, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=5119, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@prometheus ~]# blkid /dev/vdb1 
/dev/vdb1: UUID="450b41db-e9a4-49df-beb5-d4ccef7461a3" TYPE="xfs" PARTLABEL="primary" PARTUUID="d189b7ce-c419-4457-9281-13b9f988caa6" 

Test the partition and mount it


#add an auxiliary space under /lsc
vi /etc/fstab
#RT 3485324
UUID=450b41db-e9a4-49df-beb5-d4ccef7461a3 /lsc xfs defaults 0 0
mount /dev/vdb1 /lsc
df -h
umount /lsc
mount -a
df -h

Misc settings for comfortable editing

 
vi ~/.vimrc

1 syntax enable
2 set tabstop=4
3 set softtabstop=4
4 set expandtab
5 set number
6 set showcmd " show command in bottom bar
7 set cursorline " highlight current line
8 filetype indent on " load filetype-specific indent files
9 set wildmenu
10 set showmatch
11 set incsearch " search as characters are entered
12 set hlsearch " highlight matches
13 nnoremap <leader><space> :nohlsearch<CR>
14 " set foldenable " enable folding
15 " set foldlevelstart=10 " open most folds by default
16 " set foldnestmax=10 " 10 nested fold max
17 " space open/closes folds
18 nnoremap <space> za
19 " set foldmethod=indent " fold based on indent level

vi ~/.bashrc
alias ls='LC_ALL=C ls --color=auto --literal --human-readable --group-directories-first --classify'

Configuring collectd

General configuration

systemctl stop collectd
systemctl enable collectd
ssh prometheus.uio.no cp /etc/collectd.conf /etc/collectd.conf.orig
scp ~georgmar/ncmm/files/etc/collectd.conf prometheus:/etc/collectd.conf
vi /etc/collectd.conf
:s/TEMPLATEHOST\.uio\.no/prometheus.uio.no/g
:s/TEMPLATEINTHOST\.internal\.biotek/intprometheus.internal.biotek/g
:wq
/usr/sbin/collectd -T
systemctl start collectd

Set up firewall on panoptis.uio.no

ssh root@panoptis -- "firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.144/32" port port=2003 protocol="tcp" accept' && firewall-cmd --reload"
ssh root@panoptis -- "firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.144/32" port port=2004 protocol="tcp" accept' && firewall-cmd --reload"


Setup firewall on prometheus.uio.no

After looking at the cryosparc logs, it looks like cryosparc is using more ports than documented to communicate. so, enable more firewall ports.

On the public interface we specify dias.uio.no ( 129.240.235.130 ) for the source address, as intaristotle.internal.biotek uses dias for the NAT:

 
#public interface
firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="129.240.235.130/32" port port=3900 protocol="tcp" accept' && firewall-cmd --reload
firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="129.240.235.130/32" port port=3901 protocol="tcp" accept' && firewall-cmd --reload
firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="129.240.235.130/32" port port=3902 protocol="tcp" accept' && firewall-cmd --reload
firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="129.240.235.130/32" port port=3903 protocol="tcp" accept' && firewall-cmd --reload
firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="129.240.235.130/32" port port=3904 protocol="tcp" accept' && firewall-cmd --reload

On the private interface we specify the address of intaristotle.internal.biotek ( 192.168.8.109 ):

# internal interface
firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.109/32" port port=3900 protocol="tcp" accept' && firewall-cmd --reload
firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.109/32" port port=3901 protocol="tcp" accept' && firewall-cmd --reload
firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.109/32" port port=3902 protocol="tcp" accept' && firewall-cmd --reload
firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.109/32" port port=3903 protocol="tcp" accept' && firewall-cmd --reload
firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.8.109/32" port port=3904 protocol="tcp" accept' && firewall-cmd --reload

The cryosparc installer, as we will see, defines only the base port for the web application. supervisord and mongodb run at base port +1 and +2, respectively.

We specified that the base port is 3900, so we have to open ports 3900, 3901, 3902, 3903,for the web app, mongodb, supervisord and wsgi, respectively.


Setting up NFS on the master node

[root@pythagoras ~]# vi /etc/exports.d/intprometheus.internal.biotek.exports
/div/pythagoras/u6 192.168.8.144(rw,sync)
/div/pythagoras/u4 192.168.8.144(rw,sync)
/div/pythagoras/u1 192.168.8.144(rw,sync)
/div/pythagoras/u2 192.168.8.144(rw,sync)

[root@pythagoras ~]# exportfs -a

[root@perun ~]# vi /etc/exports
/storage/hudelarea 192.168.15.2(rw,sync) 129.240.189.230(rw,sync) 129.240.237.151(rw,sync) 192.168.8.38(rw,sync) 192.168.15.4(rw,sync,no_root_squash) 192.168.8.144(rw,sync)
/storage/hudelarea2 192.168.15.2(rw,sync) 129.240.189.230(rw,sync) 192.168.8.38(rw,sync) 129.240.237.151(rw,sync) 192.168.8.144(rw,sync)

[root@pythagoras ~]# exportfs -a

Automatically mounting all the partitions needed

[root@prometheus ~]# vi /etc/fstab

#
# /etc/fstab
# Created by anaconda on Mon Jul 8 11:28:47 2019
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/internvg-root / xfs defaults 0 0
UUID=c12a7820-0b28-408e-8cea-0f3ebaeb864a /boot xfs defaults 0 0
/dev/mapper/internvg-opt /opt xfs defaults 0 0
/dev/mapper/internvg-tmp /tmp xfs defaults 0 0
/dev/mapper/internvg-usr /usr xfs defaults 0 0
/dev/mapper/internvg-var /var xfs defaults 0 0
/dev/mapper/internvg-swap swap swap defaults 0 0

#RT 3485324
UUID=450b41db-e9a4-49df-beb5-d4ccef7461a3 /lsc xfs defaults 0 0
192.168.8.132:/div/pythagoras/u6 /div/pythagoras/u6 nfs rw,nodev,nosuid,intr,nfsvers=3
192.168.8.132:/div/pythagoras/u4 /div/pythagoras/u4 nfs rw,nodev,nosuid,intr,nfsvers=3
192.168.8.132:/div/pythagoras/u1 /div/pythagoras/u1 nfs rw,nodev,nosuid,intr,nfsvers=3
192.168.8.132:/div/pythagoras/u2 /div/pythagoras/u2 nfs rw,nodev,nosuid,intr,nfsvers=3

192.168.8.22:/storage/hudelarea /storage/hudelarea nfs rw,nodev,nosuid,intr,nfsvers=3
192.168.8.22:/storage/hudelarea2 /storage/hudelarea2 nfs rw,nodev,nosuid,intr,nfsvers=3

[root@prometheus ~]# mount -a

Installing cryoSPARC

Set up cryoSPARC master node

Full instructions here: https://cryosparc.com/docs/reference/install/


[root@intaristotle ~]# /usr/sbin/useradd --home-dir /storage/hudelarea2 -g ncmm-luecke-fg --no-create-home --shell /bin/bash --uid 1010 cryosparc_user
[root@prometheus ~]# /usr/sbin/useradd --home-dir /storage/hudelarea2  -g ncmm-luecke-fg --no-create-home --shell /bin/bash --uid 1010 cryosparc_user
[root@prometheus ~]# mkdir /storage/hudelarea2/.ssh
[root@prometheus ~]# chmod 700 /storage/hudelarea2/.ssh
[root@prometheus ~]# chown cryosparc_user:ncmm-luecke-fg /storage/hudelarea2/.ssh/
[root@prometheus ~]# su - cryosparc_user

Setting up the password-less ssh auth

Make sure that there is no password on the ssh key, cryoSPARC does not interact with ssh to ask you, the user, for a password.

[cryosparc_user@prometheus ~ ]$ ssh-keygen -t rsa 
[cryosparc_user@prometheus ~ ]$ mkdir .ssh/sockets
[cryosparc_user@prometheus ~ ]$ chmod 700 .ssh/sockets
[cryosparc_user@prometheus ~ ]$ cat .ssh/id_rsa.pub > .ssh/authorized_keys && chmod 600 .ssh/authorized_keys

Setting up the master node

[root@prometheus ~]# /usr/bin/mkdir -p /lsc/sources/cryosparc/2.8/1048c1bc-99d6-11e9-9456-bb2320f16355/
[root@prometheus ~]# /usr/bin/cd /lsc/sources/cryosparc/2.8/1048c1bc-99d6-11e9-9456-bb2320f16355/
[root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# cat > $PWD/env.sources 
export LICENSE_ID="1048c1bc-99d6-11e9-9456-bb2320f16355"
[root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# source env.sources
[root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# curl -L https://get.cryosparc.com/download/master-latest/$LICENSE_ID [Open URL] > cryosparc2_master.tar.gz
[root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# for file in *.tar.gz; do tar zxvf $file; done
[root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# cd cryosparc2_master/
[root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_MASTER_HOSTNAME=$(/usr/bin/hostname -f)
[root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_USER=cryosparc_user
[root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_VERSION=2.9
[root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_USER_HOME=/lsc/cryosparc/$CRYOSPARC_VERSION/$LICENSE_ID
[root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_INSTALL_PATH=$CRYOSPARC_USER_HOME
[root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_DB_PATH=$CRYOSPARC_INSTALL_PATH/cryosparc2_database
[root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_MASTER_PORT_NUMBER=39000 
[root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_USER_EMAIL=george.marselis@ncmm.uio.no
[root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# export CRYOSPARC_USER_PASSWORD=###REDACTED###
[root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# cd /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master
[root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# ./install.sh --license $LICENSE_ID --hostname $CRYOSPARC_MASTER_HOSTNAME --dbpath $CRYOSPARC_DB_PATH --port $CRYOSPARC_MASTER_PORT_NUMBER
[root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# cat > ~/.bashrc
export CRYOSPARC_MASTER_HOSTNAME=$(/usr/bin/hostname -f)
export CRYOSPARC_USER=cryosparc_user
export CRYOSPARC_VERSION=2.9
export CRYOSPARC_USER_HOME=/lsc/cryosparc/$CRYOSPARC_VERSION/$LICENSE_ID
export CRYOSPARC_INSTALL_PATH=$CRYOSPARC_USER_HOME
export CRYOSPARC_DB_PATH=$CRYOSPARC_INSTALL_PATH/cryosparc2_database
export CRYOSPARC_MASTER_PORT_NUMBER=80 
[root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# source ~/.bashrc
[root@prometheus 1048c1bc-99d6-11e9-9456-bb2320f16355]# mkdir run && touch run/command_core.log && touch run/database.log && touch run/webapp.log

Setup systemd service for cryosparc master

We create a hand-made systemd service unit file, since cryosparc does not provide one.

cat > /etc/systemd/system/cryosparc.service

[Unit]
Description           = cryoSPARC
After                 = network.target

[Service]
Type       = forking
RestartSec = 10
User       = cryosparc_user
ExecStart  = /bin/bash /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/bin/cryosparcm start
ExecStop   = /bin/bash /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/bin/cryosparcm stop
PIDFile    = /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/run/supervisord.pid
#StartLimitIntervalSec = 10

[Install]
WantedBy=multi-user.target

Set up cryosparc worker

Prepare the host

[root@intaristotle ~]# /usr/bin/mkdir -p /lsc/sources/cryosparc/2.8/1048c1bc-99d6-11e9-9456-bb2320f16355/
[root@intaristotle ~]# /usr/bin/cd /lsc/sources/cryosparc/2.8/1048c1bc-99d6-11e9-9456-bb2320f16355/
[root@intaristotle 1048c1bc-99d6-11e9-9456-bb2320f16355]# cat > $PWD/env.sources 
export LICENSE_ID="1048c1bc-99d6-11e9-9456-bb2320f16355"
[root@intaristotle 1048c1bc-99d6-11e9-9456-bb2320f16355]# source env.sources
[root@intaristotle 1048c1bc-99d6-11e9-9456-bb2320f16355]# curl -L https://get.cryosparc.com/download/worker-latest/$LICENSE_ID [Open URL] > cryosparc2_worker.tar.gz
[root@intaristotle 1048c1bc-99d6-11e9-9456-bb2320f16355]# su - cryosparc_user
[cryosparc_user@intaristotle cryosparc2_worker]$ export MASTER_HOSTNAME=prometheus.uio.no
[cryosparc_user@intaristotle cryosparc2_worker]$ export PORT_NUM=3900
[cryosparc_user@intaristotle cryosparc2_worker]$ export WORKER_HOSTNAME=intaristotle.internal.biotek
[cryosparc_user@intaristotle cryosparc2_worker]$ export CUDA_PATH=/lsc/nvidia/cuda/8.0-GA1/


Connect the cryosparc worker software


[root@intaristotle cryosparc2_worker]# usermod -a -G vglusers cryosparc_user # allow cryosparc_user to run /usr/bin/nvidia-smi to enumerate the graphics cards
[root@intaristotle cryosparc2_worker]# bin/cryosparcw connect --worker intaristotle.internal.biotek --master intprometheus.internal.biotek --port 3900 --ssdpath /storage/scratch --sshstr cryosparc_user@intaristotle.internal.biotek
---------------------------------------------------------------
CRYOSPARC CONNECT --------------------------------------------
---------------------------------------------------------------
Attempting to register worker intaristotle.internal.biotek to command intprometheus.internal.biotek:3902
Connecting as unix user root
Will register using ssh string: cryosparc_user@intaristotle.internal.biotek
If this is incorrect, you should re-run this command with the flag --sshstr <ssh string> 
---------------------------------------------------------------
Connected to master.
---------------------------------------------------------------
Current connected workers:
---------------------------------------------------------------
Autodetecting available GPUs...
Detected 4 CUDA devices.

id pci-bus name
---------------------------------------------------------------
0 0000:02:00.0 GeForce GTX 1080 Ti
1 0000:03:00.0 GeForce GTX 1080 Ti
2 0000:83:00.0 GeForce GTX 1080 Ti
3 0000:84:00.0 GeForce GTX 1080 Ti
---------------------------------------------------------------
All devices will be enabled now. 
This can be changed later using --update
---------------------------------------------------------------
Worker will be registered with SSD cache location /storage/scratch 
---------------------------------------------------------------
Autodetecting the amount of RAM available...
This machine has 128.64GB RAM .
---------------------------------------------------------------
---------------------------------------------------------------
Registering worker...
Done.

You can now launch jobs on the master node and they will be scheduled
on to this worker node if resource requirements are met.
---------------------------------------------------------------
Final configuration for intaristotle.internal.biotek
lane : default
name : intaristotle.internal.biotek
title : Worker node intaristotle.internal.biotek
resource_slots : {u'GPU': [0, 1, 2, 3], u'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], u'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]}
hostname : intaristotle.internal.biotek
worker_bin_path : /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_worker/bin/cryosparcw
cache_path : /storage/scratch
cache_quota_mb : None
resource_fixed : {u'SSD': True}
cache_reserve_mb : 10000
type : node
ssh_str : cryosparc_user@intaristotle.internal.biotek
desc : None
---------------------------------------------------------------

Done

Additional Info

How to delete a worker node

Connect to the master node (prometheus.uio.no) and

[cryosparc_user@prometheus ~] /lsc/cryosparc/2.9/1048c1bc-99d6-11e9-9456-bb2320f16355/cryosparc2_master/bin/cryosparcm cluster remove intaristotle.internal.biotek