Difference between revisions of "NCMM CryoEM Computing platform"

From medicin.ncmm.IT
Jump to: navigation, search
(relion/{ 2.1, 3.0b })
(updated cryosparc version)
 
(122 intermediate revisions by 3 users not shown)
Line 1: Line 1:
This is the (growing) documentation for the software installed in the NCMM computational nodes
+
There are two groups that work with CryoEM data: the [https://www.med.uio.no/ncmm/english/groups/luecke-group/index.html Luecke Group] and the [https://www.med.uio.no/ncmm/english/groups/sekulic-group/ Sekulic Group]:
 +
 
 +
The [https://www.med.uio.no/ncmm/english/groups/luecke-group/index.html Luecke Group] CryoEM setup data processing platform consists of two physical servers and one VM:
 +
* intaristotle.internal.biotek GPU server: 64 [https://ark.intel.com/products/91766/Intel-Xeon-Processor-E5-2683-v4-40M-Cache-2-10-GHz- Intel Broadwell cores], 128 Gbytes of RAM, 4 x [https://www.nvidia.com/en-us/geforce/products/10series/geforce-gtx-1080-ti/ NVIDIA GTX 1080 GPU cards], 6 Tb of SSD scratch space
 +
* perun.uio.no file server: 28 [https://ark.intel.com/products/91754/Intel-Xeon-Processor-E5-2680-v4-35M-Cache-2-40-GHz- Intel Broadwell cores], 32 Gigs of RAM, 212 Tb of local disk space used as file server for the storage of CryoEM data. The machine is also used as a gateway to internal GPU/CPU processing nodes.
 +
* cryosparc.uio.no VM: 2 x Intel Core Processor (Haswell, IBRS) vcores, 8 Gigs of RAM and 72 Gigs of disk space to run a discrete CryoSPARC web instance
 +
All nodes have 10 Gigabit Ethernet connectivity for dedicated NAS/NFS disk space between the file server and the current (and future) GPU compute nodes.
 +
 
 +
The [https://www.med.uio.no/ncmm/english/groups/sekulic-group/ Sekulic Group]  CryoEM setup data processing platform consists of one physical server, one VM and a dedicated file share on the main pythagoras file server :
 +
* sekulicgpu1.internal.biotek GPU server: 64 CPU core system Dual [https://ark.intel.com/content/www/us/en/ark/products/192440/intel-xeon-gold-6242-processor-22m-cache-2-80-ghz.html Intel Xeon Gold 6242 CPU @ 2.80GHz], 6 x [https://www.nvidia.com/en-us/data-center/tesla-t4/ T4 Nvidia cards] for a total of 15360 CUDA cores plus 1920 Tensor cores, 768 Gigs of RAM. The system also arrived with 416GB of storage for the OS and 3.49 TB of storage for scratch storage.
 +
* cryosparc2.uio.no VM: 4 x Intel Core Processor (Haswell, IBRS) vcores, 16 Gb of RAM and 170 Gigs of disk space to run a discrete CryoSPARC web interface instance
 +
 
  
 
= Aristotle/CryoMP Installation Documentation ( intaristotle.internal.biotek/192.168.8.109 ) =
 
= Aristotle/CryoMP Installation Documentation ( intaristotle.internal.biotek/192.168.8.109 ) =
Line 5: Line 16:
 
[https://rt.uio.no/Ticket/Update.html?id=3135527 RT ticket #3135527]
 
[https://rt.uio.no/Ticket/Update.html?id=3135527 RT ticket #3135527]
  
== Software Installed, List of ==
+
= Live computational resource usage information (cpu and memory) =
 +
 
 +
{{#widget:Image|url=http://panoptis.uio.no/S/I}}
 +
 
 +
{{#widget:Image|url=http://panoptis.uio.no/S/F}}
 +
 
 +
Software Installed, List of =
  
* ccp4/4.7.0
+
{| class="wikitable" style="text-align: center;"
* chimera/1.13.1
+
|-
* ctffind4/4.1.10
+
|  '''Application'''                                                                                        || '''Version on intaristotle.internal.biotek server''' ||  '''Status''' ||  '''Comments''' ||
* eman2/2.2
+
|-
* external
+
|  [https://www.scripps.edu/tainer/arvai/adxv.html adxv]                        ||        1.9.13                ||                          Done      ||  -            ||
** openssl/1.0.2o
+
|-
* frealign/9.11
+
|  [http://www.ccp4.ac.uk/ ccp4]                                                              ||        7.1.003              ||                          Done      ||  -            ||
* gautomatch/0.53
+
|-
* gctf/1.18
+
|  [https://www.cgl.ucsf.edu/chimera/ chimera]                                        ||        1.14                  ||                          Done      ||  -            ||
* modules/4.20
+
|-
* mpich/3.0.4
+
|  [https://cistem.org/ cistem]                                                                    ||        1.0.0-beta          ||                          Done      ||  -            ||
* motioncor2/1.2.1
+
|-
* nvidia/
+
|  [https://sphire.mpg.de/wiki/doku.php?id=pipeline:window:cryolo cryolo]                                                                    ||        1.5          ||                          Done      ||  -            ||
** cuda/{ 8.0-GA1, 8.0-GA2, 9.0, 9.1, 9.2 }
+
|-
* openmpi/3.1.2
+
|  [http://grigoriefflab.janelia.org/ctffind4 ctffind4]                                ||        4.1.10                 ||                          Done      ||  -            ||
* phenix/1.14-3260
+
|-
* relion/{ 2.1, 3.0b }
+
|  [http://prometheus.uio.no:3900/ cryosparc]                                        ||        3.1                  ||                          Done      ||  [https://wiki.uio.no/medicin/ncmm/IT/index.php/Prometheus.uio.no Documentation]                  ||
* sphire/1.1
+
|-
* xchimera/0.8
+
|  [http://www.msg.ucsf.edu/local/programs/eman/ eman2]                    ||        2.31 ||                          Done      ||  -            ||
 +
|-
 +
|  [http://www.fftw.org/ fftw, double precision]                                        ||        2.1.5                  ||                          Done      ||  -            ||
 +
|-
 +
|  [http://www.fftw.org/ fftw, single precision]                                          ||        2.1.5                  ||                          Done      ||  -            ||
 +
|-
 +
|  [http://www.fftw.org/ fftw, double precision]                                        ||        3.3.8                  ||                          Done      ||  -            ||
 +
|-
 +
|  [http://www.fftw.org/ fftw, single precision]                                          ||        3.3.8                  ||                          Done      ||  -            ||
 +
|-
 +
|[https://bio3d.colorado.edu/imod/download.html#Development imod] ||    4.9.12                    ||                            Done      ||  -            ||
 +
|-
 +
|  [https://www.openssl.org/ openssl]                                                        ||        1.0.2                  ||                          Done      ||  -            ||
 +
|-
 +
|  [http://grigoriefflab.janelia.org/frealign frealign]                                  ||        9.11                   ||                          Done      ||  -            ||
 +
|-
 +
|  [https://www.mrc-lmb.cam.ac.uk/kzhang/ gautomatch]                      ||        0.56                  ||                          Done      ||  -            ||
 +
|-
 +
|  [https://www.mrc-lmb.cam.ac.uk/kzhang/ gctf]                                    ||        1.18                   ||                          Done      ||  -            ||
 +
|-
 +
|  [https://modules.readthedocs.io/en/latest/ modules]                          ||        4.20                   ||                          Done      ||  -            ||
 +
|-
 +
|  [http://msg.ucsf.edu/em/software/motioncor2.html motioncor2]        ||        1.2.1                  ||                          Done      ||  -            ||
 +
|-
 +
|  [https://www.mpich.org/ mpich]                                                            ||        3.0.4                 ||                          Done      ||  -            ||
 +
|-
 +
|  [https://developer.nvidia.com/cuda-zone nvidia/cuda]                        || 8.0-GA1, 8.0-GA2, 9.0, 9.1, 9.2 ||          Done      ||  -            ||
 +
|-
 +
|  [https://www.open-mpi.org/ openmpi]                                                  ||      3.1.3[+cuda]        ||                          Done      ||  -            ||
 +
|-
 +
|  [https://www.open-mpi.org/ openmpi]                                                  ||      4.0.0[+cuda]      ||                          Done      ||  -            ||
 +
|-
 +
|  [https://www.phenix-online.org/ phenix]                                                ||      1.18.2-3874      ||                          Done      ||  -            ||
 +
|-
 +
|  [https://www2.mrc-lmb.cam.ac.uk/relion/index.php?title=Main_Page relion]            ||      3.0.4, 3.0.6 ||                          Done      ||  -            ||
 +
|-
 +
|  [https://github.com/I2PC/scipion scipion]                    ||        2.1              ||                          Done      ||  -            ||
 +
|-
 +
|  [http://sphire.mpg.de/ sphire]                                      ||        1.3                ||                          Done      ||  -            ||
 +
|-
 +
| [http://cb.csail.mit.edu/cb/topaz/ topaz]                      ||        0.2.3            ||                          Done      ||  -            ||
 +
|-
 +
|  [https://www.rbvi.ucsf.edu/chimerax/ xchimera]          ||        0.8               ||                          Done      ||  -            ||
 +
|-
 +
|  [http://xds.mpimf-heidelberg.mpg.de/ xds]                  ||        2021-03-31 ||                          Done      ||  -            ||
 +
|-
 +
|}
  
== Top Directory ( /lsc ) ==
+
= Top Directory ( /lsc ) =
  
 
Top directory for all installed software is <code>/lsc</code>
 
Top directory for all installed software is <code>/lsc</code>
  
=== Source Files ( in case you need to re-compile something ) ===
+
== Source Files ( in case you need to re-compile something ) ==
  
 
Source for all installed programs can be found under <code>/lsc/sources</code>
 
Source for all installed programs can be found under <code>/lsc/sources</code>
  
=== Dependencies for interactive 3D applications ===
+
== Dependencies for interactive 3D applications ==
  
==== VirtualGL: Running accelerated 3D graphics remotely via OpenGL ====
+
=== VirtualGL: Running accelerated 3D graphics remotely via OpenGL ===
  
 
<blockquote>Located under <code>/opt/VirtualGL</code>
 
<blockquote>Located under <code>/opt/VirtualGL</code>
Line 64: Line 127:
 
The “-d $DISPLAY” argument is there just in case something has corrupted your local environment.
 
The “-d $DISPLAY” argument is there just in case something has corrupted your local environment.
  
== How was each piece of software compiled (flags, switches) ==
+
= How was each piece of software compiled (flags, switches) =
 +
 
 +
== advx ==
  
=== ccp4 ===
+
binary distribution from [https://www.scripps.edu/tainer/arvai/adxv.html here]. Do not forget to download the [https://www.scripps.edu/tainer/arvai/adxv/fonts.tar.gz relevant fonts]
  
<pre>cd /lsc/sources/ccp4/4.7.0
+
== ccp4 ==
mkdir -p /lsc/ccp4/4.7.0
+
 
cp -r /lsc/sources/ccp4/4.7.0/* /lsc/ccp4/4.7.0
+
<pre>cd /lsc/sources/ccp4/7.0.074/
cd /lsc/ccp4/4.7.0
+
mkdir -p /lsc/ccp4/7.0.074/
 +
cp -r /lsc/sources/ccp4/7.0.074/* /lsc/ccp4/7.0.074/
 +
cd /lsc/ccp4/7.0.074/
 
./BINARY.setup
 
./BINARY.setup
# mucho bullshito, but yeah, welcome to scientific applications</pre>
+
</pre>
=== chimera ===
+
 
 +
== arp/warp ==
 +
 
 +
<pre>
 +
module load ccp4/7.0.074
 +
/lsc/ccp4/7.0.074/start
 +
cd /lsc/sources/ccp4/arp_warp_8.0
 +
./install.sh
 +
</pre>
 +
 
 +
== chimera ==
  
 
<pre>cd /lsc/sources/chimera/1.13.1
 
<pre>cd /lsc/sources/chimera/1.13.1
Line 84: Line 161:
 
# installer copies files into destination
 
# installer copies files into destination
 
&gt; Installation is done; press return.</pre>
 
&gt; Installation is done; press return.</pre>
=== ctffind4 ===
 
  
 +
 +
== cistem ==
 +
 +
<pre>cd /lsc/sources/cistem/1.0/cistem-1.0.0-beta
 +
./configure --prefix=/lsc/cistem/1.0
 +
make
 +
make install
 +
</pre>
 +
 +
== cryolo ==
 +
 +
=== 1.2.1 ===
 +
 +
<pre>
 +
cd /lsc/sources/cryolo/1.2.1/
 +
scl enable rh-python36 bash
 +
conda create --name cryolo -c anaconda python=3.6 pyqt=5 cudnn=7.1.2
 +
conda activate cryolo
 +
conda install numpy==1.14.5
 +
pip install ./cryolo-1.2.1.tar.gz
 +
</pre>
 +
 +
=== 1.3.6 ===
 +
 +
<pre>
 +
cd /lsc/sources/cryolo
 +
wget ftp://ftp.gwdg.de/pub/misc/sphire/crYOLO_V1_3_6/cryolo-1.3.6.tar.gz
 +
wget ftp://ftp.gwdg.de/pub/misc/sphire/crYOLO_BM_V1_2_2/cryoloBM-1.2.2.tar.gz
 +
tar zxvf cryolo-1.3.6.tar.gz
 +
ln -s cryolo-1.4.0 1.4.0
 +
conda env remove --name cryolo # Due to conda, cryolo cannot support previous installations
 +
conda create --prefix /lsc/sphire/1.3.6/.conda/envs/ -c anaconda python=3.6 pyqt=5 cudnn=7.1.2 cython
 +
source activate cryolo
 +
conda install numpy==1.15.4
 +
pip install cryolo-1.3.6.tar.gz[gpu]
 +
pip install cryoloBM-1.2.2.tar.gz
 +
 +
</pre>
 +
 +
<pre>conda activate /lsc/sphire/1.3.6/.conda/envs/cryolo</pre> to activate
 +
 +
=== 1.4.0 ===
 +
 +
<pre>
 +
cd /lsc/sources/cryolo
 +
wget ftp://ftp.gwdg.de/pub/misc/sphire/crYOLO_V1_4_0/cryolo-1.4.0.tar.gz
 +
wget ftp://ftp.gwdg.de/pub/misc/sphire/crYOLO_BM_V1_2_3/cryoloBM-1.2.3.tar.gz
 +
tar zxvf cryolo-1.4.0.tar.gz
 +
ln -s cryolo-1.4.0 1.4.0
 +
conda create --prefix /lsc/cryolo/1.4.0 -c anaconda python=3.6 pyqt=5 cudnn=7.1.2 cython
 +
conda activate /lsc/cryolo/1.4.0
 +
conda install numpy==1.15.4
 +
pip install cryolo-1.4.0.tar.gz[gpu]
 +
pip install cryoloBM-1.2.3.tar.gz
 +
 +
</pre>
 +
 +
<pre>conda activate /lsc/cryolo/1.4.0</pre> to activate
 +
 +
=== 1.5.0 ===
 +
 +
<pre>
 +
cd /lsc/sources/cryolo
 +
/lsc/anaconda/5.3.1/bin/conda  create --prefix=/lsc/cryolo/1.5.0 -c conda -c conda-forge -c anaconda python=3.6 pyqt=5 cudnn=7.1.2 numpy==1.14.5 cython wxPython==4.0.4 intel-openmp==2019.4
 +
/lsc/anaconda/5.3.1/bin/conda activate /lsc/cryolo/1.5.0
 +
. /lsc/anaconda/5.3.1/etc/profile.d/conda.sh
 +
 +
</pre>
 +
 +
<pre>/lsc/anaconda/5.3.1/bin/conda activate /lsc/cryolo/1.5.0</pre> to activate
 +
 +
== cryosparc ==
 +
* installation see RT #3978772
 +
* update (here for <code>sekulicgpu1</code>/<code>cryosparc2</code>)
 +
<pre>
 +
[root@cryosparc2 cryosparc]# su cryosparc_user
 +
bash-4.2$ cryosparcm backup
 +
 +
Backing up to /lsc/cryosparc/2.15.0/.../cryosparc2_database/backup/cryosparc_backup_2021_02_09_13h33.archive
 +
[...]
 +
bash-4.2$ cryosparcm update --check
 +
CryoSPARC current version v3.0.1
 +
          update starting on Tue Feb  9 13:44:40 CET 2021
 +
 +
  current version v3.0.1
 +
      new version v3.1.0
 +
 +
Update available!
 +
bash-4.2$ cryosparcm update
 +
CryoSPARC current version v3.0.1
 +
          update starting on Tue Feb  9 13:44:58 CET 2021
 +
 +
No version specified - updating to latest version.
 +
[...]
 +
</pre>
 +
 +
Notes
 +
* this updates both master and worker
 +
* if update has to be re-done for any reason (e.g., update worked on master, but something was stuck on worker, etc.), this here can be useful (but usually should not be required!):
 +
 +
<pre>
 +
bash-4.2$ cryosparcm stop
 +
CryoSPARC is running.
 +
Stopping cryoSPARC
 +
Shut down
 +
bash-4.2$ cryosparcm update --override
 +
[...]
 +
</pre>
 +
 +
== ctffind4 ==
 +
 +
binary blob here https://grigoriefflab.umassmed.edu/ctffind4
 +
 +
== 4.1.10 ==
 
<pre>cd /lsc/sources/ctffind/4.1.10/ctffind-4.1.10
 
<pre>cd /lsc/sources/ctffind/4.1.10/ctffind-4.1.10
 
./configure --prefix=/lsc/ctffind4/4.1.10 --enable-latest-instruction-set
 
./configure --prefix=/lsc/ctffind4/4.1.10 --enable-latest-instruction-set
 
make -j 64 all
 
make -j 64 all
 
make -j 64 install</pre>
 
make -j 64 install</pre>
=== eman2 ===
+
 
 +
== 4.1.13 ==
 +
 
 +
This is not working: this new version of ctf find requires the wxWidgets and /usr/bin/wx-config --query-version returns 2.8 (needs version 3.0.0 and above). Will use static binary instead for now.
 +
 
 +
Suspended till we upgrade the machine to something newer
 +
 
 +
<pre>yum install wxGTK3-webview.x86_64 wxGTK3-media.x86_64 wxGTK3-gl.x86_64 wxGTK3.x86_64 wxGTK-media.x86_64 wxGTK-gl.x86_64 wxGTK.x86_64 wxBase3.x86_64 wxBase.x86_64 mathgl-wx-devel.x86_64 mathgl-wx.x86_64 erlang-wx.x86_64 compat-wxGTK3-gtk2-media.x86_64 compat-wxGTK3-gtk2-gl.x86_64 compat-wxGTK3-gtk2.x86_64 compat-wxBase3-gtk2.x86_64
 +
cd /lsc/sources/ctffind/4.1.13/ctffind-4.1.13
 +
/configure --prefix=/lsc/ctffind/4.1.13 --enable-latest-instruction-set  --enable-openmp --with-wx-config=/usr/bin/wx-config --with-fftw-dir=/lsc/fftw/3.3.8
 +
make -j 64 all
 +
make -j 64 install</pre>
 +
 
 +
== 4.1.13 + cuda ==
 +
 
 +
binary distribution
 +
 
 +
== eman2 ==
 +
 
 +
=== 2.2 ===
  
 
<pre>cd /lsc/sources/eman2/2.2/
 
<pre>cd /lsc/sources/eman2/2.2/
Line 96: Line 305:
 
&gt; EMAN2 will now be installed into this location:
 
&gt; EMAN2 will now be installed into this location:
 
&gt; [/root/EMAN2] &gt;&gt;&gt; /lsc/eman2/2.2
 
&gt; [/root/EMAN2] &gt;&gt;&gt; /lsc/eman2/2.2
# installer does the rest of the work, re-installing a bunch of python modules
+
# installer does the rest of the work, re-installing a bunch of python modules via anaconda
via anaconda, cuz apparently the built-in package managers are less elitist
 
# *sigh* scientific computation, alright
 
 
&gt; Do you wish the installer to prepend the EMAN2 install location
 
&gt; Do you wish the installer to prepend the EMAN2 install location
 
&gt; to PATH in your /root/.bashrc ? [yes|no]
 
&gt; to PATH in your /root/.bashrc ? [yes|no]
Line 105: Line 312:
 
&gt; export PATH=/opt/eman/2.2/bin:$PATH
 
&gt; export PATH=/opt/eman/2.2/bin:$PATH
 
# covered in the relevant environment module</pre>
 
# covered in the relevant environment module</pre>
=== external/openssl ===
+
 
 +
=== 2.3 ===
 +
 
 +
<pre>cd /lsc/sources/eman2/2.3/
 +
./eman2.3.linux64.sh
 +
&gt; EMAN2 will now be installed into this location:
 +
&gt; [/root/EMAN2] &gt;&gt;&gt; /lsc/eman2/2.3
 +
# installer does the rest of the work, re-installing a bunch of python modules via anaconda
 +
&gt; Do you wish the installer to prepend the EMAN2 install location
 +
&gt; to PATH in your /root/.bashrc ? [yes|no]
 +
&gt; [no] &gt;&gt;&gt; no
 +
&gt; You may wish to edit your .bashrc to prepend the EMAN2 install location to PATH:
 +
&gt; export PATH=/opt/eman/2.3/bin:$PATH
 +
# covered in the relevant environment module</pre>
 +
 
 +
=== 2.31 ===
 +
 
 +
<pre>cd /lsc/sources/eman2/2.31/
 +
eman2.31_sphire1.3_huge.linux64.sh
 +
&gt; EMAN2 will now be installed into this location:
 +
&gt; [/root/EMAN2] &gt;&gt;&gt; /lsc/eman2/2.31
 +
# installer does the rest of the work, re-installing a bunch of python modules via anaconda
 +
&gt; Do you wish the installer to prepend the EMAN2 install location
 +
&gt; to PATH in your /root/.bashrc ? [yes|no]
 +
&gt; [no] &gt;&gt;&gt; no
 +
&gt; You may wish to edit your .bashrc to prepend the EMAN2 install location to PATH:
 +
&gt; export PATH=/opt/eman/2.31/bin:$PATH
 +
# covered in the relevant environment module</pre>
 +
 
 +
== Dynamo ==
 +
 
 +
https://wiki.dynamo.biozentrum.unibas.ch/w/index.php/Main_Page
 +
 
 +
=== 1401 ===
 +
 
 +
== external/fftw ==
 +
 
 +
=== 2.1.5 ===
 +
<pre>
 +
cd /lsc/sources/fftw/2.1.5/fftw-2.1.5
 +
./configure --prefix=/lsc/fftw/2.1.5 --enable-threads --enable-mpi --enable-i386-hacks
 +
make
 +
make install
 +
</pre>
 +
 
 +
=== 3.3.8 ===
 +
<pre>
 +
cd /lsc/sources/fftw/3.3.8/fftw-3.3.8
 +
./configure CC=/usr/bin/gcc-4.8.5 --prefix=/lsc/fftw/3.3.8 --enable-single --enable-sse --enable-sse2 --enable-avx --enable-avx2 --enable-avx-128-fma --enable-generic-simd128 --enable-generic-simd256 --enable-fma --enable-mpi --enable-threads --with-g77-wrappers --with-combined-threads
 +
make
 +
make install
 +
</pre>
 +
 
 +
== external/openssl ==
  
 
<pre>cd /lsc/sources/openssl/openssl-1.0.2o/
 
<pre>cd /lsc/sources/openssl/openssl-1.0.2o/
Line 112: Line 372:
 
make -j 64 install
 
make -j 64 install
 
#done</pre>
 
#done</pre>
=== frealign ===
+
 
 +
== frealign ==
  
 
<pre>cd /lsc/sources/frealign/9.11/frealign_v9.11
 
<pre>cd /lsc/sources/frealign/9.11/frealign_v9.11
Line 120: Line 381:
 
# you just need to add the relevant bin path for the application to work
 
# you just need to add the relevant bin path for the application to work
 
# the relevant module covers the details</pre>
 
# the relevant module covers the details</pre>
=== gautomatch ===
+
 
 +
== gautomatch ==
  
 
<pre>cd /lsc/sources/gautomatch/Gautomatch_v0.53
 
<pre>cd /lsc/sources/gautomatch/Gautomatch_v0.53
 
mkdir /lsc/gautomatch/0.53
 
mkdir /lsc/gautomatch/0.53
 
cp -r /lsc/sources/gautomatch/Gautomatch_v0.53 /lsc/gautomatch/0.53</pre>
 
cp -r /lsc/sources/gautomatch/Gautomatch_v0.53 /lsc/gautomatch/0.53</pre>
=== gctf ===
+
 
 +
== gctf ==
  
 
<pre>cd /lsc/sources/gctf/1.18
 
<pre>cd /lsc/sources/gctf/1.18
Line 134: Line 397:
 
./configure --prefix=/lsc/modules --enable-doc-install --enable-example-modulefiles --enable-compat-version --enable-versioning --enable-quarantine-support --enable-auto-handling  
 
./configure --prefix=/lsc/modules --enable-doc-install --enable-example-modulefiles --enable-compat-version --enable-versioning --enable-quarantine-support --enable-auto-handling  
 
# since --enable-versioning is turned on, the module will autoversion itself and place all its files under
 
# since --enable-versioning is turned on, the module will autoversion itself and place all its files under
# /lsc/modules/4.2.0</pre>
+
# /lsc/modules/4.2.0
=== mpich ===
+
</pre>
 +
 
 +
== imod ==
 +
 
 +
<pre> Follow instructions as described in RT 3509886
 +
</pre>
 +
 
 +
== mpich ==
  
 
<pre>default loaded by CentOS 7.
 
<pre>default loaded by CentOS 7.
 
Environmental module was altered to make sure that it conflicts with the openmpi modules</pre>
 
Environmental module was altered to make sure that it conflicts with the openmpi modules</pre>
=== motioncor2 ===
+
 
 +
== motioncor2 ==
  
 
<pre>cd /lsc/sources/motioncor2/1.2.1
 
<pre>cd /lsc/sources/motioncor2/1.2.1
Line 146: Line 417:
 
ln -s /lsc/motioncor2/1.2.1/MotionCor2_1.2.1-Cuda80 /lsc/motioncor2/1.2.1/bin/motioncor2
 
ln -s /lsc/motioncor2/1.2.1/MotionCor2_1.2.1-Cuda80 /lsc/motioncor2/1.2.1/bin/motioncor2
 
# environmental module takes care of setting up the bin path </pre>
 
# environmental module takes care of setting up the bin path </pre>
=== nvidia/driver ===
+
 
 +
== nvidia/driver ==
  
 
<pre>cd /lsc/sources/nvidia/drivers/396.54;
 
<pre>cd /lsc/sources/nvidia/drivers/396.54;
 
./NVIDIA-Linux-x86_64-396.54.run</pre>
 
./NVIDIA-Linux-x86_64-396.54.run</pre>
=== nvidia/cuda ===
+
 
 +
 
 +
Update: In order to combat the issue with creating remote OpenGL context (X11 forwarding over ssh), I upgraded the driver to 415.23 but had no luck. In nvidia-smi, the new driver [https://stackoverflow.com/questions/53422407/different-cuda-versions-shown-by-nvcc-and-nvidia-smi reports that is nvidia CUDA 10 capable], which is fine as it can use the CUDA 8 run-time environment, as [https://docs.nvidia.com/deploy/cuda-compatibility/index.html NVIDIA guarantees binary compatibility]
 +
 
 +
== nvidia/cuda ==
  
 
CUDA frameworks installed:
 
CUDA frameworks installed:
Line 159: Line 435:
 
* 9.1
 
* 9.1
 
* 9.2
 
* 9.2
 +
* 10.2
  
<pre>Default CUDA framework for all applications is 8.0GA1</pre>
+
<pre>Default CUDA framework for most applications is 8.0GA1</pre>
  
NVidia Driver installed is 396.54, short-term support as of 2018-10-31, as per above.
+
The Nvidia driver version for the kernel installed is Driver Version: 440.44 of CUDA Version: 10.2
 
 
Default driver installed with CUDA 8.0-GA1 is 375.64 (not installed, obviously)
 
  
 
During loading different CUDA versions, NVIDIA driver remains the same. Things seem to work, but may require future additional detailed testing
 
During loading different CUDA versions, NVIDIA driver remains the same. Things seem to work, but may require future additional detailed testing
Line 195: Line 470:
 
./9.2.148_396.37_linux --silent --toolkit=/lsc/nvidia/cuda/9.2 --samples --samplespath=/lsc/nvidia/cuda/9.2/samples --run-nvidia-xconfig --tmpdir=/tmp</pre>
 
./9.2.148_396.37_linux --silent --toolkit=/lsc/nvidia/cuda/9.2 --samples --samplespath=/lsc/nvidia/cuda/9.2/samples --run-nvidia-xconfig --tmpdir=/tmp</pre>
  
=== openmpi ===
+
==== 10.2 ====
 +
 
 +
<pre>mkdir /lsc/nvidia/cuda/10.2;
 +
wget http://developer.download.nvidia.com/compute/cuda/10.2/Prod/local_installers/cuda_10.2.89_440.33.01_linux.run;
 +
sh ./cuda_10.2.89_440.33.01_linux.run --silent --toolkitpath=/lsc/nvidia/cuda/10.2 --toolkit=/lsc/nvidia/cuda/10.2 --samples --samplespath=/lsc/nvidia/cuda/10.2/samples --run-nvidia-xconfig --tmpdir=/storage/scratch/tmp</pre>
 +
 
 +
== openmpi ==
  
 
<pre>cd /lsc/sources/openmpi/3.1.2;
 
<pre>cd /lsc/sources/openmpi/3.1.2;
 
./configure --prefix /lsc/openmpi/3.1.2 --enable-binaries --enable-mpi-fortran --with-cuda=/lsc/nvidia/cuda/8.0-GA1 --with-devel-headers </pre>
 
./configure --prefix /lsc/openmpi/3.1.2 --enable-binaries --enable-mpi-fortran --with-cuda=/lsc/nvidia/cuda/8.0-GA1 --with-devel-headers </pre>
==== Default: 3.1.2 ====
+
=== Default: 3.1.2 ===
  
 
<pre>The default version loaded is the stable-as-of 2018-10-31 3.1.2</pre>
 
<pre>The default version loaded is the stable-as-of 2018-10-31 3.1.2</pre>
=== phenix ===
 
  
<pre>cd /lsc/sources/phenix/1.14/phenix-installer-1.14-3260-intel-linux-2.6-x86_64-centos6 ;
+
== phenix ==
./install --prefix=/lsc/phenix/1.14-3260 --openmp --makedirs </pre>
+
 
=== relion ===
+
<pre>cd /lsc/sources/phenix/$version/phenix-installer-$version-intel-linux-2.6-x86_64-centos6 ;
 +
./install --prefix=/lsc/phenix --openmp --makedirs </pre>
 +
 
 +
== relion ==
  
==== 2.1 ====
+
=== 2.1 ===
  
 
<pre>cd /lsc/sources/relion/2.1-mpich ;
 
<pre>cd /lsc/sources/relion/2.1-mpich ;
Line 215: Line 498:
 
make -j 64
 
make -j 64
 
make -j 64 install</pre>
 
make -j 64 install</pre>
==== 3.0b ====
+
 
 +
=== 3.0b ===
  
 
<pre>cd /lsc/sources/relion/3.0b-mpich ;
 
<pre>cd /lsc/sources/relion/3.0b-mpich ;
Line 222: Line 506:
 
make -j 64
 
make -j 64
 
make -j 64 install</pre>
 
make -j 64 install</pre>
=== sphire ===
 
  
 +
=== 3.0.4 ===
 +
 +
<pre>
 +
cd /lsc/sources/relion/3.0.4 ;
 +
mkdir build && cd build;
 +
cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/3.0.4 --DGUI=ON -DCUDA=ON -DCudaTexture=ON -DFORCE_OWN_TBB=ON -DFORCE_OWN_FLTK=ON -DFORCE_OWN_FFTW=ON -DCUDA_ARCH=61 -DBUILD_SHARED_LIBS=ON ..
 +
make -j 64
 +
make install
 +
</pre>
 +
 +
=== 3.0.6 ===
 +
 +
<pre>
 +
cd /lsc/sources/relion/3.0.6 ;
 +
mkdir build && cd build;
 +
cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/3.0.6 --DGUI=ON -DCUDA=ON -DCudaTexture=ON -DFORCE_OWN_TBB=ON -DFORCE_OWN_FLTK=ON -DFORCE_OWN_FFTW=ON -DCUDA_ARCH=61 -DBUILD_SHARED_LIBS=ON ..
 +
make -j 64
 +
make install
 +
</pre>
 +
 +
=== 3.0.8 ===
 +
 +
<pre>
 +
cd /lsc/sources/relion/3.0.8 ;
 +
mkdir build && cd build;
 +
cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/3.0.8 --DGUI=ON -DCUDA=ON -DCudaTexture=ON -DFORCE_OWN_TBB=ON -DFORCE_OWN_FLTK=ON -DFORCE_OWN_FFTW=ON -DCUDA_ARCH=61 -DBUILD_SHARED_LIBS=ON ..
 +
make -j 64
 +
make install
 +
</pre>
 +
 +
=== 3.1-beta ===
 +
 +
<pre>
 +
cd /lsc/sources/relion/relion.git
 +
git checkout ver3.1
 +
mkdir build && cd build
 +
module load relion/3.1-beta
 +
cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/3.1-beta --DGUI=ON -DCUDA=ON -DCudaTexture=ON -DFORCE_OWN_TBB=ON -DFORCE_OWN_FLTK=ON -DFORCE_OWN_FFTW=ON -DCUDA_ARCH=61 -DBUILD_SHARED_LIBS=ON ..
 +
make -j 64
 +
make install
 +
</pre>
 +
 +
=== 3.1 ===
 +
 +
<pre>
 +
cd /lsc/sources/relion/3.1
 +
mkdir build && cd build
 +
module load relion/3.1
 +
cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/3.1 --DGUI=ON -DCUDA=ON -DCudaTexture=ON -DFORCE_OWN_TBB=ON -DFORCE_OWN_FLTK=ON -DFORCE_OWN_FFTW=ON -DCUDA_ARCH=61 -DBUILD_SHARED_LIBS=ON ..
 +
make -j 64
 +
make install
 +
</pre>
 +
 +
 +
List of CUDA architectures here : https://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/
 +
 +
== scipion ==
 +
 +
=== 2.0 ===
 +
 +
Note: The comments for scipion and the Sekulic GPU server are applicable to module scipion/2.0source. Its installation is described in RT 3879628. The instructions below are just basic steps and do not achieve production installation.
 +
 +
==== Cloning/Getting software ====
 +
mkdir -p /lsc/scipion/2.0
 +
git clone https://github.com/I2PC/scipion.git
 +
cd scipion
 +
 +
==== Dependencies ====
 +
yum install -y gcc gcc-g++ cmake java-1.8.0-openjdk-devel.x86_64 libXft-devel.x86_64 openssl-devel.x86_64 libXext-devel.x86_64  libxml++.x86_64 libquadmath-devel.x86_64  libxslt.x86_64 openmpi-devel.x86_64  gsl-devel.x86_64  libX11.x86_64  gcc-gfortran.x86_64 git
 +
 +
==== Configuring ====
 +
 +
./scipion config
 +
scipion installp -p scipion-em-relion
 +
 +
 +
=== 2.1 ===
 +
 +
Note: same as above for xmipp2
 +
 +
==== Cloning/Getting software ====
 +
<pre>
 +
mkdir -p /lsc/scipion/2.0
 +
git clone https://github.com/I2PC/scipion.git
 +
cd scipion
 +
</pre>
 +
 +
==== Dependencies ====
 +
yum install -y gcc gcc-g++ cmake java-1.8.0-openjdk-devel.x86_64 libXft-devel.x86_64 openssl-devel.x86_64 libXext-devel.x86_64  libxml++.x86_64 libquadmath-devel.x86_64  libxslt.x86_64 openmpi-devel.x86_64  gsl-devel.x86_64  libX11.x86_64  gcc-gfortran.x86_64 git
 +
 +
==== Configuring and installing modules from the command-line ====
 +
 +
<pre>./scipion config
 +
./scipion installp --help | awk -p '{ print $1 }' | grep 'scipion-em' | grep -E -v 'Configuration|xmipp2' > modules.out
 +
./scipion installp -p scipion-em-xmipp
 +
for module in $( < module.out ); do ./scipion installp -p $module ); done
 +
rm module.out
 +
</pre>
 +
 +
 +
see [https://rt.uio.no/Ticket/Display.html?id=4049861 ticket 4049861]: some modules cannot compile due to python2 issues:
 +
 +
<pre>
 +
./scipion installp -p scipion-em-cryodrgn 3.0.0b4 # python2 issues
 +
./scipion installp -p scipion-em-dynamo 3.1.4 # python2 issues
 +
./scipion installp -p scipion-em-fsc3d 3.0.5 # python2 issues
 +
./scipion installp -p scipion-em-imod 3.0.3 # python2 issues
 +
./scipion installp -p scipion-em-novactf 3.0.0 # python2 issues
 +
./scipion installp -p scipion-em-pyseg 0.2 # python2 issues
 +
./scipion installp -p scipion-em-sidesplitter 3.0.3 # python2 issues
 +
./scipion installp -p scipion-em-tomo 3.0.5 # python2 issues
 +
</pre>
 +
 +
== sphire ==
 +
 +
=== 1.1 ===
 
<pre>cd /lsc/sources/sphire/1.1 ;
 
<pre>cd /lsc/sources/sphire/1.1 ;
 
./sphire_1_1_linux.sh
 
./sphire_1_1_linux.sh
 
# follow questions, install under /lsc/sphire/1.1</pre>
 
# follow questions, install under /lsc/sphire/1.1</pre>
 +
 +
=== 1.3 ===
 +
 +
sphire 1.3 is included with the installation of eman 2.31.
 +
 +
<pre>module load eman 2.31</pre>
 +
 +
== topaz ==
 +
 +
=== 0.2.3 ===
 +
<pre>
 +
[root@intaristotle ~]# conda create --prefix=/lsc/topaz/0.2.3
 +
Collecting package metadata: done
 +
Solving environment: done
 +
 +
## Package Plan ##
 +
 +
  environment location: /lsc/topaz/0.2.3
 +
 +
 +
 +
Proceed ([y]/n)? y
 +
 +
Preparing transaction: done
 +
Verifying transaction: done
 +
Executing transaction: done
 +
#
 +
# To activate this environment, use
 +
#
 +
#    $ conda activate /lsc/topaz/0.2.3
 +
#
 +
# To deactivate an active environment, use
 +
#
 +
#    $ conda deactivate
 +
 +
[root@intaristotle ~]# conda activate /lsc/topaz/0.2.3
 +
(/lsc/topaz/0.2.3) [root@intaristotle ~]# module load nvidia/cuda/9.2
 +
(/lsc/topaz/0.2.3) [root@intaristotle ~]# conda install topaz cudatoolkit=9.2 -c tbepler -c pytorch
 +
Collecting package metadata: done
 +
Solving environment: done
 +
 +
## Package Plan ##
 +
 +
  environment location: /lsc/topaz/0.2.3
 +
 +
  added / updated specs:
 +
    - cudatoolkit=9.2
 +
    - topaz
 +
 +
 +
The following packages will be downloaded:
 +
 +
    package                    |            build
 +
    ---------------------------|-----------------
 +
    certifi-2020.4.5.1        |          py38_0        159 KB
 +
    freetype-2.9.1            |      h8a8886c_1        822 KB
 +
    intel-openmp-2020.0        |              166        916 KB
 +
    joblib-0.14.1              |            py_0        202 KB
 +
    jpeg-9b                    |      h024ee3a_2        248 KB
 +
    libpng-1.6.37              |      hbc83047_0        364 KB
 +
    libtiff-4.1.0              |      h2733197_0        589 KB
 +
    mkl-2020.0                |              166      202.1 MB
 +
    mkl-service-2.3.0          |  py38he904b0f_0          68 KB
 +
    mkl_fft-1.0.15            |  py38ha843d7b_0        178 KB
 +
    mkl_random-1.1.0          |  py38h962f231_0        398 KB
 +
    ninja-1.9.0                |  py38hfd86e86_0        1.6 MB
 +
    numpy-1.18.1              |  py38h4f9e942_0          5 KB
 +
    numpy-base-1.18.1          |  py38hde5b4d6_1        5.3 MB
 +
    olefile-0.46              |            py_0          33 KB
 +
    pandas-1.0.3              |  py38h0573a6f_0        11.5 MB
 +
    pillow-7.0.0              |  py38hb39fc2d_0        664 KB
 +
    pip-20.0.2                |          py38_1        1.9 MB
 +
    python-3.8.2              |      hcf32534_0        57.8 MB
 +
    python-dateutil-2.8.1      |            py_0        224 KB
 +
    pytorch-1.5.0              |py3.8_cuda9.2.148_cudnn7.6.3_0      344.4 MB  pytorch
 +
    pytz-2019.3                |            py_0        231 KB
 +
    scikit-learn-0.22.1        |  py38hd81dba3_0        7.3 MB
 +
    scipy-1.4.1                |  py38h0b6359f_0        19.1 MB
 +
    setuptools-46.1.3          |          py38_0        656 KB
 +
    six-1.14.0                |          py38_0          27 KB
 +
    topaz-0.2.3                |            py_0        26.0 MB  tbepler
 +
    torchvision-0.6.0          |        py38_cu92        11.7 MB  pytorch
 +
    wheel-0.34.2              |          py38_0          49 KB
 +
    zstd-1.3.7                |      h0b5b093_0        887 KB
 +
    ------------------------------------------------------------
 +
                                          Total:      695.2 MB
 +
 +
The following NEW packages will be INSTALLED:
 +
 +
  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
 +
  blas              pkgs/main/linux-64::blas-1.0-mkl
 +
  ca-certificates    pkgs/main/linux-64::ca-certificates-2020.1.1-0
 +
  certifi            pkgs/main/linux-64::certifi-2020.4.5.1-py38_0
 +
  cudatoolkit        pkgs/main/linux-64::cudatoolkit-9.2-0
 +
  freetype          pkgs/main/linux-64::freetype-2.9.1-h8a8886c_1
 +
  intel-openmp      pkgs/main/linux-64::intel-openmp-2020.0-166
 +
  joblib            pkgs/main/noarch::joblib-0.14.1-py_0
 +
  jpeg              pkgs/main/linux-64::jpeg-9b-h024ee3a_2
 +
  ld_impl_linux-64  pkgs/main/linux-64::ld_impl_linux-64-2.33.1-h53a641e_7
 +
  libedit            pkgs/main/linux-64::libedit-3.1.20181209-hc058e9b_0
 +
  libffi            pkgs/main/linux-64::libffi-3.2.1-hd88cf55_4
 +
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
 +
  libgfortran-ng    pkgs/main/linux-64::libgfortran-ng-7.3.0-hdf63c60_0
 +
  libpng            pkgs/main/linux-64::libpng-1.6.37-hbc83047_0
 +
  libstdcxx-ng      pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
 +
  libtiff            pkgs/main/linux-64::libtiff-4.1.0-h2733197_0
 +
  mkl                pkgs/main/linux-64::mkl-2020.0-166
 +
  mkl-service        pkgs/main/linux-64::mkl-service-2.3.0-py38he904b0f_0
 +
  mkl_fft            pkgs/main/linux-64::mkl_fft-1.0.15-py38ha843d7b_0
 +
  mkl_random        pkgs/main/linux-64::mkl_random-1.1.0-py38h962f231_0
 +
  ncurses            pkgs/main/linux-64::ncurses-6.2-he6710b0_1
 +
  ninja              pkgs/main/linux-64::ninja-1.9.0-py38hfd86e86_0
 +
  numpy              pkgs/main/linux-64::numpy-1.18.1-py38h4f9e942_0
 +
  numpy-base        pkgs/main/linux-64::numpy-base-1.18.1-py38hde5b4d6_1
 +
  olefile            pkgs/main/noarch::olefile-0.46-py_0
 +
  openssl            pkgs/main/linux-64::openssl-1.1.1g-h7b6447c_0
 +
  pandas            pkgs/main/linux-64::pandas-1.0.3-py38h0573a6f_0
 +
  pillow            pkgs/main/linux-64::pillow-7.0.0-py38hb39fc2d_0
 +
  pip                pkgs/main/linux-64::pip-20.0.2-py38_1
 +
  python            pkgs/main/linux-64::python-3.8.2-hcf32534_0
 +
  python-dateutil    pkgs/main/noarch::python-dateutil-2.8.1-py_0
 +
  pytorch            pytorch/linux-64::pytorch-1.5.0-py3.8_cuda9.2.148_cudnn7.6.3_0
 +
  pytz              pkgs/main/noarch::pytz-2019.3-py_0
 +
  readline          pkgs/main/linux-64::readline-8.0-h7b6447c_0
 +
  scikit-learn      pkgs/main/linux-64::scikit-learn-0.22.1-py38hd81dba3_0
 +
  scipy              pkgs/main/linux-64::scipy-1.4.1-py38h0b6359f_0
 +
  setuptools        pkgs/main/linux-64::setuptools-46.1.3-py38_0
 +
  six                pkgs/main/linux-64::six-1.14.0-py38_0
 +
  sqlite            pkgs/main/linux-64::sqlite-3.31.1-h62c20be_1
 +
  tk                pkgs/main/linux-64::tk-8.6.8-hbc83047_0
 +
  topaz              tbepler/noarch::topaz-0.2.3-py_0
 +
  torchvision        pytorch/linux-64::torchvision-0.6.0-py38_cu92
 +
  wheel              pkgs/main/linux-64::wheel-0.34.2-py38_0
 +
  xz                pkgs/main/linux-64::xz-5.2.5-h7b6447c_0
 +
  zlib              pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3
 +
  zstd              pkgs/main/linux-64::zstd-1.3.7-h0b5b093_0
 +
 +
 +
Proceed ([y]/n)? y
 +
 +
 +
Downloading and Extracting Packages
 +
mkl-2020.0          | 202.1 MB  | ################################################################################################# | 100%
 +
pytorch-1.5.0        | 344.4 MB  | ################################################################################################# | 100%
 +
jpeg-9b              | 248 KB    | ################################################################################################# | 100%
 +
python-dateutil-2.8. | 224 KB    | ################################################################################################# | 100%
 +
pip-20.0.2          | 1.9 MB    | ################################################################################################# | 100%
 +
joblib-0.14.1        | 202 KB    | ################################################################################################# | 100%
 +
certifi-2020.4.5.1  | 159 KB    | ################################################################################################# | 100%
 +
python-3.8.2        | 57.8 MB  | ################################################################################################# | 100%
 +
setuptools-46.1.3    | 656 KB    | ################################################################################################# | 100%
 +
pytz-2019.3          | 231 KB    | ################################################################################################# | 100%
 +
mkl_fft-1.0.15      | 178 KB    | ################################################################################################# | 100%
 +
six-1.14.0          | 27 KB    | ################################################################################################# | 100%
 +
numpy-1.18.1        | 5 KB      | ################################################################################################# | 100%
 +
libtiff-4.1.0        | 589 KB    | ################################################################################################# | 100%
 +
zstd-1.3.7          | 887 KB    | ################################################################################################# | 100%
 +
ninja-1.9.0          | 1.6 MB    | ################################################################################################# | 100%
 +
scikit-learn-0.22.1  | 7.3 MB    | ################################################################################################# | 100%
 +
pillow-7.0.0        | 664 KB    | ################################################################################################# | 100%
 +
numpy-base-1.18.1    | 5.3 MB    | ################################################################################################# | 100%
 +
freetype-2.9.1      | 822 KB    | ################################################################################################# | 100%
 +
scipy-1.4.1          | 19.1 MB  | ################################################################################################# | 100%
 +
pandas-1.0.3        | 11.5 MB  | ################################################################################################# | 100%
 +
mkl_random-1.1.0    | 398 KB    | ################################################################################################# | 100%
 +
mkl-service-2.3.0    | 68 KB    | ################################################################################################# | 100%
 +
libpng-1.6.37        | 364 KB    | ################################################################################################# | 100%
 +
wheel-0.34.2        | 49 KB    | ################################################################################################# | 100%
 +
torchvision-0.6.0    | 11.7 MB  | ################################################################################################# | 100%
 +
topaz-0.2.3          | 26.0 MB  | ################################################################################################# | 100%
 +
olefile-0.46        | 33 KB    | ################################################################################################# | 100%
 +
intel-openmp-2020.0  | 916 KB    | ################################################################################################# | 100%
 +
Preparing transaction: done
 +
Verifying transaction: done
 +
Executing transaction: done
 +
(/lsc/topaz/0.2.3) [root@intaristotle ~]# topaz
 +
Traceback (most recent call last):
 +
  File "/lsc/topaz/0.2.3/bin/topaz", line 6, in <module>
 +
    from pkg_resources import load_entry_point
 +
  File "/lsc/topaz/0.2.3/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3259, in <module>
 +
    def _initialize_master_working_set():
 +
  File "/lsc/topaz/0.2.3/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3242, in _call_aside
 +
    f(*args, **kwargs)
 +
  File "/lsc/topaz/0.2.3/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3271, in _initialize_master_working_set
 +
    working_set = WorkingSet._build_master()
 +
  File "/lsc/topaz/0.2.3/lib/python3.8/site-packages/pkg_resources/__init__.py", line 584, in _build_master
 +
    ws.require(__requires__)
 +
  File "/lsc/topaz/0.2.3/lib/python3.8/site-packages/pkg_resources/__init__.py", line 901, in require
 +
    needed = self.resolve(parse_requirements(requirements))
 +
  File "/lsc/topaz/0.2.3/lib/python3.8/site-packages/pkg_resources/__init__.py", line 787, in resolve
 +
    raise DistributionNotFound(req, requirers)
 +
pkg_resources.DistributionNotFound: The 'future' distribution was not found and is required by torch
 +
(/lsc/topaz/0.2.3) [root@intaristotle ~]# pip install future
 +
Collecting future
 +
  Downloading future-0.18.2.tar.gz (829 kB)
 +
    |████████████████████████████████| 829 kB 4.3 MB/s
 +
Building wheels for collected packages: future
 +
  Building wheel for future (setup.py) ... done
 +
  Created wheel for future: filename=future-0.18.2-py3-none-any.whl size=491058 sha256=f1d7892b69b66f8a96624ddc534bfc5dcf52e8042aa760743760f036db892c26
 +
  Stored in directory: /root/.cache/pip/wheels/8e/70/28/3d6ccd6e315f65f245da085482a2e1c7d14b90b30f239e2cf4
 +
Successfully built future
 +
Installing collected packages: future
 +
Successfully installed future-0.18.2
 +
(/lsc/topaz/0.2.3) [root@intaristotle ~]# topaz
 +
usage: topaz [-h] [--version] <command> ...
 +
topaz: error: the following arguments are required: <command>
 +
 +
</pre>
 +
 +
=== 0.2.4 ===
 +
* Installed on <code>sekulicgpu1</code> and <code>cryosparc2</code>
 +
 +
<pre>
 +
[root@sekulicgpu1 topaz]# mkdir /lsc/venvs
 +
[root@sekulicgpu1 topaz]# chown cryosparc_user /lsc/venvs
 +
[root@sekulicgpu1 topaz]# su cryosparc_user
 +
[cryosparc_user@sekulicgpu1 topaz]$ /lsc/python/3.6.11/bin/python3.6 -m venv /lsc/venvs/topaz
 +
[cryosparc_user@sekulicgpu1 topaz]$ source /lsc/venvs/topaz/bin/activate
 +
(topaz) [cryosparc_user@sekulicgpu1 topaz]$ mkdir /lsc/venvs/tmp
 +
(topaz) [cryosparc_user@sekulicgpu1 topaz]$ env TMPDIR=/lsc/venvs/tmp pip3 install topaz-em
 +
[...]
 +
Successfully installed dataclasses-0.8 future-0.18.2 joblib-1.0.1 numpy-1.19.5 pandas-1.1.5 pillow-8.1.0 python-dateutil-2.8.1 pytz-2021.1 scikit-learn-0.24.1 scipy-1.5.4 six-1.15.0 threadpoolctl-2.1.0 topaz-em-0.2.4 torch-1.7.1 torchvision-0.8.2 typing-extensions-3.7.4.3
 +
(topaz) [cryosparc_user@sekulicgpu1 topaz]$ rmdir /lsc/venvs/tmp
 +
</pre>
 +
 +
On <code>cryosparc2</code>, first install Python 3.6.11 (for consistency with <code>sekulicgpu1</code>), depends on 3 <code>-devel</code> packages:
 +
<pre>
 +
[root@cryosparc2 ~]# yum install zlib-devel openssl-devel bzip2-devel
 +
[root@cryosparc2 ~]# mkdir -p /lsc/sources/python /lsc/python/3.6.11
 +
[root@cryosparc2 ~]# chown cryosparc_user /lsc/sources/python /lsc/python/3.6.11
 +
[root@cryosparc2 ~]# su cryosparc_user
 +
bash-4.2$ cd /lsc/sources/python/
 +
bash-4.2$ wget https://www.python.org/ftp/python/3.6.11/Python-3.6.11.tar.xz
 +
bash-4.2$ tar xf Python-3.6.11.tar.xz
 +
bash-4.2$ cd Python-3.6.11
 +
bash-4.2$ ./configure --prefix=/lsc/python/3.6.11 --enable-optimizations && make && make install
 +
</pre>
 +
 +
and then same steps for <code>topaz</code> as on <code>sekulicgpu1</code>:
 +
<pre>
 +
[root@cryosparc2 ~]# mkdir /lsc/venvs
 +
[root@cryosparc2 ~]# chown cryosparc_user /lsc/venvs
 +
[root@cryosparc2 ~]# su cryosparc_user
 +
bash-4.2$ /lsc/python/3.6.11/bin/python3.6 -m venv /lsc/venvs/topaz
 +
bash-4.2$ source /lsc/venvs/topaz/bin/activate
 +
(topaz) bash-4.2$ mkdir /lsc/venvs/tmp
 +
(topaz) bash-4.2$ env TMPDIR=/lsc/venvs/tmp pip3 install topaz-em
 +
[...]
 +
Successfully installed dataclasses-0.8 future-0.18.2 joblib-1.0.1 numpy-1.19.5 pandas-1.1.5 pillow-8.1.0 python-dateutil-2.8.1 pytz-2021.1 scikit-learn-0.24.1 scipy-1.5.4 six-1.15.0 threadpoolctl-2.1.0 topaz-em-0.2.4 torch-1.7.1 torchvision-0.8.2 typing-extensions-3.7.4.3
 +
(topaz) bash-4.2$ rmdir /lsc/venvs/tmp
 +
</pre>
 +
 +
path to topaz: <code>/lsc/venvs/topaz/bin/topaz</code>
 +
 +
verify GPU support:
 +
<pre>
 +
[haroldgu@sekulicgpu1 ~]$ /lsc/venvs/topaz/bin/python -c 'import torch ; print(torch.cuda.is_available())'
 +
True
 +
</pre>
 +
 
== How is each piece of software run ( really basic, just front UI, text or graphics) ==
 
== How is each piece of software run ( really basic, just front UI, text or graphics) ==
  
Line 243: Line 902:
 
<pre>module load eman
 
<pre>module load eman
 
eman2.py</pre>
 
eman2.py</pre>
 +
 +
=== eman2/2.3 ===
 +
 +
<pre>module load eman/2.3
 +
eman2.py</pre>
 +
 
=== external ===
 
=== external ===
  
Line 290: Line 955:
 
=== phenix/1.14-3260 ===
 
=== phenix/1.14-3260 ===
  
<pre>module load phenix
+
<pre>module load phenix/1.14-3260
 +
phenix</pre>
 +
 
 +
=== phenix/1.18.2-3874 ===
 +
 
 +
<pre>module load phenix/1.18.2-3874
 
phenix</pre>
 
phenix</pre>
=== relion/{ 2.1, 3.0b } ===
 
  
<pre>for version in 2.1 3.0b;  
+
=== relion/{ 2.1, 3.0b, 3.1 } ===
 +
 
 +
<pre>for version in 2.1 3.0b 3.1;  
 
do
 
do
 
     module switch relion/$version;
 
     module switch relion/$version;
Line 304: Line 975:
 
<pre>module load sphire
 
<pre>module load sphire
 
sphire</pre>
 
sphire</pre>
 +
 +
=== sphire/1.3 ===
 +
 +
<pre>module load sphire/1.3
 +
sphire</pre>
 +
 
=== xchimera/0.8 ===
 
=== xchimera/0.8 ===
  
Line 309: Line 986:
 
== Environmental Modules ==
 
== Environmental Modules ==
  
 +
=== Download ===
 +
 +
Download from here:
 +
 +
<pre>http://modules.sourceforge.net/</pre>
 +
 +
=== Installation ===
 +
 +
<pre>cd /lsc/sources/modules/4.2.0</pre>
 +
<pre> ./configure --prefix=/lsc/modules/4.2.0 --enable-doc-install --enable-example-modulefiles --enable-compat-version --enable-versioning --enable-quarantine-support --enable-auto-handling --with-tclsh=/usr/bin/tclsh --with-pager=/usr/bin/less && make && make install</pre>
 +
 +
This will place symbolic links from <pre>/etc/profile.d/modules.sh</pre> to <pre>/lsc/modules/4.2.0/init/bash</pre>
 +
 +
The configure log is under <pre>compat/config.log</pre> in case you need to see what went wrong
 +
 +
In the event you need to re-initialize the modules subsystem, for any reason run the following as root:
 +
 +
<pre>ln -sf /lsc/modules/4.2.0/init/bash /etc/profile.d/modules.sh</pre>
 +
 +
A soft link is preferred, as hard links do not work across filesystems
 +
 +
=== How to use ===
 
To see what environment modules are available:
 
To see what environment modules are available:
  
Line 324: Line 1,023:
  
 
=== Pushing them to the nodes via Ansible + git ===
 
=== Pushing them to the nodes via Ansible + git ===
 +
 +
== Running Test Jobs on Aristotle to test the software ==
 +
 +
=== Relion ===
 +
 +
 +
As per the [https://hpc.nih.gov/apps/RELION/relion30_tutorial.pdf Relion 3.0 tutorial]:
 +
 +
You will need the following test sets:
 +
 +
ftp://ftp.mrc-lmb.cam.ac.uk/pub/scheres/relion30_tutorial_data.tar
 +
ftp://ftp.mrc-lmb.cam.ac.uk/pub/scheres/relion30_tutorial_precalculated_results.tar.gz
 +
 +
 +
These test sets are already downloaded and untar'ed under <code>intaristotle.internal.biotek:/lsc/relion/test_data</code>
 +
 +
<code>module load relion/</code>'''$version''' you need and wait for relion to open on your X server.
 +
 +
==== 3D Classification ====
 +
 +
Follow the instructions in the tutorial regarding 3D classification

Latest revision as of 12:38, 18 March 2021

There are two groups that work with CryoEM data: the Luecke Group and the Sekulic Group:

The Luecke Group CryoEM setup data processing platform consists of two physical servers and one VM:

  • intaristotle.internal.biotek GPU server: 64 Intel Broadwell cores, 128 Gbytes of RAM, 4 x NVIDIA GTX 1080 GPU cards, 6 Tb of SSD scratch space
  • perun.uio.no file server: 28 Intel Broadwell cores, 32 Gigs of RAM, 212 Tb of local disk space used as file server for the storage of CryoEM data. The machine is also used as a gateway to internal GPU/CPU processing nodes.
  • cryosparc.uio.no VM: 2 x Intel Core Processor (Haswell, IBRS) vcores, 8 Gigs of RAM and 72 Gigs of disk space to run a discrete CryoSPARC web instance

All nodes have 10 Gigabit Ethernet connectivity for dedicated NAS/NFS disk space between the file server and the current (and future) GPU compute nodes.

The Sekulic Group CryoEM setup data processing platform consists of one physical server, one VM and a dedicated file share on the main pythagoras file server :

  • sekulicgpu1.internal.biotek GPU server: 64 CPU core system Dual Intel Xeon Gold 6242 CPU @ 2.80GHz, 6 x T4 Nvidia cards for a total of 15360 CUDA cores plus 1920 Tensor cores, 768 Gigs of RAM. The system also arrived with 416GB of storage for the OS and 3.49 TB of storage for scratch storage.
  • cryosparc2.uio.no VM: 4 x Intel Core Processor (Haswell, IBRS) vcores, 16 Gb of RAM and 170 Gigs of disk space to run a discrete CryoSPARC web interface instance


Contents

Aristotle/CryoMP Installation Documentation ( intaristotle.internal.biotek/192.168.8.109 )

RT ticket #3135527

Live computational resource usage information (cpu and memory)

Software Installed, List of

Application Version on intaristotle.internal.biotek server Status Comments
adxv 1.9.13 Done -
ccp4 7.1.003 Done -
chimera 1.14 Done -
cistem 1.0.0-beta Done -
cryolo 1.5 Done -
ctffind4 4.1.10 Done -
cryosparc 3.1 Done Documentation
eman2 2.31 Done -
fftw, double precision 2.1.5 Done -
fftw, single precision 2.1.5 Done -
fftw, double precision 3.3.8 Done -
fftw, single precision 3.3.8 Done -
imod 4.9.12 Done -
openssl 1.0.2 Done -
frealign 9.11 Done -
gautomatch 0.56 Done -
gctf 1.18 Done -
modules 4.20 Done -
motioncor2 1.2.1 Done -
mpich 3.0.4 Done -
nvidia/cuda 8.0-GA1, 8.0-GA2, 9.0, 9.1, 9.2 Done -
openmpi 3.1.3[+cuda] Done -
openmpi 4.0.0[+cuda] Done -
phenix 1.18.2-3874 Done -
relion 3.0.4, 3.0.6 Done -
scipion 2.1 Done -
sphire 1.3 Done -
topaz 0.2.3 Done -
xchimera 0.8 Done -
xds 2021-03-31 Done -

Top Directory ( /lsc )

Top directory for all installed software is /lsc

Source Files ( in case you need to re-compile something )

Source for all installed programs can be found under /lsc/sources

Dependencies for interactive 3D applications

VirtualGL: Running accelerated 3D graphics remotely via OpenGL

Located under /opt/VirtualGL
Loaded by default

VirtualGL is an open source toolkit that gives any Unix or Linux remote display software the ability to run OpenGL applications with full 3D hardware acceleration.

It is used to display 3D applications on a laptop that does not have a powerful 3D graphics card: Instead of letting the laptop render the 3D application, VirtualGL uses the (more powerful) graphics cards of the server to pre-render the application and send back the result.

This app is very useful in 3D applications like XChimera/Chimera, PyMol and the like

In order to take full advantage of this application, you need to be connected to the NCMM internal network, and not UiO.

Open the command-line/terminal of your choice and type

vglconnect intaristotle.internal.biotek

vgl connect will connect you to the internal interface for aristotle and set up the environment for executing a graphical application.

Let’s say we need to run chimera. So, we load the module

module load chimera/1.13.1

and then we use vglrun to run the chimera application

vglrun -d $DISPLAY chimera

The “-d $DISPLAY” argument is there just in case something has corrupted your local environment.

How was each piece of software compiled (flags, switches)

advx

binary distribution from here. Do not forget to download the relevant fonts

ccp4

cd /lsc/sources/ccp4/7.0.074/
mkdir -p /lsc/ccp4/7.0.074/
cp -r /lsc/sources/ccp4/7.0.074/* /lsc/ccp4/7.0.074/
cd /lsc/ccp4/7.0.074/
./BINARY.setup

arp/warp

module load ccp4/7.0.074
/lsc/ccp4/7.0.074/start
cd /lsc/sources/ccp4/arp_warp_8.0
./install.sh

chimera

cd /lsc/sources/chimera/1.13.1
./chimera-1.13.1-linux_x86_64.bin #self-extracts into two files
# answer questions
> Enter install location: /lsc/chimera/1.13.1
> Install desktop menu (icon has to be done by user)? no 
> Install symbolic link to chimera executable for command line use in which directory? [hit Enter for default (0)]: 0
# installer copies files into destination
> Installation is done; press return.


cistem

cd /lsc/sources/cistem/1.0/cistem-1.0.0-beta
./configure --prefix=/lsc/cistem/1.0
make
make install

cryolo

1.2.1

cd /lsc/sources/cryolo/1.2.1/
scl enable rh-python36 bash
conda create --name cryolo -c anaconda python=3.6 pyqt=5 cudnn=7.1.2
conda activate cryolo
conda install numpy==1.14.5
pip install ./cryolo-1.2.1.tar.gz

1.3.6

cd /lsc/sources/cryolo
wget ftp://ftp.gwdg.de/pub/misc/sphire/crYOLO_V1_3_6/cryolo-1.3.6.tar.gz
wget ftp://ftp.gwdg.de/pub/misc/sphire/crYOLO_BM_V1_2_2/cryoloBM-1.2.2.tar.gz
tar zxvf cryolo-1.3.6.tar.gz
ln -s cryolo-1.4.0 1.4.0
conda env remove --name cryolo # Due to conda, cryolo cannot support previous installations
conda create --prefix /lsc/sphire/1.3.6/.conda/envs/ -c anaconda python=3.6 pyqt=5 cudnn=7.1.2 cython
source activate cryolo
conda install numpy==1.15.4
pip install cryolo-1.3.6.tar.gz[gpu]
pip install cryoloBM-1.2.2.tar.gz

conda activate /lsc/sphire/1.3.6/.conda/envs/cryolo
to activate

1.4.0

cd /lsc/sources/cryolo
wget ftp://ftp.gwdg.de/pub/misc/sphire/crYOLO_V1_4_0/cryolo-1.4.0.tar.gz
wget ftp://ftp.gwdg.de/pub/misc/sphire/crYOLO_BM_V1_2_3/cryoloBM-1.2.3.tar.gz
tar zxvf cryolo-1.4.0.tar.gz
ln -s cryolo-1.4.0 1.4.0
conda create --prefix /lsc/cryolo/1.4.0 -c anaconda python=3.6 pyqt=5 cudnn=7.1.2 cython
conda activate /lsc/cryolo/1.4.0
conda install numpy==1.15.4
pip install cryolo-1.4.0.tar.gz[gpu]
pip install cryoloBM-1.2.3.tar.gz

conda activate /lsc/cryolo/1.4.0
to activate

1.5.0

cd /lsc/sources/cryolo
/lsc/anaconda/5.3.1/bin/conda  create --prefix=/lsc/cryolo/1.5.0 -c conda -c conda-forge -c anaconda python=3.6 pyqt=5 cudnn=7.1.2 numpy==1.14.5 cython wxPython==4.0.4 intel-openmp==2019.4
/lsc/anaconda/5.3.1/bin/conda activate /lsc/cryolo/1.5.0
. /lsc/anaconda/5.3.1/etc/profile.d/conda.sh

/lsc/anaconda/5.3.1/bin/conda activate /lsc/cryolo/1.5.0
to activate

cryosparc

  • installation see RT #3978772
  • update (here for sekulicgpu1/cryosparc2)
[root@cryosparc2 cryosparc]# su cryosparc_user
bash-4.2$ cryosparcm backup

Backing up to /lsc/cryosparc/2.15.0/.../cryosparc2_database/backup/cryosparc_backup_2021_02_09_13h33.archive
[...]
bash-4.2$ cryosparcm update --check
CryoSPARC current version v3.0.1
          update starting on Tue Feb  9 13:44:40 CET 2021

  current version v3.0.1
      new version v3.1.0

Update available!
bash-4.2$ cryosparcm update
CryoSPARC current version v3.0.1
          update starting on Tue Feb  9 13:44:58 CET 2021

No version specified - updating to latest version.
[...]

Notes

  • this updates both master and worker
  • if update has to be re-done for any reason (e.g., update worked on master, but something was stuck on worker, etc.), this here can be useful (but usually should not be required!):
bash-4.2$ cryosparcm stop
CryoSPARC is running.
Stopping cryoSPARC 
Shut down
bash-4.2$ cryosparcm update --override
[...]

ctffind4

binary blob here https://grigoriefflab.umassmed.edu/ctffind4

4.1.10

cd /lsc/sources/ctffind/4.1.10/ctffind-4.1.10
./configure --prefix=/lsc/ctffind4/4.1.10 --enable-latest-instruction-set
make -j 64 all
make -j 64 install

4.1.13

This is not working: this new version of ctf find requires the wxWidgets and /usr/bin/wx-config --query-version returns 2.8 (needs version 3.0.0 and above). Will use static binary instead for now.

Suspended till we upgrade the machine to something newer

yum install wxGTK3-webview.x86_64 wxGTK3-media.x86_64 wxGTK3-gl.x86_64 wxGTK3.x86_64 wxGTK-media.x86_64 wxGTK-gl.x86_64 wxGTK.x86_64 wxBase3.x86_64 wxBase.x86_64 mathgl-wx-devel.x86_64 mathgl-wx.x86_64 erlang-wx.x86_64 compat-wxGTK3-gtk2-media.x86_64 compat-wxGTK3-gtk2-gl.x86_64 compat-wxGTK3-gtk2.x86_64 compat-wxBase3-gtk2.x86_64
cd /lsc/sources/ctffind/4.1.13/ctffind-4.1.13
/configure --prefix=/lsc/ctffind/4.1.13 --enable-latest-instruction-set  --enable-openmp --with-wx-config=/usr/bin/wx-config --with-fftw-dir=/lsc/fftw/3.3.8
make -j 64 all
make -j 64 install

4.1.13 + cuda

binary distribution

eman2

2.2

cd /lsc/sources/eman2/2.2/
./eman2.22.linux64.sh
> EMAN2 will now be installed into this location:
> [/root/EMAN2] >>> /lsc/eman2/2.2
# installer does the rest of the work, re-installing a bunch of python modules via anaconda
> Do you wish the installer to prepend the EMAN2 install location
> to PATH in your /root/.bashrc ? [yes|no]
> [no] >>> no
> You may wish to edit your .bashrc to prepend the EMAN2 install location to PATH:
> export PATH=/opt/eman/2.2/bin:$PATH
# covered in the relevant environment module

2.3

cd /lsc/sources/eman2/2.3/
./eman2.3.linux64.sh
> EMAN2 will now be installed into this location:
> [/root/EMAN2] >>> /lsc/eman2/2.3
# installer does the rest of the work, re-installing a bunch of python modules via anaconda
> Do you wish the installer to prepend the EMAN2 install location
> to PATH in your /root/.bashrc ? [yes|no]
> [no] >>> no
> You may wish to edit your .bashrc to prepend the EMAN2 install location to PATH:
> export PATH=/opt/eman/2.3/bin:$PATH
# covered in the relevant environment module

2.31

cd /lsc/sources/eman2/2.31/
eman2.31_sphire1.3_huge.linux64.sh
> EMAN2 will now be installed into this location:
> [/root/EMAN2] >>> /lsc/eman2/2.31
# installer does the rest of the work, re-installing a bunch of python modules via anaconda
> Do you wish the installer to prepend the EMAN2 install location
> to PATH in your /root/.bashrc ? [yes|no]
> [no] >>> no
> You may wish to edit your .bashrc to prepend the EMAN2 install location to PATH:
> export PATH=/opt/eman/2.31/bin:$PATH
# covered in the relevant environment module

Dynamo

https://wiki.dynamo.biozentrum.unibas.ch/w/index.php/Main_Page

1401

external/fftw

2.1.5

cd /lsc/sources/fftw/2.1.5/fftw-2.1.5
./configure --prefix=/lsc/fftw/2.1.5 --enable-threads --enable-mpi --enable-i386-hacks
make
make install

3.3.8

cd /lsc/sources/fftw/3.3.8/fftw-3.3.8
./configure CC=/usr/bin/gcc-4.8.5 --prefix=/lsc/fftw/3.3.8 --enable-single --enable-sse --enable-sse2 --enable-avx --enable-avx2 --enable-avx-128-fma --enable-generic-simd128 --enable-generic-simd256 --enable-fma --enable-mpi --enable-threads --with-g77-wrappers --with-combined-threads
make
make install

external/openssl

cd /lsc/sources/openssl/openssl-1.0.2o/
./config --prefix=/lsc/external/openssl/1.0.2o
make -j 64
make -j 64 install
#done

frealign

cd /lsc/sources/frealign/9.11/frealign_v9.11
mkdir -p /lsc/frealign/9.11
cp -r /lsc/sources/frealign/9.11/frealign_v9.11/* /lsc/frealign/9.11
# Ignore the ./INSTALL file, it does nothing
# you just need to add the relevant bin path for the application to work
# the relevant module covers the details

gautomatch

cd /lsc/sources/gautomatch/Gautomatch_v0.53
mkdir /lsc/gautomatch/0.53
cp -r /lsc/sources/gautomatch/Gautomatch_v0.53 /lsc/gautomatch/0.53

gctf

cd /lsc/sources/gctf/1.18
mkdir /lsc/gctf/1.18
cp -r /lsc/sources/gctf/1.18/* /lsc/gctf/1.18
# environmental module takes care of loading up cuda 8.0, (required) adding the bin path, and changing the LD_PRELOAD path ### modules 
cd /lsc/sources/modules/4.2.0
./configure --prefix=/lsc/modules --enable-doc-install --enable-example-modulefiles --enable-compat-version --enable-versioning --enable-quarantine-support --enable-auto-handling 
# since --enable-versioning is turned on, the module will autoversion itself and place all its files under
# /lsc/modules/4.2.0

imod

 Follow instructions as described in RT 3509886

mpich

default loaded by CentOS 7.
Environmental module was altered to make sure that it conflicts with the openmpi modules

motioncor2

cd /lsc/sources/motioncor2/1.2.1
mkdir -p /lsc/motioncor2/1.2.1/bin
cp -r /lsc/sources/motioncor2/1.2.1/* /lsc/motioncor2/1.2.1a
ln -s /lsc/motioncor2/1.2.1/MotionCor2_1.2.1-Cuda80 /lsc/motioncor2/1.2.1/bin/motioncor2
# environmental module takes care of setting up the bin path 

nvidia/driver

cd /lsc/sources/nvidia/drivers/396.54;
./NVIDIA-Linux-x86_64-396.54.run


Update: In order to combat the issue with creating remote OpenGL context (X11 forwarding over ssh), I upgraded the driver to 415.23 but had no luck. In nvidia-smi, the new driver reports that is nvidia CUDA 10 capable, which is fine as it can use the CUDA 8 run-time environment, as NVIDIA guarantees binary compatibility

nvidia/cuda

CUDA frameworks installed:

  • 8.0-GA1
  • 8.0-GA2
  • 9.0
  • 9.1
  • 9.2
  • 10.2
Default CUDA framework for most applications is 8.0GA1

The Nvidia driver version for the kernel installed is Driver Version: 440.44 of CUDA Version: 10.2

During loading different CUDA versions, NVIDIA driver remains the same. Things seem to work, but may require future additional detailed testing

All CUDA installations require gcc 4.8 (hence the reason we went with CentOS)

All mentioned versions are currently installed, but we regard 8.0-GA1 as the base release, due to software that demands 8.0-GA1 and has no alternatives.

Installations:

8.0-GA1

cd /lsc/sources/nvidia/cuda/8.0-GA1 ;
./cuda_8.0.44_linux-run --silent --toolkit=/lsc/nvidia/cuda/8.0-GA1 --samples --samplespath=/lsc/nvidia/cuda/8.0-GA1/samples --run-nvidia-xconfig --tmpdir=/tmp

8.0-GA2

cd /lsc/sources/nvidia/cuda/8.0-GA2 ;
./cuda_8.0.61_375.26_linux-run --silent --toolkit=/lsc/nvidia/cuda/8.0-GA2 --samples --samplespath=/lsc/nvidia/cuda/8.0-GA2/samples --run-nvidia-xconfig --tmpdir=/tmp

9.0

cd /lsc/sources/nvidia/cuda/9.0 ;
./cuda_9.0.176_384.81_linux-run --silent --toolkit=/lsc/nvidia/cuda/9.0 --samples --samplespath=/lsc/nvidia/cuda/9.0/samples --run-nvidia-xconfig --tmpdir=/tmp

9.1

cd /lsc/sources/nvidia/cuda/9.1 ;
./cuda_9.1.85_387.26_linux --silent --toolkit=/lsc/nvidia/cuda/9.1 --samples --samplespath=/lsc/nvidia/cuda/9.1/samples --run-nvidia-xconfig --tmpdir=/tmp

9.2

cd /lsc/sources/nvidia/cuda/9.2 ;
./9.2.148_396.37_linux --silent --toolkit=/lsc/nvidia/cuda/9.2 --samples --samplespath=/lsc/nvidia/cuda/9.2/samples --run-nvidia-xconfig --tmpdir=/tmp

10.2

mkdir /lsc/nvidia/cuda/10.2;
wget http://developer.download.nvidia.com/compute/cuda/10.2/Prod/local_installers/cuda_10.2.89_440.33.01_linux.run;
sh ./cuda_10.2.89_440.33.01_linux.run --silent --toolkitpath=/lsc/nvidia/cuda/10.2 --toolkit=/lsc/nvidia/cuda/10.2 --samples --samplespath=/lsc/nvidia/cuda/10.2/samples --run-nvidia-xconfig --tmpdir=/storage/scratch/tmp

openmpi

cd /lsc/sources/openmpi/3.1.2;
./configure --prefix /lsc/openmpi/3.1.2 --enable-binaries --enable-mpi-fortran --with-cuda=/lsc/nvidia/cuda/8.0-GA1 --with-devel-headers 

Default: 3.1.2

The default version loaded is the stable-as-of 2018-10-31 3.1.2

phenix

cd /lsc/sources/phenix/$version/phenix-installer-$version-intel-linux-2.6-x86_64-centos6 ;
./install --prefix=/lsc/phenix --openmp --makedirs 

relion

2.1

cd /lsc/sources/relion/2.1-mpich ;
mkdir build;
cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/2.1
make -j 64
make -j 64 install

3.0b

cd /lsc/sources/relion/3.0b-mpich ;
mkdir build;
cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/3.0b
make -j 64
make -j 64 install

3.0.4

cd /lsc/sources/relion/3.0.4 ;
mkdir build && cd build;
cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/3.0.4 --DGUI=ON -DCUDA=ON -DCudaTexture=ON -DFORCE_OWN_TBB=ON -DFORCE_OWN_FLTK=ON -DFORCE_OWN_FFTW=ON -DCUDA_ARCH=61 -DBUILD_SHARED_LIBS=ON ..
make -j 64
make install

3.0.6

cd /lsc/sources/relion/3.0.6 ;
mkdir build && cd build;
cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/3.0.6 --DGUI=ON -DCUDA=ON -DCudaTexture=ON -DFORCE_OWN_TBB=ON -DFORCE_OWN_FLTK=ON -DFORCE_OWN_FFTW=ON -DCUDA_ARCH=61 -DBUILD_SHARED_LIBS=ON ..
make -j 64
make install

3.0.8

cd /lsc/sources/relion/3.0.8 ;
mkdir build && cd build;
cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/3.0.8 --DGUI=ON -DCUDA=ON -DCudaTexture=ON -DFORCE_OWN_TBB=ON -DFORCE_OWN_FLTK=ON -DFORCE_OWN_FFTW=ON -DCUDA_ARCH=61 -DBUILD_SHARED_LIBS=ON ..
make -j 64
make install

3.1-beta

cd /lsc/sources/relion/relion.git
git checkout ver3.1
mkdir build && cd build
module load relion/3.1-beta
cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/3.1-beta --DGUI=ON -DCUDA=ON -DCudaTexture=ON -DFORCE_OWN_TBB=ON -DFORCE_OWN_FLTK=ON -DFORCE_OWN_FFTW=ON -DCUDA_ARCH=61 -DBUILD_SHARED_LIBS=ON ..
make -j 64
make install

3.1

cd /lsc/sources/relion/3.1
mkdir build && cd build
module load relion/3.1
cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/3.1 --DGUI=ON -DCUDA=ON -DCudaTexture=ON -DFORCE_OWN_TBB=ON -DFORCE_OWN_FLTK=ON -DFORCE_OWN_FFTW=ON -DCUDA_ARCH=61 -DBUILD_SHARED_LIBS=ON ..
make -j 64
make install


List of CUDA architectures here : https://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/

scipion

2.0

Note: The comments for scipion and the Sekulic GPU server are applicable to module scipion/2.0source. Its installation is described in RT 3879628. The instructions below are just basic steps and do not achieve production installation.

Cloning/Getting software

mkdir -p /lsc/scipion/2.0
git clone https://github.com/I2PC/scipion.git
cd scipion

Dependencies

yum install -y gcc gcc-g++ cmake java-1.8.0-openjdk-devel.x86_64 libXft-devel.x86_64 openssl-devel.x86_64 libXext-devel.x86_64  libxml++.x86_64 libquadmath-devel.x86_64  libxslt.x86_64 openmpi-devel.x86_64  gsl-devel.x86_64  libX11.x86_64  gcc-gfortran.x86_64 git

Configuring

./scipion config
scipion installp -p scipion-em-relion


2.1

Note: same as above for xmipp2

Cloning/Getting software

 
 mkdir -p /lsc/scipion/2.0
 git clone https://github.com/I2PC/scipion.git
 cd scipion

Dependencies

yum install -y gcc gcc-g++ cmake java-1.8.0-openjdk-devel.x86_64 libXft-devel.x86_64 openssl-devel.x86_64 libXext-devel.x86_64  libxml++.x86_64 libquadmath-devel.x86_64  libxslt.x86_64 openmpi-devel.x86_64  gsl-devel.x86_64  libX11.x86_64  gcc-gfortran.x86_64 git

Configuring and installing modules from the command-line

./scipion config
./scipion installp --help | awk -p '{ print $1 }' | grep 'scipion-em' | grep -E -v 'Configuration|xmipp2' > modules.out
./scipion installp -p scipion-em-xmipp
for module in $( < module.out ); do ./scipion installp -p $module ); done
rm module.out


see ticket 4049861: some modules cannot compile due to python2 issues:

./scipion installp -p scipion-em-cryodrgn 3.0.0b4 # python2 issues
./scipion installp -p scipion-em-dynamo 3.1.4 # python2 issues
./scipion installp -p scipion-em-fsc3d 3.0.5 # python2 issues
./scipion installp -p scipion-em-imod 3.0.3 # python2 issues
./scipion installp -p scipion-em-novactf 3.0.0 # python2 issues
./scipion installp -p scipion-em-pyseg 0.2 # python2 issues
./scipion installp -p scipion-em-sidesplitter 3.0.3 # python2 issues
./scipion installp -p scipion-em-tomo 3.0.5 # python2 issues

sphire

1.1

cd /lsc/sources/sphire/1.1 ;
./sphire_1_1_linux.sh
# follow questions, install under /lsc/sphire/1.1

1.3

sphire 1.3 is included with the installation of eman 2.31.

module load eman 2.31

topaz

0.2.3

[root@intaristotle ~]# conda create --prefix=/lsc/topaz/0.2.3
Collecting package metadata: done
Solving environment: done

## Package Plan ##

  environment location: /lsc/topaz/0.2.3



Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate /lsc/topaz/0.2.3
#
# To deactivate an active environment, use
#
#     $ conda deactivate

[root@intaristotle ~]# conda activate /lsc/topaz/0.2.3
(/lsc/topaz/0.2.3) [root@intaristotle ~]# module load nvidia/cuda/9.2
(/lsc/topaz/0.2.3) [root@intaristotle ~]# conda install topaz cudatoolkit=9.2 -c tbepler -c pytorch
Collecting package metadata: done
Solving environment: done

## Package Plan ##

  environment location: /lsc/topaz/0.2.3

  added / updated specs:
    - cudatoolkit=9.2
    - topaz


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    certifi-2020.4.5.1         |           py38_0         159 KB
    freetype-2.9.1             |       h8a8886c_1         822 KB
    intel-openmp-2020.0        |              166         916 KB
    joblib-0.14.1              |             py_0         202 KB
    jpeg-9b                    |       h024ee3a_2         248 KB
    libpng-1.6.37              |       hbc83047_0         364 KB
    libtiff-4.1.0              |       h2733197_0         589 KB
    mkl-2020.0                 |              166       202.1 MB
    mkl-service-2.3.0          |   py38he904b0f_0          68 KB
    mkl_fft-1.0.15             |   py38ha843d7b_0         178 KB
    mkl_random-1.1.0           |   py38h962f231_0         398 KB
    ninja-1.9.0                |   py38hfd86e86_0         1.6 MB
    numpy-1.18.1               |   py38h4f9e942_0           5 KB
    numpy-base-1.18.1          |   py38hde5b4d6_1         5.3 MB
    olefile-0.46               |             py_0          33 KB
    pandas-1.0.3               |   py38h0573a6f_0        11.5 MB
    pillow-7.0.0               |   py38hb39fc2d_0         664 KB
    pip-20.0.2                 |           py38_1         1.9 MB
    python-3.8.2               |       hcf32534_0        57.8 MB
    python-dateutil-2.8.1      |             py_0         224 KB
    pytorch-1.5.0              |py3.8_cuda9.2.148_cudnn7.6.3_0       344.4 MB  pytorch
    pytz-2019.3                |             py_0         231 KB
    scikit-learn-0.22.1        |   py38hd81dba3_0         7.3 MB
    scipy-1.4.1                |   py38h0b6359f_0        19.1 MB
    setuptools-46.1.3          |           py38_0         656 KB
    six-1.14.0                 |           py38_0          27 KB
    topaz-0.2.3                |             py_0        26.0 MB  tbepler
    torchvision-0.6.0          |        py38_cu92        11.7 MB  pytorch
    wheel-0.34.2               |           py38_0          49 KB
    zstd-1.3.7                 |       h0b5b093_0         887 KB
    ------------------------------------------------------------
                                           Total:       695.2 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
  blas               pkgs/main/linux-64::blas-1.0-mkl
  ca-certificates    pkgs/main/linux-64::ca-certificates-2020.1.1-0
  certifi            pkgs/main/linux-64::certifi-2020.4.5.1-py38_0
  cudatoolkit        pkgs/main/linux-64::cudatoolkit-9.2-0
  freetype           pkgs/main/linux-64::freetype-2.9.1-h8a8886c_1
  intel-openmp       pkgs/main/linux-64::intel-openmp-2020.0-166
  joblib             pkgs/main/noarch::joblib-0.14.1-py_0
  jpeg               pkgs/main/linux-64::jpeg-9b-h024ee3a_2
  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.33.1-h53a641e_7
  libedit            pkgs/main/linux-64::libedit-3.1.20181209-hc058e9b_0
  libffi             pkgs/main/linux-64::libffi-3.2.1-hd88cf55_4
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
  libgfortran-ng     pkgs/main/linux-64::libgfortran-ng-7.3.0-hdf63c60_0
  libpng             pkgs/main/linux-64::libpng-1.6.37-hbc83047_0
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
  libtiff            pkgs/main/linux-64::libtiff-4.1.0-h2733197_0
  mkl                pkgs/main/linux-64::mkl-2020.0-166
  mkl-service        pkgs/main/linux-64::mkl-service-2.3.0-py38he904b0f_0
  mkl_fft            pkgs/main/linux-64::mkl_fft-1.0.15-py38ha843d7b_0
  mkl_random         pkgs/main/linux-64::mkl_random-1.1.0-py38h962f231_0
  ncurses            pkgs/main/linux-64::ncurses-6.2-he6710b0_1
  ninja              pkgs/main/linux-64::ninja-1.9.0-py38hfd86e86_0
  numpy              pkgs/main/linux-64::numpy-1.18.1-py38h4f9e942_0
  numpy-base         pkgs/main/linux-64::numpy-base-1.18.1-py38hde5b4d6_1
  olefile            pkgs/main/noarch::olefile-0.46-py_0
  openssl            pkgs/main/linux-64::openssl-1.1.1g-h7b6447c_0
  pandas             pkgs/main/linux-64::pandas-1.0.3-py38h0573a6f_0
  pillow             pkgs/main/linux-64::pillow-7.0.0-py38hb39fc2d_0
  pip                pkgs/main/linux-64::pip-20.0.2-py38_1
  python             pkgs/main/linux-64::python-3.8.2-hcf32534_0
  python-dateutil    pkgs/main/noarch::python-dateutil-2.8.1-py_0
  pytorch            pytorch/linux-64::pytorch-1.5.0-py3.8_cuda9.2.148_cudnn7.6.3_0
  pytz               pkgs/main/noarch::pytz-2019.3-py_0
  readline           pkgs/main/linux-64::readline-8.0-h7b6447c_0
  scikit-learn       pkgs/main/linux-64::scikit-learn-0.22.1-py38hd81dba3_0
  scipy              pkgs/main/linux-64::scipy-1.4.1-py38h0b6359f_0
  setuptools         pkgs/main/linux-64::setuptools-46.1.3-py38_0
  six                pkgs/main/linux-64::six-1.14.0-py38_0
  sqlite             pkgs/main/linux-64::sqlite-3.31.1-h62c20be_1
  tk                 pkgs/main/linux-64::tk-8.6.8-hbc83047_0
  topaz              tbepler/noarch::topaz-0.2.3-py_0
  torchvision        pytorch/linux-64::torchvision-0.6.0-py38_cu92
  wheel              pkgs/main/linux-64::wheel-0.34.2-py38_0
  xz                 pkgs/main/linux-64::xz-5.2.5-h7b6447c_0
  zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3
  zstd               pkgs/main/linux-64::zstd-1.3.7-h0b5b093_0


Proceed ([y]/n)? y


Downloading and Extracting Packages
mkl-2020.0           | 202.1 MB  | ################################################################################################# | 100%
pytorch-1.5.0        | 344.4 MB  | ################################################################################################# | 100%
jpeg-9b              | 248 KB    | ################################################################################################# | 100%
python-dateutil-2.8. | 224 KB    | ################################################################################################# | 100%
pip-20.0.2           | 1.9 MB    | ################################################################################################# | 100%
joblib-0.14.1        | 202 KB    | ################################################################################################# | 100%
certifi-2020.4.5.1   | 159 KB    | ################################################################################################# | 100%
python-3.8.2         | 57.8 MB   | ################################################################################################# | 100%
setuptools-46.1.3    | 656 KB    | ################################################################################################# | 100%
pytz-2019.3          | 231 KB    | ################################################################################################# | 100%
mkl_fft-1.0.15       | 178 KB    | ################################################################################################# | 100%
six-1.14.0           | 27 KB     | ################################################################################################# | 100%
numpy-1.18.1         | 5 KB      | ################################################################################################# | 100%
libtiff-4.1.0        | 589 KB    | ################################################################################################# | 100%
zstd-1.3.7           | 887 KB    | ################################################################################################# | 100%
ninja-1.9.0          | 1.6 MB    | ################################################################################################# | 100%
scikit-learn-0.22.1  | 7.3 MB    | ################################################################################################# | 100%
pillow-7.0.0         | 664 KB    | ################################################################################################# | 100%
numpy-base-1.18.1    | 5.3 MB    | ################################################################################################# | 100%
freetype-2.9.1       | 822 KB    | ################################################################################################# | 100%
scipy-1.4.1          | 19.1 MB   | ################################################################################################# | 100%
pandas-1.0.3         | 11.5 MB   | ################################################################################################# | 100%
mkl_random-1.1.0     | 398 KB    | ################################################################################################# | 100%
mkl-service-2.3.0    | 68 KB     | ################################################################################################# | 100%
libpng-1.6.37        | 364 KB    | ################################################################################################# | 100%
wheel-0.34.2         | 49 KB     | ################################################################################################# | 100%
torchvision-0.6.0    | 11.7 MB   | ################################################################################################# | 100%
topaz-0.2.3          | 26.0 MB   | ################################################################################################# | 100%
olefile-0.46         | 33 KB     | ################################################################################################# | 100%
intel-openmp-2020.0  | 916 KB    | ################################################################################################# | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(/lsc/topaz/0.2.3) [root@intaristotle ~]# topaz
Traceback (most recent call last):
  File "/lsc/topaz/0.2.3/bin/topaz", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/lsc/topaz/0.2.3/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3259, in <module>
    def _initialize_master_working_set():
  File "/lsc/topaz/0.2.3/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3242, in _call_aside
    f(*args, **kwargs)
  File "/lsc/topaz/0.2.3/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3271, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/lsc/topaz/0.2.3/lib/python3.8/site-packages/pkg_resources/__init__.py", line 584, in _build_master
    ws.require(__requires__)
  File "/lsc/topaz/0.2.3/lib/python3.8/site-packages/pkg_resources/__init__.py", line 901, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/lsc/topaz/0.2.3/lib/python3.8/site-packages/pkg_resources/__init__.py", line 787, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'future' distribution was not found and is required by torch
(/lsc/topaz/0.2.3) [root@intaristotle ~]# pip install future
Collecting future
  Downloading future-0.18.2.tar.gz (829 kB)
     |████████████████████████████████| 829 kB 4.3 MB/s
Building wheels for collected packages: future
  Building wheel for future (setup.py) ... done
  Created wheel for future: filename=future-0.18.2-py3-none-any.whl size=491058 sha256=f1d7892b69b66f8a96624ddc534bfc5dcf52e8042aa760743760f036db892c26
  Stored in directory: /root/.cache/pip/wheels/8e/70/28/3d6ccd6e315f65f245da085482a2e1c7d14b90b30f239e2cf4
Successfully built future
Installing collected packages: future
Successfully installed future-0.18.2
(/lsc/topaz/0.2.3) [root@intaristotle ~]# topaz
usage: topaz [-h] [--version] <command> ...
topaz: error: the following arguments are required: <command>

0.2.4

  • Installed on sekulicgpu1 and cryosparc2
[root@sekulicgpu1 topaz]# mkdir /lsc/venvs
[root@sekulicgpu1 topaz]# chown cryosparc_user /lsc/venvs
[root@sekulicgpu1 topaz]# su cryosparc_user
[cryosparc_user@sekulicgpu1 topaz]$ /lsc/python/3.6.11/bin/python3.6 -m venv /lsc/venvs/topaz
[cryosparc_user@sekulicgpu1 topaz]$ source /lsc/venvs/topaz/bin/activate
(topaz) [cryosparc_user@sekulicgpu1 topaz]$ mkdir /lsc/venvs/tmp
(topaz) [cryosparc_user@sekulicgpu1 topaz]$ env TMPDIR=/lsc/venvs/tmp pip3 install topaz-em
[...]
Successfully installed dataclasses-0.8 future-0.18.2 joblib-1.0.1 numpy-1.19.5 pandas-1.1.5 pillow-8.1.0 python-dateutil-2.8.1 pytz-2021.1 scikit-learn-0.24.1 scipy-1.5.4 six-1.15.0 threadpoolctl-2.1.0 topaz-em-0.2.4 torch-1.7.1 torchvision-0.8.2 typing-extensions-3.7.4.3
(topaz) [cryosparc_user@sekulicgpu1 topaz]$ rmdir /lsc/venvs/tmp

On cryosparc2, first install Python 3.6.11 (for consistency with sekulicgpu1), depends on 3 -devel packages:

[root@cryosparc2 ~]# yum install zlib-devel openssl-devel bzip2-devel
[root@cryosparc2 ~]# mkdir -p /lsc/sources/python /lsc/python/3.6.11
[root@cryosparc2 ~]# chown cryosparc_user /lsc/sources/python /lsc/python/3.6.11
[root@cryosparc2 ~]# su cryosparc_user
bash-4.2$ cd /lsc/sources/python/
bash-4.2$ wget https://www.python.org/ftp/python/3.6.11/Python-3.6.11.tar.xz
bash-4.2$ tar xf Python-3.6.11.tar.xz
bash-4.2$ cd Python-3.6.11
bash-4.2$ ./configure --prefix=/lsc/python/3.6.11 --enable-optimizations && make && make install

and then same steps for topaz as on sekulicgpu1:

[root@cryosparc2 ~]# mkdir /lsc/venvs
[root@cryosparc2 ~]# chown cryosparc_user /lsc/venvs
[root@cryosparc2 ~]# su cryosparc_user
bash-4.2$ /lsc/python/3.6.11/bin/python3.6 -m venv /lsc/venvs/topaz
bash-4.2$ source /lsc/venvs/topaz/bin/activate
(topaz) bash-4.2$ mkdir /lsc/venvs/tmp
(topaz) bash-4.2$ env TMPDIR=/lsc/venvs/tmp pip3 install topaz-em
[...]
Successfully installed dataclasses-0.8 future-0.18.2 joblib-1.0.1 numpy-1.19.5 pandas-1.1.5 pillow-8.1.0 python-dateutil-2.8.1 pytz-2021.1 scikit-learn-0.24.1 scipy-1.5.4 six-1.15.0 threadpoolctl-2.1.0 topaz-em-0.2.4 torch-1.7.1 torchvision-0.8.2 typing-extensions-3.7.4.3
(topaz) bash-4.2$ rmdir /lsc/venvs/tmp

path to topaz: /lsc/venvs/topaz/bin/topaz

verify GPU support:

[haroldgu@sekulicgpu1 ~]$ /lsc/venvs/topaz/bin/python -c 'import torch ; print(torch.cuda.is_available())'
True

How is each piece of software run ( really basic, just front UI, text or graphics)

ccp4/4.7.0

chimera/1.13.1

module load chimera
chimera

ctffind4/4.1.10

module load ctffind4
ctffind4

eman2/2.2

module load eman
eman2.py

eman2/2.3

module load eman/2.3
eman2.py

external

openssl/1.0.2o

Nothing to see here, this is just a support library

frealign/9.11

module load frealign
frealign

gautomatch/0.53

module load gautomatch
gautomatch

gctf/1.18

module load gctf
gctf

mpich/3.0.4

module load mpich
mpirun

motioncor2/1.2.1

nvidia/

driver

modinfo nvidia

cuda/{ 8.0-GA1, 8.0-GA2, 9.0, 9.1, 9.2 }

There is really nothing you can do with the CUDA libraries directly

You can only verify that version of CUDA works with the following for loop in bash:

for version in 8.0-GA1 8.0-GA2 9.0 9.1 9.2; 
do
    module switch nvidia/cuda/$version; # press q to quit the nbody simulation below
    /lsc/nvidia/cuda/$version/samples/5_Simulations/nbody/nbody -hostmem -numdevices=$(lspci | grep -i nvidia | grep -i vga); # n-body gravitational attraction simulation
done && module purge;

openmpi/3.1.2

module load openmpi
mpirun -j 64 date

phenix/1.14-3260

module load phenix/1.14-3260
phenix

phenix/1.18.2-3874

module load phenix/1.18.2-3874
phenix

relion/{ 2.1, 3.0b, 3.1 }

for version in 2.1 3.0b 3.1; 
do
    module switch relion/$version;
    /lsc/relion/$version/bin/relion
done;

sphire/1.1

module load sphire
sphire

sphire/1.3

module load sphire/1.3
sphire

xchimera/0.8

not yet available   

Environmental Modules

Download

Download from here:

http://modules.sourceforge.net/

Installation

cd /lsc/sources/modules/4.2.0
 ./configure --prefix=/lsc/modules/4.2.0 --enable-doc-install --enable-example-modulefiles --enable-compat-version --enable-versioning --enable-quarantine-support --enable-auto-handling --with-tclsh=/usr/bin/tclsh --with-pager=/usr/bin/less && make && make install
This will place symbolic links from
/etc/profile.d/modules.sh
to
/lsc/modules/4.2.0/init/bash
The configure log is under
compat/config.log
in case you need to see what went wrong

In the event you need to re-initialize the modules subsystem, for any reason run the following as root:

ln -sf /lsc/modules/4.2.0/init/bash /etc/profile.d/modules.sh

A soft link is preferred, as hard links do not work across filesystems

How to use

To see what environment modules are available:

module avail

To load a module:

module load <module>

To see the already loaded modules:

module list

To unload a module:

module unload <module>

Git Repo

Pushing them to the nodes via Ansible + git

Running Test Jobs on Aristotle to test the software

Relion

As per the Relion 3.0 tutorial:

You will need the following test sets:

ftp://ftp.mrc-lmb.cam.ac.uk/pub/scheres/relion30_tutorial_data.tar
ftp://ftp.mrc-lmb.cam.ac.uk/pub/scheres/relion30_tutorial_precalculated_results.tar.gz


These test sets are already downloaded and untar'ed under intaristotle.internal.biotek:/lsc/relion/test_data

module load relion/$version you need and wait for relion to open on your X server.

3D Classification

Follow the instructions in the tutorial regarding 3D classification