Difference between revisions of "NCMM CryoEM Computing platform"
(→3.0.8 and 3.1-beta) |
(→Live computational resource usage information (CPU and memory)) |
||
(61 intermediate revisions by 4 users not shown) | |||
Line 1: | Line 1: | ||
− | The [https://www.med.uio.no/ncmm/english/groups/luecke-group/index.html Luecke Group] CryoEM setup data processing platform consists of | + | There are two groups that work with CryoEM data: the [https://www.med.uio.no/ncmm/english/groups/luecke-group/index.html Luecke Group] and the [https://www.med.uio.no/ncmm/english/groups/sekulic-group/ Sekulic Group]: |
− | * intaristotle. | + | |
+ | The [https://www.med.uio.no/ncmm/english/groups/luecke-group/index.html Luecke Group] CryoEM setup data processing platform consists of two physical servers and one VM: | ||
+ | * intaristotle.internal.biotek GPU server: 64 [https://ark.intel.com/products/91766/Intel-Xeon-Processor-E5-2683-v4-40M-Cache-2-10-GHz- Intel Broadwell cores], 128 Gbytes of RAM, 4 x [https://www.nvidia.com/en-us/geforce/products/10series/geforce-gtx-1080-ti/ NVIDIA GTX 1080 GPU cards], 6 Tb of SSD scratch space | ||
* perun.uio.no file server: 28 [https://ark.intel.com/products/91754/Intel-Xeon-Processor-E5-2680-v4-35M-Cache-2-40-GHz- Intel Broadwell cores], 32 Gigs of RAM, 212 Tb of local disk space used as file server for the storage of CryoEM data. The machine is also used as a gateway to internal GPU/CPU processing nodes. | * perun.uio.no file server: 28 [https://ark.intel.com/products/91754/Intel-Xeon-Processor-E5-2680-v4-35M-Cache-2-40-GHz- Intel Broadwell cores], 32 Gigs of RAM, 212 Tb of local disk space used as file server for the storage of CryoEM data. The machine is also used as a gateway to internal GPU/CPU processing nodes. | ||
− | All nodes have 10 Gigabit Ethernet connectivity for dedicated NAS/NFS disk space between the file server and the current (and future) GPU compute nodes. | + | * cryosparc.uio.no VM: 2 x Intel Core Processor (Haswell, IBRS) vcores, 8 Gigs of RAM and 72 Gigs of disk space to run a discrete CryoSPARC web instance |
+ | All nodes have 10 Gigabit Ethernet connectivity for dedicated NAS/NFS disk space between the file server and the current (and future) GPU compute nodes. | ||
+ | |||
+ | The [https://www.med.uio.no/ncmm/english/groups/sekulic-group/ Sekulic Group] CryoEM setup data processing platform consists of one physical server, one VM and a dedicated file share on the main pythagoras file server : | ||
+ | * sekulicgpu1.internal.biotek GPU server: 64 CPU core system Dual [https://ark.intel.com/content/www/us/en/ark/products/192440/intel-xeon-gold-6242-processor-22m-cache-2-80-ghz.html Intel Xeon Gold 6242 CPU @ 2.80GHz], 6 x [https://www.nvidia.com/en-us/data-center/tesla-t4/ T4 Nvidia cards] for a total of 15360 CUDA cores plus 1920 Tensor cores, 768 Gigs of RAM. The system also arrived with 416GB of storage for the OS and 3.49 TB of storage for scratch storage. | ||
+ | * cryosparc2.uio.no VM: 4 x Intel Core Processor (Haswell, IBRS) vcores, 16 Gb of RAM and 170 Gigs of disk space to run a discrete CryoSPARC web interface instance | ||
+ | |||
= Aristotle/CryoMP Installation Documentation ( intaristotle.internal.biotek/192.168.8.109 ) = | = Aristotle/CryoMP Installation Documentation ( intaristotle.internal.biotek/192.168.8.109 ) = | ||
[https://rt.uio.no/Ticket/Update.html?id=3135527 RT ticket #3135527] | [https://rt.uio.no/Ticket/Update.html?id=3135527 RT ticket #3135527] | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
= Software Installed, List of = | = Software Installed, List of = | ||
Line 18: | Line 20: | ||
{| class="wikitable" style="text-align: center;" | {| class="wikitable" style="text-align: center;" | ||
|- | |- | ||
− | | '''Application''' | + | | '''Application''' || '''Version on intaristotle.internal.biotek server''' || '''Status''' || '''Comments''' || |
|- | |- | ||
− | | [https://www.scripps.edu/tainer/arvai/adxv.html adxv] | + | | [https://www.scripps.edu/tainer/arvai/adxv.html adxv] || 1.9.13 || Done || - || |
|- | |- | ||
− | | [http://www.ccp4.ac.uk/ ccp4] | + | | [http://www.ccp4.ac.uk/ ccp4] || 7.1.003 || Done || - || |
|- | |- | ||
− | | [https://www.cgl.ucsf.edu/chimera/ chimera] | + | | [https://www.cgl.ucsf.edu/chimera/ chimera] || 1.14 || Done || - || |
|- | |- | ||
− | | [https://cistem.org/ cistem] | + | | [https://cistem.org/ cistem] || 1.0.0-beta || Done || - || |
|- | |- | ||
− | | [ | + | | [https://sphire.mpg.de/wiki/doku.php?id=pipeline:window:cryolo cryolo] || 1.5 || Done || - || |
|- | |- | ||
− | | [http:// | + | | [http://grigoriefflab.janelia.org/ctffind4 ctffind4] || 4.1.10 || Done || - || |
|- | |- | ||
− | | [http:// | + | | [http://prometheus.uio.no:3900/ cryosparc] || 3.1 || Done || [https://wiki.uio.no/medicin/ncmm/IT/index.php/Prometheus.uio.no Documentation] || |
|- | |- | ||
− | | [http://www. | + | | [http://www.msg.ucsf.edu/local/programs/eman/ eman2] || 2.31 || Done || - || |
|- | |- | ||
− | | [http://www.fftw.org/ fftw, | + | | [http://www.fftw.org/ fftw, double precision] || 2.1.5 || Done || - || |
|- | |- | ||
− | | [http://www.fftw.org/ fftw, | + | | [http://www.fftw.org/ fftw, single precision] || 2.1.5 || Done || - || |
|- | |- | ||
− | | [http://www.fftw.org/ fftw, | + | | [http://www.fftw.org/ fftw, double precision] || 3.3.8 || Done || - || |
|- | |- | ||
− | |[ | + | | [http://www.fftw.org/ fftw, single precision] || 3.3.8 || Done || - || |
− | | | ||
− | |Done | ||
− | | - | ||
− | | | ||
|- | |- | ||
− | | | + | |[https://bio3d.colorado.edu/imod/download.html#Development imod] || 4.9.12 || Done || - || |
|- | |- | ||
− | | [ | + | | [https://www.openssl.org/ openssl] || 1.0.2 || Done || - || |
|- | |- | ||
− | | [ | + | | [http://grigoriefflab.janelia.org/frealign frealign] || 9.11 || Done || - || |
|- | |- | ||
− | | [https://www.mrc-lmb.cam.ac.uk/kzhang/ | + | | [https://www.mrc-lmb.cam.ac.uk/kzhang/ gautomatch] || 0.56 || Done || - || |
|- | |- | ||
+ | | [https://www.mrc-lmb.cam.ac.uk/kzhang/ gctf] || 1.18 || Done || - || | ||
|- | |- | ||
− | | [https://modules.readthedocs.io/en/latest/ modules] | + | | [https://modules.readthedocs.io/en/latest/ modules] || 4.20 || Done || - || |
|- | |- | ||
− | | [http://msg.ucsf.edu/em/software/motioncor2.html motioncor2] || 1.2.1 | + | | [http://msg.ucsf.edu/em/software/motioncor2.html motioncor2] || 1.2.1 || Done || - || |
|- | |- | ||
− | | [https://www.mpich.org/ mpich] | + | | [https://www.mpich.org/ mpich] || 3.0.4 || Done || - || |
|- | |- | ||
− | | [https://developer.nvidia.com/cuda-zone nvidia/cuda] | + | | [https://developer.nvidia.com/cuda-zone nvidia/cuda] || 8.0-GA1, 8.0-GA2, 9.0, 9.1, 9.2 || Done || - || |
|- | |- | ||
− | | [https://www.open-mpi.org/ openmpi] | + | | [https://www.open-mpi.org/ openmpi] || 3.1.3[+cuda] || Done || - || |
|- | |- | ||
− | | [https://www.open-mpi.org/ openmpi] | + | | [https://www.open-mpi.org/ openmpi] || 4.0.0[+cuda] || Done || - || |
|- | |- | ||
− | | [https://www.phenix-online.org/ phenix] | + | | [https://www.phenix-online.org/ phenix] || 1.18.2-3874 || Done || - || |
|- | |- | ||
| [https://www2.mrc-lmb.cam.ac.uk/relion/index.php?title=Main_Page relion] || 3.0.4, 3.0.6 || Done || - || | | [https://www2.mrc-lmb.cam.ac.uk/relion/index.php?title=Main_Page relion] || 3.0.4, 3.0.6 || Done || - || | ||
|- | |- | ||
− | | [https://github.com/I2PC/scipion scipion] | + | | [https://github.com/I2PC/scipion scipion] || 2.1 || Done || - || |
|- | |- | ||
− | | [http://sphire.mpg.de/ sphire] | + | | [http://sphire.mpg.de/ sphire] || 1.3 || Done || - || |
|- | |- | ||
− | | | + | | [http://cb.csail.mit.edu/cb/topaz/ topaz] || 0.2.3 || Done || - || |
|- | |- | ||
− | | [http://xds.mpimf-heidelberg.mpg.de/ xds] | + | | [https://www.rbvi.ucsf.edu/chimerax/ xchimera] || 0.8 || Done || - || |
+ | |- | ||
+ | | [http://xds.mpimf-heidelberg.mpg.de/ xds] || 2021-03-31 || Done || - || | ||
|- | |- | ||
|} | |} | ||
Line 121: | Line 122: | ||
= How was each piece of software compiled (flags, switches) = | = How was each piece of software compiled (flags, switches) = | ||
+ | |||
+ | == advx == | ||
+ | |||
+ | binary distribution from [https://www.scripps.edu/tainer/arvai/adxv.html here]. Do not forget to download the [https://www.scripps.edu/tainer/arvai/adxv/fonts.tar.gz relevant fonts] | ||
== ccp4 == | == ccp4 == | ||
Line 131: | Line 136: | ||
</pre> | </pre> | ||
− | + | == arp/warp == | |
<pre> | <pre> | ||
Line 173: | Line 178: | ||
</pre> | </pre> | ||
− | === 1.3.6 === | + | === 1.3.6 === |
<pre> | <pre> | ||
Line 190: | Line 195: | ||
</pre> | </pre> | ||
− | <pre>conda activate /lsc/sphire/1.3.6/.conda/envs/cryolo</pre> to activate | + | <pre>conda activate /lsc/sphire/1.3.6/.conda/envs/cryolo</pre> to activate. |
− | === 1.4.0 === | + | === 1.4.0 === |
<pre> | <pre> | ||
Line 208: | Line 213: | ||
</pre> | </pre> | ||
− | <pre>conda activate /lsc/cryolo/1.4.0</pre> to activate | + | <pre>conda activate /lsc/cryolo/1.4.0</pre> to activate. |
+ | |||
+ | === 1.5.0 === | ||
+ | |||
+ | <pre> | ||
+ | cd /lsc/sources/cryolo | ||
+ | /lsc/anaconda/5.3.1/bin/conda create --prefix=/lsc/cryolo/1.5.0 -c conda -c conda-forge -c anaconda python=3.6 pyqt=5 cudnn=7.1.2 numpy==1.14.5 cython wxPython==4.0.4 intel-openmp==2019.4 | ||
+ | /lsc/anaconda/5.3.1/bin/conda activate /lsc/cryolo/1.5.0 | ||
+ | . /lsc/anaconda/5.3.1/etc/profile.d/conda.sh | ||
+ | |||
+ | </pre> | ||
+ | |||
+ | <pre>/lsc/anaconda/5.3.1/bin/conda activate /lsc/cryolo/1.5.0</pre> to activate. | ||
+ | |||
+ | == cryosparc == | ||
+ | * installation see RT #3978772 | ||
+ | * update (here for <code>sekulicgpu1</code>/<code>cryosparc2</code>) | ||
+ | <pre> | ||
+ | [root@cryosparc2 cryosparc]# su cryosparc_user | ||
+ | bash-4.2$ cryosparcm backup | ||
+ | |||
+ | Backing up to /lsc/cryosparc/2.15.0/.../cryosparc2_database/backup/cryosparc_backup_2021_02_09_13h33.archive | ||
+ | [...] | ||
+ | bash-4.2$ cryosparcm update --check | ||
+ | CryoSPARC current version v3.0.1 | ||
+ | update starting on Tue Feb 9 13:44:40 CET 2021 | ||
+ | |||
+ | current version v3.0.1 | ||
+ | new version v3.1.0 | ||
+ | |||
+ | Update available! | ||
+ | bash-4.2$ cryosparcm update | ||
+ | CryoSPARC current version v3.0.1 | ||
+ | update starting on Tue Feb 9 13:44:58 CET 2021 | ||
+ | |||
+ | No version specified - updating to latest version. | ||
+ | [...] | ||
+ | </pre> | ||
+ | |||
+ | Notes | ||
+ | * this updates both master and worker | ||
+ | * if update has to be re-done for any reason (e.g., update worked on master, but something was stuck on worker, etc.), this here can be useful (but usually should not be required!): | ||
+ | |||
+ | <pre> | ||
+ | bash-4.2$ cryosparcm stop | ||
+ | CryoSPARC is running. | ||
+ | Stopping cryoSPARC | ||
+ | Shut down | ||
+ | bash-4.2$ cryosparcm update --override | ||
+ | [...] | ||
+ | </pre> | ||
== ctffind4 == | == ctffind4 == | ||
+ | binary blob here https://grigoriefflab.umassmed.edu/ctffind4 | ||
+ | |||
+ | == 4.1.10 == | ||
<pre>cd /lsc/sources/ctffind/4.1.10/ctffind-4.1.10 | <pre>cd /lsc/sources/ctffind/4.1.10/ctffind-4.1.10 | ||
./configure --prefix=/lsc/ctffind4/4.1.10 --enable-latest-instruction-set | ./configure --prefix=/lsc/ctffind4/4.1.10 --enable-latest-instruction-set | ||
make -j 64 all | make -j 64 all | ||
make -j 64 install</pre> | make -j 64 install</pre> | ||
+ | |||
+ | == 4.1.13 == | ||
+ | |||
+ | This is not working: this new version of ctf find requires the wxWidgets and /usr/bin/wx-config --query-version returns 2.8 (needs version 3.0.0 and above). Will use static binary instead for now. | ||
+ | |||
+ | Suspended till we upgrade the machine to something newer | ||
+ | |||
+ | <pre>yum install wxGTK3-webview.x86_64 wxGTK3-media.x86_64 wxGTK3-gl.x86_64 wxGTK3.x86_64 wxGTK-media.x86_64 wxGTK-gl.x86_64 wxGTK.x86_64 wxBase3.x86_64 wxBase.x86_64 mathgl-wx-devel.x86_64 mathgl-wx.x86_64 erlang-wx.x86_64 compat-wxGTK3-gtk2-media.x86_64 compat-wxGTK3-gtk2-gl.x86_64 compat-wxGTK3-gtk2.x86_64 compat-wxBase3-gtk2.x86_64 | ||
+ | cd /lsc/sources/ctffind/4.1.13/ctffind-4.1.13 | ||
+ | /configure --prefix=/lsc/ctffind/4.1.13 --enable-latest-instruction-set --enable-openmp --with-wx-config=/usr/bin/wx-config --with-fftw-dir=/lsc/fftw/3.3.8 | ||
+ | make -j 64 all | ||
+ | make -j 64 install</pre> | ||
+ | |||
+ | == 4.1.13 + cuda == | ||
+ | |||
+ | binary distribution | ||
== eman2 == | == eman2 == | ||
+ | |||
+ | === 2.2 === | ||
<pre>cd /lsc/sources/eman2/2.2/ | <pre>cd /lsc/sources/eman2/2.2/ | ||
Line 230: | Line 306: | ||
> export PATH=/opt/eman/2.2/bin:$PATH | > export PATH=/opt/eman/2.2/bin:$PATH | ||
# covered in the relevant environment module</pre> | # covered in the relevant environment module</pre> | ||
+ | |||
+ | === 2.3 === | ||
+ | |||
+ | <pre>cd /lsc/sources/eman2/2.3/ | ||
+ | ./eman2.3.linux64.sh | ||
+ | > EMAN2 will now be installed into this location: | ||
+ | > [/root/EMAN2] >>> /lsc/eman2/2.3 | ||
+ | # installer does the rest of the work, re-installing a bunch of python modules via anaconda | ||
+ | > Do you wish the installer to prepend the EMAN2 install location | ||
+ | > to PATH in your /root/.bashrc ? [yes|no] | ||
+ | > [no] >>> no | ||
+ | > You may wish to edit your .bashrc to prepend the EMAN2 install location to PATH: | ||
+ | > export PATH=/opt/eman/2.3/bin:$PATH | ||
+ | # covered in the relevant environment module</pre> | ||
+ | |||
+ | === 2.31 === | ||
+ | |||
+ | <pre>cd /lsc/sources/eman2/2.31/ | ||
+ | eman2.31_sphire1.3_huge.linux64.sh | ||
+ | > EMAN2 will now be installed into this location: | ||
+ | > [/root/EMAN2] >>> /lsc/eman2/2.31 | ||
+ | # installer does the rest of the work, re-installing a bunch of python modules via anaconda | ||
+ | > Do you wish the installer to prepend the EMAN2 install location | ||
+ | > to PATH in your /root/.bashrc ? [yes|no] | ||
+ | > [no] >>> no | ||
+ | > You may wish to edit your .bashrc to prepend the EMAN2 install location to PATH: | ||
+ | > export PATH=/opt/eman/2.31/bin:$PATH | ||
+ | # covered in the relevant environment module</pre> | ||
+ | |||
+ | == Dynamo == | ||
+ | |||
+ | https://wiki.dynamo.biozentrum.unibas.ch/w/index.php/Main_Page | ||
+ | |||
+ | === 1401 === | ||
== external/fftw == | == external/fftw == | ||
Line 308: | Line 418: | ||
− | Update: In order to | + | Update: In order to tackle the issue related to creating remote OpenGL context (X11 forwarding over ssh), the driver to 415.23 was upgraded but this did not yield the desired result. In nvidia-smi, the new driver [https://stackoverflow.com/questions/53422407/different-cuda-versions-shown-by-nvcc-and-nvidia-smi is reported to be compatible with CUDA 10], which is fine as it can use the CUDA 8 run-time environment, as [https://docs.nvidia.com/deploy/cuda-compatibility/index.html NVIDIA guarantees binary compatibility] |
== nvidia/cuda == | == nvidia/cuda == | ||
Line 319: | Line 429: | ||
* 9.1 | * 9.1 | ||
* 9.2 | * 9.2 | ||
+ | * 10.2 | ||
− | <pre>Default CUDA framework for | + | <pre>Default CUDA framework for most applications is 8.0GA1</pre> |
− | + | The Nvidia driver version for the kernel installed is Driver Version: 440.44 of CUDA Version: 10.2 | |
− | + | When different CUDA versions are loaded, the NVIDIA driver remains the same. Things seem to work, but may require additional detailed testing in the future. | |
− | + | All CUDA installations require GCC 4.8 (hence the reason we went with CentOS). | |
− | |||
− | All CUDA installations require | ||
All mentioned versions are currently installed, but we regard 8.0-GA1 as the base release, due to software that demands 8.0-GA1 and has no alternatives. | All mentioned versions are currently installed, but we regard 8.0-GA1 as the base release, due to software that demands 8.0-GA1 and has no alternatives. | ||
Line 355: | Line 464: | ||
./9.2.148_396.37_linux --silent --toolkit=/lsc/nvidia/cuda/9.2 --samples --samplespath=/lsc/nvidia/cuda/9.2/samples --run-nvidia-xconfig --tmpdir=/tmp</pre> | ./9.2.148_396.37_linux --silent --toolkit=/lsc/nvidia/cuda/9.2 --samples --samplespath=/lsc/nvidia/cuda/9.2/samples --run-nvidia-xconfig --tmpdir=/tmp</pre> | ||
− | === openmpi | + | ==== 10.2 ==== |
+ | |||
+ | <pre>mkdir /lsc/nvidia/cuda/10.2; | ||
+ | wget http://developer.download.nvidia.com/compute/cuda/10.2/Prod/local_installers/cuda_10.2.89_440.33.01_linux.run; | ||
+ | sh ./cuda_10.2.89_440.33.01_linux.run --silent --toolkitpath=/lsc/nvidia/cuda/10.2 --toolkit=/lsc/nvidia/cuda/10.2 --samples --samplespath=/lsc/nvidia/cuda/10.2/samples --run-nvidia-xconfig --tmpdir=/storage/scratch/tmp</pre> | ||
+ | |||
+ | == openmpi == | ||
<pre>cd /lsc/sources/openmpi/3.1.2; | <pre>cd /lsc/sources/openmpi/3.1.2; | ||
./configure --prefix /lsc/openmpi/3.1.2 --enable-binaries --enable-mpi-fortran --with-cuda=/lsc/nvidia/cuda/8.0-GA1 --with-devel-headers </pre> | ./configure --prefix /lsc/openmpi/3.1.2 --enable-binaries --enable-mpi-fortran --with-cuda=/lsc/nvidia/cuda/8.0-GA1 --with-devel-headers </pre> | ||
− | + | === Default: 3.1.2 === | |
<pre>The default version loaded is the stable-as-of 2018-10-31 3.1.2</pre> | <pre>The default version loaded is the stable-as-of 2018-10-31 3.1.2</pre> | ||
− | |||
− | + | == phenix == | |
− | |||
− | |||
− | ==== 2.1 | + | <pre>cd /lsc/sources/phenix/$version/phenix-installer-$version-intel-linux-2.6-x86_64-centos6 ; |
+ | ./install --prefix=/lsc/phenix --openmp --makedirs </pre> | ||
+ | |||
+ | == relion == | ||
+ | |||
+ | === 2.1 === | ||
<pre>cd /lsc/sources/relion/2.1-mpich ; | <pre>cd /lsc/sources/relion/2.1-mpich ; | ||
Line 375: | Line 492: | ||
make -j 64 | make -j 64 | ||
make -j 64 install</pre> | make -j 64 install</pre> | ||
− | + | ||
+ | === 3.0b === | ||
<pre>cd /lsc/sources/relion/3.0b-mpich ; | <pre>cd /lsc/sources/relion/3.0b-mpich ; | ||
Line 383: | Line 501: | ||
make -j 64 install</pre> | make -j 64 install</pre> | ||
− | + | === 3.0.4 === | |
− | |||
<pre> | <pre> | ||
cd /lsc/sources/relion/3.0.4 ; | cd /lsc/sources/relion/3.0.4 ; | ||
− | mkdir build; | + | mkdir build && cd build; |
cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/3.0.4 --DGUI=ON -DCUDA=ON -DCudaTexture=ON -DFORCE_OWN_TBB=ON -DFORCE_OWN_FLTK=ON -DFORCE_OWN_FFTW=ON -DCUDA_ARCH=61 -DBUILD_SHARED_LIBS=ON .. | cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/3.0.4 --DGUI=ON -DCUDA=ON -DCudaTexture=ON -DFORCE_OWN_TBB=ON -DFORCE_OWN_FLTK=ON -DFORCE_OWN_FFTW=ON -DCUDA_ARCH=61 -DBUILD_SHARED_LIBS=ON .. | ||
make -j 64 | make -j 64 | ||
Line 394: | Line 511: | ||
</pre> | </pre> | ||
− | + | === 3.0.6 === | |
<pre> | <pre> | ||
cd /lsc/sources/relion/3.0.6 ; | cd /lsc/sources/relion/3.0.6 ; | ||
− | mkdir build; | + | mkdir build && cd build; |
cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/3.0.6 --DGUI=ON -DCUDA=ON -DCudaTexture=ON -DFORCE_OWN_TBB=ON -DFORCE_OWN_FLTK=ON -DFORCE_OWN_FFTW=ON -DCUDA_ARCH=61 -DBUILD_SHARED_LIBS=ON .. | cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/3.0.6 --DGUI=ON -DCUDA=ON -DCudaTexture=ON -DFORCE_OWN_TBB=ON -DFORCE_OWN_FLTK=ON -DFORCE_OWN_FFTW=ON -DCUDA_ARCH=61 -DBUILD_SHARED_LIBS=ON .. | ||
make -j 64 | make -j 64 | ||
Line 404: | Line 521: | ||
</pre> | </pre> | ||
− | + | === 3.0.8 === | |
<pre> | <pre> | ||
cd /lsc/sources/relion/3.0.8 ; | cd /lsc/sources/relion/3.0.8 ; | ||
− | mkdir build; | + | mkdir build && cd build; |
cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/3.0.8 --DGUI=ON -DCUDA=ON -DCudaTexture=ON -DFORCE_OWN_TBB=ON -DFORCE_OWN_FLTK=ON -DFORCE_OWN_FFTW=ON -DCUDA_ARCH=61 -DBUILD_SHARED_LIBS=ON .. | cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/3.0.8 --DGUI=ON -DCUDA=ON -DCudaTexture=ON -DFORCE_OWN_TBB=ON -DFORCE_OWN_FLTK=ON -DFORCE_OWN_FFTW=ON -DCUDA_ARCH=61 -DBUILD_SHARED_LIBS=ON .. | ||
make -j 64 | make -j 64 | ||
Line 414: | Line 531: | ||
</pre> | </pre> | ||
− | + | === 3.1-beta === | |
<pre> | <pre> | ||
− | cd /lsc/sources/relion/3.1-beta | + | cd /lsc/sources/relion/relion.git |
− | + | git checkout ver3.1 | |
+ | mkdir build && cd build | ||
+ | module load relion/3.1-beta | ||
cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/3.1-beta --DGUI=ON -DCUDA=ON -DCudaTexture=ON -DFORCE_OWN_TBB=ON -DFORCE_OWN_FLTK=ON -DFORCE_OWN_FFTW=ON -DCUDA_ARCH=61 -DBUILD_SHARED_LIBS=ON .. | cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/3.1-beta --DGUI=ON -DCUDA=ON -DCudaTexture=ON -DFORCE_OWN_TBB=ON -DFORCE_OWN_FLTK=ON -DFORCE_OWN_FFTW=ON -DCUDA_ARCH=61 -DBUILD_SHARED_LIBS=ON .. | ||
make -j 64 | make -j 64 | ||
Line 424: | Line 543: | ||
</pre> | </pre> | ||
− | + | === 3.1 === | |
+ | |||
+ | <pre> | ||
+ | cd /lsc/sources/relion/3.1 | ||
+ | mkdir build && cd build | ||
+ | module load relion/3.1 | ||
+ | cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/3.1 --DGUI=ON -DCUDA=ON -DCudaTexture=ON -DFORCE_OWN_TBB=ON -DFORCE_OWN_FLTK=ON -DFORCE_OWN_FFTW=ON -DCUDA_ARCH=61 -DBUILD_SHARED_LIBS=ON .. | ||
+ | make -j 64 | ||
+ | make install | ||
+ | </pre> | ||
− | |||
− | + | List of CUDA architectures: https://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/ | |
− | ===== Cloning/Getting software | + | == scipion == |
+ | |||
+ | === 2.0 === | ||
+ | |||
+ | Note: The comments for scipion and the Sekulic GPU server are applicable to the module scipion/2.0source. Its installation is described in [https://rt.uio.no/Ticket/Display.html?id=3879628 RT 3879628]. The instructions below are just basic steps not aimed at production installation. | ||
+ | |||
+ | ==== Cloning/Getting software ==== | ||
mkdir -p /lsc/scipion/2.0 | mkdir -p /lsc/scipion/2.0 | ||
git clone https://github.com/I2PC/scipion.git | git clone https://github.com/I2PC/scipion.git | ||
cd scipion | cd scipion | ||
− | + | ==== Dependencies ==== | |
yum install -y gcc gcc-g++ cmake java-1.8.0-openjdk-devel.x86_64 libXft-devel.x86_64 openssl-devel.x86_64 libXext-devel.x86_64 libxml++.x86_64 libquadmath-devel.x86_64 libxslt.x86_64 openmpi-devel.x86_64 gsl-devel.x86_64 libX11.x86_64 gcc-gfortran.x86_64 git | yum install -y gcc gcc-g++ cmake java-1.8.0-openjdk-devel.x86_64 libXft-devel.x86_64 openssl-devel.x86_64 libXext-devel.x86_64 libxml++.x86_64 libquadmath-devel.x86_64 libxslt.x86_64 openmpi-devel.x86_64 gsl-devel.x86_64 libX11.x86_64 gcc-gfortran.x86_64 git | ||
− | + | ==== Configuring ==== | |
./scipion config | ./scipion config | ||
− | scipion installp -p | + | scipion installp -p scipion-em-relion |
+ | |||
+ | |||
+ | === 2.1 === | ||
+ | |||
+ | Note: same as above for xmipp2 | ||
− | === | + | ==== Cloning/Getting software ==== |
+ | <pre> | ||
+ | mkdir -p /lsc/scipion/2.0 | ||
+ | git clone https://github.com/I2PC/scipion.git | ||
+ | cd scipion | ||
+ | </pre> | ||
− | ==== 1.1 ==== | + | ==== Dependencies ==== |
+ | yum install -y gcc gcc-g++ cmake java-1.8.0-openjdk-devel.x86_64 libXft-devel.x86_64 openssl-devel.x86_64 libXext-devel.x86_64 libxml++.x86_64 libquadmath-devel.x86_64 libxslt.x86_64 openmpi-devel.x86_64 gsl-devel.x86_64 libX11.x86_64 gcc-gfortran.x86_64 git | ||
+ | |||
+ | ==== Configuring and installing modules from the command-line ==== | ||
+ | |||
+ | <pre>./scipion config | ||
+ | ./scipion installp --help | awk -p '{ print $1 }' | grep 'scipion-em' | grep -E -v 'Configuration|xmipp2' > modules.out | ||
+ | ./scipion installp -p scipion-em-xmipp | ||
+ | for module in $( < module.out ); do ./scipion installp -p $module ); done | ||
+ | rm module.out | ||
+ | </pre> | ||
+ | |||
+ | |||
+ | see [https://rt.uio.no/Ticket/Display.html?id=4049861 ticket 4049861]: some modules cannot compile due to python2 issues: | ||
+ | |||
+ | <pre> | ||
+ | ./scipion installp -p scipion-em-cryodrgn 3.0.0b4 # python2 issues | ||
+ | ./scipion installp -p scipion-em-dynamo 3.1.4 # python2 issues | ||
+ | ./scipion installp -p scipion-em-fsc3d 3.0.5 # python2 issues | ||
+ | ./scipion installp -p scipion-em-imod 3.0.3 # python2 issues | ||
+ | ./scipion installp -p scipion-em-novactf 3.0.0 # python2 issues | ||
+ | ./scipion installp -p scipion-em-pyseg 0.2 # python2 issues | ||
+ | ./scipion installp -p scipion-em-sidesplitter 3.0.3 # python2 issues | ||
+ | ./scipion installp -p scipion-em-tomo 3.0.5 # python2 issues | ||
+ | </pre> | ||
+ | |||
+ | == sphire == | ||
+ | |||
+ | === 1.1 === | ||
<pre>cd /lsc/sources/sphire/1.1 ; | <pre>cd /lsc/sources/sphire/1.1 ; | ||
./sphire_1_1_linux.sh | ./sphire_1_1_linux.sh | ||
# follow questions, install under /lsc/sphire/1.1</pre> | # follow questions, install under /lsc/sphire/1.1</pre> | ||
− | == How is each piece of software run ( really basic, just front UI, text or graphics) == | + | === 1.3 === |
+ | |||
+ | sphire 1.3 is included with the installation of eman 2.31. | ||
+ | |||
+ | <pre>module load eman 2.31</pre> | ||
+ | |||
+ | == topaz == | ||
+ | |||
+ | === 0.2.3 === | ||
+ | <pre> | ||
+ | [root@intaristotle ~]# conda create --prefix=/lsc/topaz/0.2.3 | ||
+ | Collecting package metadata: done | ||
+ | Solving environment: done | ||
+ | |||
+ | ## Package Plan ## | ||
+ | |||
+ | environment location: /lsc/topaz/0.2.3 | ||
+ | |||
+ | |||
+ | |||
+ | Proceed ([y]/n)? y | ||
+ | |||
+ | Preparing transaction: done | ||
+ | Verifying transaction: done | ||
+ | Executing transaction: done | ||
+ | # | ||
+ | # To activate this environment, use | ||
+ | # | ||
+ | # $ conda activate /lsc/topaz/0.2.3 | ||
+ | # | ||
+ | # To deactivate an active environment, use | ||
+ | # | ||
+ | # $ conda deactivate | ||
+ | |||
+ | [root@intaristotle ~]# conda activate /lsc/topaz/0.2.3 | ||
+ | (/lsc/topaz/0.2.3) [root@intaristotle ~]# module load nvidia/cuda/9.2 | ||
+ | (/lsc/topaz/0.2.3) [root@intaristotle ~]# conda install topaz cudatoolkit=9.2 -c tbepler -c pytorch | ||
+ | Collecting package metadata: done | ||
+ | Solving environment: done | ||
+ | |||
+ | ## Package Plan ## | ||
+ | |||
+ | environment location: /lsc/topaz/0.2.3 | ||
+ | |||
+ | added / updated specs: | ||
+ | - cudatoolkit=9.2 | ||
+ | - topaz | ||
+ | |||
+ | |||
+ | The following packages will be downloaded: | ||
+ | |||
+ | package | build | ||
+ | ---------------------------|----------------- | ||
+ | certifi-2020.4.5.1 | py38_0 159 KB | ||
+ | freetype-2.9.1 | h8a8886c_1 822 KB | ||
+ | intel-openmp-2020.0 | 166 916 KB | ||
+ | joblib-0.14.1 | py_0 202 KB | ||
+ | jpeg-9b | h024ee3a_2 248 KB | ||
+ | libpng-1.6.37 | hbc83047_0 364 KB | ||
+ | libtiff-4.1.0 | h2733197_0 589 KB | ||
+ | mkl-2020.0 | 166 202.1 MB | ||
+ | mkl-service-2.3.0 | py38he904b0f_0 68 KB | ||
+ | mkl_fft-1.0.15 | py38ha843d7b_0 178 KB | ||
+ | mkl_random-1.1.0 | py38h962f231_0 398 KB | ||
+ | ninja-1.9.0 | py38hfd86e86_0 1.6 MB | ||
+ | numpy-1.18.1 | py38h4f9e942_0 5 KB | ||
+ | numpy-base-1.18.1 | py38hde5b4d6_1 5.3 MB | ||
+ | olefile-0.46 | py_0 33 KB | ||
+ | pandas-1.0.3 | py38h0573a6f_0 11.5 MB | ||
+ | pillow-7.0.0 | py38hb39fc2d_0 664 KB | ||
+ | pip-20.0.2 | py38_1 1.9 MB | ||
+ | python-3.8.2 | hcf32534_0 57.8 MB | ||
+ | python-dateutil-2.8.1 | py_0 224 KB | ||
+ | pytorch-1.5.0 |py3.8_cuda9.2.148_cudnn7.6.3_0 344.4 MB pytorch | ||
+ | pytz-2019.3 | py_0 231 KB | ||
+ | scikit-learn-0.22.1 | py38hd81dba3_0 7.3 MB | ||
+ | scipy-1.4.1 | py38h0b6359f_0 19.1 MB | ||
+ | setuptools-46.1.3 | py38_0 656 KB | ||
+ | six-1.14.0 | py38_0 27 KB | ||
+ | topaz-0.2.3 | py_0 26.0 MB tbepler | ||
+ | torchvision-0.6.0 | py38_cu92 11.7 MB pytorch | ||
+ | wheel-0.34.2 | py38_0 49 KB | ||
+ | zstd-1.3.7 | h0b5b093_0 887 KB | ||
+ | ------------------------------------------------------------ | ||
+ | Total: 695.2 MB | ||
+ | |||
+ | The following NEW packages will be INSTALLED: | ||
+ | |||
+ | _libgcc_mutex pkgs/main/linux-64::_libgcc_mutex-0.1-main | ||
+ | blas pkgs/main/linux-64::blas-1.0-mkl | ||
+ | ca-certificates pkgs/main/linux-64::ca-certificates-2020.1.1-0 | ||
+ | certifi pkgs/main/linux-64::certifi-2020.4.5.1-py38_0 | ||
+ | cudatoolkit pkgs/main/linux-64::cudatoolkit-9.2-0 | ||
+ | freetype pkgs/main/linux-64::freetype-2.9.1-h8a8886c_1 | ||
+ | intel-openmp pkgs/main/linux-64::intel-openmp-2020.0-166 | ||
+ | joblib pkgs/main/noarch::joblib-0.14.1-py_0 | ||
+ | jpeg pkgs/main/linux-64::jpeg-9b-h024ee3a_2 | ||
+ | ld_impl_linux-64 pkgs/main/linux-64::ld_impl_linux-64-2.33.1-h53a641e_7 | ||
+ | libedit pkgs/main/linux-64::libedit-3.1.20181209-hc058e9b_0 | ||
+ | libffi pkgs/main/linux-64::libffi-3.2.1-hd88cf55_4 | ||
+ | libgcc-ng pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0 | ||
+ | libgfortran-ng pkgs/main/linux-64::libgfortran-ng-7.3.0-hdf63c60_0 | ||
+ | libpng pkgs/main/linux-64::libpng-1.6.37-hbc83047_0 | ||
+ | libstdcxx-ng pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0 | ||
+ | libtiff pkgs/main/linux-64::libtiff-4.1.0-h2733197_0 | ||
+ | mkl pkgs/main/linux-64::mkl-2020.0-166 | ||
+ | mkl-service pkgs/main/linux-64::mkl-service-2.3.0-py38he904b0f_0 | ||
+ | mkl_fft pkgs/main/linux-64::mkl_fft-1.0.15-py38ha843d7b_0 | ||
+ | mkl_random pkgs/main/linux-64::mkl_random-1.1.0-py38h962f231_0 | ||
+ | ncurses pkgs/main/linux-64::ncurses-6.2-he6710b0_1 | ||
+ | ninja pkgs/main/linux-64::ninja-1.9.0-py38hfd86e86_0 | ||
+ | numpy pkgs/main/linux-64::numpy-1.18.1-py38h4f9e942_0 | ||
+ | numpy-base pkgs/main/linux-64::numpy-base-1.18.1-py38hde5b4d6_1 | ||
+ | olefile pkgs/main/noarch::olefile-0.46-py_0 | ||
+ | openssl pkgs/main/linux-64::openssl-1.1.1g-h7b6447c_0 | ||
+ | pandas pkgs/main/linux-64::pandas-1.0.3-py38h0573a6f_0 | ||
+ | pillow pkgs/main/linux-64::pillow-7.0.0-py38hb39fc2d_0 | ||
+ | pip pkgs/main/linux-64::pip-20.0.2-py38_1 | ||
+ | python pkgs/main/linux-64::python-3.8.2-hcf32534_0 | ||
+ | python-dateutil pkgs/main/noarch::python-dateutil-2.8.1-py_0 | ||
+ | pytorch pytorch/linux-64::pytorch-1.5.0-py3.8_cuda9.2.148_cudnn7.6.3_0 | ||
+ | pytz pkgs/main/noarch::pytz-2019.3-py_0 | ||
+ | readline pkgs/main/linux-64::readline-8.0-h7b6447c_0 | ||
+ | scikit-learn pkgs/main/linux-64::scikit-learn-0.22.1-py38hd81dba3_0 | ||
+ | scipy pkgs/main/linux-64::scipy-1.4.1-py38h0b6359f_0 | ||
+ | setuptools pkgs/main/linux-64::setuptools-46.1.3-py38_0 | ||
+ | six pkgs/main/linux-64::six-1.14.0-py38_0 | ||
+ | sqlite pkgs/main/linux-64::sqlite-3.31.1-h62c20be_1 | ||
+ | tk pkgs/main/linux-64::tk-8.6.8-hbc83047_0 | ||
+ | topaz tbepler/noarch::topaz-0.2.3-py_0 | ||
+ | torchvision pytorch/linux-64::torchvision-0.6.0-py38_cu92 | ||
+ | wheel pkgs/main/linux-64::wheel-0.34.2-py38_0 | ||
+ | xz pkgs/main/linux-64::xz-5.2.5-h7b6447c_0 | ||
+ | zlib pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3 | ||
+ | zstd pkgs/main/linux-64::zstd-1.3.7-h0b5b093_0 | ||
+ | |||
+ | |||
+ | Proceed ([y]/n)? y | ||
+ | |||
+ | |||
+ | Downloading and Extracting Packages | ||
+ | mkl-2020.0 | 202.1 MB | ################################################################################################# | 100% | ||
+ | pytorch-1.5.0 | 344.4 MB | ################################################################################################# | 100% | ||
+ | jpeg-9b | 248 KB | ################################################################################################# | 100% | ||
+ | python-dateutil-2.8. | 224 KB | ################################################################################################# | 100% | ||
+ | pip-20.0.2 | 1.9 MB | ################################################################################################# | 100% | ||
+ | joblib-0.14.1 | 202 KB | ################################################################################################# | 100% | ||
+ | certifi-2020.4.5.1 | 159 KB | ################################################################################################# | 100% | ||
+ | python-3.8.2 | 57.8 MB | ################################################################################################# | 100% | ||
+ | setuptools-46.1.3 | 656 KB | ################################################################################################# | 100% | ||
+ | pytz-2019.3 | 231 KB | ################################################################################################# | 100% | ||
+ | mkl_fft-1.0.15 | 178 KB | ################################################################################################# | 100% | ||
+ | six-1.14.0 | 27 KB | ################################################################################################# | 100% | ||
+ | numpy-1.18.1 | 5 KB | ################################################################################################# | 100% | ||
+ | libtiff-4.1.0 | 589 KB | ################################################################################################# | 100% | ||
+ | zstd-1.3.7 | 887 KB | ################################################################################################# | 100% | ||
+ | ninja-1.9.0 | 1.6 MB | ################################################################################################# | 100% | ||
+ | scikit-learn-0.22.1 | 7.3 MB | ################################################################################################# | 100% | ||
+ | pillow-7.0.0 | 664 KB | ################################################################################################# | 100% | ||
+ | numpy-base-1.18.1 | 5.3 MB | ################################################################################################# | 100% | ||
+ | freetype-2.9.1 | 822 KB | ################################################################################################# | 100% | ||
+ | scipy-1.4.1 | 19.1 MB | ################################################################################################# | 100% | ||
+ | pandas-1.0.3 | 11.5 MB | ################################################################################################# | 100% | ||
+ | mkl_random-1.1.0 | 398 KB | ################################################################################################# | 100% | ||
+ | mkl-service-2.3.0 | 68 KB | ################################################################################################# | 100% | ||
+ | libpng-1.6.37 | 364 KB | ################################################################################################# | 100% | ||
+ | wheel-0.34.2 | 49 KB | ################################################################################################# | 100% | ||
+ | torchvision-0.6.0 | 11.7 MB | ################################################################################################# | 100% | ||
+ | topaz-0.2.3 | 26.0 MB | ################################################################################################# | 100% | ||
+ | olefile-0.46 | 33 KB | ################################################################################################# | 100% | ||
+ | intel-openmp-2020.0 | 916 KB | ################################################################################################# | 100% | ||
+ | Preparing transaction: done | ||
+ | Verifying transaction: done | ||
+ | Executing transaction: done | ||
+ | (/lsc/topaz/0.2.3) [root@intaristotle ~]# topaz | ||
+ | Traceback (most recent call last): | ||
+ | File "/lsc/topaz/0.2.3/bin/topaz", line 6, in <module> | ||
+ | from pkg_resources import load_entry_point | ||
+ | File "/lsc/topaz/0.2.3/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3259, in <module> | ||
+ | def _initialize_master_working_set(): | ||
+ | File "/lsc/topaz/0.2.3/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3242, in _call_aside | ||
+ | f(*args, **kwargs) | ||
+ | File "/lsc/topaz/0.2.3/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3271, in _initialize_master_working_set | ||
+ | working_set = WorkingSet._build_master() | ||
+ | File "/lsc/topaz/0.2.3/lib/python3.8/site-packages/pkg_resources/__init__.py", line 584, in _build_master | ||
+ | ws.require(__requires__) | ||
+ | File "/lsc/topaz/0.2.3/lib/python3.8/site-packages/pkg_resources/__init__.py", line 901, in require | ||
+ | needed = self.resolve(parse_requirements(requirements)) | ||
+ | File "/lsc/topaz/0.2.3/lib/python3.8/site-packages/pkg_resources/__init__.py", line 787, in resolve | ||
+ | raise DistributionNotFound(req, requirers) | ||
+ | pkg_resources.DistributionNotFound: The 'future' distribution was not found and is required by torch | ||
+ | (/lsc/topaz/0.2.3) [root@intaristotle ~]# pip install future | ||
+ | Collecting future | ||
+ | Downloading future-0.18.2.tar.gz (829 kB) | ||
+ | |████████████████████████████████| 829 kB 4.3 MB/s | ||
+ | Building wheels for collected packages: future | ||
+ | Building wheel for future (setup.py) ... done | ||
+ | Created wheel for future: filename=future-0.18.2-py3-none-any.whl size=491058 sha256=f1d7892b69b66f8a96624ddc534bfc5dcf52e8042aa760743760f036db892c26 | ||
+ | Stored in directory: /root/.cache/pip/wheels/8e/70/28/3d6ccd6e315f65f245da085482a2e1c7d14b90b30f239e2cf4 | ||
+ | Successfully built future | ||
+ | Installing collected packages: future | ||
+ | Successfully installed future-0.18.2 | ||
+ | (/lsc/topaz/0.2.3) [root@intaristotle ~]# topaz | ||
+ | usage: topaz [-h] [--version] <command> ... | ||
+ | topaz: error: the following arguments are required: <command> | ||
+ | |||
+ | </pre> | ||
+ | |||
+ | === 0.2.4 === | ||
+ | * Installed on <code>sekulicgpu1</code> and <code>cryosparc2</code> | ||
+ | |||
+ | <pre> | ||
+ | [root@sekulicgpu1 topaz]# mkdir /lsc/venvs | ||
+ | [root@sekulicgpu1 topaz]# chown cryosparc_user /lsc/venvs | ||
+ | [root@sekulicgpu1 topaz]# su cryosparc_user | ||
+ | [cryosparc_user@sekulicgpu1 topaz]$ /lsc/python/3.6.11/bin/python3.6 -m venv /lsc/venvs/topaz | ||
+ | [cryosparc_user@sekulicgpu1 topaz]$ source /lsc/venvs/topaz/bin/activate | ||
+ | (topaz) [cryosparc_user@sekulicgpu1 topaz]$ mkdir /lsc/venvs/tmp | ||
+ | (topaz) [cryosparc_user@sekulicgpu1 topaz]$ env TMPDIR=/lsc/venvs/tmp pip3 install topaz-em | ||
+ | [...] | ||
+ | Successfully installed dataclasses-0.8 future-0.18.2 joblib-1.0.1 numpy-1.19.5 pandas-1.1.5 pillow-8.1.0 python-dateutil-2.8.1 pytz-2021.1 scikit-learn-0.24.1 scipy-1.5.4 six-1.15.0 threadpoolctl-2.1.0 topaz-em-0.2.4 torch-1.7.1 torchvision-0.8.2 typing-extensions-3.7.4.3 | ||
+ | (topaz) [cryosparc_user@sekulicgpu1 topaz]$ rmdir /lsc/venvs/tmp | ||
+ | </pre> | ||
+ | |||
+ | On <code>cryosparc2</code>, first install Python 3.6.11 (for consistency with <code>sekulicgpu1</code>), depends on 3 <code>-devel</code> packages: | ||
+ | <pre> | ||
+ | [root@cryosparc2 ~]# yum install zlib-devel openssl-devel bzip2-devel | ||
+ | [root@cryosparc2 ~]# mkdir -p /lsc/sources/python /lsc/python/3.6.11 | ||
+ | [root@cryosparc2 ~]# chown cryosparc_user /lsc/sources/python /lsc/python/3.6.11 | ||
+ | [root@cryosparc2 ~]# su cryosparc_user | ||
+ | bash-4.2$ cd /lsc/sources/python/ | ||
+ | bash-4.2$ wget https://www.python.org/ftp/python/3.6.11/Python-3.6.11.tar.xz | ||
+ | bash-4.2$ tar xf Python-3.6.11.tar.xz | ||
+ | bash-4.2$ cd Python-3.6.11 | ||
+ | bash-4.2$ ./configure --prefix=/lsc/python/3.6.11 --enable-optimizations && make && make install | ||
+ | </pre> | ||
+ | |||
+ | and then same steps for <code>topaz</code> as on <code>sekulicgpu1</code>: | ||
+ | <pre> | ||
+ | [root@cryosparc2 ~]# mkdir /lsc/venvs | ||
+ | [root@cryosparc2 ~]# chown cryosparc_user /lsc/venvs | ||
+ | [root@cryosparc2 ~]# su cryosparc_user | ||
+ | bash-4.2$ /lsc/python/3.6.11/bin/python3.6 -m venv /lsc/venvs/topaz | ||
+ | bash-4.2$ source /lsc/venvs/topaz/bin/activate | ||
+ | (topaz) bash-4.2$ mkdir /lsc/venvs/tmp | ||
+ | (topaz) bash-4.2$ env TMPDIR=/lsc/venvs/tmp pip3 install topaz-em | ||
+ | [...] | ||
+ | Successfully installed dataclasses-0.8 future-0.18.2 joblib-1.0.1 numpy-1.19.5 pandas-1.1.5 pillow-8.1.0 python-dateutil-2.8.1 pytz-2021.1 scikit-learn-0.24.1 scipy-1.5.4 six-1.15.0 threadpoolctl-2.1.0 topaz-em-0.2.4 torch-1.7.1 torchvision-0.8.2 typing-extensions-3.7.4.3 | ||
+ | (topaz) bash-4.2$ rmdir /lsc/venvs/tmp | ||
+ | </pre> | ||
+ | |||
+ | path to topaz: <code>/lsc/venvs/topaz/bin/topaz</code> | ||
+ | |||
+ | verify GPU support: | ||
+ | <pre> | ||
+ | [haroldgu@sekulicgpu1 ~]$ /lsc/venvs/topaz/bin/python -c 'import torch ; print(torch.cuda.is_available())' | ||
+ | True | ||
+ | </pre> | ||
+ | |||
+ | == How is each piece of software run (really basic, just front UI, text or graphics) == | ||
=== ccp4/4.7.0 === | === ccp4/4.7.0 === | ||
Line 466: | Line 896: | ||
<pre>module load eman | <pre>module load eman | ||
eman2.py</pre> | eman2.py</pre> | ||
+ | |||
+ | === eman2/2.3 === | ||
+ | |||
+ | <pre>module load eman/2.3 | ||
+ | eman2.py</pre> | ||
+ | |||
=== external === | === external === | ||
==== openssl/1.0.2o ==== | ==== openssl/1.0.2o ==== | ||
− | + | A principal SSL library. | |
=== frealign/9.11 === | === frealign/9.11 === | ||
Line 497: | Line 933: | ||
==== cuda/{ 8.0-GA1, 8.0-GA2, 9.0, 9.1, 9.2 } ==== | ==== cuda/{ 8.0-GA1, 8.0-GA2, 9.0, 9.1, 9.2 } ==== | ||
− | <pre>There is really nothing you can do with the CUDA libraries directly | + | <pre>There is really nothing you can do with the CUDA libraries directly. |
− | You can only verify that version of CUDA works with the following for loop in bash: | + | You can only verify that the current version of CUDA works with the following for loop in bash: |
for version in 8.0-GA1 8.0-GA2 9.0 9.1 9.2; | for version in 8.0-GA1 8.0-GA2 9.0 9.1 9.2; | ||
Line 513: | Line 949: | ||
=== phenix/1.14-3260 === | === phenix/1.14-3260 === | ||
− | <pre>module load phenix | + | <pre>module load phenix/1.14-3260 |
phenix</pre> | phenix</pre> | ||
− | |||
− | <pre>for version in 2.1 3.0b; | + | === phenix/1.18.2-3874 === |
+ | |||
+ | <pre>module load phenix/1.18.2-3874 | ||
+ | phenix</pre> | ||
+ | |||
+ | === relion/{ 2.1, 3.0b, 3.1 } === | ||
+ | |||
+ | <pre>for version in 2.1 3.0b 3.1; | ||
do | do | ||
module switch relion/$version; | module switch relion/$version; | ||
Line 527: | Line 969: | ||
<pre>module load sphire | <pre>module load sphire | ||
sphire</pre> | sphire</pre> | ||
+ | |||
+ | === sphire/1.3 === | ||
+ | |||
+ | <pre>module load sphire/1.3 | ||
+ | sphire</pre> | ||
+ | |||
=== xchimera/0.8 === | === xchimera/0.8 === | ||
Line 539: | Line 987: | ||
=== Installation === | === Installation === | ||
− | |||
<pre>cd /lsc/sources/modules/4.2.0</pre> | <pre>cd /lsc/sources/modules/4.2.0</pre> | ||
Line 590: | Line 1,037: | ||
==== 3D Classification ==== | ==== 3D Classification ==== | ||
− | Follow the instructions in the tutorial regarding 3D classification | + | Follow the instructions in the tutorial regarding 3D classification. |
Latest revision as of 09:55, 4 May 2023
There are two groups that work with CryoEM data: the Luecke Group and the Sekulic Group:
The Luecke Group CryoEM setup data processing platform consists of two physical servers and one VM:
- intaristotle.internal.biotek GPU server: 64 Intel Broadwell cores, 128 Gbytes of RAM, 4 x NVIDIA GTX 1080 GPU cards, 6 Tb of SSD scratch space
- perun.uio.no file server: 28 Intel Broadwell cores, 32 Gigs of RAM, 212 Tb of local disk space used as file server for the storage of CryoEM data. The machine is also used as a gateway to internal GPU/CPU processing nodes.
- cryosparc.uio.no VM: 2 x Intel Core Processor (Haswell, IBRS) vcores, 8 Gigs of RAM and 72 Gigs of disk space to run a discrete CryoSPARC web instance
All nodes have 10 Gigabit Ethernet connectivity for dedicated NAS/NFS disk space between the file server and the current (and future) GPU compute nodes.
The Sekulic Group CryoEM setup data processing platform consists of one physical server, one VM and a dedicated file share on the main pythagoras file server :
- sekulicgpu1.internal.biotek GPU server: 64 CPU core system Dual Intel Xeon Gold 6242 CPU @ 2.80GHz, 6 x T4 Nvidia cards for a total of 15360 CUDA cores plus 1920 Tensor cores, 768 Gigs of RAM. The system also arrived with 416GB of storage for the OS and 3.49 TB of storage for scratch storage.
- cryosparc2.uio.no VM: 4 x Intel Core Processor (Haswell, IBRS) vcores, 16 Gb of RAM and 170 Gigs of disk space to run a discrete CryoSPARC web interface instance
Contents
- 1 Aristotle/CryoMP Installation Documentation ( intaristotle.internal.biotek/192.168.8.109 )
- 2 Software Installed, List of
- 3 Top Directory ( /lsc )
- 4 How was each piece of software compiled (flags, switches)
- 4.1 advx
- 4.2 ccp4
- 4.3 arp/warp
- 4.4 chimera
- 4.5 cistem
- 4.6 cryolo
- 4.7 cryosparc
- 4.8 ctffind4
- 4.9 4.1.10
- 4.10 4.1.13
- 4.11 4.1.13 + cuda
- 4.12 eman2
- 4.13 Dynamo
- 4.14 external/fftw
- 4.15 external/openssl
- 4.16 frealign
- 4.17 gautomatch
- 4.18 gctf
- 4.19 imod
- 4.20 mpich
- 4.21 motioncor2
- 4.22 nvidia/driver
- 4.23 nvidia/cuda
- 4.24 openmpi
- 4.25 phenix
- 4.26 relion
- 4.27 scipion
- 4.28 sphire
- 4.29 topaz
- 4.30 How is each piece of software run (really basic, just front UI, text or graphics)
- 4.30.1 ccp4/4.7.0
- 4.30.2 chimera/1.13.1
- 4.30.3 ctffind4/4.1.10
- 4.30.4 eman2/2.2
- 4.30.5 eman2/2.3
- 4.30.6 external
- 4.30.7 frealign/9.11
- 4.30.8 gautomatch/0.53
- 4.30.9 gctf/1.18
- 4.30.10 mpich/3.0.4
- 4.30.11 motioncor2/1.2.1
- 4.30.12 nvidia/
- 4.30.13 openmpi/3.1.2
- 4.30.14 phenix/1.14-3260
- 4.30.15 phenix/1.18.2-3874
- 4.30.16 relion/{ 2.1, 3.0b, 3.1 }
- 4.30.17 sphire/1.1
- 4.30.18 sphire/1.3
- 4.30.19 xchimera/0.8
- 4.31 Environmental Modules
- 4.32 Running Test Jobs on Aristotle to test the software
Aristotle/CryoMP Installation Documentation ( intaristotle.internal.biotek/192.168.8.109 )
Software Installed, List of
Application | Version on intaristotle.internal.biotek server | Status | Comments | |
adxv | 1.9.13 | Done | - | |
ccp4 | 7.1.003 | Done | - | |
chimera | 1.14 | Done | - | |
cistem | 1.0.0-beta | Done | - | |
cryolo | 1.5 | Done | - | |
ctffind4 | 4.1.10 | Done | - | |
cryosparc | 3.1 | Done | Documentation | |
eman2 | 2.31 | Done | - | |
fftw, double precision | 2.1.5 | Done | - | |
fftw, single precision | 2.1.5 | Done | - | |
fftw, double precision | 3.3.8 | Done | - | |
fftw, single precision | 3.3.8 | Done | - | |
imod | 4.9.12 | Done | - | |
openssl | 1.0.2 | Done | - | |
frealign | 9.11 | Done | - | |
gautomatch | 0.56 | Done | - | |
gctf | 1.18 | Done | - | |
modules | 4.20 | Done | - | |
motioncor2 | 1.2.1 | Done | - | |
mpich | 3.0.4 | Done | - | |
nvidia/cuda | 8.0-GA1, 8.0-GA2, 9.0, 9.1, 9.2 | Done | - | |
openmpi | 3.1.3[+cuda] | Done | - | |
openmpi | 4.0.0[+cuda] | Done | - | |
phenix | 1.18.2-3874 | Done | - | |
relion | 3.0.4, 3.0.6 | Done | - | |
scipion | 2.1 | Done | - | |
sphire | 1.3 | Done | - | |
topaz | 0.2.3 | Done | - | |
xchimera | 0.8 | Done | - | |
xds | 2021-03-31 | Done | - |
Top Directory ( /lsc )
Top directory for all installed software is /lsc
Source Files ( in case you need to re-compile something )
Source for all installed programs can be found under /lsc/sources
Dependencies for interactive 3D applications
VirtualGL: Running accelerated 3D graphics remotely via OpenGL
Located under /opt/VirtualGL
Loaded by default
VirtualGL is an open source toolkit that gives any Unix or Linux remote display software the ability to run OpenGL applications with full 3D hardware acceleration.
It is used to display 3D applications on a laptop that does not have a powerful 3D graphics card: Instead of letting the laptop render the 3D application, VirtualGL uses the (more powerful) graphics cards of the server to pre-render the application and send back the result.
This app is very useful in 3D applications like XChimera/Chimera, PyMol and the like
In order to take full advantage of this application, you need to be connected to the NCMM internal network, and not UiO.
Open the command-line/terminal of your choice and type
vglconnect intaristotle.internal.biotek
vgl connect will connect you to the internal interface for aristotle and set up the environment for executing a graphical application.
Let’s say we need to run chimera. So, we load the module
module load chimera/1.13.1
and then we use vglrun to run the chimera application
vglrun -d $DISPLAY chimera
The “-d $DISPLAY” argument is there just in case something has corrupted your local environment.
How was each piece of software compiled (flags, switches)
advx
binary distribution from here. Do not forget to download the relevant fonts
ccp4
cd /lsc/sources/ccp4/7.0.074/ mkdir -p /lsc/ccp4/7.0.074/ cp -r /lsc/sources/ccp4/7.0.074/* /lsc/ccp4/7.0.074/ cd /lsc/ccp4/7.0.074/ ./BINARY.setup
arp/warp
module load ccp4/7.0.074 /lsc/ccp4/7.0.074/start cd /lsc/sources/ccp4/arp_warp_8.0 ./install.sh
chimera
cd /lsc/sources/chimera/1.13.1 ./chimera-1.13.1-linux_x86_64.bin #self-extracts into two files # answer questions > Enter install location: /lsc/chimera/1.13.1 > Install desktop menu (icon has to be done by user)? no > Install symbolic link to chimera executable for command line use in which directory? [hit Enter for default (0)]: 0 # installer copies files into destination > Installation is done; press return.
cistem
cd /lsc/sources/cistem/1.0/cistem-1.0.0-beta ./configure --prefix=/lsc/cistem/1.0 make make install
cryolo
1.2.1
cd /lsc/sources/cryolo/1.2.1/ scl enable rh-python36 bash conda create --name cryolo -c anaconda python=3.6 pyqt=5 cudnn=7.1.2 conda activate cryolo conda install numpy==1.14.5 pip install ./cryolo-1.2.1.tar.gz
1.3.6
cd /lsc/sources/cryolo wget ftp://ftp.gwdg.de/pub/misc/sphire/crYOLO_V1_3_6/cryolo-1.3.6.tar.gz wget ftp://ftp.gwdg.de/pub/misc/sphire/crYOLO_BM_V1_2_2/cryoloBM-1.2.2.tar.gz tar zxvf cryolo-1.3.6.tar.gz ln -s cryolo-1.4.0 1.4.0 conda env remove --name cryolo # Due to conda, cryolo cannot support previous installations conda create --prefix /lsc/sphire/1.3.6/.conda/envs/ -c anaconda python=3.6 pyqt=5 cudnn=7.1.2 cython source activate cryolo conda install numpy==1.15.4 pip install cryolo-1.3.6.tar.gz[gpu] pip install cryoloBM-1.2.2.tar.gz
conda activate /lsc/sphire/1.3.6/.conda/envs/cryoloto activate.
1.4.0
cd /lsc/sources/cryolo wget ftp://ftp.gwdg.de/pub/misc/sphire/crYOLO_V1_4_0/cryolo-1.4.0.tar.gz wget ftp://ftp.gwdg.de/pub/misc/sphire/crYOLO_BM_V1_2_3/cryoloBM-1.2.3.tar.gz tar zxvf cryolo-1.4.0.tar.gz ln -s cryolo-1.4.0 1.4.0 conda create --prefix /lsc/cryolo/1.4.0 -c anaconda python=3.6 pyqt=5 cudnn=7.1.2 cython conda activate /lsc/cryolo/1.4.0 conda install numpy==1.15.4 pip install cryolo-1.4.0.tar.gz[gpu] pip install cryoloBM-1.2.3.tar.gz
conda activate /lsc/cryolo/1.4.0to activate.
1.5.0
cd /lsc/sources/cryolo /lsc/anaconda/5.3.1/bin/conda create --prefix=/lsc/cryolo/1.5.0 -c conda -c conda-forge -c anaconda python=3.6 pyqt=5 cudnn=7.1.2 numpy==1.14.5 cython wxPython==4.0.4 intel-openmp==2019.4 /lsc/anaconda/5.3.1/bin/conda activate /lsc/cryolo/1.5.0 . /lsc/anaconda/5.3.1/etc/profile.d/conda.sh
/lsc/anaconda/5.3.1/bin/conda activate /lsc/cryolo/1.5.0to activate.
cryosparc
- installation see RT #3978772
- update (here for
sekulicgpu1
/cryosparc2
)
[root@cryosparc2 cryosparc]# su cryosparc_user bash-4.2$ cryosparcm backup Backing up to /lsc/cryosparc/2.15.0/.../cryosparc2_database/backup/cryosparc_backup_2021_02_09_13h33.archive [...] bash-4.2$ cryosparcm update --check CryoSPARC current version v3.0.1 update starting on Tue Feb 9 13:44:40 CET 2021 current version v3.0.1 new version v3.1.0 Update available! bash-4.2$ cryosparcm update CryoSPARC current version v3.0.1 update starting on Tue Feb 9 13:44:58 CET 2021 No version specified - updating to latest version. [...]
Notes
- this updates both master and worker
- if update has to be re-done for any reason (e.g., update worked on master, but something was stuck on worker, etc.), this here can be useful (but usually should not be required!):
bash-4.2$ cryosparcm stop CryoSPARC is running. Stopping cryoSPARC Shut down bash-4.2$ cryosparcm update --override [...]
ctffind4
binary blob here https://grigoriefflab.umassmed.edu/ctffind4
4.1.10
cd /lsc/sources/ctffind/4.1.10/ctffind-4.1.10 ./configure --prefix=/lsc/ctffind4/4.1.10 --enable-latest-instruction-set make -j 64 all make -j 64 install
4.1.13
This is not working: this new version of ctf find requires the wxWidgets and /usr/bin/wx-config --query-version returns 2.8 (needs version 3.0.0 and above). Will use static binary instead for now.
Suspended till we upgrade the machine to something newer
yum install wxGTK3-webview.x86_64 wxGTK3-media.x86_64 wxGTK3-gl.x86_64 wxGTK3.x86_64 wxGTK-media.x86_64 wxGTK-gl.x86_64 wxGTK.x86_64 wxBase3.x86_64 wxBase.x86_64 mathgl-wx-devel.x86_64 mathgl-wx.x86_64 erlang-wx.x86_64 compat-wxGTK3-gtk2-media.x86_64 compat-wxGTK3-gtk2-gl.x86_64 compat-wxGTK3-gtk2.x86_64 compat-wxBase3-gtk2.x86_64 cd /lsc/sources/ctffind/4.1.13/ctffind-4.1.13 /configure --prefix=/lsc/ctffind/4.1.13 --enable-latest-instruction-set --enable-openmp --with-wx-config=/usr/bin/wx-config --with-fftw-dir=/lsc/fftw/3.3.8 make -j 64 all make -j 64 install
4.1.13 + cuda
binary distribution
eman2
2.2
cd /lsc/sources/eman2/2.2/ ./eman2.22.linux64.sh > EMAN2 will now be installed into this location: > [/root/EMAN2] >>> /lsc/eman2/2.2 # installer does the rest of the work, re-installing a bunch of python modules via anaconda > Do you wish the installer to prepend the EMAN2 install location > to PATH in your /root/.bashrc ? [yes|no] > [no] >>> no > You may wish to edit your .bashrc to prepend the EMAN2 install location to PATH: > export PATH=/opt/eman/2.2/bin:$PATH # covered in the relevant environment module
2.3
cd /lsc/sources/eman2/2.3/ ./eman2.3.linux64.sh > EMAN2 will now be installed into this location: > [/root/EMAN2] >>> /lsc/eman2/2.3 # installer does the rest of the work, re-installing a bunch of python modules via anaconda > Do you wish the installer to prepend the EMAN2 install location > to PATH in your /root/.bashrc ? [yes|no] > [no] >>> no > You may wish to edit your .bashrc to prepend the EMAN2 install location to PATH: > export PATH=/opt/eman/2.3/bin:$PATH # covered in the relevant environment module
2.31
cd /lsc/sources/eman2/2.31/ eman2.31_sphire1.3_huge.linux64.sh > EMAN2 will now be installed into this location: > [/root/EMAN2] >>> /lsc/eman2/2.31 # installer does the rest of the work, re-installing a bunch of python modules via anaconda > Do you wish the installer to prepend the EMAN2 install location > to PATH in your /root/.bashrc ? [yes|no] > [no] >>> no > You may wish to edit your .bashrc to prepend the EMAN2 install location to PATH: > export PATH=/opt/eman/2.31/bin:$PATH # covered in the relevant environment module
Dynamo
https://wiki.dynamo.biozentrum.unibas.ch/w/index.php/Main_Page
1401
external/fftw
2.1.5
cd /lsc/sources/fftw/2.1.5/fftw-2.1.5 ./configure --prefix=/lsc/fftw/2.1.5 --enable-threads --enable-mpi --enable-i386-hacks make make install
3.3.8
cd /lsc/sources/fftw/3.3.8/fftw-3.3.8 ./configure CC=/usr/bin/gcc-4.8.5 --prefix=/lsc/fftw/3.3.8 --enable-single --enable-sse --enable-sse2 --enable-avx --enable-avx2 --enable-avx-128-fma --enable-generic-simd128 --enable-generic-simd256 --enable-fma --enable-mpi --enable-threads --with-g77-wrappers --with-combined-threads make make install
external/openssl
cd /lsc/sources/openssl/openssl-1.0.2o/ ./config --prefix=/lsc/external/openssl/1.0.2o make -j 64 make -j 64 install #done
frealign
cd /lsc/sources/frealign/9.11/frealign_v9.11 mkdir -p /lsc/frealign/9.11 cp -r /lsc/sources/frealign/9.11/frealign_v9.11/* /lsc/frealign/9.11 # Ignore the ./INSTALL file, it does nothing # you just need to add the relevant bin path for the application to work # the relevant module covers the details
gautomatch
cd /lsc/sources/gautomatch/Gautomatch_v0.53 mkdir /lsc/gautomatch/0.53 cp -r /lsc/sources/gautomatch/Gautomatch_v0.53 /lsc/gautomatch/0.53
gctf
cd /lsc/sources/gctf/1.18 mkdir /lsc/gctf/1.18 cp -r /lsc/sources/gctf/1.18/* /lsc/gctf/1.18 # environmental module takes care of loading up cuda 8.0, (required) adding the bin path, and changing the LD_PRELOAD path ### modules cd /lsc/sources/modules/4.2.0 ./configure --prefix=/lsc/modules --enable-doc-install --enable-example-modulefiles --enable-compat-version --enable-versioning --enable-quarantine-support --enable-auto-handling # since --enable-versioning is turned on, the module will autoversion itself and place all its files under # /lsc/modules/4.2.0
imod
Follow instructions as described in RT 3509886
mpich
default loaded by CentOS 7. Environmental module was altered to make sure that it conflicts with the openmpi modules
motioncor2
cd /lsc/sources/motioncor2/1.2.1 mkdir -p /lsc/motioncor2/1.2.1/bin cp -r /lsc/sources/motioncor2/1.2.1/* /lsc/motioncor2/1.2.1a ln -s /lsc/motioncor2/1.2.1/MotionCor2_1.2.1-Cuda80 /lsc/motioncor2/1.2.1/bin/motioncor2 # environmental module takes care of setting up the bin path
nvidia/driver
cd /lsc/sources/nvidia/drivers/396.54; ./NVIDIA-Linux-x86_64-396.54.run
Update: In order to tackle the issue related to creating remote OpenGL context (X11 forwarding over ssh), the driver to 415.23 was upgraded but this did not yield the desired result. In nvidia-smi, the new driver is reported to be compatible with CUDA 10, which is fine as it can use the CUDA 8 run-time environment, as NVIDIA guarantees binary compatibility
nvidia/cuda
CUDA frameworks installed:
- 8.0-GA1
- 8.0-GA2
- 9.0
- 9.1
- 9.2
- 10.2
Default CUDA framework for most applications is 8.0GA1
The Nvidia driver version for the kernel installed is Driver Version: 440.44 of CUDA Version: 10.2
When different CUDA versions are loaded, the NVIDIA driver remains the same. Things seem to work, but may require additional detailed testing in the future.
All CUDA installations require GCC 4.8 (hence the reason we went with CentOS).
All mentioned versions are currently installed, but we regard 8.0-GA1 as the base release, due to software that demands 8.0-GA1 and has no alternatives.
Installations:
8.0-GA1
cd /lsc/sources/nvidia/cuda/8.0-GA1 ; ./cuda_8.0.44_linux-run --silent --toolkit=/lsc/nvidia/cuda/8.0-GA1 --samples --samplespath=/lsc/nvidia/cuda/8.0-GA1/samples --run-nvidia-xconfig --tmpdir=/tmp
8.0-GA2
cd /lsc/sources/nvidia/cuda/8.0-GA2 ; ./cuda_8.0.61_375.26_linux-run --silent --toolkit=/lsc/nvidia/cuda/8.0-GA2 --samples --samplespath=/lsc/nvidia/cuda/8.0-GA2/samples --run-nvidia-xconfig --tmpdir=/tmp
9.0
cd /lsc/sources/nvidia/cuda/9.0 ; ./cuda_9.0.176_384.81_linux-run --silent --toolkit=/lsc/nvidia/cuda/9.0 --samples --samplespath=/lsc/nvidia/cuda/9.0/samples --run-nvidia-xconfig --tmpdir=/tmp
9.1
cd /lsc/sources/nvidia/cuda/9.1 ; ./cuda_9.1.85_387.26_linux --silent --toolkit=/lsc/nvidia/cuda/9.1 --samples --samplespath=/lsc/nvidia/cuda/9.1/samples --run-nvidia-xconfig --tmpdir=/tmp
9.2
cd /lsc/sources/nvidia/cuda/9.2 ; ./9.2.148_396.37_linux --silent --toolkit=/lsc/nvidia/cuda/9.2 --samples --samplespath=/lsc/nvidia/cuda/9.2/samples --run-nvidia-xconfig --tmpdir=/tmp
10.2
mkdir /lsc/nvidia/cuda/10.2; wget http://developer.download.nvidia.com/compute/cuda/10.2/Prod/local_installers/cuda_10.2.89_440.33.01_linux.run; sh ./cuda_10.2.89_440.33.01_linux.run --silent --toolkitpath=/lsc/nvidia/cuda/10.2 --toolkit=/lsc/nvidia/cuda/10.2 --samples --samplespath=/lsc/nvidia/cuda/10.2/samples --run-nvidia-xconfig --tmpdir=/storage/scratch/tmp
openmpi
cd /lsc/sources/openmpi/3.1.2; ./configure --prefix /lsc/openmpi/3.1.2 --enable-binaries --enable-mpi-fortran --with-cuda=/lsc/nvidia/cuda/8.0-GA1 --with-devel-headers
Default: 3.1.2
The default version loaded is the stable-as-of 2018-10-31 3.1.2
phenix
cd /lsc/sources/phenix/$version/phenix-installer-$version-intel-linux-2.6-x86_64-centos6 ; ./install --prefix=/lsc/phenix --openmp --makedirs
relion
2.1
cd /lsc/sources/relion/2.1-mpich ; mkdir build; cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/2.1 make -j 64 make -j 64 install
3.0b
cd /lsc/sources/relion/3.0b-mpich ; mkdir build; cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/3.0b make -j 64 make -j 64 install
3.0.4
cd /lsc/sources/relion/3.0.4 ; mkdir build && cd build; cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/3.0.4 --DGUI=ON -DCUDA=ON -DCudaTexture=ON -DFORCE_OWN_TBB=ON -DFORCE_OWN_FLTK=ON -DFORCE_OWN_FFTW=ON -DCUDA_ARCH=61 -DBUILD_SHARED_LIBS=ON .. make -j 64 make install
3.0.6
cd /lsc/sources/relion/3.0.6 ; mkdir build && cd build; cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/3.0.6 --DGUI=ON -DCUDA=ON -DCudaTexture=ON -DFORCE_OWN_TBB=ON -DFORCE_OWN_FLTK=ON -DFORCE_OWN_FFTW=ON -DCUDA_ARCH=61 -DBUILD_SHARED_LIBS=ON .. make -j 64 make install
3.0.8
cd /lsc/sources/relion/3.0.8 ; mkdir build && cd build; cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/3.0.8 --DGUI=ON -DCUDA=ON -DCudaTexture=ON -DFORCE_OWN_TBB=ON -DFORCE_OWN_FLTK=ON -DFORCE_OWN_FFTW=ON -DCUDA_ARCH=61 -DBUILD_SHARED_LIBS=ON .. make -j 64 make install
3.1-beta
cd /lsc/sources/relion/relion.git git checkout ver3.1 mkdir build && cd build module load relion/3.1-beta cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/3.1-beta --DGUI=ON -DCUDA=ON -DCudaTexture=ON -DFORCE_OWN_TBB=ON -DFORCE_OWN_FLTK=ON -DFORCE_OWN_FFTW=ON -DCUDA_ARCH=61 -DBUILD_SHARED_LIBS=ON .. make -j 64 make install
3.1
cd /lsc/sources/relion/3.1 mkdir build && cd build module load relion/3.1 cmake -DCMAKE_INSTALL_PREFIX=/lsc/relion/3.1 --DGUI=ON -DCUDA=ON -DCudaTexture=ON -DFORCE_OWN_TBB=ON -DFORCE_OWN_FLTK=ON -DFORCE_OWN_FFTW=ON -DCUDA_ARCH=61 -DBUILD_SHARED_LIBS=ON .. make -j 64 make install
List of CUDA architectures: https://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/
scipion
2.0
Note: The comments for scipion and the Sekulic GPU server are applicable to the module scipion/2.0source. Its installation is described in RT 3879628. The instructions below are just basic steps not aimed at production installation.
Cloning/Getting software
mkdir -p /lsc/scipion/2.0 git clone https://github.com/I2PC/scipion.git cd scipion
Dependencies
yum install -y gcc gcc-g++ cmake java-1.8.0-openjdk-devel.x86_64 libXft-devel.x86_64 openssl-devel.x86_64 libXext-devel.x86_64 libxml++.x86_64 libquadmath-devel.x86_64 libxslt.x86_64 openmpi-devel.x86_64 gsl-devel.x86_64 libX11.x86_64 gcc-gfortran.x86_64 git
Configuring
./scipion config scipion installp -p scipion-em-relion
2.1
Note: same as above for xmipp2
Cloning/Getting software
mkdir -p /lsc/scipion/2.0 git clone https://github.com/I2PC/scipion.git cd scipion
Dependencies
yum install -y gcc gcc-g++ cmake java-1.8.0-openjdk-devel.x86_64 libXft-devel.x86_64 openssl-devel.x86_64 libXext-devel.x86_64 libxml++.x86_64 libquadmath-devel.x86_64 libxslt.x86_64 openmpi-devel.x86_64 gsl-devel.x86_64 libX11.x86_64 gcc-gfortran.x86_64 git
Configuring and installing modules from the command-line
./scipion config ./scipion installp --help | awk -p '{ print $1 }' | grep 'scipion-em' | grep -E -v 'Configuration|xmipp2' > modules.out ./scipion installp -p scipion-em-xmipp for module in $( < module.out ); do ./scipion installp -p $module ); done rm module.out
see ticket 4049861: some modules cannot compile due to python2 issues:
./scipion installp -p scipion-em-cryodrgn 3.0.0b4 # python2 issues ./scipion installp -p scipion-em-dynamo 3.1.4 # python2 issues ./scipion installp -p scipion-em-fsc3d 3.0.5 # python2 issues ./scipion installp -p scipion-em-imod 3.0.3 # python2 issues ./scipion installp -p scipion-em-novactf 3.0.0 # python2 issues ./scipion installp -p scipion-em-pyseg 0.2 # python2 issues ./scipion installp -p scipion-em-sidesplitter 3.0.3 # python2 issues ./scipion installp -p scipion-em-tomo 3.0.5 # python2 issues
sphire
1.1
cd /lsc/sources/sphire/1.1 ; ./sphire_1_1_linux.sh # follow questions, install under /lsc/sphire/1.1
1.3
sphire 1.3 is included with the installation of eman 2.31.
module load eman 2.31
topaz
0.2.3
[root@intaristotle ~]# conda create --prefix=/lsc/topaz/0.2.3 Collecting package metadata: done Solving environment: done ## Package Plan ## environment location: /lsc/topaz/0.2.3 Proceed ([y]/n)? y Preparing transaction: done Verifying transaction: done Executing transaction: done # # To activate this environment, use # # $ conda activate /lsc/topaz/0.2.3 # # To deactivate an active environment, use # # $ conda deactivate [root@intaristotle ~]# conda activate /lsc/topaz/0.2.3 (/lsc/topaz/0.2.3) [root@intaristotle ~]# module load nvidia/cuda/9.2 (/lsc/topaz/0.2.3) [root@intaristotle ~]# conda install topaz cudatoolkit=9.2 -c tbepler -c pytorch Collecting package metadata: done Solving environment: done ## Package Plan ## environment location: /lsc/topaz/0.2.3 added / updated specs: - cudatoolkit=9.2 - topaz The following packages will be downloaded: package | build ---------------------------|----------------- certifi-2020.4.5.1 | py38_0 159 KB freetype-2.9.1 | h8a8886c_1 822 KB intel-openmp-2020.0 | 166 916 KB joblib-0.14.1 | py_0 202 KB jpeg-9b | h024ee3a_2 248 KB libpng-1.6.37 | hbc83047_0 364 KB libtiff-4.1.0 | h2733197_0 589 KB mkl-2020.0 | 166 202.1 MB mkl-service-2.3.0 | py38he904b0f_0 68 KB mkl_fft-1.0.15 | py38ha843d7b_0 178 KB mkl_random-1.1.0 | py38h962f231_0 398 KB ninja-1.9.0 | py38hfd86e86_0 1.6 MB numpy-1.18.1 | py38h4f9e942_0 5 KB numpy-base-1.18.1 | py38hde5b4d6_1 5.3 MB olefile-0.46 | py_0 33 KB pandas-1.0.3 | py38h0573a6f_0 11.5 MB pillow-7.0.0 | py38hb39fc2d_0 664 KB pip-20.0.2 | py38_1 1.9 MB python-3.8.2 | hcf32534_0 57.8 MB python-dateutil-2.8.1 | py_0 224 KB pytorch-1.5.0 |py3.8_cuda9.2.148_cudnn7.6.3_0 344.4 MB pytorch pytz-2019.3 | py_0 231 KB scikit-learn-0.22.1 | py38hd81dba3_0 7.3 MB scipy-1.4.1 | py38h0b6359f_0 19.1 MB setuptools-46.1.3 | py38_0 656 KB six-1.14.0 | py38_0 27 KB topaz-0.2.3 | py_0 26.0 MB tbepler torchvision-0.6.0 | py38_cu92 11.7 MB pytorch wheel-0.34.2 | py38_0 49 KB zstd-1.3.7 | h0b5b093_0 887 KB ------------------------------------------------------------ Total: 695.2 MB The following NEW packages will be INSTALLED: _libgcc_mutex pkgs/main/linux-64::_libgcc_mutex-0.1-main blas pkgs/main/linux-64::blas-1.0-mkl ca-certificates pkgs/main/linux-64::ca-certificates-2020.1.1-0 certifi pkgs/main/linux-64::certifi-2020.4.5.1-py38_0 cudatoolkit pkgs/main/linux-64::cudatoolkit-9.2-0 freetype pkgs/main/linux-64::freetype-2.9.1-h8a8886c_1 intel-openmp pkgs/main/linux-64::intel-openmp-2020.0-166 joblib pkgs/main/noarch::joblib-0.14.1-py_0 jpeg pkgs/main/linux-64::jpeg-9b-h024ee3a_2 ld_impl_linux-64 pkgs/main/linux-64::ld_impl_linux-64-2.33.1-h53a641e_7 libedit pkgs/main/linux-64::libedit-3.1.20181209-hc058e9b_0 libffi pkgs/main/linux-64::libffi-3.2.1-hd88cf55_4 libgcc-ng pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0 libgfortran-ng pkgs/main/linux-64::libgfortran-ng-7.3.0-hdf63c60_0 libpng pkgs/main/linux-64::libpng-1.6.37-hbc83047_0 libstdcxx-ng pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0 libtiff pkgs/main/linux-64::libtiff-4.1.0-h2733197_0 mkl pkgs/main/linux-64::mkl-2020.0-166 mkl-service pkgs/main/linux-64::mkl-service-2.3.0-py38he904b0f_0 mkl_fft pkgs/main/linux-64::mkl_fft-1.0.15-py38ha843d7b_0 mkl_random pkgs/main/linux-64::mkl_random-1.1.0-py38h962f231_0 ncurses pkgs/main/linux-64::ncurses-6.2-he6710b0_1 ninja pkgs/main/linux-64::ninja-1.9.0-py38hfd86e86_0 numpy pkgs/main/linux-64::numpy-1.18.1-py38h4f9e942_0 numpy-base pkgs/main/linux-64::numpy-base-1.18.1-py38hde5b4d6_1 olefile pkgs/main/noarch::olefile-0.46-py_0 openssl pkgs/main/linux-64::openssl-1.1.1g-h7b6447c_0 pandas pkgs/main/linux-64::pandas-1.0.3-py38h0573a6f_0 pillow pkgs/main/linux-64::pillow-7.0.0-py38hb39fc2d_0 pip pkgs/main/linux-64::pip-20.0.2-py38_1 python pkgs/main/linux-64::python-3.8.2-hcf32534_0 python-dateutil pkgs/main/noarch::python-dateutil-2.8.1-py_0 pytorch pytorch/linux-64::pytorch-1.5.0-py3.8_cuda9.2.148_cudnn7.6.3_0 pytz pkgs/main/noarch::pytz-2019.3-py_0 readline pkgs/main/linux-64::readline-8.0-h7b6447c_0 scikit-learn pkgs/main/linux-64::scikit-learn-0.22.1-py38hd81dba3_0 scipy pkgs/main/linux-64::scipy-1.4.1-py38h0b6359f_0 setuptools pkgs/main/linux-64::setuptools-46.1.3-py38_0 six pkgs/main/linux-64::six-1.14.0-py38_0 sqlite pkgs/main/linux-64::sqlite-3.31.1-h62c20be_1 tk pkgs/main/linux-64::tk-8.6.8-hbc83047_0 topaz tbepler/noarch::topaz-0.2.3-py_0 torchvision pytorch/linux-64::torchvision-0.6.0-py38_cu92 wheel pkgs/main/linux-64::wheel-0.34.2-py38_0 xz pkgs/main/linux-64::xz-5.2.5-h7b6447c_0 zlib pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3 zstd pkgs/main/linux-64::zstd-1.3.7-h0b5b093_0 Proceed ([y]/n)? y Downloading and Extracting Packages mkl-2020.0 | 202.1 MB | ################################################################################################# | 100% pytorch-1.5.0 | 344.4 MB | ################################################################################################# | 100% jpeg-9b | 248 KB | ################################################################################################# | 100% python-dateutil-2.8. | 224 KB | ################################################################################################# | 100% pip-20.0.2 | 1.9 MB | ################################################################################################# | 100% joblib-0.14.1 | 202 KB | ################################################################################################# | 100% certifi-2020.4.5.1 | 159 KB | ################################################################################################# | 100% python-3.8.2 | 57.8 MB | ################################################################################################# | 100% setuptools-46.1.3 | 656 KB | ################################################################################################# | 100% pytz-2019.3 | 231 KB | ################################################################################################# | 100% mkl_fft-1.0.15 | 178 KB | ################################################################################################# | 100% six-1.14.0 | 27 KB | ################################################################################################# | 100% numpy-1.18.1 | 5 KB | ################################################################################################# | 100% libtiff-4.1.0 | 589 KB | ################################################################################################# | 100% zstd-1.3.7 | 887 KB | ################################################################################################# | 100% ninja-1.9.0 | 1.6 MB | ################################################################################################# | 100% scikit-learn-0.22.1 | 7.3 MB | ################################################################################################# | 100% pillow-7.0.0 | 664 KB | ################################################################################################# | 100% numpy-base-1.18.1 | 5.3 MB | ################################################################################################# | 100% freetype-2.9.1 | 822 KB | ################################################################################################# | 100% scipy-1.4.1 | 19.1 MB | ################################################################################################# | 100% pandas-1.0.3 | 11.5 MB | ################################################################################################# | 100% mkl_random-1.1.0 | 398 KB | ################################################################################################# | 100% mkl-service-2.3.0 | 68 KB | ################################################################################################# | 100% libpng-1.6.37 | 364 KB | ################################################################################################# | 100% wheel-0.34.2 | 49 KB | ################################################################################################# | 100% torchvision-0.6.0 | 11.7 MB | ################################################################################################# | 100% topaz-0.2.3 | 26.0 MB | ################################################################################################# | 100% olefile-0.46 | 33 KB | ################################################################################################# | 100% intel-openmp-2020.0 | 916 KB | ################################################################################################# | 100% Preparing transaction: done Verifying transaction: done Executing transaction: done (/lsc/topaz/0.2.3) [root@intaristotle ~]# topaz Traceback (most recent call last): File "/lsc/topaz/0.2.3/bin/topaz", line 6, in <module> from pkg_resources import load_entry_point File "/lsc/topaz/0.2.3/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3259, in <module> def _initialize_master_working_set(): File "/lsc/topaz/0.2.3/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3242, in _call_aside f(*args, **kwargs) File "/lsc/topaz/0.2.3/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3271, in _initialize_master_working_set working_set = WorkingSet._build_master() File "/lsc/topaz/0.2.3/lib/python3.8/site-packages/pkg_resources/__init__.py", line 584, in _build_master ws.require(__requires__) File "/lsc/topaz/0.2.3/lib/python3.8/site-packages/pkg_resources/__init__.py", line 901, in require needed = self.resolve(parse_requirements(requirements)) File "/lsc/topaz/0.2.3/lib/python3.8/site-packages/pkg_resources/__init__.py", line 787, in resolve raise DistributionNotFound(req, requirers) pkg_resources.DistributionNotFound: The 'future' distribution was not found and is required by torch (/lsc/topaz/0.2.3) [root@intaristotle ~]# pip install future Collecting future Downloading future-0.18.2.tar.gz (829 kB) |████████████████████████████████| 829 kB 4.3 MB/s Building wheels for collected packages: future Building wheel for future (setup.py) ... done Created wheel for future: filename=future-0.18.2-py3-none-any.whl size=491058 sha256=f1d7892b69b66f8a96624ddc534bfc5dcf52e8042aa760743760f036db892c26 Stored in directory: /root/.cache/pip/wheels/8e/70/28/3d6ccd6e315f65f245da085482a2e1c7d14b90b30f239e2cf4 Successfully built future Installing collected packages: future Successfully installed future-0.18.2 (/lsc/topaz/0.2.3) [root@intaristotle ~]# topaz usage: topaz [-h] [--version] <command> ... topaz: error: the following arguments are required: <command>
0.2.4
- Installed on
sekulicgpu1
andcryosparc2
[root@sekulicgpu1 topaz]# mkdir /lsc/venvs [root@sekulicgpu1 topaz]# chown cryosparc_user /lsc/venvs [root@sekulicgpu1 topaz]# su cryosparc_user [cryosparc_user@sekulicgpu1 topaz]$ /lsc/python/3.6.11/bin/python3.6 -m venv /lsc/venvs/topaz [cryosparc_user@sekulicgpu1 topaz]$ source /lsc/venvs/topaz/bin/activate (topaz) [cryosparc_user@sekulicgpu1 topaz]$ mkdir /lsc/venvs/tmp (topaz) [cryosparc_user@sekulicgpu1 topaz]$ env TMPDIR=/lsc/venvs/tmp pip3 install topaz-em [...] Successfully installed dataclasses-0.8 future-0.18.2 joblib-1.0.1 numpy-1.19.5 pandas-1.1.5 pillow-8.1.0 python-dateutil-2.8.1 pytz-2021.1 scikit-learn-0.24.1 scipy-1.5.4 six-1.15.0 threadpoolctl-2.1.0 topaz-em-0.2.4 torch-1.7.1 torchvision-0.8.2 typing-extensions-3.7.4.3 (topaz) [cryosparc_user@sekulicgpu1 topaz]$ rmdir /lsc/venvs/tmp
On cryosparc2
, first install Python 3.6.11 (for consistency with sekulicgpu1
), depends on 3 -devel
packages:
[root@cryosparc2 ~]# yum install zlib-devel openssl-devel bzip2-devel [root@cryosparc2 ~]# mkdir -p /lsc/sources/python /lsc/python/3.6.11 [root@cryosparc2 ~]# chown cryosparc_user /lsc/sources/python /lsc/python/3.6.11 [root@cryosparc2 ~]# su cryosparc_user bash-4.2$ cd /lsc/sources/python/ bash-4.2$ wget https://www.python.org/ftp/python/3.6.11/Python-3.6.11.tar.xz bash-4.2$ tar xf Python-3.6.11.tar.xz bash-4.2$ cd Python-3.6.11 bash-4.2$ ./configure --prefix=/lsc/python/3.6.11 --enable-optimizations && make && make install
and then same steps for topaz
as on sekulicgpu1
:
[root@cryosparc2 ~]# mkdir /lsc/venvs [root@cryosparc2 ~]# chown cryosparc_user /lsc/venvs [root@cryosparc2 ~]# su cryosparc_user bash-4.2$ /lsc/python/3.6.11/bin/python3.6 -m venv /lsc/venvs/topaz bash-4.2$ source /lsc/venvs/topaz/bin/activate (topaz) bash-4.2$ mkdir /lsc/venvs/tmp (topaz) bash-4.2$ env TMPDIR=/lsc/venvs/tmp pip3 install topaz-em [...] Successfully installed dataclasses-0.8 future-0.18.2 joblib-1.0.1 numpy-1.19.5 pandas-1.1.5 pillow-8.1.0 python-dateutil-2.8.1 pytz-2021.1 scikit-learn-0.24.1 scipy-1.5.4 six-1.15.0 threadpoolctl-2.1.0 topaz-em-0.2.4 torch-1.7.1 torchvision-0.8.2 typing-extensions-3.7.4.3 (topaz) bash-4.2$ rmdir /lsc/venvs/tmp
path to topaz: /lsc/venvs/topaz/bin/topaz
verify GPU support:
[haroldgu@sekulicgpu1 ~]$ /lsc/venvs/topaz/bin/python -c 'import torch ; print(torch.cuda.is_available())' True
How is each piece of software run (really basic, just front UI, text or graphics)
ccp4/4.7.0
chimera/1.13.1
module load chimera chimera
ctffind4/4.1.10
module load ctffind4 ctffind4
eman2/2.2
module load eman eman2.py
eman2/2.3
module load eman/2.3 eman2.py
external
openssl/1.0.2o
A principal SSL library.
frealign/9.11
module load frealign frealign
gautomatch/0.53
module load gautomatch gautomatch
gctf/1.18
module load gctf gctf
mpich/3.0.4
module load mpich mpirun
motioncor2/1.2.1
nvidia/
driver
modinfo nvidia
cuda/{ 8.0-GA1, 8.0-GA2, 9.0, 9.1, 9.2 }
There is really nothing you can do with the CUDA libraries directly. You can only verify that the current version of CUDA works with the following for loop in bash: for version in 8.0-GA1 8.0-GA2 9.0 9.1 9.2; do module switch nvidia/cuda/$version; # press q to quit the nbody simulation below /lsc/nvidia/cuda/$version/samples/5_Simulations/nbody/nbody -hostmem -numdevices=$(lspci | grep -i nvidia | grep -i vga); # n-body gravitational attraction simulation done && module purge;
openmpi/3.1.2
module load openmpi mpirun -j 64 date
phenix/1.14-3260
module load phenix/1.14-3260 phenix
phenix/1.18.2-3874
module load phenix/1.18.2-3874 phenix
relion/{ 2.1, 3.0b, 3.1 }
for version in 2.1 3.0b 3.1; do module switch relion/$version; /lsc/relion/$version/bin/relion done;
sphire/1.1
module load sphire sphire
sphire/1.3
module load sphire/1.3 sphire
xchimera/0.8
not yet available
Environmental Modules
Download
Download from here:
http://modules.sourceforge.net/
Installation
cd /lsc/sources/modules/4.2.0
./configure --prefix=/lsc/modules/4.2.0 --enable-doc-install --enable-example-modulefiles --enable-compat-version --enable-versioning --enable-quarantine-support --enable-auto-handling --with-tclsh=/usr/bin/tclsh --with-pager=/usr/bin/less && make && make installThis will place symbolic links from
/etc/profile.d/modules.shto
/lsc/modules/4.2.0/init/bashThe configure log is under
compat/config.login case you need to see what went wrong
In the event you need to re-initialize the modules subsystem, for any reason run the following as root:
ln -sf /lsc/modules/4.2.0/init/bash /etc/profile.d/modules.sh
A soft link is preferred, as hard links do not work across filesystems
How to use
To see what environment modules are available:
module avail
To load a module:
module load <module>
To see the already loaded modules:
module list
To unload a module:
module unload <module>
Git Repo
Pushing them to the nodes via Ansible + git
Running Test Jobs on Aristotle to test the software
Relion
As per the Relion 3.0 tutorial:
You will need the following test sets:
ftp://ftp.mrc-lmb.cam.ac.uk/pub/scheres/relion30_tutorial_data.tar ftp://ftp.mrc-lmb.cam.ac.uk/pub/scheres/relion30_tutorial_precalculated_results.tar.gz
These test sets are already downloaded and untar'ed under intaristotle.internal.biotek:/lsc/relion/test_data
module load relion/
$version you need and wait for relion to open on your X server.
3D Classification
Follow the instructions in the tutorial regarding 3D classification.