Getting started

From mn.kjemi.vaspwiki
Jump to: navigation, search

Getting started

User access

Go to www.metacenter.no > User Login (Passwords) > 1. Request a user account > HPC

Fill in the form with your UiO username and choose the appropriate project. If you already have an account through for instance FYS-MENA41111, the project manager can in principle directly add you to the project without requesting a user account.

The project manager can give you access to the VASP license through the VASP portal and thereafter informing sigma2@uninett.no to give your user the necessary group affiliations (vasp5).

Project Manager Group Resources
NN4604K Jonathan M. Polfus Electrochemistry Saga
NN9136K Lasse Vines LENS Saga
NN9180K Clas Persson Structure Physics Saga

Login

ssh and scp

ssh can be used to login to the cluster with a terminal app on UNIX, macOS or Windows Subsystem for Linux

ssh [username]@saga.sigma2.no

Hostnames and usernames can be saved by creating or editing the config file using for instance the vim text editor

vim ~/.ssh/config

Press i to insert text

Host saga
   HostName = saga.sigma2.no
   user [username]

Press escape to exit insert mode, and then :wq to 'write' and 'quit'.

Login is then simplified

ssh saga

Copy files from cluster to current folder (.)

scp saga:[path]/[filename] .

Copy files to the home directory on the cluster

scp [filename] saga:

PuTTY and WinSCP

PuTTY is an ssh client for Windows that can be downloaded from www.putty.org.

WinSCP is an scp client for Windows that can be downloaded from winscp.net.

Working in UNIX

Basic commands

Learn to use the TAB key to autocomplete commands, paths and filenames.

Learn to view the manual for any command: man [command]

Navigating the filesystem
Command Description
ls List files and directories in current directory
mkdir [directoryname] Make a directory
cd [directoryname] Change current path to another directory
pwd Display current path
cd .. Change path to parent directory
cp [path]/[filename] . Copy file to current directory
cp [filename] [directoryname] Copy file to another directory
mv [filename] [path] Move file to another directory
mv [filename] [newfilename] Change the name of a file
cp -r [directory] [path] Copy directory and its contents to new location
rm [filename] Delete file
rm -r [directoryname] Delete directory and its contents
Special paths and symbols
Symbol Description
. Current directory
.. Parent directory
~ Home directory
/ Root directory
* Any text pattern
 ? Any symbol
Reading text files
Command Description
less View text file. View live updates to end of file: Shift-F
head Display top 10 lines of text file. Specify number of lines: -n [count]
tail Display bottom 10 lines of text file. Specify number of lines: -n [count]
grep [pattern] Display each line in text file containing text specific text/symbol pattern.
diff [file1] [file2] Compare files line by line and display the difference between them.
cat Concatenate text files sequentially. Write the output to a file: > [filename]

Filesystem on Saga

Home directory: mainly for personal files, scripts etc. due to limited storage space

/cluster/home/[username]

Project folder: Run jobs and store output files in a subfolder called your username

/cluster/projects/[project number]/[username]

Check disk usage and storage quota for home directory and project folder

dusage

Check disk usage of current folder and subfolders

du -hs

Find and remove files matching [filename] in current folder and subfolders

find . -name "[filename]" -exec rm {} \;

Logfile

Excel sheet for tracking jobs can be downloaded here: Media:Logfile.xlsx It also includes conversion between supercell size, number of atoms and k-point grid for a given k-point density.

Job script

Sigma2 NRIS HPC clusters use Slurm as workload manager and job scheduler. Documentation is available on documentation.sigma2.no/

Computations are run by submitting a job script to the queue

sbatch jobfile

View the queue for a user or project

squeue -u [username] squeue -p [project number]

Job scripts for Saga are available in the project folder

/cluster/projects/nn4604k/jobfile

The job script below contains several SBATCH options

Saga has 40 CPUs per node and it is recommended to use all CPUs on a node and a 1-2 nodes for optimal performance documentation.sigma2.no/hpc_machines/saga

The module system is restored to avoid potential conflicts. There exist several compiled versions of VASP and specific versions are compiled for increased performance with gamma K-point grid

module restore system
module load VASP/5.4.4-intel-2019a-std
#module load VASP/5.4.4-intel-2019a-gam 

Available modules on the cluster can be listed with

module avail

Files are copied to a temporary job folder ($SCRATCH). The WAVECAR file is copied if it is available.

cp [filenames] $SCRATCH
if [ -f WAVECAR ]
then
cp WAVECAR $SCRATCH
fi

Output files are copied from the job folder to the submit folder when job completes. Savefile will also copy files if the job crashes.

savefile [filenames] 

The job is started from the $SCRATCH folder and the elapsed time is recorded to slurm output

cd $SCRATCH
time srun vasp

The progression of the job can be viewed in the slurm output file slurm-[slurm-id] in the submit folder once the job is running.

The working folder can be accessed while the job is running /cluster/work/jobs/[slurm-id]


Example job script

#!/bin/bash

#SBATCH --job-name=[jobname]
#SBATCH --account=[project number]
#SBATCH --time=10:00:00
#SBATCH --mem-per-cpu=4600M
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=40

set -o errexit 
set -o nounset

module restore system
module load VASP/5.4.4-intel-2019a-std
#module load VASP/5.4.4-intel-2019a-gam 
 
cp INCAR POSCAR POTCAR KPOINTS $SCRATCH
if [ -f WAVECAR ]
then
cp WAVECAR $SCRATCH
fi

savefile CONTCAR OUTCAR DOSCAR CHGCAR WAVECAR 

cd $SCRATCH
time srun vasp

Aliases and settings

Define useful shortcuts by adding aliases to ~/.bash_profile

alias l='ls -l'
alias la='ls -l -a'
alias ..='cd ..'
alias nn='cd /cluster/projects/[project number]'
alias q='squeue -u [username]'

You can also add commands such as automatically changing directory to the project folder upon login.

Changes will take effect on next login or manually with source ~/.bash_profile .


vim can be configured in ~/.vimrc . Turn on line numbers to easily jump to a specific line using  :[line number]

set number

Shell scripting

Shell scripting can be used to submit several jobs as exemplified below

#!/bin/bash

for i in $(seq 0.0 0.2 1.4); do
   for j in $(seq 0.0 0.2 1.4); do
      dir="$i-$j"
      echo $dir
      mkdir $dir
      cp INCAR POTCAR KPOINTS jobfile $dir
      cp "prefix_$dir.POSCAR.vasp" $dir/POSCAR
      cd $dir
      sbatch jobfile
      cd ..
   done
done

First VASP calculation

Five files required for a VASP simulation
File Description
POSCAR Structure file that specifies the unit cell vectors, types of atoms and their positions
POTCAR Pseudopotentials for the atoms (same order as in POSCAR)
INCAR Input parameters for the calculation including relaxation algorithms, functionals, convergence criteria, etc.
KPOINTS k-point grid
jobfile job script

Cell volume relaxation of BaZrO3 unit cell

Go to the project folder: cd /cluster/projects/nn4604k

Or if you have defined an alias: nn

Create a subfolder with your username (if you have not done so already)

mkdir [username]

cd [username]

mkdir test

cd test

Copy all the files from the 'vasptest' folder (POSCAR, KPOINTS and INCAR) to your current folder (remember to use TAB to autocomplete the path)

cp ../../vasptest/* .

Copy the necessary POTCAR files (press the UP arrow key to bring up your previous command)

cp ../../potpaw_PBE.54/Ba_sv/POTCAR POTCAR_Ba_sv

cp ../../potpaw_PBE.54/Zr_sv/POTCAR POTCAR_Zr_sv

cp ../../potpaw_PBE.54/O/POTCAR POTCAR_O

Check the order of the atoms in POSCAR file

head POSCAR

Concatenate the POTCAR files in the same order to a file named POTCAR

cat POTCAR_Ba_sv POTCAR_Zr_sv POTCAR_O > POTCAR

Check the order of the POTCAR

grep TITEL POTCAR

Copy job script to current folder

cp ../../jobfile .

Submit the job

sbatch jobfile

Check the queue

squeue -u [username]

Or if you have defined an alias: q

When the job starts you can follow the progress in the slurm log file

less slurm-[slurm-id]

After three ionic relaxation steps, the required accuracy has been reached and the output files defined job script are copied to the current folder.

3 F= -.41724097E+02 E0= -.41724097E+02  d E =-.431449E-02
curvature:  -0.24 expect dE=-0.515E-06 dE for cont linesearch -0.515E-06
trial: gam= 0.00000 g(F)=  0.940E-64 g(S)=  0.213E-05 ort =-0.198E-03 (trialstep = 0.895E+00)
search vector abs. value=  0.213E-05
reached required accuracy - stopping structural energy minimisation
writing wavefunctions

Check the lattice parameter in the POSCAR (input) and output (CONTCAR)

head POSCAR

BaZrO3                           
  1.00000000000000     
    4.2200000000000000    0.0000000000000000    0.0000000000000000
    0.0000000000000000    4.2200000000000000    0.0000000000000000
    0.0000000000000000    0.0000000000000000    4.2200000000000000
...

head CONTCAR

BaZrO3                                  
  1.00000000000000     
    4.2353879522908331   -0.0000000000000000   -0.0000000000000000
    0.0000000000000000    4.2353879522908331   -0.0000000000000000
    0.0000000000000000   -0.0000000000000000    4.2353879522908331
...

The optimized cubic lattice parameter is 4.2354 Å.