From Documentation
Jump to: navigation, search
Note: Some of the information on this page is for our legacy systems only. The page is scheduled for an update to make it applicable to Graham.

VASP: The Vienna Ab initio Simulation Package (VASP) is a computer program for atomic scale materials modelling, e.g. electronic structure calculations and quantum-mechanical molecular dynamics, from first principles.

Installing vasp.5.4.1.05Feb16 for CPU only

You should prepare the tarball (ie. vasp.5.4.1.05Feb16.tar.gz) and download the patch(es) from [1] Unpack the .gz file and patches

tar xf vasp.5.4.1.tar.gz
gunzip patch.5.4.1.*****

Then put the patches under root folder (which contains "makefile" file) and add patches to source code one by one:

patch -p0 < patch.5.4.1.14032016
patch -p0 < patch.5.4.1.03082016
patch -p0 < patch.5.4.1.***

After patching, just copy the makefile.include.linux_intel from "arch" folder to the root folder and rename it with "makefile.include"

cp arch/makefile.include.linux_intel ./makefile.include

Then modify this makefile.include as below:

# Precompiler options
CPP_OPTIONS= -DMPI -DHOST=\"IFC91_ompi\" -DIFC \
             -DCACHE_SIZE=4000 -DPGF90 -Davoidalloc \
             -DMPI_BLOCK=8000 -DscaLAPACK -Duse_collective \
             -Duse_bse_te -Duse_shmem -Dtbdyn

CPP        = fpp -f_com=no -free -w0  $*$(FUFFIX) $*$(SUFFIX) $(CPP_OPTIONS)

FC         = mpif90
FCL        = mpif90 -mkl=sequential

FREE       = -free -names lowercase

FFLAGS     = -assume byterecl
OFLAG      = -O2
OFLAG_IN   = $(OFLAG)
DEBUG      = -O0

MKL_PATH   = $(MKLROOT)/lib/intel64
BLAS       =
LAPACK     =
BLACS      = -lmkl_blacs_openmpi_lp64
SCALAPACK  = $(MKL_PATH)/libmkl_scalapack_lp64.a $(BLACS)

OBJECTS    = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o

INCS       =-I$(MKLROOT)/include/fftw

LLIBS      = $(SCALAPACK) $(LAPACK) $(BLAS)

OBJECTS_O1 += fft3dfurth.o fftw3d.o fftmpi.o fftmpiw.o
OBJECTS_O2 += fft3dlib.o

# For what used to be vasp.5.lib
CPP_LIB    = $(CPP)
FC_LIB     = $(FC)
CC_LIB     = icc
CFLAGS_LIB = -O
FFLAGS_LIB = -O1
FREE_LIB   = $(FREE)

OBJECTS_LIB= linpack_double.o getshmem.o

# Normally no need to change this
SRCDIR     = ../../src
BINDIR     = ../../bin

To get the best performance on newer clusters, you should use newer intel compiler and openmpi libraray to build VASP. To change the environment, just unload the default ones and load the new ones:

module unload intel mkl openmpi 
module load intel/15.0.3 
module load openmpi/intel1503-std/1.8.7 

Then you can start to build it by running:

make all

The binary files can be found under "bin" folder.

There is also a READ_Hu_5.4.1 file for how to build VASP-5.4.1 under

/work/jemmyhu/VASP

Installing VASP GPU version (vasp.5.4.1.05Feb16)

Download the path from VASP Patches Put the patch in vasp root folder and run:

patch -p0 < patch.5.4.1.14032016
patch -p0 < patch.5.4.1.03082016

Before actually compiling, perform the following module operations.

module unload intel mkl openmpi 
module load intel/15.0.3 
module load openmpi/intel1503-std/1.8.7 

Copy the makefile.include

cp arch/makefile.include.linux_intel_cuda ./makefile.include

and modify the makefile.include:

# Precompiler options 
CPP_OPTIONS= -DMPI -DHOST=\"LinuxIFC\" -DIFC \
             -DCACHE_SIZE=4000 -DPGF90 -Davoidalloc \
             -DMPI_BLOCK=8000 -Duse_collective \
             -Duse_bse_te -Duse_shmem -Dkind8 

CPP = fpp -f_com=no -free -w0 $*$(FUFFIX) $*$(SUFFIX) $(CPP_OPTIONS)

FC = mpif90 
FCL = mpif90 -mkl=sequential -lstdc++

FREE = -free -names lowercase

FFLAGS = -assume byterecl 
OFLAG = -O2 
OFLAG_IN = $(OFLAG) 
DEBUG = -O0

MKL_PATH = $(MKLROOT)/lib/intel64 
BLAS = 
LAPACK = 
BLACS = -lmkl_blacs_openmpi_lp64 
SCALAPACK = $(MKL_PATH)/libmkl_scalapack_lp64.a $(BLACS)

OBJECTS = fftmpiw.o fftmpi_map.o fft3dlib.o fftw3d.o

INCS =-I$(MKLROOT)/include/fftw

LLIBS = $(SCALAPACK) $(LAPACK) $(BLAS)

#OBJECTS_O1 += fft3dfurth.o fftw3d.o fftmpi.o fftmpiw.o 
OBJECTS_O1 += fftw3d.o fftmpi.o fftmpiw.o 
OBJECTS_O2 += fft3dlib.o

# For what used to be vasp.5.lib 
CPP_LIB = $(CPP) 
FC_LIB = $(FC) 
CC_LIB = icc 
CFLAGS_LIB = -O 
FFLAGS_LIB = -O1 
FREE_LIB = $(FREE)

OBJECTS_LIB= linpack_double.o getshmem.o

# Normally no need to change this 
SRCDIR = ../../src 
BINDIR = ../../bin

#================================================ 
# GPU Stuff

CPP_GPU = -DCUDA_GPU -DRPROMU_CPROJ_OVERLAP -DUSE_PINNED_MEMORY -DCUFFT_MIN=28

OBJECTS_GPU = fftmpiw.o fftmpi_map.o fft3dlib.o fftw3d_gpu.o fftmpiw_gpu.o

CUDA_ROOT := /opt/sharcnet/cuda/7.5.18
NVCC := $(CUDA_ROOT)/bin/nvcc 
CUDA_LIB := -L$(CUDA_ROOT)/lib64 -lnvToolsExt -lcudart -lcuda -lcufft -lcublas

GENCODE_ARCH := -gencode=arch=compute_20,code=\"sm_20,compute_20\" -gencode=arch=compute_37,code=\"sm_37,compute_37\" -gencode=arch=compute_35,code=\"sm_35,compute_35\"

MPI_INC = /opt/sharcnet/openmpi/1.8.7/intel-15.0.3/std/include

Then run:

make gpu
make gpu_ncl

to built the GPU ports of the standard and non-collinear versions, respectively.

  • N.B.: Unfortunately at this time there is no GPU port of the gamma-only version yet.

Running vasp.5.4.1.05Feb16 CPU job

After compiling VASP, you should be able to find the binary programs under bin folder. (e.g. /home/$USER/vasp.5.4.1.05Feb16/bin)

Before submitting jobs to the scheduler, modules should be set to the ones that you used to build VASP. Please run the following commands to change modules:

module unload intel mkl openmpi 
module load intel/15.0.3 
module load openmpi/intel1503-std/1.8.7

You should go to the folder with input files and submit the job to the cluster from that folder. For example, we want to use 16 cpu cores with 2GB memory each core:

sqsub -q mpi -n 16 --mpp=2g -r 4h -o vasp-output.txt /home/$USER/vasp.5.4.1.05Feb16/bin/vasp_std

VASP need to be submitted into mpi queue. InfiniBand networking is highly recommended to run VASP across multiple nodes. If the cluster use Ethernet interconnection (see https://www.sharcnet.ca/my/systems), you should specify to use only one node by adding "-N 1" after "-n <number>" flag.