From Documentation
Revision as of 17:09, 4 January 2018 by Ppomorsk (Talk | contribs)

Jump to: navigation, search

Pawel Pomorski

High Performance Computing Programming Specialist

University of Waterloo


User:Ppomorsk/Toward Exascale Simulations of 3D Radiative Transfer for Cloudy Atmospheres

Important Pages

Software which I help maintain

Help Wiki stats

Wiki stats page is here.

This wiki has 6,498 users, among them 0 active users (users who performed an action within last 91 days). The total number of page edits is 19,165. The total number of page views is Template:NUMBEROFVIEWS. There are 498 pages and 415 articles.

NAMD instructions

These instructions for monk, current as of Feb.2014, with default modules:



module unload openmpi
module load openmpi/intel/1.7.4
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/sharcnet/intel/12.1.3/icc/composer_xe_2011_sp1.9.293/compiler/lib/intel64

OpenMPI 1.7.4 is compiled with CUDA support. Environment variable is needed to locate all the libraries at runtime.

Get all the required source files:

tar xvfz NAMD_2.9_Source.tar.gz 
cd NAMD_2.9_Source

Download and install precompiled TCL and FFTW libraries: (working in NAMD source directory)

 tar xzf fftw-linux-x86_64.tar.gz
 mv linux-x86_64 fftw
 tar xzf tcl8.5.9-linux-x86_64.tar.gz
 tar xzf tcl8.5.9-linux-x86_64-threaded.tar.gz
 mv tcl8.5.9-linux-x86_64 tcl
 mv tcl8.5.9-linux-x86_64-threaded tcl-threaded

Compile charm, choosing which parallelization is to be used (threaded, MPI, MPI-SMP):

tar xvf charm-6.4.0.tar 
cd charm-6.4.0

Compile charm - MPI build:

env MPICXX=mpicxx ./build charm++ mpi-linux-x86_64  --no-build-shared --with-production

Compile charm - MPI-SMP build:

env MPICXX=mpicxx ./build charm++ mpi-linux-x86_64 smp --no-build-shared --with-production

Compile charm - threaded build:

./build charm++ multicore-linux64 --no-build-shared  icc8 --with-production

Configure NAMD and compile (with CUDA support enabled) - replace CHARM_ARCHITECTURE with one of the three options above:

./config  Linux-x86_64-icc   --charm-arch CHARM_ARCHITECTURE --with-cuda --cuda-prefix /opt/sharcnet/cuda/5.5.22/toolkit/
cd Linux-x86_64-icc

Test charm if needed:

From the directory charm is in,

cd architecture_directory/tests/charm++/megatest
make pgm
mpirun -n 4 ./pgm

Python installation instructions

Even though Python modules are provided by SHARCNET, sometimes there may be a need to compile your own Python and Numpy. Here are instructions on how to do it (tested in September, 2014):

To get the source tarballs, do:

wget --no-check-certificate

and unpack these somewhere in your directories.

These were built under modules:

module unload intel 
module unload mkl 
module unload openmpi 
module load gcc/4.8.2 
module load openmpi/gcc/1.8.1

(openmpi is not necessary for this built, but I would suggest loading the gcc/1.8.1 anyway)

Python was built with:

./configure --enable-shared --prefix=~/software_installs/python/2.7.8/gcc/installdir
make install

Then set ( in .bashrc for permanent change):

export PATH=~/software_installs/python/2.7.8/gcc/installdir/bin:$PATH
export LD_LIBRARY_PATH=~/software_installs/python/2.7.8/gcc/installdir/lib:$LD_LIBRARY_PATH

Numpy was built with:

python build --fcompiler=gnu95
python install --prefix=~/software_installs/numpy/1.8.2/gcc/installdir

Also did:

ln -sf ~/software_installs/numpy/1.8.2/gcc/installdir/lib/python2.7/site-packages/numpy/core/include/numpy ~/software_installs/python/2.7.8/gcc/installdir/include/python2.7

and finally

export PYTHONPATH=~/software_installs/numpy/1.8.2/gcc/installdir/lib/python2.7/site-packages/

CNVnator instructions

Install root prerequisite

Note: compiling this package takes a long time, so it's best to use /tmp directory for faster disk access.

tar xvfz root_v6.06.06.source.tar.gz
module unload intel openmpi mkl
module load gcc/4.9.2
module load python/gcc/2.7.8
mkdir builddir
cd builddir
cmake ../ -DCMAKE_INSTALL_PREFIX=/work/lianglab/bin/install_root -Dgnuinstall=ON 
cmake --build .                 
cmake --build . --target install

Install CNVnator

(load same modules as for root above)

module unload intel openmpi mkl
module load gcc/4.9.2
module load python/gcc/2.7.8
cd CNVnator_v0.3/src/samtools/
cd ..

now edit the Makefile so it has

ROOTLIBS  = -L$(ROOTSYS)/lib/root -lCore -lRIO -lNet -lHist -lGraf -lGraf3d \
               -lGpad -lTree -lRint -lMatrix -lPhysics \
               -lMathCore -lThread -lGui
CXX    = g++ -std=c++11 $(ROOTFLAGS) -DCNVNATOR_VERSION=\"$(VERSION)\"
SAMDIR = samtools
INC    = -I$(ROOTSYS)/include/root -I$(SAMDIR)
SAMLIB = $(SAMDIR)/libbam.a

Note that -lCint library was removed from ROOTLIBS.

Finally, run:

export ROOTSYS=/work/lianglab/bin/install_root
export LD_LIBRARY_PATH=/work/lianglab/bin/install_root/lib/root:$LD_LIBRARY_PATH

This will produce the executable. In the future, run the above export LD_LIBRARY_PATH command to be able to run it.

Checking global work location

USER=ppomorsk which-global-work

Octopus instructions

Install cblas

module load openblas/0.2.20 
module load cuda/8.0.44
module load boost/1.60.0
FC=ifort CC=icc CXX=icpc cmake ..  -DNetlib_BLAS_LIBRARY=$EBROOTOPENBLAS   -DNetlib_INCLUDE_DIRS=$EBROOTOPENBLAS/include -
make install