https://www.sharcnet.ca/help/api.php?action=feedcontributions&feedformat=atom&user=EdwardDocumentation - User contributions [en]2026-04-09T19:04:44ZUser contributionsMediaWiki 1.25.2https://www.sharcnet.ca/help/index.php?title=VALGRIND&diff=17969VALGRIND2019-06-06T14:41:49Z<p>Edward: </p>
<hr />
<div>{{Template:CCDelete}}<br />
{{GrahamUpdate}}<br />
{{Software<br />
|package_name=VALGRIND<br />
|package_description=Memory debugger<br />
|package_idnumber=132<br />
}}<br />
'''Valgrind''' is a powerful tool for analyzing programs, memory debugging, memory leak detection and profiling. It is freely available under GNU license. Version 3.5.0 is available on most SHARCNET systems.<br />
<br />
'''NOTE''' To avoid spurious warnings it is important to not use too new of a version of GCC or OpenMPI. We recommend<br />
<br />
* gcc/4.8.2<br />
* openmpi/gcc-debug/1.8.3 modules<br />
<br />
= Overview =<br />
<br />
Valgrind is a dynamic binary instrumentation framework that dynamically translates executables to add instrumentation and track all memory and register usages by a program. The advantages of this approach are that<br />
<br />
* it can be directly run on any executable, and<br />
* dynamic translation allows ultimate instrumentation<br />
<br />
while the disadvantages are<br />
<br />
* 5-100 x slow down depending on tool,<br />
* 12-18 x increase in size of translated code, and<br />
* corner cases may exist between translated code and original.<br />
<br />
Several tools have been built upon this framework. These include<br />
<br />
* ''memcheck'' - memory error detector<br />
* ''cachegrind'' - cache and branch-prediction profiler<br />
* ''callgrind'' - call-graph generating cache and branch prediction profiler<br />
* ''helgrind'' - thread error detector<br />
* ''DRD'' - thread error detector<br />
* ''Massif'' - heap profiler<br />
* ''DHAT'' - dynamic heap analysis tool<br />
* ''SGCheck'' - experimental stack and global array overrun detector<br />
* ''BBV'' - experimental basic block vector generation tool<br />
<br />
You are welcome to use any or all of these, but we have only used ''memcheck'' and ''cachegrind'' and only support ''memcheck''.<br />
<br />
= Usage =<br />
<br />
The primary used tool is ''memcheck''. This is the default tool and the only one we discuss here. Documentation for other tools can be found on the [http://www.valgrind.org valgrind website]. The memcheck tool detects several common memory errors<br />
<br />
* overrunning and underrunning heap blocks,<br />
* overrunning top of stack,<br />
* continuing to access released memory,<br />
* using uninitialized values,<br />
* incorrectly using memory copying routines,<br />
* incorrectly paired allocation/release calls,<br />
* relasing unallocated memory, and<br />
* not releasing memory.<br />
<br />
We recommend running all new code under valgrind on small test cases (small due to the aforementioned ~10x slowdown). This can save hours and hours of debugging. Running the program under valgrind can be as simple as compiling with debugging information (adding the <tt>-g</tt> flag) and running as <tt>valgrind <program> <arguements></tt>.<br />
<br />
== Serial code ==<br />
<br />
Consider the following ''bug.c'' code<br />
<br />
<source lang="C"><br />
#include <stdio.h><br />
<br />
int main() {<br />
double array[10];<br />
<br />
// Execution depends on uninitialized value<br />
if (array[4] < 0.0)<br />
printf("the results is negative\n");<br />
else if (array[4] == 0.0)<br />
printf("the results is zero\n");<br />
else if (array[4] > 0.0)<br />
printf("the results is positive\n");<br />
else<br />
printf("the results are special\n");<br />
<br />
return 0;<br />
}<br />
</source><br />
<br />
It has an uninitialized value bug. Running this under valgrind<br />
<br />
<source lang="bash"><br />
cc -Wall -g bug.c -o bug<br />
valgrind ./bug<br />
</source><br />
<br />
reports this<br />
<br />
==5955== Conditional jump or move depends on uninitialised value(s)<br />
==5955== at 0x400511: main (bug.c:7)<br />
==5955== <br />
==5955== Conditional jump or move depends on uninitialised value(s)<br />
==5955== at 0x40052C: main (bug.c:9)<br />
==5955== <br />
==5955== Conditional jump or move depends on uninitialised value(s)<br />
==5955== at 0x400536: main (bug.c:9)<br />
==5955== <br />
==5955== Conditional jump or move depends on uninitialised value(s)<br />
==5955== at 0x400551: main (bug.c:11)<br />
<br />
Note that valgrind only reports uninitialized values usage once they lead to non-determinism (i.e., when the program encounters a branch whose choice depends on the result of an uninitiated value). This means you the first report you get is frequently not in the calculation done on the uninitialized values but rather the convergence test or print statement at the end of calculations (print statements contains a bunch of branches in order to handling printing of each different digit).<br />
<br />
=== Tracking down uninitialized values ===<br />
<br />
More typically a program will be composed of multiple routines that all work on the data. To this end, consider the following ''bug.c'' code<br />
<br />
<source lang="C"><br />
#include <stdio.h><br />
<br />
void initialize_sequence(double* array, const int array_length) {<br />
int i;<br />
<br />
for (i=1; i<array_length+1; ++i)<br />
array[i] = i;<br />
}<br />
<br />
double sum(const double* array, const int array_length) {<br />
double array_sum;<br />
int i;<br />
<br />
for (i=0; i<array_length; ++i)<br />
array_sum += array[i];<br />
<br />
return array_sum;<br />
}<br />
<br />
int main() {<br />
double array[10];<br />
const int array_length = sizeof(array)/sizeof(array[0]);<br />
<br />
double array_sum;<br />
<br />
initialize_sequence(array,array_length);<br />
array_sum = sum(array,array_length);<br />
<br />
printf("the sum of 0..%d is %f\n", array_length-1, array_sum);<br />
<br />
return 0;<br />
}<br />
</source><br />
<br />
It has both indexing and uninitialized value bugs. Despite this, directly running the code<br />
<br />
<source lang="bash"><br />
cc -Wall -g bug.c -o bug<br />
./bug<br />
</source><br />
<br />
will mostly likely produce the correct answer on most machines most of the time<br />
<br />
the sum of 0..9 is 45.000000<br />
<br />
Running under valgrind<br />
<br />
<source lang="bash"><br />
valgrind ./bug<br />
</source><br />
<br />
reliably gives many warnings of the following form<br />
<br />
==15930== Conditional jump or move depends on uninitialised value(s)<br />
==15930== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)<br />
==15930== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)<br />
==15930== by 0x5318189: printf (in /lib64/libc-2.12.so)<br />
==15930== by 0x400722: main (bug.c:29)<br />
<br />
This is a typical numeric code example were the warnings first occur at a print statement because this is the first time the uninitialized value leads to non-determinism (i.e., the program's behaviour is random as it is taking or does not taking a branch based on a result computed from something that was not set by the programmer).<br />
<br />
We now know the problem is that something unset went into computing the value of ''array_sum'', so we should trace the ''array_sum'' calculation backwards through the program. This quickly gets difficult as we then find ourselves also tracing back all variables that went into the ''array_sum'' calculation, and then all variables that went into those variables, and so on.<br />
<br />
Fortunately Valgrind provides a <tt>--track-origins=yes</tt> flag to ease our search by telling us which variables are the source of the problem. Re-running with this flag<br />
<br />
<source lang="bash"><br />
valgrind --track-origins=yes ./bug<br />
</source><br />
<br />
gives warning messages augmented to include the source of the uninitialized value that went into the computation of ''array_sum''<br />
<br />
==17589== Conditional jump or move depends on uninitialised value(s)<br />
==17589== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)<br />
==17589== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)<br />
==17589== by 0x5318189: printf (in /lib64/libc-2.12.so)<br />
==17589== by 0x400722: main (bug.c:29)<br />
==17589== Uninitialised value was created by a stack allocation<br />
==17589== at 0x400632: sum (bug.c:10)<br />
<br />
Now we know the source of the uninitialized value in the ''array_sum'' computation was a local variable in the ''sum'' routine. Looking into ''sum'' we hopefully figure out without too much difficulty that we forgot to initialize ''array_sum'' to zero. If our program is producing the correct answer, it is only because we are getting lucky and ''array_sum'' happens to be allocated from memory that is initially zero.<br />
<br />
Correcting the ''sum'' routine<br />
<br />
<source lang="bash"><br />
double sum(const double* array, const int array_length) {<br />
double array_sum;<br />
int i;<br />
<br />
array_sum = 0.0;<br />
for (i=0; i<array_length; ++i)<br />
array_sum += array[i];<br />
<br />
return array_sum;<br />
}<br />
</source><br />
<br />
and re-running under valgrind reveals we are still getting uninitialized values warnings associated with the ''array_sum'' computation. Now (using the <tt>--track-origins=yes</tt>) they are of the form<br />
<br />
==18613== Conditional jump or move depends on uninitialised value(s)<br />
==18613== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)<br />
==18613== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)<br />
==18613== by 0x5318189: printf (in /lib64/libc-2.12.so)<br />
==18613== by 0x40072C: main (bug.c:30)<br />
==18613== Uninitialised value was created by a stack allocation<br />
==18613== at 0x400690: main (bug.c:21)<br />
<br />
Valgrind is now telling us that a local variable inside ''main'' went into the computation of ''array_sum'' despite never being set. This must be ''array'' itself as ''array_length'' is clearly set to the compiler computed <tt>sizeof(array)/sizeof(array[0])</tt> value.<br />
<br />
Looking at ''main'' it is clear ''array'' should have been fully initialized by ''initialize_sequence''. Careful examination of this routine reveals the final error. The initialization loop was done using Fortran indices (1...''array_length'') instead of C (0...''array_length-1''). Correcting this<br />
<br />
<source lang="bash"><br />
void initialize_sequence(double* array, const int array_length) {<br />
int i;<br />
<br />
for (i=0; i<array_length; ++i)<br />
array[i] = i;<br />
}<br />
</source><br />
<br />
finally produces code the runs under valgrind without any warnings.<br />
<br />
'''NOTE''' The Valgrind warnings were being generated because the ''array[0]'' value was not being initialized. The code was also incorrect in that it was initializing ''array[array_length]'' which is one past the end of ''array''. If you put this later error back into the program and run under Valgrind you will discover it does not produce any warnings. This shows that even codes that successfully under Valgrind can still contain errors.<br />
<br />
=== Calling Valgrind from your program ===<br />
<br />
In some cases it can be helpful to call Valgrind directly from your program in order to check the status of various bits of memory. This is easily done by including the ''valgrind/memcheck.h'' header file<br />
<br />
<source lang="C"><br />
#include <valgrind/memcheck.h><br />
</source><br />
<br />
and then adding calls to the appropriate [http://valgrind.org/docs/manual/mc-manual.html#mc-manual.clientreqs client check routines]. For example, here is a modified ''sum'' routine for the above program that prints a warning if it is called with an ''array'' that is not fully initialized<br />
<br />
<source lang="C"><br />
double sum(const double* array, const int array_length) {<br />
double array_sum;<br />
int i;<br />
<br />
if (VALGRIND_CHECK_MEM_IS_DEFINED(array,sizeof(double)*array_length)) <br />
fprintf(stderr,"sum called with an array that is not fully defined...\n"); <br />
<br />
for (i=0; i<array_length; ++i)<br />
array_sum += array[i];<br />
<br />
return array_sum;<br />
}<br />
</source><br />
<br />
Running the ''bug.c'' code under Valgrind with this modification produces the following output<br />
<br />
==29765== Uninitialised byte(s) found during client check request<br />
==29765== at 0x400745: sum (bug.c:15)<br />
==29765== by 0x400832: main (bug.c:31)<br />
==29765== Address 0x7fefff8e8 is on thread 1's stack<br />
==29765== <br />
sum called with an array that is not fully defined...<br />
<br />
== MPI code ==<br />
<br />
Consider the following ''bug.c'' MPI code<br />
<br />
<source lang="C"><br />
#include <mpi.h><br />
<br />
int main(int argc,char *argv[]){<br />
int rank, size;<br />
int value;<br />
<br />
MPI_Init(&argc, &argv);<br />
<br />
MPI_Comm_rank(MPI_COMM_WORLD, &rank);<br />
MPI_Comm_size(MPI_COMM_WORLD, &size);<br />
<br />
if (rank == 0 && size > 1) {<br />
MPI_Request request;<br />
<br />
MPI_Irecv(&value, 1, MPI_INT, 1, 0, MPI_COMM_WORLD, &request);<br />
value = 0;<br />
MPI_Wait (&request, MPI_STATUS_IGNORE);<br />
}<br />
else if (rank == 1) {<br />
MPI_Request request;<br />
<br />
MPI_Isend(&value, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &request);<br />
value = 1;<br />
MPI_Wait(&request, MPI_STATUS_IGNORE);<br />
} <br />
<br />
MPI_Finalize();<br />
<br />
return 0;<br />
}<br />
</source><br />
<br />
It has an uninitialized value bug and two race condition bugs around the use of value.<br />
<br />
# The first race condition is that rank==0 sets value=0 while at the same time doing a non-blocking receive into value (bug:16).<br />
# The uninitialized value problem is that rank==1 starts a send of value to rank==0 without ever setting value (bug:22).<br />
# The second race condition is that rank==1 sets value=1 while at the same time doing a non-blocking send of value (bug:23).<br />
<br />
=== Basic Functionality ===<br />
<br />
If we just straight up compile and run this<br />
<br />
<source lang="bash"><br />
module unload intel openmpi<br />
module load gcc/4.8.2 openmpi/gcc/1.8.3<br />
mpicc -Wall -g bug.c -o bug<br />
mpirun -np 2 valgrind ./bug<br />
</source><br />
<br />
we presumably get a report about the uninitialized value, but it is buried in tens of thousands of other bogus error messages.<br />
<br />
This solution is to link against the valgrind openmpi debug wrapper library too<br />
<br />
<source lang="bash"><br />
mpicc -Wall -g bug.c -L/usr/lib64/valgrind -lmpiwrap-amd64-linux -Xlinker -rpath=/usr/lib64/valgrind -o bug<br />
mpirun -np 2 valgrind ./bug<br />
</source><br />
<br />
Now there are only a few bogus errors reported of the form<br />
<br />
==12598== Syscall param write(buf) points to uninitialised byte(s)<br />
==12598== at 0x53916FD: ??? (in /lib64/libpthread-2.12.so)<br />
==12598== by 0x8AC8E40: send_bytes (oob_tcp_sendrecv.c:84)<br />
==12598== by 0x8AC9471: mca_oob_tcp_send_handler (oob_tcp_sendrecv.c:205)<br />
==12598== by 0x616FA23: opal_libevent2021_event_base_loop (in /opt/sharcnet/openmpi/1.8.1/gcc-debug/lib/libopen-pal.so.6.1.1)<br />
==12598== by 0x5B7E78D: orte_progress_thread_engine (ess_base_std_app.c:456)<br />
==12598== by 0x538A9D0: start_thread (in /lib64/libpthread-2.12.so)<br />
==12598== by 0x8AB86FF: ???<br />
<br />
We know this isn't an issue with ''bug.c'' as the backtrace shows it is occurring in a pure OpenMPI thread (the backtrace starts with ''start_thread'' and all subsequent routines are not from ''bug.c''). Skipping over these we now see Valgrind is picking up on sending the uninitialized value<br />
<br />
==12599== Uninitialised byte(s) found during client check request<br />
==12599== at 0x4E3641D: check_mem_is_defined_untyped (libmpiwrap.c:952)<br />
==12599== by 0x4E5BBC5: generic_Isend (libmpiwrap.c:908)<br />
==12599== by 0x4E5BEE9: PMPI_Isend (libmpiwrap.c:1393)<br />
==12599== by 0x400B02: main (bug.c:22)<br />
==12599== Address 0x7fefff5c4 is on thread 1's stack<br />
<br />
'''NOTE''' If we want to avoid recompiling our program, we can also preload the ''mpiwrap-amd64-linux'' library instead of linking against it<br />
<br />
<source lang="bash"><br />
mpicc -Wall -g bug.c -o bug<br />
LD_LIBRARY_PATH="$(which mpirun | sed -e 's|/bin/.*$|/lib|')${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" \<br />
LD_PRELOAD=/usr/lib64/valgrind/libmpiwrap-amd64-linux.so mpirun -np 2 valgrind ./bug<br />
</source><br />
<br />
=== Breaking out output by rank ===<br />
<br />
By default the Valgrind output from all the MPI ranks gets intermingled together on stderr. Although they can be distinguished for the most part as each line is prefixed with the process number, it is frequently nice to write them out to individual files. This can be done by using the ''mpirun'' <tt>--output-filename <filename></tt> option. It captures the output of each rank to ''<filename>'' suffixed with the ''MPI_COMM_WORLD'' rank<br />
<br />
<source lang="bash"><br />
mpirun --output-filename bug.log -np 2 valgrind ./bug<br />
</source><br />
<br />
To redirect just the Valgrind output we need to use the Valgrind <tt>--log-file=<filename></tt> option with the special <tt>%q{<environment-variable>}</tt> syntax<br />
<br />
<source lang="bash"><br />
mpirun -np 2 valgrind --log-file=bug.log-%q{OMPI_COMM_WORLD_RANK} ./bug<br />
</source><br />
<br />
Both of the above can also be done by wrapping the Valgrind call with Bash to perform redirection and/or environment variable expansion<br />
<br />
<source lang="bash"><br />
mpirun -np 2 /bin/bash -c 'exec valgrind ./bug > bug.log-$OMPI_COMM_WORLD_RANK 2>&1'<br />
</source><br />
<br />
and<br />
<br />
<source lang="bash"><br />
mpirun -np 2 /bin/bash -c 'exec valgrind --log-file=bug.log-$OMPI_COMM_WORLD_RANK ./bug'<br />
</source><br />
<br />
=== Advanced Functionality ===<br />
<br />
Now, if on top of this, we also bring in the valgrind enabled openmpi debug library (i.e., the ''debug'' version of our ''openmpi'' module), things get really sweet<br />
<br />
<source lang="bash"><br />
module unload gcc openmpi<br />
module load gcc/4.8.2 openmpi/gcc-debug/1.8.3<br />
mpicc -Wall -g bug.c -L/usr/lib64/valgrind -lmpiwrap-amd64-linux -Xlinker -rpath=/usr/lib64/valgrind -o bug<br />
</source><br />
<br />
Now all the bugs in the code are detected and reported<br />
<br />
==27774== Uninitialised byte(s) found during client check request<br />
==27774== at 0x4E3641D: check_mem_is_defined_untyped (libmpiwrap.c:952)<br />
==27774== by 0x4E5BBC5: generic_Isend (libmpiwrap.c:908)<br />
==27774== by 0x4E5BEE9: PMPI_Isend (libmpiwrap.c:1393)<br />
==27774== by 0x402713: main (bug.c:22)<br />
==27774== Address 0x7fefff6f4 is on thread 1's stack<br />
<br />
==27773== Invalid write of size 4<br />
==27773== at 0x4026A0: main (bug.c:16)<br />
==27773== Address 0x7fefff6f4 is on thread 1's stack<br />
<br />
==27774== Invalid write of size 4<br />
==27774== at 0x40271B: main (bug.c:23)<br />
==27774== Address 0x7fefff6f4 is on thread 1's stack<br />
<br />
=== Suppression File ===<br />
<br />
There is also a valgrind suppression option <tt>--suppressions="$(which mpirun | sed -e 's|/bin/.*|/share/openmpi/openmpi-valgrind.supp|')"</tt>. We have not observed any cases where this makes a difference yet though.<br />
<br />
=References=<br />
<br />
o Valgrind Homepage<br><br />
http://www.valgrind.org<br />
<br />
o Valgrind's Tool Suite<br><br />
http://valgrind.org/info/tools.html<br />
<br />
o kcachegrind (sharcnet does not have)<br><br />
http://kcachegrind.sourceforge.net/html/Download.html<br />
<br />
[[Category:Software packages]]</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=User:Ppomorsk/Test&diff=17964User:Ppomorsk/Test2019-06-06T14:37:38Z<p>Edward: </p>
<hr />
<div>{{Template:CCMissing}}<br />
The header titles get automatically converted into tab titles.<br />
<br />
<br />
<hr><br />
<big>'''PyCUDA''' makes it possible to easily use CUDA inside Python code.</big><br />
<br />
Documentation can be found on the [http://mathema.tician.de/software/pycuda package webpage].<br />
<br />
This package is not currently installed as SHARCNET-supported software, but it's easy for users to install it on their own following instructions below. If any difficulties are encountered when following these instructions, please ask SHARCNET staff for help.<br />
<br />
See also: [[PyOpenCL]]<br />
<br />
==SHARCNET installation instructions==<br />
<br />
===Monk cluster===<br />
(These instructions were tested on Oct 22, 2014 )<br />
<br />
1. Unload the Intel compiler module (loaded by default), so that GCC becomes the default compiler. Also, use a later python version.<br />
<source lang="bash"><br />
module unload intel<br />
module unload openmpi<br />
module load gcc/4.8.2<br />
module load openmpi/gcc/1.8.1<br />
module load python/gcc/2.7.8<br />
</source><br />
Note: openmpi module is loaded because the python module needs it (it is not actually used by PyCuda)<br />
<br />
<br />
2. Create some directory you want to build the package in, cd into it, then get the PyCUDA source code:<br />
<br />
<source lang="bash"><br />
git clone http://git.tiker.net/trees/pycuda.git<br />
cd pycuda<br />
git submodule init<br />
git submodule update<br />
<br />
wget https://pypi.python.org/packages/source/p/pycuda/pycuda-2014.1.tar.gz#md5=fdc2f59e57ab7256a7e0df0d9d943022<br />
tar xfz pycuda-2014.1.tar.gz<br />
cd pycuda-2014.1<br />
<br />
</source><br />
<br />
3. At this point decide where you want the package to be installed. In this example will use a directory called python_packages in the home directory. If this directory does not yet exist, make it with:<br />
<br />
<source lang="bash"><br />
mkdir -p ~/python_packages/lib/python/<br />
</source><br />
Doing it this way creates the required subdirectories as well.<br />
<br />
4. Edit file Makefile.in to add --home flag pointing to the directory you created to the setup install line, so that it reads:<br />
<source lang="bash"><br />
${PYTHON_EXE} setup.py install --home=~/python_packages<br />
</source><br />
<br />
5.You now need to update the PYTHONPATH variable to point to the library directory:<br />
<source lang="bash"><br />
export PYTHONPATH=~/python_packages/lib/python/:$PYTHONPATH<br />
</source><br />
<br />
6. Configure and compile, providing a path to the CUDA files on monk:<br />
<br />
<source lang="bash"><br />
python configure.py --cuda-root=/opt/sharcnet/cuda/6.0.37/toolkit<br />
make install<br />
</source><br />
<br />
7. Do the first test of the installation to make sure the pycuda module can be imported, by starting python and executing:<br />
<br />
<source lang="bash"><br />
import pycuda<br />
</source><br />
<br />
If no errors are reported, everything worked and the package is ready for use.<br />
<br />
8. Add the lines:<br />
<br />
<source lang="bash"><br />
module unload intel<br />
module unload openmpi<br />
module load gcc/4.8.2<br />
module load openmpi/gcc/1.8.1<br />
module load python/gcc/2.7.8<br />
export PYTHONPATH=~/python_packages/lib/python/:$PYTHONPATH<br />
</source><br />
<br />
to your ~/.bashrc file so that this variable is set automatically for you on every login.<br />
<br />
9. Test PyCUDA on a development node which has a GPU (the login node does not have one so PyCUDA tests will produce an error). To do this, execute on monk login node:<br />
<br />
<source lang="bash"><br />
ssh mon54<br />
</source><br />
<br />
Then go to the directory where you put the PyCuda source code, and execute:<br />
<br />
<source lang="bash"><br />
python test/test_driver.py<br />
</source><br />
<br />
Gives error:<br />
<br />
[ppomorsk@mon241:~/supported_sharcnet_packages/pycuda/pycuda-2014.1/test] python test_driver.py<br />
Traceback (most recent call last):<br />
File "test_driver.py", line 4, in <module><br />
from pycuda.tools import mark_cuda_test<br />
File "/home/ppomorsk/python_packages/lib/python/pycuda-2014.1-py2.7-linux-x86_64.egg/pycuda/tools.py", line 30, in <module><br />
import pycuda.driver as cuda<br />
File "/home/ppomorsk/python_packages/lib/python/pycuda-2014.1-py2.7-linux-x86_64.egg/pycuda/driver.py", line 2, in <module><br />
from pycuda._driver import * # noqa<br />
ImportError: /home/ppomorsk/python_packages/lib/python/pycuda-2014.1-py2.7-linux-x86_64.egg/pycuda/_driver.so: undefined symbol: cuStreamAttachMemAsync<br />
<br />
ldd output<br />
<br />
[ppomorsk@mon54:~/supported_sharcnet_packages/pycuda/pycuda-2014.1/test] ldd /home/ppomorsk/python_packages/lib/python/pycuda-2014.1-py2.7-linux-x86_64.egg/pycuda/_driver.so<br />
<br />
linux-vdso.so.1 => (0x00007fffac5c1000)<br />
<br />
libcuda.so.1 => /usr/lib64/libcuda.so.1 (0x00002b7dbda9e000)<br />
<br />
libcurand.so.6.0 => /opt/sharcnet/cuda/6.0.37/toolkit/lib64/libcurand.so.6.0 (0x00002b7dbea01000)<br />
<br />
libstdc++.so.6 => /opt/sharcnet/gcc/4.8.2/lib64/libstdc++.so.6 (0x00002b7dc3f08000)<br />
<br />
libm.so.6 => /lib64/libm.so.6 (0x00002b7dc4211000)<br />
<br />
libgcc_s.so.1 => /opt/sharcnet/gcc/4.8.2/lib64/libgcc_s.so.1 (0x00002b7dc4496000)<br />
<br />
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b7dc46ac000)<br />
<br />
libc.so.6 => /lib64/libc.so.6 (0x00002b7dc48c9000)<br />
<br />
libz.so.1 => /lib64/libz.so.1 (0x00002b7dc4c5e000)<br />
<br />
libdl.so.2 => /lib64/libdl.so.2 (0x00002b7dc4e74000)<br />
<br />
librt.so.1 => /lib64/librt.so.1 (0x00002b7dc5078000)<br />
/lib64/ld-linux-x86-64.so.2 (0x00002b7dbd4a9000)<br />
<br />
and the symbol does seem to exist<br />
<br />
[ppomorsk@mon54:/usr/lib64] readelf -Ws libcuda.so.331.89 | grep cuStreamAttachMemAsync<br />
43: 0000000000139890 538 FUNC GLOBAL DEFAULT 10 cuStreamAttachMemAsync<br />
<br />
<br />
try<br />
<br />
python configure.py --cuda-root=/opt/sharcnet/cuda/6.0.37/toolkit --cudadrv-lib-dir=/usr/lib64<br />
If everything is working properly, the output should look like this:<br />
<br />
http://pycuda.2962900.n2.nabble.com/PyCUDA-undefined-symbol-cuMemAllocPitch-v2-td7571641.html<br />
<br />
"cudart is the run-time interface, which is used by 'conventional' CUDA C <br />
code. (cudaMemcpy) PyCUDA uses the driver interface. (cuMemcpy) "<br />
<br />
"mismatch between cuda.h and driver" ??<br />
try older CUDA modules?<br />
<br />
<br />
python configure.py --cuda-root=/opt/sharcnet/cuda/5.5.22/toolkit --cudadrv-lib-dir=/usr/lib64<br />
<br />
with cuda/5.5.22 loaded<br />
even basic cuda test program fails with<br />
<br />
FATAL: Error inserting nvidia (/lib/modules/2.6.32-431.3.1.el6.x86_64/kernel/drivers/video/nvidia.ko): No such device<br />
<br />
<br />
--<br />
<br />
runpaths problem<br />
<br />
_driver.so has an RPATH, but none of the members contain libcuda.so. so it was build wrong, in that it doesn't correctly encode its actual dependency.<br />
<br />
[ppomorsk@mon241:~/supported_sharcnet_packages/pycuda/pycuda-2014.1/test] objdump -x /home/ppomorsk/python_packages/lib/python/pycuda-2014.1-py2.7-linux-x86_64.egg/pycuda/_driver.so | grep RPATH<br />
RPATH /opt/sharcnet/python/2.7.8/gcc/lib:/opt/sharcnet/gcc/4.8.2/lib64:/opt/sharcnet/cuda/6.0.37/cula/lib64:/opt/sharcnet/cuda/6.0.37/toolkit/lib64:/opt/sharcnet/mkl/10.3.9/mkl/lib/intel64:/opt/sharcnet/mkl/10.3.9/lib/intel64<br />
<br />
Problem with python itself??<br />
python/gcc/2.7.8 module has theano, which is GPU , so there might be some hidden dependency.<br />
<br />
Anyway, with theano python/gcc/2.7.8 ought to have an explicit CUDA dependency.<br />
<br />
<br />
<source lang="bash"><br />
<br />
[ppomorsk@mon54:~/pycuda] python test/test_driver.py <br />
============================= test session starts ==============================<br />
platform linux2 -- Python 2.6.6 -- pytest-2.3.4<br />
collected 21 items <br />
<br />
test_driver.py .....................<br />
<br />
========================== 21 passed in 61.39 seconds ==========================<br />
</source><br />
<br />
<br />
10. Try the example programs provided with the source code, found in the ''examples'' subdirectory of your pycuda source directory:<br />
<br />
<source lang="bash"><br />
python examples/dump_properties.py<br />
python examples/hello_gpu.py<br />
</source><br />
<br />
==Sample PyCUDA code==<br />
This is code from the '''hello_gpu.py''' example program. It multiplies two vectors elementwise on the GPU, and then verifies the result with a standard calculation on the CPU.<br />
<br />
<source lang="python"><br />
<br />
import pycuda.driver as drv<br />
import pycuda.tools<br />
import pycuda.autoinit<br />
import numpy<br />
import numpy.linalg as la<br />
from pycuda.compiler import SourceModule<br />
<br />
mod = SourceModule("""<br />
__global__ void multiply_them(float *dest, float *a, float *b)<br />
{<br />
const int i = threadIdx.x;<br />
dest[i] = a[i] * b[i];<br />
}<br />
""")<br />
<br />
multiply_them = mod.get_function("multiply_them")<br />
<br />
a = numpy.random.randn(400).astype(numpy.float32)<br />
b = numpy.random.randn(400).astype(numpy.float32)<br />
<br />
dest = numpy.zeros_like(a)<br />
multiply_them(<br />
drv.Out(dest), drv.In(a), drv.In(b),<br />
block=(400,1,1))<br />
<br />
print dest-a*b<br />
</source><br />
<br />
[[Category:Software packages]]<br />
<br />
<br />
<br />
<hr><br />
List of colors: [https://meta.wikimedia.org/wiki/Wiki_color_formatting_help Wiki_color_formatting_help] <br />
<br />
Simplest table<br />
<br />
{| <br />
|AA || BB || CC<br />
|-<br />
|DD || EE || FF<br />
|}<br />
<br />
Table with padding<br />
<br />
{| cellpadding=5<br />
|AA || BB || CC<br />
|-<br />
|DD || EE || FF<br />
|}<br />
<br />
Table with border<br />
<br />
{| cellpadding=5 style="border:1px solid #BBB"<br />
|- bgcolor="#fafeff"<br />
|AA || CC || EE<br />
|- bgcolor="#fafeff"<br />
|BB || DD || FF<br />
|}<br />
<br />
<br />
Notebox:<br />
<br />
{| cellpadding=5 style="border:4px solid #4682B4"<br />
|- bgcolor="#E0FFFF"<br />
| This is a notebox<br>to show border color.<br />
|}<br />
<br />
<br />
{{#switchtablink:Tab name|CUDA|CUDA}}<br />
<headertabs/><br />
= Third section header =<br />
This will be always displayed under the tab view<br />
because it's below the &lt;headertabs/&gt; tag.<br />
<br />
<br />
=== Example ===<br />
<pre>Some text above the tab view<br />
= First section header =<br />
This will be displayed on the first tab<br />
<br />
<nowiki>{{#switchtablink:Second section header|Click here to go to the next tab...}}</nowiki><br />
{{#switchtablink:Tab name|Link text|Page name}}<br />
= Second section header =<br />
This will be displayed on the second tab<br />
<headertabs/><br />
= Third section header =<br />
This will be always displayed under the tab view<br />
because it's below the &lt;headertabs/&gt; tag.<br />
</pre></div>Edwardhttps://www.sharcnet.ca/help/index.php?title=OCTAVE&diff=17959OCTAVE2019-06-06T14:34:18Z<p>Edward: </p>
<hr />
<div>{{Template:CCMissing}}<br />
{{Software<br />
|package_name=OCTAVE<br />
|package_description=Mostly compatible language with Matlab primarily intended for numerical computations<br />
|package_idnumber=28<br />
}}<br />
<br />
=Introduction=<br />
<br />
Octave is provided on SHARCNET clusters to allow serial or threaded jobs to be run in the queue as described below.<br />
<br />
==OCTAVE on the Graham national system==<br />
<br />
On the Graham system the user simply needs to call the "module load" command without manually unloading conflicting packages.<br />
<br />
$ module load octave/4.2.1<br />
<br />
To obtain information about the package's dependencies the "spider" action can be included in the call to "module":<br />
<br />
<pre><br />
$ module spider octave<br />
<br />
---------------------------------------------------------------------------------------------------------------------------------------------------------<br />
octave:<br />
---------------------------------------------------------------------------------------------------------------------------------------------------------<br />
Description:<br />
GNU Octave is a high-level interpreted language, primarily intended for numerical computations. - Homepage: http://www.gnu.org/software/octave/<br />
<br />
Versions:<br />
octave/4.2.1<br />
<br />
---------------------------------------------------------------------------------------------------------------------------------------------------------<br />
For detailed information about a specific "octave" module (including how to load the modules) use the module's full name.<br />
For example:<br />
<br />
$ module spider octave/4.2.1<br />
---------------------------------------------------------------------------------------------------------------------------------------------------------<br />
<br />
<br />
<br />
[jdesjard@gra-login1 ~]$ module spider octave/4.2.1<br />
<br />
---------------------------------------------------------------------------------------------------------------------------------------------------------<br />
octave: octave/4.2.1<br />
---------------------------------------------------------------------------------------------------------------------------------------------------------<br />
Description:<br />
GNU Octave is a high-level interpreted language, primarily intended for numerical computations. - Homepage: http://www.gnu.org/software/octave/<br />
<br />
Properties:<br />
Tools for development / Outils de développement<br />
<br />
You will need to load all module(s) on any one of the lines below before the "octave/4.2.1" module is available to load.<br />
<br />
nixpkgs/16.09 gcc/5.4.0<br />
nixpkgs/16.09 intel/2016.4<br />
nixpkgs/16.09 intel/2017.1<br />
<br />
Help:<br />
GNU Octave is a high-level interpreted language, primarily intended for numerical computations. - Homepage: http://www.gnu.org/software/octave/<br />
</pre><br />
<br />
=Job Submission=<br />
<br />
The Graham national system uses the Slurm scheduler which is different from the Torque Moab scheduler that is typical to the majority of other SHARCNET systems. In order to submit a simple "Hello World!" process to Graham the user would first need to create a submit script as follows.<br />
<br />
$ cat oct_serial.sh <br />
#!/bin/bash<br />
#SBATCH -t 0-00:01<br />
#SBATCH --mem=400<br />
octave --eval 'disp("Hello World! from Octave")'<br />
<br />
Then pass this submit script to "sbatch" to be entered into the queue:<br />
<br />
$ sbatch --account=accountname oct_serial.sh<br />
<br />
... where "accountname" is the name of the accounting group to submit to (e.g. def-groupname, rgg-projectname).<br />
<br />
=Example Job=<br />
<br />
This section shows howto submit a [http://www.gnu.org/software/octave/doc/interpreter/Executable-Octave-Programs.html sample.m] file to the serial queue that accepts [http://www.gnu.org/software/octave/doc/interpreter/Command-Line-Options.html#Command-Line-Options command line] arguments. <br />
<br />
<pre><br />
[roberpj@hnd20:~/samples/octave/args] cat sample.m <br />
#! /bin/octave -qf<br />
printf ("%s", program_name ());<br />
arg_list = argv ();<br />
for i = 1:nargin<br />
printf (" %s", arg_list{i});<br />
endfor<br />
printf ("\n");<br />
</pre><br />
<br />
To eliminate exatraneous verbosity in the output file two switches are passed:<br />
<br />
<pre><br />
[roberpj@hnd20:~/samples/octave/args] sqsub -r 60m -o ofile.%J octave -qf --no-window-system sample.m arg1 arg2 arg3 etc<br />
WARNING: no memory requirement defined; assuming 2GB per process.<br />
submitted as jobid 6937872<br />
</pre><br />
<br />
The output file from the job appears as:<br />
<br />
<pre><br />
[roberpj@hnd20:~/samples/octave/args] cat ofile.6937872.hnd50<br />
sample.m arg1 arg2 arg3 etc<br />
</pre><br />
<br />
<br />
<br />
==One the Graham national system==<br />
<br />
Rather than dedicated development as it typical on some of the other SHARCNET systems the national systems have nodes that prioritize interactive jobs allocated by call to "salloc" of the Slurm scheduler.<br />
<br />
To use Octave interactively on Graham you can do the following:<br />
<br />
Use salloc to request a single task for one hour:<br />
<br />
$ salloc --account=def-roxa88 --time=1:0:0 --ntasks=1<br />
<br />
Then you will receive notification when the reservation is granted (including jobid), for<br />
example:<br />
<br />
salloc: Granted job allocation 12345<br />
<br />
Then use srun to start a terminal to work in on the reservation:<br />
<br />
$ srun --wait 0 --pty bash<br />
<br />
Once running the terminal in the allocation you can start your work with Octave by loading<br />
the module and starting the program:<br />
<br />
<pre><br />
$ module load octave<br />
$ octave<br />
octave: X11 DISPLAY environment variable not set<br />
octave: disabling GUI features<br />
GNU Octave, version 4.2.1<br />
Copyright (C) 2017 John W. Eaton and others.<br />
This is free software; see the source code for copying conditions.<br />
There is ABSOLUTELY NO WARRANTY; not even for MERCHANTABILITY or<br />
FITNESS FOR A PARTICULAR PURPOSE. For details, type 'warranty'.<br />
<br />
Octave was configured for "x86_64-pc-linux-gnu".<br />
<br />
Additional information about Octave is available at http://www.octave.org.<br />
<br />
Please contribute if you find this software useful.<br />
For more information, visit http://www.octave.org/get-involved.html<br />
<br />
Read http://www.octave.org/bugs.html to learn how to submit bug reports.<br />
For information about changes from previous versions, type 'news'.<br />
<br />
octave:1><br />
<br />
</pre><br />
<br />
... Then once you have the "octave:#>" prompt the terminal will behave like the Matlab<br />
command line.<br />
<br />
Once you are finished with Octave use "quit" to exit the program, then "exit" to logout of<br />
the compute node (terminate srun), then "exit" again to terminate the allocation<br />
(relinquishing the allocate resources).<br />
<br />
=Running on a visualization system=<br />
<br />
Once logged in to a visualization system an instance of Octave can be launched from Applications Menu > Eduction > GNU Octave.<br />
<br />
=General notes=<br />
<br />
==Matlab Compatibility==<br />
<br />
The online wiki resources "Octave Wiki":<br />
http://wiki.octave.org/ <br />
or "Wikibook":<br />
http://en.wikibooks.org/wiki/MATLAB_Programming/Differences_between_Octave_and_MATLAB <br />
provide good introductory explanations of code compatibility between Octave and Matlab.<br />
<br />
In order to run Octave so that it interprets scripts as closely as possible to Matlab, use the --traditional flag. Taking from the above sqsub example the maximum Matlab compatible submission would be:<br />
<br />
sqsub -r 60m -o ofile.%J octave --traditional mycode.m<br />
<br />
==Reading/Writing Files==<br />
<br />
There are two strategies for handling file <i>input and output</i> described in the "Octave manual" viz ...<br><br />
http://www.gnu.org/software/octave/doc/interpreter/Input-and-Output.html#Input-and-Output<br> http://www.gnu.org/software/octave/docs.html.<br />
<br />
The following stanza demonstrates the "simple file I/O" approach:<br />
http://www.gnu.org/software/octave/doc/interpreter/Simple-File-I_002fO.html#Simple-File-I_002fO <br />
<br />
<pre><br />
save myiofile.dat A B C<br />
save ("-text", "myiofile.dat", "A", "B", "C")<br />
save ("-binary", "myiofile.dat", "A", "B", "C")<br />
load myiofile.dat<br />
load ("-text", "myiofile.dat", "A", "B", "C")<br />
load ("-binary", "myiofile.dat", "A", "B", "C")<br />
</pre><br />
<br />
where A, B and C are a potential mix of entities such as scalars, vectors or matrices. Note that for large files the binary format is strongly recommended to both minimize disk space and file read/write wallclock times.<br />
<br />
==Octave-Forge==<br />
<br />
Octave-forge packages are not pre-installed on SHARCNET clusters since none are available at this time for the operating system. Therefore any required packages must be downloaded, manually installed from source and then managed in local user accounts (as described below).<br />
<br />
However 16 of approximately 95 octave-forge packages are installed on some sharcnet visualization workstations including viz1,2,3-uwaterloo, viz2-uwo, viz6-uoguelph under /usr/share/octave/packages. As a word of caution however, all package versions will likely be significantly old since they are tied to the operating system fedora /etc/redhat-release major version and hence not contain recent critical bug fixes (which could be numerical in nature). Users are therefore strongly recommended to likewise download and install the latest version similarly as would be done on the clusters.<br />
<br />
===Sharing Packages===<br />
<br />
Note that packages installed by a single sharcnet user in their home account can be shared out to other research group members or even among general SHARCNET users. To do this simply set access permissions for group or global read access accordingly. For details on how to change <br />
the permissions see: https://www.sharcnet.ca/help/index.php/Knowledge_Base#How_do_I_give_other_users_access_to_my_files_.3F<br />
<br />
===Managing Packages===<br />
<br />
The "Octave Forge" http://octave.sourceforge.net/ project provides extra packages for use with octave that can be downloaded into a directory such as <i>~/my_octave_sources</i> then manually installed into your own sharcnet account as will be shown in the follow example. A current list of packages available for download that notable are generally only compatible with the latest major release of octave cat be found here http://octave.sourceforge.net/packages.php. <br />
<br />
In the event where the major version of octave on sharcnet you are using (for instance 3.4.x) is not the same as latest major version (at the time of writing 3.6.x) and there were programming api changes between the two versions that effect the package you want to use, then you probably will need to download an older version of a package from http://sourceforge.net/projects/octave/files/Octave%20Forge%20Packages/Individual%20Package%20Releases/. for it to work.<br />
<br />
To help estimate the package version required as a starting point, consider the major release dates of recent which are octave-3.4.3 (10/10/11), octave-3.6.0 (01/15/12), octave-3.6.1 (02/22/12) and finally octave-3.6.2 (05/31/12). Next consider you want to download the latest geometry package that was compatible with octave-3.4.3 at the time of its release, then simply display all the archived versions as shown in the following stanza and pick the last available geometry release date before the next major release octave-3.6.0 (01/15/12). To show a list of all archived geometry versions, do the following steps:<br />
<br />
<pre><br />
http://octave.sourceforge.net/<br />
Click Packages on top menu menu bar<br />
Scroll down to the miscellaneous package row and click details<br />
Click (older versions) located below "Download Package"<br />
Click Octave Forge Packages<br />
Click Individual Package Releases<br />
Please wait for the page to load ...<br />
Click "Name" at the top of first colum to sort packages alphabetically<br />
Scroll down you will find all available archived geometry packages:<br />
geometry-1.0.1.tar.gz 2011-09-27<br />
geometry-1.1.1.tar.gz 2011-10-06<br />
geometry-1.1.2.tar.gz 2011-10-09<br />
geometry-1.1.3.tar.gz 2011-10-13<br />
geometry-1.1.tar.gz 2011-10-04<br />
geometry-1.2.0.tar.gz 2011-10-22<br />
geometry-1.2.1.tar.gz 2011-11-02<br />
geometry-1.2.2.tar.gz 2011-11-04<br />
geometry-1.4.0.tar.gz 2012-01-25<br />
geometry-1.4.1.tar.gz 2012-03-24<br />
geometry-1.5.0.tar.gz 2012-06-05<br />
</pre><br />
<br />
Therefore you will download geometry-1.2.2.tar.gz (2011-11-04) since the next<br />
release geometry-1.4.0.tar.gz (2012-01-25) and then install it into octave<br />
3.4.3 as follows:<br />
<br />
<pre><br />
[roberpj@tope:~/my_octave_sources] octave<br />
GNU Octave, version 3.4.3<br />
octave:1> pkg install geometry-1.2.2.tar.gz<br />
octave:2> pkg list<br />
Package Name | Version | Installation directory<br />
---------------+---------+-----------------------<br />
geometry | 1.2.2 | /home/roberpj/octave/geometry-1.2.2<br />
octave:15> pkg load geometry<br />
octave:16> pkg describe geometry<br />
---<br />
Package name:<br />
geometry<br />
Version:<br />
1.2.2<br />
Short description:<br />
Library for geometric computing extending MatGeom functions. Useful to create,<br />
transform, manipulate and display geometric primitives.<br />
Status:<br />
Loaded<br />
</pre><br />
<br />
<b>Note:</b> Any questions regarding package version compatibility with major or minor octave release should be referred to the developers.<br />
<br />
===Package Commands===<br />
<br />
Additional examples of package commands are shown in this section. For demonstration purpose linear-algebra will first be downloaded:<br />
<br />
<pre><br />
[roberpj@iqaluk:~] mkdir my_octave_sources<br />
[roberpj@iqaluk:~] cd my_octave_sources<br />
wget http://downloads.sourceforge.net/octave/general-1.3.2.tar.gz<br />
wget http://downloads.sourceforge.net/octave/linear-algebra-2.2.0.tar.gz<br />
</pre><br />
<br />
<pre><br />
[myusername@orc-login1:~]octave<br />
octave:1> help pkg<br />
</pre><br />
<br />
To install a package such as linear-algebra do:<br />
<br />
<pre><br />
octave:2> cd ~/my_octave_sources<br />
octave:3> pkg install general-1.3.2.tar.gz<br />
octave:4> pkg install linear-algebra-2.2.0.tar.gz<br />
</pre><br />
<br />
To remove a package, from a terminal first do:<br />
<pre>cd ~/octave<br />
rm -rf linear-algebra-2.2.0</pre><br />
<br />
... then from within octave do:<br />
<pre>octave:5> pkg uninstall linear-algebra</pre><br />
<br />
To list all installed packages do:<br />
<br />
<pre>octave:6> pkg list</pre><br />
<br />
To add a named packages to your path:<br />
<br />
<pre>octave:7> pkg load name</pre><br />
<br />
To remove a named package from your path:<br />
<br />
<pre>octave:8> pkg unload</pre><br />
<br />
To list all functions provided by a package:<br />
<br />
<pre>octave:9> pkg describe -verbose all</pre><br />
<br />
To list functions provide by extra odepkg package:<br />
<br />
<pre>octave:10> pkg describe odepkg</pre><br />
<br />
To list functions details for extra financial package:<br />
<br />
<pre>octave:11> pkg describe financial -verbose</pre><br />
<br />
To get documentation for a topic such as sin do:<br />
<br />
<pre>octave:12> doc sin</pre><br />
<br />
To get help using the doc command do:<br />
<br />
<pre>octave:13> help doc</pre><br />
<br />
To get documentation for a topic such as matrix do:<br />
<br />
<pre>octave:14> doc matrix</pre><br />
<br />
To get documentation for any topic run the doc command alone:<br />
<br />
<pre>octave:15> doc</pre><br />
<br />
===Package Example===<br />
<br />
This example assumes you first create two directories in your home account one being named<br />
~/my_octave_packages <br />
and the other <br />
~/my_octave_sources <br />
then download the required three tar.gz files into the latter from <br />
http://octave.sourceforge.net/packages.php ...<br />
<br />
[roberpj@orc-login1:~] module unload octave<br />
[roberpj@orc-login1:~] module load octave<br />
[roberpj@orc-login1:~] octave<br />
GNU Octave, version 3.4.3<br />
octave:1> pkg prefix ~/my_octave_packages<br />
ans = /home/roberpj/my_octave_packages<br />
octave:2> cd my_octave_sources<br />
octave:3> ls<br />
miscellaneous-1.0.11.tar.gz optim-1.0.17.tar.gz struct-1.0.9.tar.gz<br />
octave:4> pkg install miscellaneous-1.0.11.tar.gz optim-1.0.17.tar.gz struct-1.0.9.tar.gz<br />
octave:5> pkg list<br />
Package Name | Version | Installation directory<br />
---------------|---------|-----------------------<br />
miscellaneous *| 1.0.11 | /home/roberpj/my_octave_packages/miscellaneous-1.0.11<br />
optim *| 1.0.17 | /home/roberpj/my_octave_packages/optim-1.0.17<br />
struct *| 1.0.9 | /home/roberpj/my_octave_packages/struct-1.0.9</PRE><br />
<br />
=References=<br />
<br />
o Octave Homepage<br><br />
http://www.gnu.org/software/octave/<br />
<br />
o Octave 725 Page Manual (Version 3.4.0)<br><br />
http://www.gnu.org/software/octave/doc/interpreter/<br />
<br />
o Statistic Package Function Reference<br><br />
http://octave.sourceforge.net/doc/funref_statistics.html<br />
<br />
o GNU Octave Wiki<br><br />
http://wiki.octave.org/<br />
<br />
o Matlab-Like Tools for HPC (article)<br><br />
http://www.admin-magazine.com/HPC/Articles/Matlab-Like-Tools-for-HPC</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=MPIBLAST&diff=17957MPIBLAST2019-06-06T14:32:11Z<p>Edward: </p>
<hr />
<div>{{Template:CCDelete}}<br />
{{Software<br />
|package_name=MPIBLAST<br />
|package_description=Parallel implementation of NCBI BLAST<br />
|package_idnumber=55}}<br />
{{Template:GrahamUpdate}}<br />
<br />
=Introduction=<br />
<br />
The mpiblast module must be manually loaded before submitting any mpiblast jobs. Two examples below <br />
demonstrate how to setup and submit jobs to the mpi queue.<br />
<br />
=Version Selection=<br />
<br />
===All Clusters (except Guppy)===<br />
<br />
module unload openmpi intel<br />
module load intel/12.1.3 openmpi/intel/1.6.2 mpiblast/1.6.0<br />
<br />
===Guppy Only===<br />
<br />
module unload openmpi intel<br />
module load intel/11.0.083 openmpi/intel/1.4.2 mpiblast/1.6.0<br />
<br />
=Job Submission=<br />
<br />
sqsub -r 15m -n 24 -q mpi --mpp=3g -o ofile%J mpiblast etc<br />
<br />
where the number of cores n=24 is arbitrary.<br />
<br />
=Example Jobs=<br />
<br />
==DROSOPH==<br />
<br />
Copy sample problem files (fasta database and input) from the /opt/sharcnet <i>examples</i> directory a directory under work as shown here. The fasta database used in this example can be obtained as a guest from NCBI here http://www.ncbi.nlm.nih.gov/guide/all/#downloads_ then clicking "FTP: FASTA BLAST Databases".<br />
<br />
<pre><br />
mkdir -p /work/$USER/samples/mpiblast/test1<br />
rm /work/$USER/samples/mpiblast/test1/*<br />
cd /work/$USER/samples/mpiblast/test1<br />
cp /opt/sharcnet/mpiblast/1.6.0/examples/drosoph.in drosoph.in<br />
gunzip -c /opt/sharcnet/mpiblast/1.6.0/examples/drosoph.nt.gz > drosoph.nt.$USER<br />
</pre><br />
<br />
Create a hidden configuration file using a text editor (such as vi) to define a Shared storage location between nodes and a Local storage directory available on each compute node as follows:<br />
<br />
<pre><br />
[roberpj@hnd20:/work/$USER/samples/mpiblast/test1] vi .ncbirc<br />
[NCBI]<br />
Data=/opt/sharcnet/mpiblast/1.6.0/ncbi/data<br />
[BLAST]<br />
BLASTDB=/scratch/YourSharcnetUsername/mpiblasttest1<br />
BLASTMAT=/work/YourSharcnetUsername/samples/mpiblast/test1<br />
[mpiBLAST]<br />
Shared=/scratch/YourSharcnetUsername/mpiblasttest1<br />
Local=/tmp<br />
</pre><br />
<br />
Partition the database into 8 fragments into <i>/scratch/$USER/mpiblasttest1</i> where they will be searched for when running a job in queue according to the <i>.ncbirc</i> file:<br />
<br />
<pre><br />
module load mpiblast/1.6.0<br />
mkdir /scratch/roberpj/mpiblasttest1<br />
rm -f /scratch/$USER/mpiblasttest1/*<br />
cd /work/$USER/samples/mpiblast/test1<br />
<br />
[roberpj@hnd20:/work/$USER/samples/mpiblast/test1] mpiformatdb -N 8 -i drosoph.nt.$USER -o T -p F -n /scratch/$USER/mpiblasttest1<br />
Reading input file<br />
Done, read 1534943 lines<br />
Breaking drosoph.nt into 8 fragments<br />
Executing: formatdb -p F -i drosoph.nt -N 8 -n /scratch/roberpj/mpiblasttest1/drosoph.nt -o T <br />
Created 8 fragments.<br />
<<< Please make sure the formatted database fragments are placed in /scratch/roberpj/mpiblasttest1/ before executing mpiblast. >>><br />
<br />
[roberpj@hnd20:/scratch/roberpj/mpiblasttest1] ls<br />
drosoph.nt.000.nhr drosoph.nt.002.nin drosoph.nt.004.nnd drosoph.nt.006.nni<br />
drosoph.nt.000.nin drosoph.nt.002.nnd drosoph.nt.004.nni drosoph.nt.006.nsd<br />
drosoph.nt.000.nnd drosoph.nt.002.nni drosoph.nt.004.nsd drosoph.nt.006.nsi<br />
drosoph.nt.000.nni drosoph.nt.002.nsd drosoph.nt.004.nsi drosoph.nt.006.nsq<br />
drosoph.nt.000.nsd drosoph.nt.002.nsi drosoph.nt.004.nsq drosoph.nt.007.nhr<br />
drosoph.nt.000.nsi drosoph.nt.002.nsq drosoph.nt.005.nhr drosoph.nt.007.nin<br />
drosoph.nt.000.nsq drosoph.nt.003.nhr drosoph.nt.005.nin drosoph.nt.007.nnd<br />
drosoph.nt.001.nhr drosoph.nt.003.nin drosoph.nt.005.nnd drosoph.nt.007.nni<br />
drosoph.nt.001.nin drosoph.nt.003.nnd drosoph.nt.005.nni drosoph.nt.007.nsd<br />
drosoph.nt.001.nnd drosoph.nt.003.nni drosoph.nt.005.nsd drosoph.nt.007.nsi<br />
drosoph.nt.001.nni drosoph.nt.003.nsd drosoph.nt.005.nsi drosoph.nt.007.nsq<br />
drosoph.nt.001.nsd drosoph.nt.003.nsi drosoph.nt.005.nsq drosoph.nt.dbs<br />
drosoph.nt.001.nsi drosoph.nt.003.nsq drosoph.nt.006.nhr drosoph.nt.mbf<br />
drosoph.nt.001.nsq drosoph.nt.004.nhr drosoph.nt.006.nin drosoph.nt.nal<br />
drosoph.nt.002.nhr drosoph.nt.004.nin drosoph.nt.006.nnd<br />
</pre><br />
<br />
Submit a short job with a 15m time limit on n=16 cores calculate by taking N=8 fragments + 8. If all goes well output results will be written to <i>drosoph.out</i> and the execution time will appear in ofile%J where %J is the job number:<br />
<br />
<pre><br />
[roberpj@hnd20:/work/$USER/samples/mpiblast/test1]<br />
sqsub -r 15m -n 106 -q mpi --mpp=1G -o ofile%J mpiblast -d drosoph.nt.$USER -i drosoph.in<br />
-p blastn -o drosoph.out --use-parallel-write --use-virtual-frags<br />
submitted as jobid 6966896<br />
<br />
[roberpj@hnd20:/work/roberpj/samples/mpiblast/test1] cat ofile6966896.hnd50 <br />
Total Execution Time: 1.80031<br />
</pre><br />
<br />
When submitting a mpiblast job on a cluster such as goblin that doesnt have an inifiniband interconnect better performance (at least double speedup) will be achieved running the mpi job on one compute node. For regular users of non-contributed hardware typically specify "-n 8" to reflect the max number of cores on a single node:<br />
<br />
<pre><br />
sqsub -r 15m -n 8 -N 1 -q mpi --mpp=4G -o ofile%J mpiblast -d drosoph.nt -i drosoph.in<br />
-p blastn -o drosoph.out --use-parallel-write --use-virtual-frags<br />
</pre><br />
<br />
Sample output results computed previously with BLASTN 2.2.15 [Oct-15-2006] are included in <i>/opt/sharcnet/mpiblast/1.6.0/examples/ROSOPH.out</i> to compare your newly generated <i>drosoph.out</i> file with.<br />
<br />
==UNIGENE==<br />
<br />
The main purpose of this example is to illustrate some additional options and switchs that maybe useful for debugging and for dealing with larger databases as described in official detail at http://www.mpiblast.org/Docs/Guide. The fasta database used in this example can also be downloaded from http://www.ncbi.nlm.nih.gov/guide/all/#downloads_ as a guest by clicking "FTP: UniGene" then entering the "Homo_sapiens" sub-directory. More information about UniGene alignments can be found at https://cgwb.nci.nih.gov/cgi-bin/hgTrackUi?hgsid=95443&c=chr1&g=uniGene_3 . As with Example1 above, for convenience all required files can simply be copied from the /opt/sharcnet <i>examples</i> subdirectory to work as shown here:<br />
<br />
<pre><br />
mkdir /work/$USER/samples/mpiblast/test2<br />
rm -f /work/$USER/samples/mpiblast/test2/*<br />
cd /work/$USER/samples/mpiblast/test2<br />
cp /opt/sharcnet/mpiblast/1.6.0/examples/il2ra.in il2ra.in<br />
gunzip -c /opt/sharcnet/mpiblast/1.6.0/examples/Hs.seq.uniq.gz > Hs.seq.$USER<br />
<br />
[roberpj@orc-login2:/work/roberpj/samples/mpiblast/test2] ls<br />
Hs.seq.roberpj il2ra.in<br />
</pre><br />
<br />
Create a hidden configuration file using a text editor (such as vi) to define a Shared storage location between nodes and a Local storage directory available on each compute node as followw. Note that the ncbi/data directory is not used in this example and hence can be omitted. If the Local and Shared directories are the same replace <b>--copy-via=mpi</b> with <b>--copy-via=none</b> as will be demonstrated in the below sqsub commands.<br />
<br />
<pre><br />
[username@orc-login1:/work/$USER/samples/mpiblast/test2] vi .ncbirc<br />
[NCBI]<br />
Data=/opt/sharcnet/mpiblast/1.6.0/ncbi/data<br />
[BLAST]<br />
BLASTDB=/work/YourSharcnetUsername/mpiblasttest2<br />
BLASTMAT=/work/YourSharcnetUsername/samples/mpiblast/test2<br />
[mpiBLAST]<br />
Shared=/work/YourSharcnetUsername/mpiblasttest2<br />
Local=/tmp<br />
</pre><br />
<br />
Again partition the database into 8 fragments into <i>/work/$USER/mpiblasttest2</i> where they will be searched for when a job runs in the queue according to the <i>.ncbirc</i> file, first removing previous partitioning if any such files exist. For this example assume $USER=roberpj however be sure to replace with your username!<br />
<br />
<pre><br />
module load mpiblast/1.6.0<br />
mkdir -p /work/$USER/mpiblasttest2<br />
rm -f /work/$USER/mpiblasttest2/*<br />
cd /work/$USER/samples/mpiblast/test2<br />
<br />
[roberpj@orc-login1:/work/roberpj/samples/mpiblast/test2] mpiformatdb -N 8 -i Hs.seq.$USER -o T -p F -n /work/roberpj/mpiblasttest2<br />
Reading input file<br />
Done, read 2348651 lines<br />
Breaking Hs.seq.roberpj into 8 fragments<br />
Executing: formatdb -p F -i Hs.seq.roberpj -N 8 -n /work/roberpj/mpiblasttest2/Hs.seq.roberpj -o T <br />
Created 8 fragments.<br />
<<< Please make sure the formatted database fragments are placed in /work/roberpj/mpiblasttest2/ before executing mpiblast. >>><br />
<br />
[roberpj@orc-login1:/work/roberpj/samples/mpiblast/test1] ls<br />
formatdb.log Hs.seq.roberpj il2ra.in<br />
<br />
[roberpj@orc-login1:/work/roberpj/mpiblasttest2] ls<br />
Hs.seq.roberpj.000.nhr Hs.seq.roberpj.002.nin Hs.seq.roberpj.004.nnd Hs.seq.roberpj.006.nni<br />
Hs.seq.roberpj.000.nin Hs.seq.roberpj.002.nnd Hs.seq.roberpj.004.nni Hs.seq.roberpj.006.nsd<br />
Hs.seq.roberpj.000.nnd Hs.seq.roberpj.002.nni Hs.seq.roberpj.004.nsd Hs.seq.roberpj.006.nsi<br />
Hs.seq.roberpj.000.nni Hs.seq.roberpj.002.nsd Hs.seq.roberpj.004.nsi Hs.seq.roberpj.006.nsq<br />
Hs.seq.roberpj.000.nsd Hs.seq.roberpj.002.nsi Hs.seq.roberpj.004.nsq Hs.seq.roberpj.007.nhr<br />
Hs.seq.roberpj.000.nsi Hs.seq.roberpj.002.nsq Hs.seq.roberpj.005.nhr Hs.seq.roberpj.007.nin<br />
Hs.seq.roberpj.000.nsq Hs.seq.roberpj.003.nhr Hs.seq.roberpj.005.nin Hs.seq.roberpj.007.nnd<br />
Hs.seq.roberpj.001.nhr Hs.seq.roberpj.003.nin Hs.seq.roberpj.005.nnd Hs.seq.roberpj.007.nni<br />
Hs.seq.roberpj.001.nin Hs.seq.roberpj.003.nnd Hs.seq.roberpj.005.nni Hs.seq.roberpj.007.nsd<br />
Hs.seq.roberpj.001.nnd Hs.seq.roberpj.003.nni Hs.seq.roberpj.005.nsd Hs.seq.roberpj.007.nsi<br />
Hs.seq.roberpj.001.nni Hs.seq.roberpj.003.nsd Hs.seq.roberpj.005.nsi Hs.seq.roberpj.007.nsq<br />
Hs.seq.roberpj.001.nsd Hs.seq.roberpj.003.nsi Hs.seq.roberpj.005.nsq Hs.seq.roberpj.dbs<br />
Hs.seq.roberpj.001.nsi Hs.seq.roberpj.003.nsq Hs.seq.roberpj.006.nhr Hs.seq.roberpj.mbf<br />
Hs.seq.roberpj.001.nsq Hs.seq.roberpj.004.nhr Hs.seq.roberpj.006.nin Hs.seq.roberpj.nal<br />
Hs.seq.roberpj.002.nhr Hs.seq.roberpj.004.nin Hs.seq.roberpj.006.nnd <br />
</pre><br />
<br />
Submit a couple of short jobs 15m time limit. If all goes well output results will be written to <i>biobrewA.out</i> and <i>biobrewB.out</i> and the execution time appear in corresponding ofile%J's where %J is the job number as per usual. In these examples we will submit the job from a saw specific directory which can be copied from test2 as follows:<br />
<br />
<pre><br />
[roberpj@saw-login1:~] module load mpiblast/1.6.0<br />
[roberpj@saw-login1:~] cd /work/roberpj/samples/mpiblast<br />
[roberpj@saw-login1:/work/roberpj/samples/mpiblast] cp -a test2 test2.saw<br />
<br />
[roberpj@saw-login1:/work/roberpj/samples/mpiblast/test2.saw] vi .ncbirc<br />
[roberpj@saw-login1:/work/roberpj/samples/mpiblast/test2.saw] cat .ncbirc | grep BLASTMAT<br />
BLASTMAT=/work/roberpj/samples/mpiblast/test2.saw<br />
</pre><br />
<br />
A) Submit job with profile time option choosing n=24 (8 fragments + 8 or greater):<br />
<br />
<pre><br />
[roberpj@saw-login1:/work/roberpj/samples/mpiblast/test2.saw] rm -f oTime*;<br />
sqsub -r 15m -n 24 -q mpi -o ofile%J mpiblast --use-parallel-write --copy-via=mpi<br />
-d Hs.seq.roberpj -i il2ra.in -p blastn -o biobrew.out --time-profile=oTime<br />
</pre><br />
<br />
B) Usage of the debug option again choosing n=16 (8 fragments + 8 or greater):<br />
<br />
<pre><br />
[roberpj@saw-login1:/work/roberpj/samples/mpiblast/test2.saw] rm -f oLog*;<br />
sqsub -r 15m -n 16 -q mpi -o ofile%J mpiblast --use-parallel-write --copy-via=none<br />
-d Hs.seq.$USER -i il2ra.in -p blastn -o biobrew.out --debug=oLog<br />
</pre><br />
<br />
Finally compare <i>/opt/sharcnet/mpiblast/1.6.0/examples/BIOBREW.out</i> computed previously with BLASTN 2.2.15 [Oct-15-2006] with your newly generated <i>biobrew.out</i> output file to verify the results and submit a ticket if there are any problems!<br />
<br />
<u>SUPPORTED PROGRAMS IN MPIBLAST</U><br />
<br />
As described in http://www.mpiblast.org/Docs/FAQ mpiblast supports the standard blast programs http://www.ncbi.nlm.nih.gov/BLAST/blast_program.shtml which are reproduced here for reference:<br />
<br />
<pre><br />
blastp: Compares an amino acid query sequence against a protein sequence database.<br />
blastn: Compares a nucleotide query sequence against a nucleotide sequence database.<br />
blastx: Compares a nucleotide query sequence translated in all reading frames against<br />
a protein sequence database.<br />
tblastn: Compares a protein query sequence against a nucleotide sequence database<br />
dynamically translated in all reading frames.<br />
tblastx: Compares the six-frame translations of a nucleotide query sequence against<br />
the six-frame translations of a nucleotide sequence database.<br />
</pre><br />
<br />
<u>MPIBLAST BINARIES OPTIONS</u><br />
<br />
<pre><br />
[roberpj@orc-login1:/opt/sharcnet/mpiblast/1.6.0/bin] ./mpiblast -help<br />
mpiBLAST requires the following options: -d [database] -i [query file] -p [blast program name]<br />
</pre><br />
<br />
<pre><br />
[roberpj@orc-login1:/opt/sharcnet/mpiblast/1.6.0/bin] ./mpiformatdb --help<br />
Executing: formatdb - <br />
<br />
formatdb 2.2.20 arguments:<br />
<br />
-t Title for database file [String] Optional<br />
-i Input file(s) for formatting [File In] Optional<br />
-l Logfile name: [File Out] Optional<br />
default = formatdb.log<br />
-p Type of file<br />
T - protein <br />
F - nucleotide [T/F] Optional<br />
default = T<br />
-o Parse options<br />
T - True: Parse SeqId and create indexes.<br />
F - False: Do not parse SeqId. Do not create indexes.<br />
[T/F] Optional<br />
default = F<br />
-a Input file is database in ASN.1 format (otherwise FASTA is expected)<br />
T - True, <br />
F - False.<br />
[T/F] Optional<br />
default = F<br />
-b ASN.1 database in binary mode<br />
T - binary, <br />
F - text mode.<br />
[T/F] Optional<br />
default = F<br />
-e Input is a Seq-entry [T/F] Optional<br />
default = F<br />
-n Base name for BLAST files [String] Optional<br />
-v Database volume size in millions of letters [Integer] Optional<br />
default = 4000<br />
-s Create indexes limited only to accessions - sparse [T/F] Optional<br />
default = F<br />
-V Verbose: check for non-unique string ids in the database [T/F] Optional<br />
default = F<br />
-L Create an alias file with this name<br />
use the gifile arg (below) if set to calculate db size<br />
use the BLAST db specified with -i (above) [File Out] Optional<br />
-F Gifile (file containing list of gi's) [File In] Optional<br />
-B Binary Gifile produced from the Gifile specified above [File Out] Optional<br />
-T Taxid file to set the taxonomy ids in ASN.1 deflines [File In] Optional<br />
-N Number of database volumes [Integer] Optional<br />
default = 0<br />
</pre><br />
<br />
=General Notes=<br />
<br />
==Issue1: Jobs Fail to Start==<br />
<br />
If the following error message occurs it maybe necessary to stop all mpiblast jobs you are running on the cluster and clear<br />
out all the files from /tmp then submit your job again. In the case of Example2 above the following steps work to resolve the problem, where the nodes used from the last run job were red[7-14]. Should there be any questions please open a problem ticket.<br />
<br />
<pre><br />
Purging Job Node Subset (redfin)<br />
pdsh -w red[7-14] ls /tmp/Hs* (confirm existence of file from previous runs)<br />
pdsh -w red[7-14] rm -f /tmp/Hs*<br />
pdsh -w red[7-14] ls /tmp/Hs* (confirm all files from previous runs removed)<br />
<br />
Purging Full Cluster (hound)<br />
pdsh -w hnd[1-18] -f 4 rm -f /tmp/Hs*<br />
pdsh -w hnd[1-18] -f 4 ls /tmp/Hs*<br />
</pre><br />
<br />
<pre><br />
[roberpj@red-admin:/work/roberpj/samples/mpiblast/test2.red4] cat ofile489214.red-admin.redfin.sharcnet<br />
8 0.577904 Bailing out with signal 11<br />
--------------------------------------------------------------------------<br />
MPI_ABORT was invoked on rank 8 in communicator MPI_COMM_WORLD<br />
with errorcode 0.<br />
<br />
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.<br />
You may or may not see output from other processes, depending on<br />
exactly when Open MPI kills them.<br />
--------------------------------------------------------------------------<br />
20 0.577719 Bailing out with signal 11<br />
0 0.591128 Bailing out with signal 15<br />
[red14:09194] [[36424,0],0]-[[36424,1],0] mca_oob_tcp_msg_recv: readv failed: Connection reset by peer (104)<br />
1 0.591327 Bailing out with signal 15<br />
etc<br />
</pre><br />
<br />
==Issue2: Jobs Run for a While then Die==<br />
<br />
The solution here is to filter the input sequence file. For reasons yet understood the presence of repeat sections results in many thousands of WARNING and ERROR messages rapidly written to the "sqsub -o ofile" output file presumably as mpiblast ignores sequences before eventually diing after several hours, or possibly days.<br />
<br />
<pre><br />
# cat ofile1635556.saw-admin.saw.sharcnet | grep "WARNING\|ERROR" | wc -l<br />
10560<br />
</pre><br />
<br />
<pre><br />
# cat ofile1635556.saw-admin.saw.sharcnet<br />
Selenocysteine (U) at position 60 replaced by X<br />
Selenocysteine (U) at position 42 replaced by X<br />
[blastall] WARNING: [000.000] NODE_84_length_162_cov_46.259258_1_192_-: SetUpBlastSearch failed.<br />
[blastall] ERROR: [000.000] NODE_84_length_162_cov_46.259258_1_192_-: BLASTSetUpSearch: Unable to calculate Karlin-Altschul params, check query sequence<br />
<<<< snipped out ~10000 similar WARNING and ERROR messages from this example >>>><br />
[blastall] WARNING: [000.000] NODE_65409_length_87_cov_2.367816_1_77_+: SetUpBlastSearch failed.<br />
[blastall] ERROR: [000.000] NODE_65409_length_87_cov_2.367816_1_77_+: BLASTSetUpSearch: Unable to calculate Karlin-Altschul params, check query sequence<br />
Selenocysteine (U) at position 61 replaced by X<br />
Selenocysteine (U) at position 62 replaced by X<br />
Selenocysteine (U) at position 34 replaced by X<br />
Selenocysteine (U) at position 1058 replaced by X<br />
--------------------------------------------------------------------------<br />
mpirun noticed that process rank 52 with PID 30067 on node saw214 exited on signal 9 (Killed).<br />
--------------------------------------------------------------------------<br />
1618 17 651 62909.3 Bailing out with signal 15<br />
14 3 62909.3 Bailing out with signal 15<br />
19 15 62909.3 Bailing out with signal 1536 5 7 62909.312 62909.310 62909.3 Bailing out with signal 1547 21 62909.3 Bailing out with signal 159 62909.3 Bailing out with signal 15<br />
45 62909.3 Bailing out with signal 1525 62909.3 Bailing out with signal 158 62909.3 Bailing out with signal 15<br />
50 62909.3 Bailing out with signal 1522 62909.3 Bailing out with signal 15<br />
11 62909.3 Bailing out with signal 15<br />
48 62909.323 62909.3 Bailing out with signal 15<br />
Bailing out with signal 15<br />
46 62909.3 Bailing out with signal 15<br />
etc<br />
</pre><br />
<br />
=References=<br />
<br />
o MPIBLAST Homepage<br><br />
http://www.mpiblast.org/<br />
<br />
o MPIBLAST Version History<br><br />
http://www.mpiblast.org/Downloads/Version-History<br />
<br />
[[Category:Bioinformatics]]<br />
<br />
<!--checked2016--></div>Edwardhttps://www.sharcnet.ca/help/index.php?title=MPFUN90&diff=17956MPFUN902019-06-06T14:30:29Z<p>Edward: </p>
<hr />
<div>{{Template:CCDelete}}<br />
{{Template:LegacyPage}}<br />
{{Software<br />
|package_name=MPFUN90<br />
|package_description=A Fortran-90 arbitrary precision package<br />
|package_idnumber=16<br />
}}<br />
<br />
=Introduction=<br />
<br />
MPFUN90 is a Fortran-90 arbitrary precision package (i.e. user specifies the desired precision)written by David H. Bailey et. al from the Lawrence Berkeley National Laboratory. If 64 digits of precision are sufficient for an application then users should use QD instead of MPFUN which would run much faster.<br />
<br />
=Versions=<br />
<br />
==20060123==<br />
<br />
<pre><br />
module load intel/12.1.3<br />
module load mpfun90/20060123 <br />
</pre><br />
<br />
==20100825==<br />
<br />
<pre><br />
module load intel/15.0.3<br />
module load mpfun90/intel1503/20100825<br />
</pre><br />
<br />
=Usage=<br />
<br />
==Cluster==<br />
To submit a job to the serial queue use:<br />
<br />
sqsub -r 10m --mpp=2G -o ofile.%J ./a.out<br />
<br />
==Interactive==<br />
<br />
===testmp90===<br />
<br />
This example shows how to compile and run <i>testmp90.f90</i> and <i>testmp90.f</i> using the 2006 and 2010 sharcnet mpfun90 modules:<br />
<br />
====20060123====<br />
<br />
[roberpj@orc-login2:~] module load intel/12.1.3; module load mpfun90/20060123<br />
[roberpj@orc-login2:~/samples/mpfun90/20060123] cp /opt/sharcnet/mpfun90/20060123/f90/src/testmp90.f90 .<br />
[roberpj@orc-login2:~/samples/mpfun90/20060123] ifort testmp90.f90 $CPPFLAGS $LDFLAGS -lmp<br />
[roberpj@orc-login2:~/samples/mpfun90/20060123] ./a.out<br />
Completed Test 1<br />
Completed Test 2<br />
Completed Test 3<br />
.................<br />
Completed Test 62<br />
<br />
====20100825====<br />
<br />
[roberpj@orc-login2:~] module unload intel mkl mpfun90<br />
[roberpj@orc-login2:~] module load intel/15.0.3; module load mpfun90/intel1503/20100825<br />
[roberpj@orc-login2:~/samples/mpfun90/20100825] cp $MPFUN90_ROOT/f90/testmp90.f .<br />
[roberpj@orc-login2:~/samples/mpfun90/20100825] ifort -free testmp90.f $MPFUN90_ROOT/toolkit/mpfun90.o -I$MPFUN90_ROOT/toolkit<br />
[roberpj@orc-login2:~/samples/mpfun90/20100825] ./a.out<br />
Completed Test 1<br />
Completed Test 2<br />
Completed Test 3<br />
.................<br />
Completed Test 62<br />
<br />
o Note that when using the mpfun90/20100825 (or newer sharcnet modules) when the main program contains "use mpmodule" its necessary to include "$MPFUN90_ROOT/toolkit/mpmod90.o" when compiling otherwise error messages such as <i>"undefined reference to mpdefmod_mp_mpinit_"</i> will occur. Also note this is not required with the legacy mpfun90/20060123 module since the custom built static mp library (which is no longer being provided by sharcnet) contains all mpfun objects.<br />
<br />
===factorial_100.f90===<br />
<br />
This example shows how to compile and run the program <i>factorial_100.f90</i> using the 2010 sharcnet mpfun90 module only:<br />
<br />
program factorial_100_pgm<br />
use mpmodule<br />
implicit none<br />
integer, parameter :: Nmx = 100<br />
integer :: AllocateStatus<br />
integer :: J<br />
type(mp_real) zero, one<br />
type(mp_real), dimension (:), allocatable :: fac<br />
call mpinit(800)<br />
call mpsetprec(800)<br />
call mpsetoutputprec(700)<br />
zero='0.'<br />
one ='1.'<br />
allocate(fac(0:Nmx),STAT=AllocateStatus)<br />
IF (AllocateStatus /= 0) STOP "Not enough memory for fac "<br />
fac(0)=one<br />
do J=1,Nmx<br />
fac(J)=fac(J-1)*J<br />
end do<br />
J=4<br />
write(6,1001) J<br />
1001 format(/"fac(",i3.3,") =",$)<br />
call mpwrite(6,fac(J))<br />
J=100<br />
write(6,1001) J<br />
call mpwrite(6,fac(J))<br />
write(6,1005)<br />
1005 format(/"DONE"//)<br />
end<br />
<br />
====Compile====<br />
<br />
[roberpj@orc-login2:~/samples/mpfun90/factorial_100_pgm] ifort factorial_100.f90 \<br />
$MPFUN90_ROOT/toolkit/mpmod90.o $MPFUN90_ROOT/toolkit/mpfun90.o -I$MPFUN90_ROOT/toolkit<br />
<br />
====Execute====<br />
<br />
[roberpj@orc-login2:~/samples/mpfun90/factorial_100_pgm] ./a.out<br />
fac(004) =10 ^ 1 x 2.4,<br />
fac(100) =10 ^ 157 x 9.3326215443944152681699238856266700490715968264381621468592<br />
963895217599993229915608941463976156518286253697920827223758251185210916864,<br />
DONE<br />
<br />
==Graphical==<br />
<br />
There is no graphical usage component for this software package.<br />
<br />
=Notes=<br />
<br />
==OpenMP==<br />
<br />
The Multi Precision packages MPFUN90 (and QD) use "user defined data type" for their variables.<br />
Since the OpenMP reduction clause cannot be used for "user defined data types" to run MPFUN90 or QD<br />
with OpenMP the reduction must be done manually in a PARALLEL OpenMP region.<br />
<br />
Detailed documentation on how to use OpenMP with MPFUN90 and QD are presented in the following<br />
document:<br />
<br />
https://www.sharcnet.ca/help/index.php/OpenMP_Reduction_for_User_Defined_Data_Types<br />
<br />
==Large Arrays==<br />
<br />
Declaring large arrays in an MPFUN program often leads to compile<br />
errors like:<br />
<br />
"relocation truncated to fit r_x86_64_pc32 against .bss"<br />
<br />
Even with the flag '-mcmodel large' in the compile command the error<br />
will persist, because there is a limit to the amount of data you can<br />
have allocated in static arrays for x86_64 architecture. This data<br />
segment is limited to 2GB. The data in COMMON plus any other<br />
statically declared arrays PLUS your code must fit in 2GB.<br />
<br />
In order to overcome this problem use allocatable arrays declared in<br />
a module together with a short subroutine which allocates the arrays.<br />
<br />
We illustrate these concepts by starting with a program 'betax.f90'<br />
which work fine for values of parameter 'Nsz' less than 37. Thus, if<br />
you compile it with<br />
<br />
integer, parameter :: Nsz=36<br />
<br />
it works, but for any higher values it fails with error message:<br />
<br />
"relocation truncated to fit: R_X86_64_PC32 against symbol '...'"<br />
<br />
To overcome this problem, we first copy program betax.f90 into a new file,<br />
say, betay.f90, and write a module called 'mp_data' and a subroutine called<br />
'allocate_arrays' at the top of the new file. We move all the declarations<br />
of large arrays into the module, and declare them as 'allocatable'.<br />
<br />
In the subroutine 'allocate_arrays' we do the allocations and in the main<br />
program we add the statements:<br />
<br />
USE mp_data<br />
...<br />
call allocate_arrays<br />
...<br />
<br />
Below you will find the listings for the files 'betax.f90', 'betay.f90' and<br />
the 'HIST' file containing the commands to compile these files and submit<br />
jobs:<br />
<br />
===betax.f90===<br />
<br />
<pre><br />
program BETAX<br />
use mpmodule<br />
implicit none<br />
! integer, parameter :: Nsz=12 ! compilation fails for 37<br />
integer, parameter :: Nsz=37<br />
INTEGER :: I,J,K,L<br />
type(mp_real) zero, one<br />
type(mp_real) fac(0:Nsz),fci(0:Nsz)<br />
type(mp_real) BETA2(Nsz,Nsz)<br />
type(mp_real) BETA3(Nsz,Nsz,Nsz)<br />
type(mp_real) BETA4(Nsz,Nsz,Nsz,Nsz)<br />
call mpinit(800)<br />
call mpsetprec(800)<br />
call mpsetoutputprec(700)<br />
zero='0.'<br />
one ='1.'<br />
fac(0)=one<br />
fci(0)=one<br />
do j=1,Nsz<br />
fac(j)=fac(j-1)*j<br />
fci(j)=one/fac(j)<br />
end do<br />
I=Nsz/2<br />
J=I-1<br />
K=J-1<br />
L=K-1<br />
write(6,1000) I,J,K,L<br />
1000 format(/"I,J,K,L = ",4i3)<br />
write(6,1001) I<br />
1001 format(/"fac(",i3.3,") =",$)<br />
call mpwrite(6,fac(I))<br />
BETA2(I,J) =fac(I)*fac(J)*fci(I+J)<br />
BETA3(I,J,K) =fac(I)*fac(J)*fac(K)*fci(I+J)*fci(J+K)<br />
BETA4(I,J,K,L) =fac(I)*fac(J)*fac(K)*fac(L)* &<br />
& fci(I+J)*fci(J+K)*fci(K+L)<br />
write(6,1002) I,J<br />
1002 format(/"BETA2(",i3.3,",",i3.3,") =")<br />
call mpwrite(6,BETA2(I,J))<br />
write(6,1003) I,J,K<br />
1003 format(/"BETA3(",i3.3,",",i3.3,",",i3.3,") =")<br />
call mpwrite(6,BETA3(I,J,K))<br />
write(6,1004) I,J,K,L<br />
1004 format(/"BETA4(",i3.3,",",i3.3,",",i3.3,",",i3.3,") =")<br />
call mpwrite(6,BETA4(I,J,K,L))<br />
write(6,1005)<br />
1005 format(/"DONE"//)<br />
stop<br />
end<br />
</pre><br />
<br />
====Nsz=12 (works)====<br />
<br />
<pre><br />
[roberpj@orc-login2:~] ssh orc-dev1<br />
[roberpj@orc129:~] module purge; module load ldwrapper; module load intel/15.0.3; module load mpfun90/intel1503/20100825<br />
[roberpj@orc129:~] cd ~/samples/mpfun90/betax<br />
[roberpj@orc129:~/samples/mpfun90/betax] ifort betax.f90 $MPFUN90_ROOT/toolkit/mpmod90.o $MPFUN90_ROOT/toolkit/mpfun90.o -I$MPFUN90_ROOT/toolkit<br />
[roberpj@orc129:~/samples/mpfun90/betax] ./a.out<br />
<br />
I,J,K,L = 6 5 4 3<br />
<br />
fac(006) =10 ^ 2 x 7.2,<br />
<br />
BETA2(006,005) =<br />
10 ^ -3 x 2.1645021645021645021645021645021645021645021645021645021645<br />
021645021645021645021645021645021645021645021645021645021645021645021645021645<br />
021645021645021645021645021645021645021645021645021645021645021645021645021645<br />
021645021645021645021645021645021645021645021645021645021645021645021645021645<br />
021645021645021645021645021645021645021645021645021645021645021645021645021645<br />
021645021645021645021645021645021645021645021645021645021645021645021645021645<br />
021645021645021645021645021645021645021645021645021645021645021645021645021645<br />
021645021645021645021645021645021645021645021645021645021645021645021645021645<br />
021645021645021645021645021645021645021645021645021645021645021645021645021645<br />
021645021645021645,<br />
<br />
BETA3(006,005,004) =<br />
10 ^ -7 x 1.4315490505966696442886919077395267871458347648823839300029<br />
776220252410728601204791680982157172633363109553585744061934538125014315490505<br />
966696442886919077395267871458347648823839300029776220252410728601204791680982<br />
157172633363109553585744061934538125014315490505966696442886919077395267871458<br />
347648823839300029776220252410728601204791680982157172633363109553585744061934<br />
538125014315490505966696442886919077395267871458347648823839300029776220252410<br />
728601204791680982157172633363109553585744061934538125014315490505966696442886<br />
919077395267871458347648823839300029776220252410728601204791680982157172633363<br />
109553585744061934538125014315490505966696442886919077395267871458347648823839<br />
300029776220252410,<br />
<br />
BETA4(006,005,004,003) =<br />
10 ^ -10 x 1.7042250602341305289151094139756271275545651962885522976225<br />
924071729060391191910466286883520443611146558992363981026112545386921804155364<br />
246067193912998901661033180307556724790284880987828833633822295953815228191645<br />
425205515908463754268742930874450148826566060126150829098674903663565795085069<br />
461486695046785749733595538584200715719990096407329967420670368516173504835636<br />
354910731327964888055591003436808425470556989831366248599808690511638357443346<br />
105477624752001169234729325432273278078266740398259672636089869649960352908198<br />
713187375318894593271010504570595273543119348108010239529513905931139491230194<br />
178039983028645160164434540851774411865114812960617949280080799355175772409332<br />
500035447881252869,<br />
<br />
DONE<br />
</pre><br />
<br />
====Nsz=37 (fails)====<br />
<br />
<pre><br />
[roberpj@orc-login2:~/samples/mpfun90/betax] ssh orc-dev1<br />
[roberpj@orc129:~] module purge; module load ldwrapper; module load intel/15.0.3; module load mpfun90/intel1503/20100825<br />
[roberpj@orc129:~] cd ~/samples/mpfun90/betax<br />
[roberpj@orc129:~/samples/mpfun90/betax] ifort betax.f90 $MPFUN90_ROOT/toolkit/mpmod90.o $MPFUN90_ROOT/toolkit/mpfun90.o -I$MPFUN90_ROOT/toolkit<br />
/opt/sharcnet/mpfun90/20100825/intel1503/toolkit/mpmod90.o: In function `mpdefmod_mp_mpsetprec_':<br />
mpmod90.f:(.text+0x1b): relocation truncated to fit: R_X86_64_PC32 against symbol `mpdefmod_mp_new_mpipl_' defined in COMMON section in /opt/sharcnet/mpfun90/20100825/intel1503/toolkit/mpmod90.o<br />
mpmod90.f:(.text+0xa0): relocation truncated to fit: R_X86_64_PC32 against symbol `mpdefmod_mp_new_mpwds_' defined in COMMON section in /opt/sharcnet/mpfun90/20100825/intel1503/toolkit/mpmod90.o<br />
mpmod90.f:(.text+0xa6): relocation truncated to fit: R_X86_64_PC32 against symbol `mpdefmod_mp_mpnwx_' defined in COMMON section in /opt/sharcnet/mpfun90/20100825/intel1503/toolkit/mpmod90.o<br />
mpmod90.f:(.text+0xcf): relocation truncated to fit: R_X86_64_PC32 against symbol `mpdefmod_mp_mpnwx_' defined in COMMON section in /opt/sharcnet/mpfun90/20100825/intel1503/toolkit/mpmod90.o<br />
/opt/sharcnet/mpfun90/20100825/intel1503/toolkit/mpmod90.o: In function `mpdefmod_mp_mpgetprec_':<br />
mpmod90.f:(.text+0xe6): relocation truncated to fit: R_X86_64_PC32 against symbol `mpdefmod_mp_mpnwx_' defined in COMMON section in /opt/sharcnet/mpfun90/20100825/intel1503/toolkit/mpmod90.o<br />
/opt/sharcnet/mpfun90/20100825/intel1503/toolkit/mpmod90.o: In function `mpdefmod_mp_mpsetprecwords_':<br />
mpmod90.f:(.text+0x109): relocation truncated to fit: R_X86_64_PC32 against symbol `mpdefmod_mp_new_mpwds_' defined in COMMON section in /opt/sharcnet/mpfun90/20100825/intel1503/toolkit/mpmod90.o<br />
mpmod90.f:(.text+0x18c): relocation truncated to fit: R_X86_64_PC32 against symbol `mpdefmod_mp_mpnwx_' defined in COMMON section in /opt/sharcnet/mpfun90/20100825/intel1503/toolkit/mpmod90.o<br />
mpmod90.f:(.text+0x198): relocation truncated to fit: R_X86_64_PC32 against symbol `mpdefmod_mp_mpnwx_' defined in COMMON section in /opt/sharcnet/mpfun90/20100825/intel1503/toolkit/mpmod90.o<br />
/opt/sharcnet/mpfun90/20100825/intel1503/toolkit/mpmod90.o: In function `mpdefmod_mp_mpgetprecwords_':<br />
mpmod90.f:(.text+0x1b2): relocation truncated to fit: R_X86_64_PC32 against symbol `mpdefmod_mp_mpnwx_' defined in COMMON section in /opt/sharcnet/mpfun90/20100825/intel1503/toolkit/mpmod90.o<br />
/opt/sharcnet/mpfun90/20100825/intel1503/toolkit/mpmod90.o: In function `mpdefmod_mp_mpsetoutputprec_':<br />
mpmod90.f:(.text+0x1c9): relocation truncated to fit: R_X86_64_PC32 against symbol `mpdefmod_mp_new_mpipl_' defined in COMMON section in /opt/sharcnet/mpfun90/20100825/intel1503/toolkit/mpmod90.o<br />
mpmod90.f:(.text+0x24c): additional relocation overflows omitted from the output<br />
</pre><br />
<br />
===betay.f90===<br />
<br />
Copy betay.f90 from betax.f90 and modify as follows:<br />
<br />
<pre><br />
module mp_data<br />
use mpmodule<br />
use mpfunmod<br />
implicit none<br />
integer, parameter :: Nsz =48<br />
integer :: AllocateStatus<br />
! type(mp_real) zero, one<br />
!<br />
! type(mp_real) fac(0:Nsz),fci(0:Nsz)<br />
! type(mp_real) BETA2(Nsz,Nsz)<br />
! type(mp_real) BETA3(Nsz,Nsz,Nsz)<br />
! type(mp_real) BETA4(Nsz,Nsz,Nsz,Nsz)<br />
type(mp_real) zero, one<br />
type(mp_real), dimension (:), allocatable :: fac,fci<br />
type(mp_real), dimension (:,:), allocatable :: BETA2<br />
type(mp_real), dimension (:,:,:), allocatable :: BETA3<br />
type(mp_real), dimension (:,:,:,:), allocatable :: BETA4<br />
end module mp_data<br />
subroutine allocate_arrays<br />
USE mp_data<br />
implicit none<br />
allocate(fac(0:Nsz),STAT=AllocateStatus)<br />
IF (AllocateStatus /= 0) STOP "Not enough memory for fac "<br />
allocate(fci(0:Nsz),STAT=AllocateStatus)<br />
IF (AllocateStatus /= 0) STOP "Not enough memory for fci "<br />
allocate(BETA2(Nsz,Nsz),STAT=AllocateStatus)<br />
IF (AllocateStatus /= 0) STOP "Not enough memory for BETA2"<br />
allocate(BETA3(Nsz,Nsz,Nsz),STAT=AllocateStatus)<br />
IF (AllocateStatus /= 0) STOP "Not enough memory for BETA3"<br />
allocate(BETA4(Nsz,Nsz,Nsz,Nsz),STAT=AllocateStatus)<br />
IF (AllocateStatus /= 0) STOP "Not enough memory for BETA4"<br />
print *,"Done with allocations"<br />
return<br />
END<br />
<br />
program BETAY<br />
USE mp_data<br />
implicit none<br />
INTEGER :: I,J,K,L<br />
call mpinit(800)<br />
call mpsetprec(800)<br />
call mpsetoutputprec(700)<br />
zero='0.'<br />
one ='1.'<br />
call allocate_arrays<br />
fac(0)=one<br />
fci(0)=one<br />
do j=1,Nsz<br />
fac(j)=fac(j-1)*j<br />
fci(j)=one/fac(j)<br />
end do<br />
I=Nsz/2<br />
J=I-1<br />
K=J-1<br />
L=K-1<br />
write(6,1000) I,J,K,L<br />
1000 format(/"I,J,K,L = ",4i3)<br />
write(6,1001) I<br />
1001 format(/"fac(",i3.3,") =",$)<br />
call mpwrite(6,fac(I))<br />
BETA2(I,J) =fac(I)*fac(J)*fci(I+J)<br />
BETA3(I,J,K) =fac(I)*fac(J)*fac(K)*fci(I+J)*fci(J+K)<br />
BETA4(I,J,K,L) =fac(I)*fac(J)*fac(K)*fac(L)* &<br />
& fci(I+J)*fci(J+K)*fci(K+L)<br />
write(6,1002) I,J<br />
1002 format(/"BETA2(",i3.3,",",i3.3,") =")<br />
call mpwrite(6,BETA2(I,J))<br />
write(6,1003) I,J,K<br />
1003 format(/"BETA3(",i3.3,",",i3.3,",",i3.3,") =")<br />
call mpwrite(6,BETA3(I,J,K))<br />
write(6,1004) I,J,K,L<br />
1004 format(/"BETA4(",i3.3,",",i3.3,",",i3.3,",",i3.3,") =")<br />
call mpwrite(6,BETA4(I,J,K,L))<br />
write(6,1005)<br />
1005 format(/"DONE"//)<br />
stop<br />
end<br />
</pre><br />
<br />
====Run Interactive====<br />
<br />
<pre><br />
[roberpj@orc-login2:~/samples/mpfun90/betax] ssh orc-dev1<br />
[roberpj@orc129:~] module purge; module load ldwrapper; module load intel/15.0.3; module load mpfun90/intel1503/20100825<br />
[roberpj@orc129:~] cd ~/samples/mpfun90/betax<br />
[roberpj@orc129:~/samples/mpfun90/betay] ifort betay.f90 -mcmodel medium -shared-intel $MPFUN90_ROOT/toolkit/mpmod90.o $MPFUN90_ROOT/toolkit/mpfun90.o -I$MPFUN90_ROOT/toolkit<br />
[roberpj@orc129:~/samples/mpfun90/betay] ./a.out<br />
Done with allocations<br />
<br />
I,J,K,L = 24 23 22 21<br />
<br />
fac(024) =10 ^ 23 x 6.2044840173323943936,<br />
<br />
BETA2(024,023) =<br />
10 ^ -14 x 6.2020112243197155666795543346647015590753323586760996092750<br />
322219057797634187521971369584938063040803657153278083912027225083805532623694<br />
253019502124309149392729129350380917947755526197968886374824625487894350759012<br />
041810328483619840297565282378760232413828873982152911730876431238573292437598<br />
662321548481235961397432074855738879756265023434311148398287293760405994587153<br />
193913843867324131168164219803469468792765957446870530750541069496092327458240<br />
264036867349068267612059922537556261610989123549224099029159406148147186635876<br />
682339231176161254018298581141061530140422181489818261044243997939224351270878<br />
819950204611859530447542248373743937988058087925403948261102164487732977810152<br />
365677688323239749,<br />
<br />
BETA3(024,023,022) =<br />
10 ^ -49 x 5.8275670518268372535457693390802431850085211093628371195552<br />
665374004872800254829089878748169551314149473682136937787061527746060528126189<br />
766201816798822863047397630953393452298952922317427385387709426414471466082503<br />
710520168054747139227447182197529542668023453841447481072362595230792080104688<br />
990118018686670752770192225324208683555045454680275731685429800649187636704457<br />
682218863512356150460390146982645823361128943131689947031597399022247950302452<br />
330798644776819869632357497968409458052013048605853804143958746701285425073948<br />
635138571530969417104049554300550153503104110125327389142504664160736472211972<br />
376564769573352356164657421339817982321431569247028739128974299782191260982443<br />
302487480103263037,<br />
<br />
BETA4(024,023,022,021) =<br />
10 ^ -82 x 4.9281568290835382164029311372592682488032902612395816340121<br />
089418282192742634150386978468714226783345875673304699950206823277002261928996<br />
900244023525594363488176338336719985617691508208486288075631383295355204178844<br />
640181812076780791059441681639997667611161419120033906970044689546074997396047<br />
324233844363558571961437990937177293607133483649131082039969185440365241880338<br />
469876039008981574675398716901015661349359043742699044662693828465598208328182<br />
649393881845546476915223922569122221086448855100004979240419350679011848465439<br />
752707160104858505636971713317247147406784939008134376861978582533070195429961<br />
381129842064351212761591279694768634684602143478580742025860111467351736320562<br />
750016997598905712,<br />
<br />
DONE<br />
</pre><br />
<br />
====Run In Queue====<br />
<br />
The executable requires 4G or more. Setting mpp=8G should work (logout then in again to reset to default environment): <br />
<br />
<pre><br />
sqsub -r 5m --mpp=8.0G -o ofile.%J ./a.out<br />
</pre><br />
<br />
==Math Toolkit==<br />
<br />
===Running Mathinit===<br />
This toolkit provides and extensive example of how the mpfun package can be used. Read the header of files <br />
mathinit.f90 <br />
and <br />
mathtool.f90 <br />
found under <br />
/opt/sharcnet/mpfun90/version/[flavor]/toolkit <br />
for an explanation what the package does. The Makefile in this directory shows how the mpfun libraries were linked to compile this sample program pair. The following steps shows the first initialization step followed by actually running the mathtool program to perform some examples such as performing an integration and displaying the value of pie. Note that the displayed zero cputimes are due to a bug in the mpfun current build which will be fixed in future updates on the clusters.<br />
<br />
To run Mathtool you must first generate following two input files <br />
const.dat <br />
and <br />
quadts.dat <br />
<br />
which can be accomplished by running following command:<br />
<br />
<pre><br />
[roberpj@orc-login1:~/samples/mpfun90/math] cp /opt/sharcnet/mpfun90/20100825/intel1503/toolkit/mathinit .<br />
[roberpj@orc-login1:~/samples/mpfun90/math] chmod 700 mathinit <br />
[roberpj@orc-login1:~/samples/mpfun90/math] ./mathinit<br />
mathinit: start<br />
const complete<br />
file const.dat written<br />
cpu time = 2.57360911369324 <br />
initqts: Tanh-sinh quadrature initialization<br />
0 18632<br />
1000 18632<br />
2000 18632<br />
3000 18632<br />
4000 18632<br />
5000 18632<br />
6000 18632<br />
7000 18632<br />
8000 18632<br />
initqts: Table spaced used = 8177<br />
initqts complete<br />
file quadts.dat written<br />
cpu time = 90.4362528324127 <br />
total cpu time = 93.0098619461060<br />
[roberpj@orc-login1:~/samples/mpfun90/mathinit] ls<br />
const.dat mathinit quadts.dat<br />
</pre><br />
<br />
===Running Mathtool===<br />
Use the files created in the previous section to run mathtool as follows:<br />
<br />
<pre><br />
[roberpj@orc-login1:~/samples/mpfun90/math] cp /opt/sharcnet/mpfun90/20100825/intel1503/toolkit/mathtool .<br />
[roberpj@orc-login1:~/samples/mpfun90/math] chmod 700 mathtool <br />
[roberpj@orc-login1:~/samples/mpfun90/math] ./mathtool <br />
Welcome to the Experimental Mathematician's Toolkit<br />
Initializing...<br />
<br />
Current settings:<br />
Debug level = 2<br />
Primary precision level = 100 digits<br />
Secondary precision level = 200 digits<br />
Primary epsilon = 10^ -100<br />
Secondary epsilon = 10^ -200<br />
PSLQ bound = 100<br />
PSLQ level = 1<br />
Quadrature level = 6<br />
Quadrature type = 3 (Tanh-Sinh)<br />
<br />
Sample expressions (case insensitive):<br />
e + pi + log2 + log10 + catalan Adds these pre-defined constants.<br />
result[1] + result[2] Adds result #1 to result #2.<br />
alpha = arctan[3] - 3*log[2] Defines or sets user variable alpha.<br />
fun1[x,y] = 2*sqrt[x]*erf[y] Defines user function fun1.<br />
clear[nam1] Clears definition of variable or function.<br />
integrate[1/(1+x^2), {x, 0, 1}] Integrates 1/(1+x^2) from x=0 to 1.<br />
sum[1/2^k, {k, 0, infinity}] Sums 1/2^k from k=0 to infinity.<br />
binomial[20,10]*factorial[10] Evaluates binomial coeff and factorial.<br />
zeta[3] + zetaz[1,1,2] Evaluates zeta and multi-zeta functions.<br />
table[x^k, {k, 1, 4}] Forms the list [x^1, x^2, x^3, x^4].<br />
pslq[table[x^k, {k, 0, n}]] Finds coeffs of degree-n poly for x.<br />
polyroot[1,-1,-1,{0.618}] Finds real root of 1-x-x^2=0 near 0.618.<br />
polyroot[1,2,3,{-0.33, 0.47}] Complex root of 1+2x+3x^2 near -0.33+0.47i.<br />
digits = 200 Sets working precision to 200 digits.<br />
epsilon = -190 Sets epsilon level to 10^(-190).<br />
eformat[190,180] Display using E format with 180 digits.<br />
fformat[60,50] Display using F format with 50 digits.<br />
input file.dat Inputs commands from file file.dat.<br />
output file.dat Outputs user vars and funs to file.dat.<br />
help polyroot Displays a brief explanation of polyroot.<br />
functions Displays a list of all defined functions.<br />
variables Displays a list of all defined variables.<br />
prompt Displays this message.<br />
exit Exits this program.<br />
Expressions may be continued on next line by typing \ at end of line.<br />
<br />
integrate[1/(1+x^2), {x, 0, 1}] (hit enter)<br />
quadts: Iteration 1 of 6; est error = 10^ 0; approx value =<br />
7.853157298876894309110693190063938396203990510665917998384243e-1<br />
quadts: Iteration 2 of 6; est error = 10^ 0; approx value =<br />
7.853981605125528448099840689980111720633592558858121242731082e-1<br />
quadts: Iteration 3 of 6; est error = 10^ -17; approx value =<br />
7.853981633974483082028533923930157568609007576776270726821597e-1<br />
quadts: Iteration 4 of 6; est error = 10^ -36; approx value =<br />
7.853981633974483096156608458198757190586422063114153956005278e-1<br />
quadts: Iteration 5 of 6; est error = 10^ -71; approx value =<br />
7.853981633974483096156608458198757210492923498437764552437361e-1<br />
quadts: Iteration 6 of 6; est error = 10^ -101; approx value =<br />
7.853981633974483096156608458198757210492923498437764552437361e-1<br />
Result[ 1] =<br />
7.85398163397448309615660845819875721049292349843776455243736148076954e-1<br />
CPU time = 0.1730<br />
<br />
functions (hit enter)<br />
Abs Arccos Arcsin Arctan <br />
Arctan2 Bessel Besselexp Binomial <br />
Cos Erf Exp Factorial <br />
Gamma Integrate Log Max <br />
Min Polyroot Pslq Result <br />
Sin Sqrt Sum Table <br />
Tan Zeta Zetap Zetaz <br />
<br />
variables (hit enter)<br />
E Log2 Log10 Pi <br />
Catalan Eulergamma Infinity Arg1 <br />
Arg2 Arg3 Arg4 Arg5 <br />
Arg6 Arg7 Arg8 Arg9 <br />
Debug Digits Digits2 Epsilon <br />
Epsilon2 Pslqbound Pslqlevel Quadlevel <br />
Quadtype x <br />
<br />
Pi (hit enter) <br />
Result[ 2] =<br />
3.14159265358979323846264338327950288419716939937510582097494459230781e0<br />
CPU time = 0.0000<br />
<br />
sqrt[1600] (hit enter) <br />
Result[ 1] =<br />
4.00000000000000000000000000000000000000000000000000000000000000000000e1<br />
CPU time = 0.0000<br />
<br />
help Binomial (hit enter) <br />
Binomial[m,n] computes the binomial coefficient of integers (m,n). <br />
<br />
Binomial[3,2] (hit enter) <br />
Result[ 2] =<br />
3.00000000000000000000000000000000000000000000000000000000000000000000e0<br />
CPU time = 0.0000<br />
<br />
hypo[x,y] = sqrt[x^2+y^2] (hit enter) <br />
parse: new function name = hypo <br />
CPU time = 0.0000<br />
<br />
z=hypo[3,4] (hit enter) <br />
parse: new variable name = z <br />
CPU time = 0.0000<br />
<br />
z (hit enter) <br />
Result[ 3] =<br />
5.00000000000000000000000000000000000000000000000000000000000000000000e0<br />
CPU time = 0.0000<br />
</pre><br />
<br />
=References=<br />
<br />
o High-Precision Software Directory<br><br />
http://crd.lbl.gov/~dhbailey/mpdist/index.html</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=MKL&diff=17955MKL2019-06-06T14:29:19Z<p>Edward: </p>
<hr />
<div>{{Template:CCMissing}}<br />
{{Template:GrahamUpdate}}<br />
{{Software<br />
|package_name=MKL<br />
|package_description=Intel Math Kernel Library<br />
|package_idnumber=89<br />
}}<br />
<br />
===Introduction===<br />
MKL (Intel's Math Kernel Library) is a computing math library of highly optimized, extensively threaded<br />
routines for applications that require maximum performance. Intel MKL provides comprehensive functionality<br />
support in these major areas of computation: BLAS (level 1, 2, and 3), LAPACK linear algebra routines,<br />
ScaLAPACK, BLACS, PBLAS, FFT. <br />
<br />
===Version Selection===<br />
<br />
Presently the version of MKL 10.3.9 is the default and its module (mkl/10.3.9) is loaded automatically when you login. Version 11.1.4 is also available (mkl/11.1.4)<br />
<br />
Later versions of MKL are bundled with the Intel compiler. You can still load them separately (for use with different compiler) by loading modules: intel/mkl/15.0.3, intel/mkl/15.0.6, intel/mkl/16.0.3.<br />
<br />
More details about module usage can be found here:<br />
https://www.sharcnet.ca/help/index.php/Configuring_your_software_environment_with_Modules<br />
<br />
===Job Submission ===<br />
Jobs requiring MKL should have a flag in the compiling command indicating the required libraries.<br />
See the next section which illustrates this procedure.<br />
<br />
===Examples of Job Compilation===<br />
We are assuming that the following modules are currently loaded: mkl/10.3.9 intel/12.1.3<br />
<br />
==== Fortran DGEMM example====<br />
Use following command to compile file test_dgemm.f90:<br />
<br />
<source lang="bash"><br />
$FC test_dgemm.f90 -L$MKLROOT/lib/intel64 -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lpthread -lm -openmp<br />
</source><br />
<br />
where<br />
<br />
<source lang="fortran"><br />
! file name = test_dgemm.f90<br />
<br />
program mainp1<br />
implicit none<br />
integer, parameter :: HEIGHT=4<br />
integer, parameter :: WIDTH=3<br />
integer, parameter :: K=1<br />
integer :: i, j<br />
double precision :: ColumnVector(HEIGHT,K)<br />
double precision :: RowVector(K,WIDTH)<br />
double precision :: Result(HEIGHT,WIDTH)<br />
double precision :: ALPHA, BETA<br />
character*1 :: NoTrans<br />
<br />
ALPHA = 1.0e0<br />
BETA = 0.0e0<br />
<br />
do i=1,HEIGHT<br />
ColumnVector(i,K) = i<br />
enddo<br />
<br />
do j=1,WIDTH<br />
RowVector(K,j) = j<br />
enddo<br />
<br />
call PrintMatrix(ColumnVector, HEIGHT,K)<br />
call PrintMatrix(RowVector, K, WIDTH)<br />
<br />
! To do the calculation, we will use the BLAS function dgemm. <br />
! This function calculates: C = ALPHA*A*B + BETA*C <br />
<br />
NoTrans = 'N'<br />
<br />
call dgemm(NoTrans,NoTrans,HEIGHT,WIDTH,1,ALPHA, &<br />
& ColumnVector,HEIGHT,RowVector,1,BETA,Result,HEIGHT)<br />
<br />
call PrintMatrix(Result, HEIGHT, WIDTH)<br />
stop<br />
end<br />
<br />
subroutine PrintMatrix(pMatrix,nRows,nCols)<br />
implicit none<br />
integer :: i, j, nRows, nCols<br />
double precision :: pMatrix(nRows,nCols)<br />
<br />
do i=1,nRows<br />
do j=1,nCols<br />
print *,i,j,pMatrix(i,j)<br />
enddo<br />
enddo<br />
<br />
print *," "<br />
return<br />
end<br />
<br />
</source><br />
<br />
==== Fortran ZDOTC example====<br />
Use following command to compile:<br />
<br />
<source lang="bash"><br />
$FC fortran_zdotc.f90 -L$MKLROOT/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -openmp -lpthread<br />
</source><br />
<br />
where <br />
<br />
<source lang="fortran"><br />
! file name = fortran_zdotc.f90<br />
<br />
program fortran_zdotc<br />
implicit none<br />
<br />
integer, parameter :: NN=5<br />
integer :: n, inca, incb<br />
integer :: i<br />
<br />
DOUBLE COMPLEX :: ZX(0:NN-1),ZY(0:NN-1),ZDPXY,ZDPYX,ZDOTC<br />
REAL*8 :: Di, Dn<br />
<br />
inca = 1<br />
incb = 1<br />
<br />
n = NN<br />
Dn = DBLE(n)<br />
<br />
print *,""<br />
<br />
DO i=0,n-1<br />
Di = i<br />
ZX(i) = CMPLX(Di,2.0D0*Di)<br />
ZY(i) = CMPLX(Dn-Di,2.0D0*Di)<br />
write(6,1001) ZX(i),ZY(i)<br />
1001 format("(",f6.2,",",f6.2,") (",f6.2,",",f6.2,")")<br />
END DO<br />
<br />
ZDPXY = ZDOTC(n,ZX,inca,ZY,incb)<br />
ZDPYX = ZDOTC(n,ZY,incb,ZX,inca)<br />
<br />
1002 format("(",f6.2,",",f6.2,")")<br />
print *,""<br />
print *,"<ZX,ZY>"<br />
write(6,1002) ZDPXY<br />
<br />
print *,""<br />
print *,"<ZY,ZX>"<br />
write(6,1002) ZDPYX<br />
print *,""<br />
<br />
print *,"Job completed successfully"<br />
print *,""<br />
<br />
end program fortran_zdotc<br />
</source><br />
<br />
<br />
==== C ZDOTC example====<br />
Use following command to compile: <br />
<br />
<source lang="bash"><br />
$CC main_zdotc.c -L$MKLROOT/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -openmp -lpthread<br />
</source><br />
<br />
where<br />
<br />
<source lang="c"><br />
/* file name = main_zdotc.c */<br />
<br />
/* The following example illustrates a call from a C program to the <br />
* complex BLAS Level 1 function zdotc(). This function computes <br />
* the dot product of two double-precision complex vectors. <br />
<br />
DOT_PRODUCT = <ZX,ZY> = SUM[i=0,i=n-1] { DCONJG(ZX(I)) * ZY(I) }<br />
------------- * ----<br />
Note that <ZX,ZY> = DCONJG(<ZY,ZX>)<br />
<br />
See: http://www2.math.umd.edu/~hking/Hermitian.pdf<br />
<br />
* In this example, the complex dot product is returned in the structure c. <br />
*/<br />
<br />
#include "mkl.h"<br />
#define N 5 <br />
<br />
void zdotc();<br />
<br />
int main() {<br />
int n, inca = 1, incb = 1, i;<br />
<br />
int DEBUG=1;<br />
<br />
/* typedef struct {...} MKL_Complex16; defined in "mkl.h" */<br />
<br />
MKL_Complex16 a[N], b[N], c, d;<br />
n = N;<br />
<br />
printf("\n");<br />
<br />
for ( i = 0; i < n; i++ ){<br />
a[i].real = (double)i;<br />
a[i].imag = (double)i * 2.0;<br />
<br />
b[i].real = (double)(n - i);<br />
b[i].imag = (double)i * 3.0;<br />
<br />
printf(" ( %6.2f, %6.2f) ( %6.2f, %6.2f) \n",a[i].real,a[i].imag,b[i].real,b[i].imag);<br />
}<br />
<br />
zdotc( &c, &n, a, &inca, b, &incb );<br />
zdotc( &d, &n, b, &incb, a, &inca ); <br />
<br />
printf("\n");<br />
printf("The complex dot product a|b is: ( %6.2f, %6.2f) \n", c.real, c.imag );<br />
printf("The complex dot product b|a is: ( %6.2f, %6.2f) \n", d.real, d.imag );<br />
printf("\n");<br />
printf("Job completed successfully\n");<br />
printf("\n");<br />
<br />
}<br />
</source><br />
<br />
<!-- <U>COMPILING WITH MKL - USING LEGACY COMPILE SCRIPT</U><br />
<br />
On the remaining centos5 clusters (see https://www.sharcnet.ca/my/software/show/129) the compile script can be used for codes needing to link with the MKL blas and lapack libraries (where the program extension <b>xyz</b> can be any of c/cc, cxx/CC/c++ or f77/f90/f95) as follows:<br />
<br />
<pre>compile program.xyz -llapack</pre><br />
<br />
To demonstrate this approach, consider the following example where the result from a.out can be compared with the expect vendor provided solution:<br />
<br />
<pre><br />
ssh kraken.sharcnet.ca<br />
ln -s /opt/sharcnet/acml/4.3.0/ifort-64bit/ifort64/examples/dgetrf_example.f<br />
compile dgetrf_example.f -llapack<br />
./a.out<br />
cat /opt/sharcnet/acml/4.3.0/ifort-64bit/ifort64/examples/dgetrf_example.expected <br />
</pre><br />
--><br />
<br />
===General Notes===<br />
<br />
====COMPILING WITH MKL - USING ONLINE LINK ADVISOR====<br />
<br />
More generally (on any SHARCNET cluster) the <i>Intel Math Kernel Library Link Line Advisor</i> http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/ can be used to generate linker options for more complex linking situations with MKL. For instance, to determine the link arguments for the MKL version 10.3, Linux Operating System, Intel Compiler, Intel (R) 64 architecture, Static Linking, 32 bit Integers, sequential version of MKL, ScaLAPACK library, and OpenMPI, the MKL Link Line Advisor would return the following linking recomendation (the syntax below would go into your [[Make utility and makefiles|Makefile]]):<br />
<br />
<pre><br />
$(MKLROOT)/lib/intel64/libmkl_scalapack_lp64.a -Wl,--start-group $(MKLROOT)/lib/intel64/libmkl_intel_lp64.a $(MKLROOT)/lib/intel64/libmkl_core.a $(MKLROOT)/lib/intel64/libmkl_sequential.a -Wl,--end-group $(MKLROOT)/lib/intel64/libmkl_blacs_openmpi_lp64.a -lpthread -lm<br />
</pre><br />
<br />
and the following recommendation for locating the include files<br />
<br />
<pre><br />
-I$(MKLROOT)/include<br />
</pre><br />
<br />
The $(MKLROOT) will already be set for you by the MKL module which is currently loaded. The above case is the one applicable on most SHARCNET clusters.<br />
<br />
====Using The ILP64 Vs LP64 Variants Of MKL====<br />
<br />
You should use Intel MKL ilp64 in following cases.<br><br />
1. If you are using huge data arrays (indexing exceeds 2^32-1)<br><br />
2. If you enable FORTRAN code with the /4I8 compiler option<br><br />
<br />
The ilp64 version of the MKL libraries defines integers as 64 bit. This implies codes should be compiled with -i8 OR internally be modfied to use the integer*8 type. Otherwise the standard lp64 version of MKL should be used which assumes integers are standard 32 bit.<br />
<br />
====Support for Third-Party Interfaces====<br />
<br />
<b>o GMP Functions</b><br><br />
The Intel MKL implementation of GMP arithmetic functions includes arbitrary precision arithmetic operations on integer numbers. The interfaces of such functions fully match the GNU Multiple Precision (GMP) Arithmetic Library. For specifications of these functions, please see this <a href='http://www.intel.com/software/products/mkl/docs/gnump/WebHelp/'>link</a>. If you currently use the GMP library, you need to modify INCLUDE statements in your programs to mkl_gmp.h.<br />
<br />
==== FFTW Interface Support====<br />
Intel MKL provides interface wrappers for the 2.x and 3.x FFTW (www.fftw.org) superstructure are located in the same directory on all clusters. Using hound and version 11.0.083 of the intel compiler as an example, the wrappers and corresponding fftw wrapper header files are located in the following locations:<br />
<pre><br />
[roberpj@hnd50:/opt/sharcnet/intel/11.0.083/ifc/mkl/interfaces] ls<br />
blas95 fftw2xc fftw2x_cdft fftw2xf fftw3xc fftw3xf lapack95<br />
<br />
[roberpj@hnd50:/opt/sharcnet/intel/11.0.083/ifc/mkl/include/fftw] ls<br />
fftw3.f fftw_f77.i fftw_mpi.h rfftw.h rfftw_threads.h<br />
fftw3.h fftw.h fftw_threads.h rfftw_mpi.h<br />
</pre><br />
<br />
The wrappers can be used for calling the Intel ~<i>equivilent</i>~ MKL Fourier transform functions instead of FFTW for programs that currently use FFTW without changing the program source code. Referring to the online document <a href='http://www.intel.com/software/products/mkl/docs/fftw_mkl_user_notes_2.htm'>FFTW to Intel® Math Kernel Library Wrappers Technical User Notes</a> its mentions that "FFTW2MKL wrappers are delivered as the source code that must be compiled by the user to build the wrapper library." By popular demand these wrapper have been precompiled for immediate use and located in two directories for each intel module (at present 11.0.083 and 11.1.069) as follows:<br />
<br />
<pre><br />
[roberpj@hnd50:/opt/sharcnet/intel/11.0.083/mkl/lib/em64t/interfaces] tree<br />
.<br />
|-- ilp64<br />
| |-- libfftw2xc_intel.a<br />
| |-- libfftw2xf_intel.a<br />
| |-- libfftw3xc_intel.a<br />
| |-- libfftw3xf_intel.a<br />
| |-- libmkl_blas95.a<br />
| |-- libmkl_lapack95.a<br />
| |-- mkl77_lapack.mod<br />
| |-- mkl77_lapack1.mod<br />
| |-- mkl95_blas.mod<br />
| |-- mkl95_lapack.mod<br />
| `-- mkl95_precision.mod<br />
`-- lp64<br />
|-- libfftw2xc_intel.a<br />
|-- libfftw2xf_intel.a<br />
|-- libfftw3xc_intel.a<br />
|-- libfftw3xf_intel.a<br />
|-- libmkl_blas95.a<br />
|-- libmkl_lapack95.a<br />
|-- mkl77_lapack.mod<br />
|-- mkl77_lapack1.mod<br />
|-- mkl95_blas.mod<br />
|-- mkl95_lapack.mod<br />
`-- mkl95_precision.mod<br />
</pre><br />
<br />
====Introduction to Using the MKL FFT====<br />
<br />
Intel markets two implementations of the FFT. The first being from <a href='http://software.intel.com/en-us/intel-mkl/'>MKL</a> and the other from <a href='http://software.intel.com/en-us/intel-ipp/'>IPP</a> whose differences are described <a href='http://software.intel.com/en-us/articles/mkl-ipp-choosing-an-fft/'>here</a>. Only the MKL version is installed on SHARCNET.<br />
<br />
The main FFT Computation Functions provided with MKL are DftiComputeForward and DftiComputeForward which compute the forward and backward FFT respectively. These functions along with Descriptor Manipulation Functions, Descriptor Configuration Functions and Status Checking Functions are provided in the <a href='http://www.intel.com/software/products/mkl/docs/webhelp/fft/fft_DFTF.html'>Table “FFT Functions in Intel MKL”</a>. Intel describes howto use these functions in their <a href='http://www.intel.com/software/products/mkl/docs/webhelp/appendices/mkl_appC_FFT.html'>Fourier Transform Functions Code Examples</a> document which also covers multi-threading aspects.<br />
<br />
The simplest way to explain howto MKL FFT is by compiling and running a example problem of which there are several located under <i>/opt/sharcnet/intel/11.0.083/ifc/mkl/examples</i> where the fortran samples are contained in the <i>dftf</i> sub-directory while the c program samples are contained in the <i>dftc</i> sub-directory. The problem demonstrated here is from the source <i>complex_2d_double_ex1.f90</i> which provides a MKL DFTI interface example program (Fortran-interface) to demonstrate Forward-Backward 2D complex transform for double precision data inplace. Steps to run this program are as follows:<br />
<br />
1) Copy the example directory to a test directory in your account with:<br><br />
<pre><br />
cp -r /opt/sharcnet/intel/11.0.083/ifc/mkl/examples/dftf /scratch/myusername/dftfdemo<br />
cd /scratch/myusername/dftfdemo<br />
</pre><br />
<br />
2) Next compile the example program. In this case the machine used is Silky ie) ia64 based.<br />
<pre><br />
make lib64 function=complex_2d_double_ex1 compiler=intel interface=ia64 [threading=parallel 2>&1 | tee myMake.out<br />
</pre><br />
<br />
3) The built output appears as follows, where you will note the first step is to compile <b>mkl_dfti.f90</b> into a module which is then used in the program on line 42 where the statement <i>Use MKL_DFTI</i> can be seen vizzz:<br />
<br />
<pre><br />
make lib64 function=complex_2d_double_ex1 compiler=intel interface=ia64 [threading=parallel 2>&1 | tee myMake.out<br />
rm -fr *.o *.mod<br />
make mkl_dfti.o dfti_example_support.o dfti_example_status_print.o complex_2d_double_ex1.res _IA=64 EXT=a RES_EXT=lib<br />
make[1]: Entering directory `/home/roberpj/samples/fft-intel/fft/dftf'<br />
mkdir -p ./_results/intel_ia64_parallel_64_lib<br />
ifort -w -c /opt/sharcnet/intel/11.0.083/ifc/mkl/include/mkl_dfti.f90 -o mkl_dfti.o<br />
mkdir -p ./_results/intel_ia64_parallel_64_lib<br />
ifort -w -c source/dfti_example_support.f90 -o dfti_example_support.o<br />
mkdir -p ./_results/intel_ia64_parallel_64_lib<br />
ifort -w -c source/dfti_example_status_print.f90 -o dfti_example_status_print.o<br />
mkdir -p ./_results/intel_ia64_parallel_64_lib<br />
ifort -w mkl_dfti.o dfti_example_support.o dfti_example_status_print.o source/complex_2d_double_ex1.f90 -L"/opt/sharcnet/intel/11.0.083/ifc/mkl/lib/64"<br />
"/opt/sharcnet/intel/11.0.083/ifc/mkl/lib/64"/libmkl_intel_lp64.a -Wl,--start-group "/opt/sharcnet/intel/11.0.083/ifc/mkl/lib/64"/libmkl_intel_thread.a<br />
"/opt/sharcnet/intel/11.0.083/ifc/mkl/lib/64"/libmkl_core.a -Wl,--end-group -L"/opt/sharcnet/intel/11.0.083/ifc/mkl/lib/64" -liomp5 -lpthread -o<br />
_results/intel_ia64_parallel_64_lib/complex_2d_double_ex1.out<br />
export<br />
LD_LIBRARY_PATH="/opt/sharcnet/intel/11.0.083/ifc/mkl/lib/64":/opt/sharcnet/lsf/6.2/linux2.6-glibc2.4-ia64/lib:/opt/sharcnet/lsf/6.2/linux2.6-glibc2.4-ia64/lib:/opt/sharcnet/intel/11$<br />
_results/intel_ia64_parallel_64_lib/complex_2d_double_ex1.out <data/complex_2d_double_ex1.d >_results/intel_ia64_parallel_64_lib/complex_2d_double_ex1.res<br />
make[1]: Leaving directory `/home/roberpj/samples/fft-intel/fft/dftf'<br />
</pre><br />
<br />
4) Since the program gets run automatically by the makefile, the output data can be examined by running more (or less) on the results file called <i>complex_2d_double_ex1.res</i> which gets created.<br />
<br />
<pre><br />
cat _results/intel_ia64_parallel_64_lib/complex_2d_double_ex1.res<br />
COMPLEX_2D_DOUBLE_EX1<br />
Forward-Backward 2D complex transform for double precision data<br />
<br />
Configuration parameters:<br />
<br />
DFTI_FORWARD_DOMAIN = DFTI_COMPLEX<br />
DFTI_PRECISION = DFTI_DOUBLE <br />
DFTI_DIMENSION = 2<br />
DFTI_LENGTHS = { 5, 3}<br />
DFTI_PLACEMENT = DFTI_INPLACE<br />
DFTI_INPUT_STRIDES = { 0, 1, 15}<br />
DFTI_FORWARD_SCALE = 1.0 <br />
DFTI_BACKWARD_SCALE = 1.0/real(m*n)<br />
<br />
<br />
INPUT vector X (2D columns)<br />
( 0.729, 0.486) ( -0.865, -0.577) ( -0.278, -0.186)<br />
( 0.787, 0.525) ( 0.839, 0.559) ( -0.586, -0.391)<br />
( 0.122, 0.081) ( -0.741, -0.494) ( -0.794, -0.529)<br />
( -0.655, -0.437) ( 0.580, 0.387) ( -0.866, -0.577)<br />
( -0.830, -0.554) ( -0.371, -0.247) ( -0.791, -0.527)<br />
<br />
Compute DftiComputeForward<br />
<br />
Forward OUTPUT vector X (2D columns)<br />
( -3.720, -2.480) ( 3.681, -0.995) ( 0.497, 3.780)<br />
( 2.932, -1.810) ( 1.422, 0.044) ( 3.078, -1.928)<br />
( 1.115, -2.479) ( -2.040, 1.814) ( 3.144, 1.228)<br />
( -1.859, 1.982) ( 2.343, 2.430) ( 0.890, -2.581)<br />
( -0.543, 3.403) ( -0.596, 3.583) ( 0.588, 1.295)<br />
<br />
Compute DftiComputeBackward<br />
<br />
Backward OUTPUT vector X (2D columns)<br />
( 0.729, 0.486) ( -0.865, -0.577) ( -0.278, -0.186)<br />
( 0.787, 0.525) ( 0.839, 0.559) ( -0.586, -0.391)<br />
( 0.122, 0.081) ( -0.741, -0.494) ( -0.794, -0.529)<br />
( -0.655, -0.437) ( 0.580, 0.387) ( -0.866, -0.577)<br />
( -0.830, -0.554) ( -0.371, -0.247) ( -0.791, -0.527)<br />
<br />
ACCURACY = 0.248253E-15<br />
TEST PASSED<br />
</pre><br />
<br />
====Intel MKL Examples====<br />
<br />
The Intel compiler came with many mkl examples which can be copied to your work directory to experiment with by doing the following:<br />
<br />
<pre>cp -r /opt/sharcnet/intel/current/ifc/mkl/examples /work/$USER</pre><br />
<br />
Then each example can be compiled by going into any example directory and executing:<br />
<br />
<pre>make soem64t</pre><br />
<br />
==References==<br />
<br />
o Intel Math Kernel Library Documentation (current release)<br><br />
https://www.sharcnet.ca/Software/Intel/IntelIFC/mkl/mkl_documentation.htm<br />
<br />
o Intel Math Kernel Library Documentation Home (latest release)<br><br />
http://software.intel.com/en-us/articles/intel-math-kernel-library-documentation/<br />
<br />
o Version of Intel IPP, Intel MKL and Intel TBB Installed With The Intel® Compiler<br><br />
http://software.intel.com/en-us/articles/which-version-of-ipp--mkl--tbb-is-installed-with-intel-compiler-professional-edition/<br />
<br />
o Known Limitiation In Mkl 10.1 For Linux<br><br />
http://software.intel.com/en-us/articles/known-limitations-in-mkl-101-for-linux/<br />
<br />
o MKL - BLAS, CBLAS and LAPACK Compiling/Linking Functions &Fortran and C/C++ Calls<br><br />
http://software.intel.com/en-us/articles/intel-math-kernel-library-intel-mkl-blas-cblas-and-lapack-compilinglinking-functions-fortran-and-cc-calls/<br />
<br />
o Using the ILP64 Interface vs. LP64 Interface<br><br />
https://software.intel.com/en-us/node/528682<br />
<br />
o Use of Intel MKL data types in C/C++ applications<br><br />
http://software.intel.com/en-us/articles/use-of-intel-mkl-data-types-in-cc-applications<br />
<br />
o Working with the Intel® Math Kernel Library Cluster Software<br><br />
https://software.intel.com/en-us/node/528420</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=LSDYNA&diff=17951LSDYNA2019-06-06T14:25:42Z<p>Edward: </p>
<hr />
<div>{{Template:CCMissing}}<br />
{{Software<br />
|package_name=LSDYNA<br />
|package_description=Suite of programs for transient dynamic finite element program<br />
|package_idnumber=104<br />
}}<br />
<br />
=Introduction=<br />
<br />
Before a research group can use LSDYNA on sharcnet, a license must be purchased directly from LSTC for the sharcnet license server. Alternately, if a research group resides at a institution that has sharcnet computing hardware (mac, uwo, guelph, waterloo) it maybe possible to use a pre-existing site license hosted on an accessible institutional license server. To access and use this software you must open a ticket and request to be added to the sharcnet <b><i>lsdyna</i></b> group.<br />
<br />
=Version Selection=<br />
<br />
==Graham Cluster==<br />
<br />
===License===<br />
<br />
Create and customize this file for your license server, where XXXXX should be replaced with your port number and Y with your server number:<br />
<br />
[roberpj@gra1:~] cat .licenses/ls-dyna.lic <br />
#LICENSE_TYPE: network<br />
#LICENSE_SERVER: XXXXX@licenseY.uwo.sharcnet<br />
<br />
===Module===<br />
<br />
The modules are loaded automatically in the testomp.sh and testmpi.sh script shown below in the example section:<br />
<br />
For single node smp jobs the available versions can be found by doing:<br />
<br />
[roberpj@gra-login2:~] module spider ls-dyna<br />
ls-dyna/7.1.2<br />
ls-dyna/7.1.3<br />
ls-dyna/8.1<br />
ls-dyna/9.1<br />
ls-dyna/9.2<br />
ls-dyna/10.0<br />
<br />
To load any version do for example:<br />
<br />
module load ls-dyna/9.1<br />
<br />
For multi node mpp jobs the available versions can be found by doing:<br />
<br />
[roberpj@gra-login2:~] module spider ls-dyna-mpi<br />
ls-dyna-mpi/7.1.2<br />
ls-dyna-mpi/7.1.3<br />
ls-dyna-mpi/8.1<br />
ls-dyna-mpi/9.1<br />
ls-dyna-mpi/9.2<br />
ls-dyna-mpi/10.0<br />
<br />
To load the 7.1.X modules do the following:<br />
module load openmpi/1.6.5 ls-dyna-mpi/7.1.3<br />
<br />
To load the 8.X or 9.X modules do:<br />
module load openmpi/1.8.8 ls-dyna-mpi/9.2<br />
<br />
To load the 10.X modules do:<br />
module load openmpi/1.10.7 ls-dyna-mpi/10.0<br />
<br />
==Legacy Clusters==<br />
<br />
===License===<br />
<br />
Once a license configuration is established, the research group will be given a 5 digit port number. The value should then be inserted into the appropriate departmental export statement before loading the module file as follows:<br />
<br />
o UofT Mechanical Engineering Dept<br />
<b><font color="purple">export</font></b> <font color="green">LSTC_LICENSE_SERVER</font>=<b><font color="red">Port</font></b>@license1.uwo.sharcnet<br />
<br />
o McGill Mechanical Engineering Department<br />
<b><font color="purple">export</font></b> <font color="green">LSTC_LICENSE_SERVER</font>=<b><font color="red">Port</font></b>@license2.uwo.sharcnet<br />
<br />
o UW Mechanical and Mechatronics Engineering Dept<br />
<b><font color="purple">export</font></b> <font color="green">LSTC_LICENSE_SERVER</font>=<b><font color="red">Port</font></b>@license3.uwo.sharcnet<br />
<br />
o Laurentian University Bharti School of Engineering<br />
<b><font color="purple">export</font></b> <font color="green">LSTC_LICENSE_SERVER</font>=<b><font color="red">Port</font></b>@license4.uwo.sharcnet<br />
<br />
o Fictitious Example:<br />
<br />
<b><font color="purple">export</font></b> <font color="green">LSTC_LICENSE_SERVER</font>=<b><font color="red">12345</font></b>@license1.uwo.sharcnet<br />
<br />
===Module===<br />
<br />
The next step is load the desired sharcnet lsdyna module version. Check which modules are available by running the <i>module avail</i> command then load one the modules as shown:<br />
<br />
<source lang="bash"><br />
[roberpj@orc-login2:~] module avail lsdyna<br />
------------------------------------ /opt/sharcnet/modules ------------------------------------<br />
lsdyna/hyb/r711.88920 lsdyna/mpp/r711.88920 lsdyna/smp/r611.79036<br />
lsdyna/mpp/ls971.r85718 lsdyna/mpp/r712.95028 lsdyna/smp/r711.88920<br />
lsdyna/mpp/ls980B1.011113 lsdyna/mpp/r800.95359 lsdyna/smp/r712.95028<br />
lsdyna/mpp/r611.79036 lsdyna/mpp/r901.109912 lsdyna/smp/r800.95359<br />
lsdyna/mpp/r611.80542 lsdyna/smp/ls980B1.78258 lsdyna/smp/r901.109912<br />
</source><br />
<br />
====r9.0.1 versions====<br />
<br />
o For serial or threaded jobs:<br />
<pre>module load lsdyna/smp/r901.109912</pre><br />
<br />
o For mpi jobs:<br />
<pre><br />
module unload intel openmpi lsdyna mkl<br />
module load intel/15.0.3 openmpi/intel1503-std/1.8.7 lsdyna/mpp/r901.109912<br />
</pre><br />
<br />
====r8.0.0 versions====<br />
<br />
o For serial or threaded jobs:<br />
<pre>module load lsdyna/smp/r800.95359</pre><br />
<br />
o For mpi jobs:<br />
<pre><br />
module unload intel openmpi lsdyna mkl<br />
module load intel/15.0.3 openmpi/intel1503-std/1.8.7 lsdyna/mpp/r800.95359<br />
</pre><br />
<br />
====r7.1.2 versions====<br />
<br />
o For serial or threaded jobs:<br />
<pre>module load lsdyna/smp/r712.95028</pre><br />
<br />
o For mpi jobs:<br />
<pre>module unload intel openmpi lsdyna<br />
module load intel/12.1.3 openmpi/intel/1.6.5 lsdyna/mpp/r712.95028<br />
</pre><br />
<br />
====r7.1.1 versions====<br />
<br />
o For serial or threaded jobs:<br />
<pre>module load lsdyna/smp/r711.88920</pre><br />
<br />
o For mpi jobs:<br />
<pre>module load lsdyna/mpp/r711.88920</pre><br />
<br />
====r6.1.1 versions====<br />
<br />
o For serial or threaded jobs:<br />
<pre>module load lsdyna/smp/r611.79036</pre><br />
<br />
o For mpi jobs:<br />
<pre><br />
module unload intel openmpi lsdyna<br />
<br />
module load intel/11.1.069 openmpi/intel/1.6.4 lsdyna/mpp/r611.79036<br />
or<br />
module load intel/11.1.069 openmpi/intel/1.6.4 lsdyna/mpp/r611.80542<br />
</pre><br />
<br />
where <i>r611.80542</i> provides lsdyna_s only.<br><br />
<br />
====ls980 versions====<br />
<br />
o For serial or threaded jobs:<br />
<pre>module load lsdyna/smp/ls980B1.78258</pre><br />
<br />
o For mpi jobs:<br />
<pre><br />
module unload intel openmpi lsdyna<br />
<br />
module load intel/12.1.3 openmpi/intel/1.4.5 lsdyna/mpp/ls980B1.011113<br />
or<br />
module load intel/12.1.3 openmpi/intel/1.6.2 lsdyna/smp/ls980B1.78258<br />
</pre><br />
<br />
Note1) restart capability for ls980smpB1 and ls980mppB1 is not supported<br />
Note2) the module using legacy openmpi/intel/1.4.5 will run extremely slow<br />
<br />
=Job Submission=<br />
<br />
==Graham Cluster==<br />
<br />
The submission scripts myompjob.sh and mympijob.sh for the airbag problem are shown in the <b>Example Job</b> section below, for both graham and orca. Please note that you should specify your own username if submitting to a def account (not roberpj). Alternatively you could specify your resource allocation account:<br />
<br />
Sample threaded job submit script:<br />
<br />
sbatch myompjob.sh<br />
<br />
Sample mpi job submit script:<br />
<br />
sbatch mympijob.sh<br />
<br />
==Legacy Clusters==<br />
<br />
When loading a lsdyna sharcnet legacy module on the new orca or a sharcnet legacy system, the single or double precision solvers are specified with <strong>lsdyna_s</strong> or <strong>lsdyna_d</strong> respectively, as shown in the following sqsub commands:<br />
<br />
===1cpu SERIAL Job===<br />
<pre>sqsub -r 1h -q serial -o ofile.%J --mpp=2G lsdyna_d i=airbag.deploy.k ncpu=1</pre><br />
<br />
===4cpu SMP Job===<br />
<pre>sqsub -r 1h -q threaded -n 4 -o ofile.%J --mpp=1G lsdyna_s i=airbag.deploy.k ncpu=4</pre><br />
<br />
If using an explicit solver, one can specify a conservative initial memory setting on the command line as follows:<br />
<br />
<pre><br />
export LSTC_MEMORY=auto<br />
sqsub -r 1h -q threaded -n 4 -o ofile.%J --mpp=2G lsdyna_d i=airbag.deploy.k ncpu=4 memory=754414</pre><br />
where memory is the minimum number of 8 byte words shared by all processors in double precision, or, 4 byte words in single precision.<br />
<br />
The initial value can be determined by starting a simulation interactively on the command line, and finding the output resembling: <i> Memory required to begin solution : 754414</i>. The number of words can be specified as memory=260M instead of memory=260000000, for further details see https://www.d3view.com/2006/10/a-few-words-on-memory-settings-in-ls-dyna/. <br />
<br />
===8cpu MPI Job===<br />
<br />
<pre>sqsub -r 1h -q mpi -o ofile.%J -n 8 --mpp=2G lsdyna_d i=airbag.deploy.k</pre><br />
<br />
The initial memory can also be specified for mpi jobs on the sqsub command line with "memory=" for the first master processor to decompose the problem and "memory2=" used on all processors including the master to solves the decomposed problem, where the values are specified as 4 bytes per word in single precision and 8 bytes per word in double precision. The number of words can be specified as memory=260M instead of memory=260000000 OR memory2=260M instead of memory2=260000000 for further details see https://www.d3view.com/2006/10/a-few-words-on-memory-settings-in-ls-dyna/. The initial values can be found by running simulation interactively on a orca compute node and checking the output for a line such as: <i>Memory required to begin solution (memory= 464898 memory2= 158794 )</i> which could then be implemented for a job run in the queue by doing:<br />
<br />
<pre><br />
export LSTC_MEMORY=auto<br />
sqsub -r 1h -q mpi -o ofile.%J -n 8 --mpp=2G lsdyna_d i=airbag.deploy.k memory=464898 memory2=158794<br />
</pre><br />
<br />
The specification of memory2 on sharcnet or compute canada clusters is not beneficial since the queue reserves the same memory per core for all nodes, such that the decomposition master node process cannot be allocated to have larger system memory. Therefore its sufficient to specify a single memory parameter by doing:<br />
<br />
<pre><br />
export LSTC_MEMORY=auto <br />
sqsub -r 1h -q mpi -o ofile.%J -n 8 --mpp=2G lsdyna_d i=airbag.deploy.k memory=464898<br />
</pre><br />
<br />
where the LSTC_MEMORY variable will only allow the memory to grow for explicit simulations. The following slurm examples demonstrate how to prescribe the memory parameters exactly.<br />
<br />
=Example Job=<br />
<br />
Copy the airbag example to your account:<br />
<br />
rsync -av orca.computecanada.ca:/opt/sharcnet/lsdyna/r901.109912/examples r901.109912_examples<br />
cd r901.109912_examples/examples/misc/airbag<br />
gunzip airbag.deploy.k.gz<br />
<br />
==Graham (default modules)==<br />
<br />
Please note that graham does not have the sharcnet legacy modules installed on it.<br />
<br />
===Threaded Job ===<br />
<br />
Sample submission script <b>mysmpjob1.sh</b> for 4 core single precision smp job using compute canada default cvmfs modules:<br />
<br />
<pre><br />
#!/bin/bash<br />
#SBATCH --account=def-roberpj-ab<br />
#SBATCH --nodes=1 # use one node<br />
#SBATCH --cpus-per-task=4 # number threads<br />
#SBATCH --mem=4000M # total memory<br />
#SBATCH --time=00:30:00 # hrs:min:sec<br />
#SBATCH --job-name=testsmp1 # %x = jobname<br />
#SBATCH --output=slurm-%x-%N-%j.out<br />
echo "SLURM_CPUS_PER_TASK= "$SLURM_CPUS_PER_TASK<br />
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK<br />
# ----------------------------------------------------------<br />
# IMPORTANT: to set license server do either:<br />
# 1) export LSTC_LICENSE_SERVER=port@some.license.server<br />
# 2) create ~/.licenses/ls-dyna.lic as shown above<br />
# ----------------------------------------------------------<br />
module load ls-dyna/9.1<br />
ls-dyna_s i=airbag.deploy.k ncpu=4 memory=1000M<br />
</pre><br />
<br />
where memory = 4000M / 4 (bytes/word) = 1000M words<br />
<br />
===Mpi Job ===<br />
<br />
Sample submission script <b>mympijob2.sh</b> for 4 core double precision mpp job using compute canada default cvmfs modules:<br />
<br />
<pre><br />
#!/bin/bash<br />
#SBATCH --account=def-roberpj-ab<br />
#SBATCH --ntasks=4 # number of ranks<br />
#SBATCH --mem-per-cpu=4000M # memory per rank<br />
#SBATCH --time=00:30:00 # hrs:min:sec<br />
#SBATCH --job-name=testmpi2 # %x = jobname<br />
#SBATCH --output=slurm-%x-%N-%j.out<br />
# ----------------------------------------------------------<br />
# IMPORTANT: to set license server do either:<br />
# 1) export LSTC_LICENSE_SERVER=port@some.license.server<br />
# 2) create ~/.licenses/ls-dyna.lic as shown above<br />
# ----------------------------------------------------------<br />
module load openmpi/1.8.8 ls-dyna-mpi/9.1<br />
srun ls-dyna_d i=airbag.deploy.k memory=500M memory2=500M<br />
</pre><br />
<br />
where memory = 4000M / 8 (bytes/word) = 500M words<br />
<br />
==New Orca (legacy modules)==<br />
<br />
Please note, while new orca has most compute canada cvmfs modules available by default it does not have the graham the <i>ls-dyna</i> or <i>ls-dyna-mpi</i> modules installed. Therefore currently the only way to run lsdyna is by using the legacy sharcnet <i>lsdyna</i> modules as shown in the following two smp and mpi examples. <br />
<br />
===Threaded Job ===<br />
<br />
Submission script <b>mysmpjob3.sh</b> to run 16 core single precision single node smp job with sharcnet legacy modules:<br />
<br />
<pre><br />
#!/bin/bash<br />
#SBATCH --account=def-roberpj-ab<br />
#SBATCH --nodes=1 # use one node<br />
#SBATCH --cpus-per-task=16 # number threads<br />
#SBATCH --mem=4000M # total memory<br />
#SBATCH --time=00:30:00 # hrs:min:sec<br />
#SBATCH --job-name=mysmpjob3-1node-16core # jobname<br />
#SBATCH --output=slurm-%x-%N-%j.out # %x = jobname<br />
echo "SLURM_CPUS_PER_TASK= "$SLURM_CPUS_PER_TASK<br />
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK<br />
# ----------------------------------------------------------------<br />
# IMPORTANT: to set license server for legacy modules:<br />
export LSTC_LICENSE_SERVER=31047@license3.sharcnet.ca<br />
# since ~/.licenses/ls-dyna.lic does NOT work with legacy modules<br />
# ----------------------------------------------------------------<br />
module purge --force<br />
export MODULEPATH=/opt/sharcnet/modules<br />
module load lsdyna/smp/r712.95028<br />
lsdyna_s i=airbag.deploy.k ncpu=$SLURM_CPUS_PER_TASK memory=500M<br />
</pre><br />
<br />
where memory = 4000M / 8 (bytes/word) = 500M words<br />
<br />
===MPI Job ===<br />
<br />
To run the sharcnet legacy <i>module lsdyna/mpp/r712.95028</i> efficiently over several compute nodes with many cores requires using orca's xeon nodes as shown in <b> mympijob4.sh</b> below. To run lsdyna on orca's opteron nodes with <i>module lsdyna/mpp/r712.95028</i> requires using only <b>one</b> opteron node as shown in <b> mympijob5.sh</b> below. The other legacy sharcnet lsdyna mpp modules, as shown in the Availability Table lsdyna/mpp/ls971.r85718, lsdyna/mpp/ls980B1.011113, lsdyna/mpp/r611.79036, lsdyna/mpp/r611.80542, lsdyna/mpp/r711.88920, lsdyna/mpp/r800.95359 and lsdyna/mpp/r901.109912 have yet to be tested. If you want to use one but cannot get it working please open a ticket. Note that lines containing a space between "# and SBATCH" are comments. To activate such line(s) required removal of the space ie) "#SBATCH". Its recommended to use <i>module ls-dyna-mpi</i> on graham instead since the machine is far newer, faster and reliable. <br />
<br />
====Multi Node Slurm Script====<br />
<br />
<pre><br />
[roberpj@orc-login2:~/samples/lsdyna/airbag] cat mympijob4.sh<br />
#!/bin/bash<br />
#SBATCH --account=def-roberpj-ab<br />
#SBATCH --constraint=xeon<br />
#SBATCH --ntasks=48<br />
#SBATCH --mem-per-cpu=4000M<br />
#SBATCH --time=00:30:00<br />
#SBATCH --job-name=mympijob4-Xnodes-48cores-xeon # jobname<br />
#SBATCH --output=slurm-%x-%N-%j.out # %x = jobname<br />
# -------------------------------------------------------------<br />
# To set license server when using legacy sharcnet modules:<br />
export LSTC_LICENSE_SERVER=port@some.license.server<br />
# since ~/.licenses/ls-dyna.lic does not work with legacy modules.<br />
# -------------------------------------------------------------<br />
module purge --force<br />
export MODULEPATH=/opt/sharcnet/modules<br />
module load intel/12.1.3 openmpi/intel/1.6.5 lsdyna/mpp/r712.95028<br />
MACHINEFILE=`srun hostname -s | tr '\n' ':' | sed 's/:/,/g' | sed 's/.$//'`<br />
mpirun --host $MACHINEFILE --mca plm_slurm_args '--cpu_bind=none' lsdyna_s i=airbag.deploy.k memory=250M memory2=250M<br />
</pre><br />
<br />
====Single Node Slurm Script====<br />
<br />
<pre><br />
[roberpj@orc-login2:~/samples/lsdyna/airbag] cat mympijob5.sh<br />
#!/bin/bash<br />
#SBATCH --account=def-roberpj-ab<br />
#SBATCH --constraint=opteron<br />
#SBATCH --nodes=1<br />
# SBATCH --ntasks=24<br />
# SBATCH --mem=0<br />
#SBATCH --ntasks=16<br />
#SBATCH --mem-per-cpu=1000M<br />
#SBATCH --time=00:30:00<br />
#SBATCH --job-name=mympijob5-1node-16cores-opteron # jobname<br />
#SBATCH --output=slurm-%x-%N-%j.out # %x = jobname<br />
# -------------------------------------------------------------<br />
# To set license server when using legacy sharcnet modules:<br />
export LSTC_LICENSE_SERVER=port@some.license.server<br />
# since ~/.licenses/ls-dyna.lic does not work with legacy modules.<br />
# -------------------------------------------------------------<br />
module purge --force<br />
export MODULEPATH=/opt/sharcnet/modules<br />
module load intel/12.1.3 openmpi/intel/1.6.5 lsdyna/mpp/r712.95028<br />
MACHINEFILE=`srun hostname -s | tr '\n' ':' | sed 's/:/,/g' | sed 's/.$//'`<br />
mpirun --host $MACHINEFILE lsdyna_s i=airbag.deploy.k memory=250M memory2=250M<br />
</pre><br />
<br />
where memory = 1000M / 4 (bytes/word) = 250M words<br />
<br />
==Legacy Clusters==<br />
<br />
STEP1) The following shows sqsub submission of the [http://www.dynaexamples.com/examples-manual/misc/airbag airbag] example to the mpi queue. Its recommended to first edit <i>airbag.deploy.k</i> and change <i>endtim</i> to 3.000E-00 so the job runs long enough to perform the restart in steps 2 and 3 below:<br />
<pre><br />
cp -a /opt/sharcnet/lsdyna/r611.79036/examples /scratch/$USER/test-lsdyna<br />
cd /scratch/$USER/test-lsdyna/misc/airbag<br />
gunzip airbag.deploy.k.gz<br />
cp airbag.deploy.k airbag.deploy.restart.k<br />
nano deploy.restart.k (reduce plot file creation frequency)<br />
*DATABASE_BINARY_D3PLOT<br />
$ dt lcdtdeploy.restart.k<br />
1.000E-01 <--- change from 5.000E-04 <br />
export LSTC_LICENSE_SERVER=#####@license#.uwo.sharcnet<br />
module unload intel openmpi lsdyna<br />
module load intel/11.1.069 openmpi/intel/1.6.4 lsdyna/mpp/r611.79036<br />
sqsub -r 10m -q mpi --mpp=2G -n 8 -o ofile.%J lsdyna_d i=airbag.deploy.k<br />
</pre><br />
<br />
STEP2) With the job still running, use the echo command as follows to create a file called "D3KIL" that will trigger generation of restart files at which point the file D3KIL itself will be erased. Do this once a day if data loss is critical to you OR once or twice just before the <i>sqsub -r</i> time limit is reached. Further information can be found here http://www.dynasupport.com/tutorial/ls-dyna-users-guide/sense-switch-control:<br />
<br />
<pre><br />
echo "sw3" > D3KIL<br />
sqkill job#<br />
</pre><br />
<br />
STEP3) Before the job can be restarted, the following two lines must be added to the <i>airbag.deploy.restart.k</i> file:<br />
<br />
<pre><br />
*CHANGE_CURVE_DEFINITION<br />
1<br />
</pre><br />
<br />
STEP4) Now resubmit the job as follows using "r=" to specify the restart file:<br />
<br />
<pre><br />
sqsub -r 1h -q mpi --mpp=2G -n 8 -o ofile.%J lsdyna_d i=airbag.deploy.restart.k r=d3dump01<br />
</pre><br />
<br />
=General Notes=<br />
<br />
==Graham Cluster==<br />
<br />
There are no special notes at the present time in regards to using lsdyna on graham.<br />
<br />
<br />
==Command Line Use==<br />
<br />
Following the upgrade of orca to the compute canada module stack, the orca development nodes were discontinued.<br><br />
Therefore this section is no longer relevent and remain for historical purposes only.<br />
<br />
Parallel lsdyna jobs can be run interactively on the command line (outside the queue) for short run testing purposes as follows:<br />
<br />
o Orca Development Node MPP (mpi) Example<br />
<br />
ssh orca.sharcnet.ca <br />
ssh orc-dev1 (or dev2,3,4)<br />
module unload intel openmpi lsdyna<br />
module load intel/12.1.3 openmpi/intel/1.6.2 lsdyna/mpp/r711.88920<br />
mpirun -np 8 lsdyna_d i=airbag.deploy.k<br />
<br />
o Orca Development Node SMP (threaded) Example<br />
<br />
ssh orca.sharcnet.ca <br />
ssh orc-dev1 (or dev2,3,4)<br />
module unload intel openmpi lsdyna<br />
module load lsdyna/smp/r711.88920<br />
lsdyna_d i=airbag.deploy.k ncpu=8<br />
<br />
==Memory Issues==<br />
<br />
A minumum of mpp=2G is recommended although the memory requirement of a job may suggest much less is required. For instance setting mpp=1G for the airbag test job above will result in the following error when running a job in the queue:<br />
<br />
<pre><br />
--------------------------------------------------------------------------<br />
An attempt to set processor affinity has failed - please check to<br />
ensure that your system supports such functionality. If so, then<br />
this is probably something that should be reported to the OMPI developers.<br />
--------------------------------------------------------------------------<br />
</pre><br />
<br />
Another error message which can occur after a job runs for a while if mpp is chosen to small is: <br />
<br />
<pre><br />
>>>>> Process 45 <<<<<<br />
>>>>> Signal 11 : Segmentation Violation <<<<<<br />
</pre><br />
<br />
To get an idea of the amount of memory a job used run grep on the output file:<br />
<br />
<pre><br />
[roberpj@orc-login1:~/samples/lsdyna/airbag] cat ofile.2063302.orc-admin2.orca.sharcnet | grep Memory<br />
| Distributed Memory Parallel |<br />
Memory size from command line: 500000, 500000<br />
Memory for the head node<br />
Memory installed (MB) : 32237<br />
Memory free (MB) : 11259<br />
Memory required (MB) : 0<br />
Memory required to process keyword : 458120<br />
Memory required for decomposition : 458120<br />
Memory required to begin solution (memory= 458120 memory2= 230721)<br />
Max. Memory reqd for implicit sol: max used 0<br />
Max. Memory reqd for implicit sol: incore 0<br />
Max. Memory reqd for implicit sol: oocore 0<br />
Memory required to complete solution (memory= 458120 memory2= 230721)<br />
</pre><br />
<br />
==Version and Revision==<br />
<br />
Again run grep on the output file to extract the major and minor revision:<br />
<br />
<pre><br />
[roberpj@orc-login1:~/samples/lsdyna/airbag] cat ofile.2063302.orc-admin2.orca.sharcnet | grep 'Revision\|Version'<br />
| Version : mpp s R6.1.1 Date: 01/02/2013 |<br />
| Revision: 78769 Time: 07:43:30 |<br />
| SVN Version: 80542 |<br />
</pre><br />
<br />
==Check License Status==<br />
<br />
===Running and Queued Programs===<br />
<br />
To get a summary of running and/or queued jobs use the <i>lstc_qrun</i> command as follows, where queued means the job has started running according to the sqsub command but its actually sitting waiting for license resources to come available. Once these are acquired the job will start running on the cluster and appear as a Running Program according to lstc_qrun ie)<br />
<br />
<source lang="bash"><br />
[roberpj@hnd19:~] export LSTC_LICENSE_SERVER=#####@license#.uwo.sharcnet<br />
<br />
[roberpj@hnd19:~] lstc_qrun<br />
Defaulting to server 1 specified by LSTC_LICENSE_SERVER variable<br />
Running Programs<br />
User Host Program Started # procs<br />
-----------------------------------------------------------------------------<br />
dgierczy 20277@saw61.saw.sharcn LS-DYNA_971 Mon Apr 8 17:44 1<br />
dgierczy 8570@saw32.saw.sharcn LS-DYNA_971 Mon Apr 8 17:44 1<br />
dscronin 25486@hnd6 MPPDYNA_971 Tue Apr 9 20:19 6<br />
dscronin 14897@hnd18 MPPDYNA_971 Tue Apr 9 21:48 6<br />
dscronin 14971@hnd18 MPPDYNA_971 Tue Apr 9 21:48 6<br />
dscronin 15046@hnd18 MPPDYNA_971 Tue Apr 9 21:48 6<br />
dscronin 31237@hnd16 MPPDYNA_971 Tue Apr 9 21:53 6<br />
dscronin 31313@hnd16 MPPDYNA_971 Tue Apr 9 21:54 6<br />
dscronin 6396@hnd15 MPPDYNA_971 Tue Apr 9 21:54 6<br />
csharson 28890@saw175.saw.sharc MPPDYNA_971 Wed Apr 10 16:48 6<br />
No programs queued<br />
</source><br />
<br />
To show license file expiration details, append "-R" to the command:<br />
<br />
<source lang="bash"><br />
[roberpj@hnd19:~] export LSTC_LICENSE_SERVER=#####@license#.uwo.sharcnet<br />
<br />
[roberpj@hnd19:~] lstc_qrun -R<br />
Defaulting to server 1 specified by LSTC_LICENSE_SERVER variable<br />
**** LICENSE INFORMATION ****<br />
PROGRAM EXPIRATION CPUS USED FREE MAX | QUEUE<br />
---------------- ---------- ----- ------ ------ | -----<br />
LS-DYNA_971 12/31/2013 - 966 1024 | 0<br />
dgierczy 20277@saw61.saw.sharcnet 1<br />
dgierczy 8570@saw32.saw.sharcnet 1<br />
MPPDYNA_971 12/31/2013 - 966 1024 | 0<br />
dscronin 25486@hnd6 6<br />
dscronin 14897@hnd18 6<br />
dscronin 14971@hnd18 6<br />
dscronin 15046@hnd18 6<br />
dscronin 31237@hnd16 6<br />
dscronin 31313@hnd16 6<br />
dscronin 6396@hnd15 6<br />
csharson 28890@saw175.saw.sharcnet 6<br />
LICENSE GROUP 58 966 1024 | 0<br />
<br />
PROGRAM EXPIRATION CPUS USED FREE MAX | QUEUE<br />
---------------- ---------- ----- ------ ------ | -----<br />
LS-OPT 12/31/2013 0 1024 1024 | 0<br />
LICENSE GROUP 0 1024 1024 | 0<br />
</source><br />
<br />
===Killling A Program===<br />
<br />
The queue normally kills lsdyna jobs cleanly. However its possible that licences for a job (which is no longer running in the queue and therefore no longer has any processes running on the cluster) will continue to be tied up according to the lstc_qrun command. To kill such a program determine the pid@hostname from the lstc_qrun command then run the following kill command. The following example demonstrates the procedure for username roberpj:<br />
<br />
<pre><br />
[roberpj@orc-login2:~] lstc_qrun -R | grep roberpj<br />
roberpj 45312@orc329.orca.shar MPPDYNA Wed Dec 5 15:50 12<br />
<br />
[roberpj@orc-login2:~] lstc_qkill 45312@orc329.orca.sharcnet.ca<br />
Defaulting to server 1 specified by LSTC_LICENSE_SERVER variable<br />
Program queued for termination<br />
<br />
[roberpj@orc-login2:~] lstc_qrun -R | grep roberpj<br />
[roberpj@orc-login2:~] <br />
</pre><br />
<br />
==Legacy Instructions==<br />
<br />
The following binaries remain available on saw and orca for backward compatibility testing:<br />
<br />
<pre><br />
[roberpj@orc-login2:~] cd /opt/sharcnet/local/lsdyna<br />
[roberpj@orc129:/opt/sharcnet/local/lsdyna] ls ls971*<br />
ls971_d_R3_1 ls971_d_R4_2_1 ls971_s_R3_1 ls971_s_R4_2_1 ls971_s_R5_1_1<br />
ls971_d_R4_2_0 ls971_d_R5_0 ls971_s_R4_2_0 ls971_s_R5_0<br />
</pre><br />
<br />
There are currently no sharcnet modules for these versions, hence jobs should be submitted as follows:<br />
<br />
<pre><br />
module load lsdyna<br />
export PATH=/opt/sharcnet/local/lsdyna:$PATH<br />
export LSTC_LICENSE_SERVER=XXXXX@license3.uwo.sharcnet<br />
cp /opt/sharcnet/local/lsdyna/examples/airbag.deploy.k airbag.deploy.k<br />
SERIAL JOB: sqsub -q serial -r 1d -o ofile.%J ls971_d_R4_2_1 i=airbag.deploy.k<br />
THREADED JOB: sqsub -q threaded -n 4 -r 1d -o ofile.%J ls971_d_R4_2_1 ncpu=4 para=2 i=airbag.deploy.k<br />
</pre><br />
<br />
Note! the <i>ls971_s_R3_1</i> and <i>ls971_d_R3_1</i> binaries do not work, a fix is being looked for.<br />
<br />
=References=<br />
<br />
o LSTC LS-DYNA Homepage<br><br />
http://www.lstc.com/products/ls-dyna<br />
<br />
o LSTC LS-DYNA Support (Tutorials, HowTos, Faq, Manuals, Release Notes, News, Links)<br><br />
http://www.dynasupport.com/<br />
<br />
o LS-DYNA Release Notes<br><br />
http://www.dynasupport.com/release-notes<br />
<br />
o LS-PrePost Online Documentation FAQ<br><br />
http://www.lstc.com/lspp/content/faq.shtml<br />
<br />
o LSTC Download/Install Overview Page<br><br />
Provides links to binaries for LS-DYNA SMP/MPP, LS-OPT, LS-PREPOST, LS-TASC.<br><br />
http://www.lstc.com/download<br />
<br />
==Memory Links==<br />
<br />
o Implicit: Memory notes<br> <br />
http://www.dynasupport.com/howtos/implicit/implicit-memory-notes<br />
<br />
o LS-DYNA Support Environment variables<br><br />
http://www.dynasupport.com/howtos/general/environment-variables<br />
<br />
o LS-DYNA and d3VIEW Blog (a few words on memory settings)<br><br />
http://blog2.d3view.com/a-few-words-on-memory-settings-in-ls-dyna/<br />
<br />
o Convert Words to GB ie) memory=500MW (3.73GB)<br><br />
http://deviceanalytics.com/memcalc.php</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=LSDYNA&diff=17950LSDYNA2019-06-06T14:25:30Z<p>Edward: </p>
<hr />
<div>{{Tamplate:CCMissing}}<br />
{{Software<br />
|package_name=LSDYNA<br />
|package_description=Suite of programs for transient dynamic finite element program<br />
|package_idnumber=104<br />
}}<br />
<br />
=Introduction=<br />
<br />
Before a research group can use LSDYNA on sharcnet, a license must be purchased directly from LSTC for the sharcnet license server. Alternately, if a research group resides at a institution that has sharcnet computing hardware (mac, uwo, guelph, waterloo) it maybe possible to use a pre-existing site license hosted on an accessible institutional license server. To access and use this software you must open a ticket and request to be added to the sharcnet <b><i>lsdyna</i></b> group.<br />
<br />
=Version Selection=<br />
<br />
==Graham Cluster==<br />
<br />
===License===<br />
<br />
Create and customize this file for your license server, where XXXXX should be replaced with your port number and Y with your server number:<br />
<br />
[roberpj@gra1:~] cat .licenses/ls-dyna.lic <br />
#LICENSE_TYPE: network<br />
#LICENSE_SERVER: XXXXX@licenseY.uwo.sharcnet<br />
<br />
===Module===<br />
<br />
The modules are loaded automatically in the testomp.sh and testmpi.sh script shown below in the example section:<br />
<br />
For single node smp jobs the available versions can be found by doing:<br />
<br />
[roberpj@gra-login2:~] module spider ls-dyna<br />
ls-dyna/7.1.2<br />
ls-dyna/7.1.3<br />
ls-dyna/8.1<br />
ls-dyna/9.1<br />
ls-dyna/9.2<br />
ls-dyna/10.0<br />
<br />
To load any version do for example:<br />
<br />
module load ls-dyna/9.1<br />
<br />
For multi node mpp jobs the available versions can be found by doing:<br />
<br />
[roberpj@gra-login2:~] module spider ls-dyna-mpi<br />
ls-dyna-mpi/7.1.2<br />
ls-dyna-mpi/7.1.3<br />
ls-dyna-mpi/8.1<br />
ls-dyna-mpi/9.1<br />
ls-dyna-mpi/9.2<br />
ls-dyna-mpi/10.0<br />
<br />
To load the 7.1.X modules do the following:<br />
module load openmpi/1.6.5 ls-dyna-mpi/7.1.3<br />
<br />
To load the 8.X or 9.X modules do:<br />
module load openmpi/1.8.8 ls-dyna-mpi/9.2<br />
<br />
To load the 10.X modules do:<br />
module load openmpi/1.10.7 ls-dyna-mpi/10.0<br />
<br />
==Legacy Clusters==<br />
<br />
===License===<br />
<br />
Once a license configuration is established, the research group will be given a 5 digit port number. The value should then be inserted into the appropriate departmental export statement before loading the module file as follows:<br />
<br />
o UofT Mechanical Engineering Dept<br />
<b><font color="purple">export</font></b> <font color="green">LSTC_LICENSE_SERVER</font>=<b><font color="red">Port</font></b>@license1.uwo.sharcnet<br />
<br />
o McGill Mechanical Engineering Department<br />
<b><font color="purple">export</font></b> <font color="green">LSTC_LICENSE_SERVER</font>=<b><font color="red">Port</font></b>@license2.uwo.sharcnet<br />
<br />
o UW Mechanical and Mechatronics Engineering Dept<br />
<b><font color="purple">export</font></b> <font color="green">LSTC_LICENSE_SERVER</font>=<b><font color="red">Port</font></b>@license3.uwo.sharcnet<br />
<br />
o Laurentian University Bharti School of Engineering<br />
<b><font color="purple">export</font></b> <font color="green">LSTC_LICENSE_SERVER</font>=<b><font color="red">Port</font></b>@license4.uwo.sharcnet<br />
<br />
o Fictitious Example:<br />
<br />
<b><font color="purple">export</font></b> <font color="green">LSTC_LICENSE_SERVER</font>=<b><font color="red">12345</font></b>@license1.uwo.sharcnet<br />
<br />
===Module===<br />
<br />
The next step is load the desired sharcnet lsdyna module version. Check which modules are available by running the <i>module avail</i> command then load one the modules as shown:<br />
<br />
<source lang="bash"><br />
[roberpj@orc-login2:~] module avail lsdyna<br />
------------------------------------ /opt/sharcnet/modules ------------------------------------<br />
lsdyna/hyb/r711.88920 lsdyna/mpp/r711.88920 lsdyna/smp/r611.79036<br />
lsdyna/mpp/ls971.r85718 lsdyna/mpp/r712.95028 lsdyna/smp/r711.88920<br />
lsdyna/mpp/ls980B1.011113 lsdyna/mpp/r800.95359 lsdyna/smp/r712.95028<br />
lsdyna/mpp/r611.79036 lsdyna/mpp/r901.109912 lsdyna/smp/r800.95359<br />
lsdyna/mpp/r611.80542 lsdyna/smp/ls980B1.78258 lsdyna/smp/r901.109912<br />
</source><br />
<br />
====r9.0.1 versions====<br />
<br />
o For serial or threaded jobs:<br />
<pre>module load lsdyna/smp/r901.109912</pre><br />
<br />
o For mpi jobs:<br />
<pre><br />
module unload intel openmpi lsdyna mkl<br />
module load intel/15.0.3 openmpi/intel1503-std/1.8.7 lsdyna/mpp/r901.109912<br />
</pre><br />
<br />
====r8.0.0 versions====<br />
<br />
o For serial or threaded jobs:<br />
<pre>module load lsdyna/smp/r800.95359</pre><br />
<br />
o For mpi jobs:<br />
<pre><br />
module unload intel openmpi lsdyna mkl<br />
module load intel/15.0.3 openmpi/intel1503-std/1.8.7 lsdyna/mpp/r800.95359<br />
</pre><br />
<br />
====r7.1.2 versions====<br />
<br />
o For serial or threaded jobs:<br />
<pre>module load lsdyna/smp/r712.95028</pre><br />
<br />
o For mpi jobs:<br />
<pre>module unload intel openmpi lsdyna<br />
module load intel/12.1.3 openmpi/intel/1.6.5 lsdyna/mpp/r712.95028<br />
</pre><br />
<br />
====r7.1.1 versions====<br />
<br />
o For serial or threaded jobs:<br />
<pre>module load lsdyna/smp/r711.88920</pre><br />
<br />
o For mpi jobs:<br />
<pre>module load lsdyna/mpp/r711.88920</pre><br />
<br />
====r6.1.1 versions====<br />
<br />
o For serial or threaded jobs:<br />
<pre>module load lsdyna/smp/r611.79036</pre><br />
<br />
o For mpi jobs:<br />
<pre><br />
module unload intel openmpi lsdyna<br />
<br />
module load intel/11.1.069 openmpi/intel/1.6.4 lsdyna/mpp/r611.79036<br />
or<br />
module load intel/11.1.069 openmpi/intel/1.6.4 lsdyna/mpp/r611.80542<br />
</pre><br />
<br />
where <i>r611.80542</i> provides lsdyna_s only.<br><br />
<br />
====ls980 versions====<br />
<br />
o For serial or threaded jobs:<br />
<pre>module load lsdyna/smp/ls980B1.78258</pre><br />
<br />
o For mpi jobs:<br />
<pre><br />
module unload intel openmpi lsdyna<br />
<br />
module load intel/12.1.3 openmpi/intel/1.4.5 lsdyna/mpp/ls980B1.011113<br />
or<br />
module load intel/12.1.3 openmpi/intel/1.6.2 lsdyna/smp/ls980B1.78258<br />
</pre><br />
<br />
Note1) restart capability for ls980smpB1 and ls980mppB1 is not supported<br />
Note2) the module using legacy openmpi/intel/1.4.5 will run extremely slow<br />
<br />
=Job Submission=<br />
<br />
==Graham Cluster==<br />
<br />
The submission scripts myompjob.sh and mympijob.sh for the airbag problem are shown in the <b>Example Job</b> section below, for both graham and orca. Please note that you should specify your own username if submitting to a def account (not roberpj). Alternatively you could specify your resource allocation account:<br />
<br />
Sample threaded job submit script:<br />
<br />
sbatch myompjob.sh<br />
<br />
Sample mpi job submit script:<br />
<br />
sbatch mympijob.sh<br />
<br />
==Legacy Clusters==<br />
<br />
When loading a lsdyna sharcnet legacy module on the new orca or a sharcnet legacy system, the single or double precision solvers are specified with <strong>lsdyna_s</strong> or <strong>lsdyna_d</strong> respectively, as shown in the following sqsub commands:<br />
<br />
===1cpu SERIAL Job===<br />
<pre>sqsub -r 1h -q serial -o ofile.%J --mpp=2G lsdyna_d i=airbag.deploy.k ncpu=1</pre><br />
<br />
===4cpu SMP Job===<br />
<pre>sqsub -r 1h -q threaded -n 4 -o ofile.%J --mpp=1G lsdyna_s i=airbag.deploy.k ncpu=4</pre><br />
<br />
If using an explicit solver, one can specify a conservative initial memory setting on the command line as follows:<br />
<br />
<pre><br />
export LSTC_MEMORY=auto<br />
sqsub -r 1h -q threaded -n 4 -o ofile.%J --mpp=2G lsdyna_d i=airbag.deploy.k ncpu=4 memory=754414</pre><br />
where memory is the minimum number of 8 byte words shared by all processors in double precision, or, 4 byte words in single precision.<br />
<br />
The initial value can be determined by starting a simulation interactively on the command line, and finding the output resembling: <i> Memory required to begin solution : 754414</i>. The number of words can be specified as memory=260M instead of memory=260000000, for further details see https://www.d3view.com/2006/10/a-few-words-on-memory-settings-in-ls-dyna/. <br />
<br />
===8cpu MPI Job===<br />
<br />
<pre>sqsub -r 1h -q mpi -o ofile.%J -n 8 --mpp=2G lsdyna_d i=airbag.deploy.k</pre><br />
<br />
The initial memory can also be specified for mpi jobs on the sqsub command line with "memory=" for the first master processor to decompose the problem and "memory2=" used on all processors including the master to solves the decomposed problem, where the values are specified as 4 bytes per word in single precision and 8 bytes per word in double precision. The number of words can be specified as memory=260M instead of memory=260000000 OR memory2=260M instead of memory2=260000000 for further details see https://www.d3view.com/2006/10/a-few-words-on-memory-settings-in-ls-dyna/. The initial values can be found by running simulation interactively on a orca compute node and checking the output for a line such as: <i>Memory required to begin solution (memory= 464898 memory2= 158794 )</i> which could then be implemented for a job run in the queue by doing:<br />
<br />
<pre><br />
export LSTC_MEMORY=auto<br />
sqsub -r 1h -q mpi -o ofile.%J -n 8 --mpp=2G lsdyna_d i=airbag.deploy.k memory=464898 memory2=158794<br />
</pre><br />
<br />
The specification of memory2 on sharcnet or compute canada clusters is not beneficial since the queue reserves the same memory per core for all nodes, such that the decomposition master node process cannot be allocated to have larger system memory. Therefore its sufficient to specify a single memory parameter by doing:<br />
<br />
<pre><br />
export LSTC_MEMORY=auto <br />
sqsub -r 1h -q mpi -o ofile.%J -n 8 --mpp=2G lsdyna_d i=airbag.deploy.k memory=464898<br />
</pre><br />
<br />
where the LSTC_MEMORY variable will only allow the memory to grow for explicit simulations. The following slurm examples demonstrate how to prescribe the memory parameters exactly.<br />
<br />
=Example Job=<br />
<br />
Copy the airbag example to your account:<br />
<br />
rsync -av orca.computecanada.ca:/opt/sharcnet/lsdyna/r901.109912/examples r901.109912_examples<br />
cd r901.109912_examples/examples/misc/airbag<br />
gunzip airbag.deploy.k.gz<br />
<br />
==Graham (default modules)==<br />
<br />
Please note that graham does not have the sharcnet legacy modules installed on it.<br />
<br />
===Threaded Job ===<br />
<br />
Sample submission script <b>mysmpjob1.sh</b> for 4 core single precision smp job using compute canada default cvmfs modules:<br />
<br />
<pre><br />
#!/bin/bash<br />
#SBATCH --account=def-roberpj-ab<br />
#SBATCH --nodes=1 # use one node<br />
#SBATCH --cpus-per-task=4 # number threads<br />
#SBATCH --mem=4000M # total memory<br />
#SBATCH --time=00:30:00 # hrs:min:sec<br />
#SBATCH --job-name=testsmp1 # %x = jobname<br />
#SBATCH --output=slurm-%x-%N-%j.out<br />
echo "SLURM_CPUS_PER_TASK= "$SLURM_CPUS_PER_TASK<br />
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK<br />
# ----------------------------------------------------------<br />
# IMPORTANT: to set license server do either:<br />
# 1) export LSTC_LICENSE_SERVER=port@some.license.server<br />
# 2) create ~/.licenses/ls-dyna.lic as shown above<br />
# ----------------------------------------------------------<br />
module load ls-dyna/9.1<br />
ls-dyna_s i=airbag.deploy.k ncpu=4 memory=1000M<br />
</pre><br />
<br />
where memory = 4000M / 4 (bytes/word) = 1000M words<br />
<br />
===Mpi Job ===<br />
<br />
Sample submission script <b>mympijob2.sh</b> for 4 core double precision mpp job using compute canada default cvmfs modules:<br />
<br />
<pre><br />
#!/bin/bash<br />
#SBATCH --account=def-roberpj-ab<br />
#SBATCH --ntasks=4 # number of ranks<br />
#SBATCH --mem-per-cpu=4000M # memory per rank<br />
#SBATCH --time=00:30:00 # hrs:min:sec<br />
#SBATCH --job-name=testmpi2 # %x = jobname<br />
#SBATCH --output=slurm-%x-%N-%j.out<br />
# ----------------------------------------------------------<br />
# IMPORTANT: to set license server do either:<br />
# 1) export LSTC_LICENSE_SERVER=port@some.license.server<br />
# 2) create ~/.licenses/ls-dyna.lic as shown above<br />
# ----------------------------------------------------------<br />
module load openmpi/1.8.8 ls-dyna-mpi/9.1<br />
srun ls-dyna_d i=airbag.deploy.k memory=500M memory2=500M<br />
</pre><br />
<br />
where memory = 4000M / 8 (bytes/word) = 500M words<br />
<br />
==New Orca (legacy modules)==<br />
<br />
Please note, while new orca has most compute canada cvmfs modules available by default it does not have the graham the <i>ls-dyna</i> or <i>ls-dyna-mpi</i> modules installed. Therefore currently the only way to run lsdyna is by using the legacy sharcnet <i>lsdyna</i> modules as shown in the following two smp and mpi examples. <br />
<br />
===Threaded Job ===<br />
<br />
Submission script <b>mysmpjob3.sh</b> to run 16 core single precision single node smp job with sharcnet legacy modules:<br />
<br />
<pre><br />
#!/bin/bash<br />
#SBATCH --account=def-roberpj-ab<br />
#SBATCH --nodes=1 # use one node<br />
#SBATCH --cpus-per-task=16 # number threads<br />
#SBATCH --mem=4000M # total memory<br />
#SBATCH --time=00:30:00 # hrs:min:sec<br />
#SBATCH --job-name=mysmpjob3-1node-16core # jobname<br />
#SBATCH --output=slurm-%x-%N-%j.out # %x = jobname<br />
echo "SLURM_CPUS_PER_TASK= "$SLURM_CPUS_PER_TASK<br />
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK<br />
# ----------------------------------------------------------------<br />
# IMPORTANT: to set license server for legacy modules:<br />
export LSTC_LICENSE_SERVER=31047@license3.sharcnet.ca<br />
# since ~/.licenses/ls-dyna.lic does NOT work with legacy modules<br />
# ----------------------------------------------------------------<br />
module purge --force<br />
export MODULEPATH=/opt/sharcnet/modules<br />
module load lsdyna/smp/r712.95028<br />
lsdyna_s i=airbag.deploy.k ncpu=$SLURM_CPUS_PER_TASK memory=500M<br />
</pre><br />
<br />
where memory = 4000M / 8 (bytes/word) = 500M words<br />
<br />
===MPI Job ===<br />
<br />
To run the sharcnet legacy <i>module lsdyna/mpp/r712.95028</i> efficiently over several compute nodes with many cores requires using orca's xeon nodes as shown in <b> mympijob4.sh</b> below. To run lsdyna on orca's opteron nodes with <i>module lsdyna/mpp/r712.95028</i> requires using only <b>one</b> opteron node as shown in <b> mympijob5.sh</b> below. The other legacy sharcnet lsdyna mpp modules, as shown in the Availability Table lsdyna/mpp/ls971.r85718, lsdyna/mpp/ls980B1.011113, lsdyna/mpp/r611.79036, lsdyna/mpp/r611.80542, lsdyna/mpp/r711.88920, lsdyna/mpp/r800.95359 and lsdyna/mpp/r901.109912 have yet to be tested. If you want to use one but cannot get it working please open a ticket. Note that lines containing a space between "# and SBATCH" are comments. To activate such line(s) required removal of the space ie) "#SBATCH". Its recommended to use <i>module ls-dyna-mpi</i> on graham instead since the machine is far newer, faster and reliable. <br />
<br />
====Multi Node Slurm Script====<br />
<br />
<pre><br />
[roberpj@orc-login2:~/samples/lsdyna/airbag] cat mympijob4.sh<br />
#!/bin/bash<br />
#SBATCH --account=def-roberpj-ab<br />
#SBATCH --constraint=xeon<br />
#SBATCH --ntasks=48<br />
#SBATCH --mem-per-cpu=4000M<br />
#SBATCH --time=00:30:00<br />
#SBATCH --job-name=mympijob4-Xnodes-48cores-xeon # jobname<br />
#SBATCH --output=slurm-%x-%N-%j.out # %x = jobname<br />
# -------------------------------------------------------------<br />
# To set license server when using legacy sharcnet modules:<br />
export LSTC_LICENSE_SERVER=port@some.license.server<br />
# since ~/.licenses/ls-dyna.lic does not work with legacy modules.<br />
# -------------------------------------------------------------<br />
module purge --force<br />
export MODULEPATH=/opt/sharcnet/modules<br />
module load intel/12.1.3 openmpi/intel/1.6.5 lsdyna/mpp/r712.95028<br />
MACHINEFILE=`srun hostname -s | tr '\n' ':' | sed 's/:/,/g' | sed 's/.$//'`<br />
mpirun --host $MACHINEFILE --mca plm_slurm_args '--cpu_bind=none' lsdyna_s i=airbag.deploy.k memory=250M memory2=250M<br />
</pre><br />
<br />
====Single Node Slurm Script====<br />
<br />
<pre><br />
[roberpj@orc-login2:~/samples/lsdyna/airbag] cat mympijob5.sh<br />
#!/bin/bash<br />
#SBATCH --account=def-roberpj-ab<br />
#SBATCH --constraint=opteron<br />
#SBATCH --nodes=1<br />
# SBATCH --ntasks=24<br />
# SBATCH --mem=0<br />
#SBATCH --ntasks=16<br />
#SBATCH --mem-per-cpu=1000M<br />
#SBATCH --time=00:30:00<br />
#SBATCH --job-name=mympijob5-1node-16cores-opteron # jobname<br />
#SBATCH --output=slurm-%x-%N-%j.out # %x = jobname<br />
# -------------------------------------------------------------<br />
# To set license server when using legacy sharcnet modules:<br />
export LSTC_LICENSE_SERVER=port@some.license.server<br />
# since ~/.licenses/ls-dyna.lic does not work with legacy modules.<br />
# -------------------------------------------------------------<br />
module purge --force<br />
export MODULEPATH=/opt/sharcnet/modules<br />
module load intel/12.1.3 openmpi/intel/1.6.5 lsdyna/mpp/r712.95028<br />
MACHINEFILE=`srun hostname -s | tr '\n' ':' | sed 's/:/,/g' | sed 's/.$//'`<br />
mpirun --host $MACHINEFILE lsdyna_s i=airbag.deploy.k memory=250M memory2=250M<br />
</pre><br />
<br />
where memory = 1000M / 4 (bytes/word) = 250M words<br />
<br />
==Legacy Clusters==<br />
<br />
STEP1) The following shows sqsub submission of the [http://www.dynaexamples.com/examples-manual/misc/airbag airbag] example to the mpi queue. Its recommended to first edit <i>airbag.deploy.k</i> and change <i>endtim</i> to 3.000E-00 so the job runs long enough to perform the restart in steps 2 and 3 below:<br />
<pre><br />
cp -a /opt/sharcnet/lsdyna/r611.79036/examples /scratch/$USER/test-lsdyna<br />
cd /scratch/$USER/test-lsdyna/misc/airbag<br />
gunzip airbag.deploy.k.gz<br />
cp airbag.deploy.k airbag.deploy.restart.k<br />
nano deploy.restart.k (reduce plot file creation frequency)<br />
*DATABASE_BINARY_D3PLOT<br />
$ dt lcdtdeploy.restart.k<br />
1.000E-01 <--- change from 5.000E-04 <br />
export LSTC_LICENSE_SERVER=#####@license#.uwo.sharcnet<br />
module unload intel openmpi lsdyna<br />
module load intel/11.1.069 openmpi/intel/1.6.4 lsdyna/mpp/r611.79036<br />
sqsub -r 10m -q mpi --mpp=2G -n 8 -o ofile.%J lsdyna_d i=airbag.deploy.k<br />
</pre><br />
<br />
STEP2) With the job still running, use the echo command as follows to create a file called "D3KIL" that will trigger generation of restart files at which point the file D3KIL itself will be erased. Do this once a day if data loss is critical to you OR once or twice just before the <i>sqsub -r</i> time limit is reached. Further information can be found here http://www.dynasupport.com/tutorial/ls-dyna-users-guide/sense-switch-control:<br />
<br />
<pre><br />
echo "sw3" > D3KIL<br />
sqkill job#<br />
</pre><br />
<br />
STEP3) Before the job can be restarted, the following two lines must be added to the <i>airbag.deploy.restart.k</i> file:<br />
<br />
<pre><br />
*CHANGE_CURVE_DEFINITION<br />
1<br />
</pre><br />
<br />
STEP4) Now resubmit the job as follows using "r=" to specify the restart file:<br />
<br />
<pre><br />
sqsub -r 1h -q mpi --mpp=2G -n 8 -o ofile.%J lsdyna_d i=airbag.deploy.restart.k r=d3dump01<br />
</pre><br />
<br />
=General Notes=<br />
<br />
==Graham Cluster==<br />
<br />
There are no special notes at the present time in regards to using lsdyna on graham.<br />
<br />
<br />
==Command Line Use==<br />
<br />
Following the upgrade of orca to the compute canada module stack, the orca development nodes were discontinued.<br><br />
Therefore this section is no longer relevent and remain for historical purposes only.<br />
<br />
Parallel lsdyna jobs can be run interactively on the command line (outside the queue) for short run testing purposes as follows:<br />
<br />
o Orca Development Node MPP (mpi) Example<br />
<br />
ssh orca.sharcnet.ca <br />
ssh orc-dev1 (or dev2,3,4)<br />
module unload intel openmpi lsdyna<br />
module load intel/12.1.3 openmpi/intel/1.6.2 lsdyna/mpp/r711.88920<br />
mpirun -np 8 lsdyna_d i=airbag.deploy.k<br />
<br />
o Orca Development Node SMP (threaded) Example<br />
<br />
ssh orca.sharcnet.ca <br />
ssh orc-dev1 (or dev2,3,4)<br />
module unload intel openmpi lsdyna<br />
module load lsdyna/smp/r711.88920<br />
lsdyna_d i=airbag.deploy.k ncpu=8<br />
<br />
==Memory Issues==<br />
<br />
A minumum of mpp=2G is recommended although the memory requirement of a job may suggest much less is required. For instance setting mpp=1G for the airbag test job above will result in the following error when running a job in the queue:<br />
<br />
<pre><br />
--------------------------------------------------------------------------<br />
An attempt to set processor affinity has failed - please check to<br />
ensure that your system supports such functionality. If so, then<br />
this is probably something that should be reported to the OMPI developers.<br />
--------------------------------------------------------------------------<br />
</pre><br />
<br />
Another error message which can occur after a job runs for a while if mpp is chosen to small is: <br />
<br />
<pre><br />
>>>>> Process 45 <<<<<<br />
>>>>> Signal 11 : Segmentation Violation <<<<<<br />
</pre><br />
<br />
To get an idea of the amount of memory a job used run grep on the output file:<br />
<br />
<pre><br />
[roberpj@orc-login1:~/samples/lsdyna/airbag] cat ofile.2063302.orc-admin2.orca.sharcnet | grep Memory<br />
| Distributed Memory Parallel |<br />
Memory size from command line: 500000, 500000<br />
Memory for the head node<br />
Memory installed (MB) : 32237<br />
Memory free (MB) : 11259<br />
Memory required (MB) : 0<br />
Memory required to process keyword : 458120<br />
Memory required for decomposition : 458120<br />
Memory required to begin solution (memory= 458120 memory2= 230721)<br />
Max. Memory reqd for implicit sol: max used 0<br />
Max. Memory reqd for implicit sol: incore 0<br />
Max. Memory reqd for implicit sol: oocore 0<br />
Memory required to complete solution (memory= 458120 memory2= 230721)<br />
</pre><br />
<br />
==Version and Revision==<br />
<br />
Again run grep on the output file to extract the major and minor revision:<br />
<br />
<pre><br />
[roberpj@orc-login1:~/samples/lsdyna/airbag] cat ofile.2063302.orc-admin2.orca.sharcnet | grep 'Revision\|Version'<br />
| Version : mpp s R6.1.1 Date: 01/02/2013 |<br />
| Revision: 78769 Time: 07:43:30 |<br />
| SVN Version: 80542 |<br />
</pre><br />
<br />
==Check License Status==<br />
<br />
===Running and Queued Programs===<br />
<br />
To get a summary of running and/or queued jobs use the <i>lstc_qrun</i> command as follows, where queued means the job has started running according to the sqsub command but its actually sitting waiting for license resources to come available. Once these are acquired the job will start running on the cluster and appear as a Running Program according to lstc_qrun ie)<br />
<br />
<source lang="bash"><br />
[roberpj@hnd19:~] export LSTC_LICENSE_SERVER=#####@license#.uwo.sharcnet<br />
<br />
[roberpj@hnd19:~] lstc_qrun<br />
Defaulting to server 1 specified by LSTC_LICENSE_SERVER variable<br />
Running Programs<br />
User Host Program Started # procs<br />
-----------------------------------------------------------------------------<br />
dgierczy 20277@saw61.saw.sharcn LS-DYNA_971 Mon Apr 8 17:44 1<br />
dgierczy 8570@saw32.saw.sharcn LS-DYNA_971 Mon Apr 8 17:44 1<br />
dscronin 25486@hnd6 MPPDYNA_971 Tue Apr 9 20:19 6<br />
dscronin 14897@hnd18 MPPDYNA_971 Tue Apr 9 21:48 6<br />
dscronin 14971@hnd18 MPPDYNA_971 Tue Apr 9 21:48 6<br />
dscronin 15046@hnd18 MPPDYNA_971 Tue Apr 9 21:48 6<br />
dscronin 31237@hnd16 MPPDYNA_971 Tue Apr 9 21:53 6<br />
dscronin 31313@hnd16 MPPDYNA_971 Tue Apr 9 21:54 6<br />
dscronin 6396@hnd15 MPPDYNA_971 Tue Apr 9 21:54 6<br />
csharson 28890@saw175.saw.sharc MPPDYNA_971 Wed Apr 10 16:48 6<br />
No programs queued<br />
</source><br />
<br />
To show license file expiration details, append "-R" to the command:<br />
<br />
<source lang="bash"><br />
[roberpj@hnd19:~] export LSTC_LICENSE_SERVER=#####@license#.uwo.sharcnet<br />
<br />
[roberpj@hnd19:~] lstc_qrun -R<br />
Defaulting to server 1 specified by LSTC_LICENSE_SERVER variable<br />
**** LICENSE INFORMATION ****<br />
PROGRAM EXPIRATION CPUS USED FREE MAX | QUEUE<br />
---------------- ---------- ----- ------ ------ | -----<br />
LS-DYNA_971 12/31/2013 - 966 1024 | 0<br />
dgierczy 20277@saw61.saw.sharcnet 1<br />
dgierczy 8570@saw32.saw.sharcnet 1<br />
MPPDYNA_971 12/31/2013 - 966 1024 | 0<br />
dscronin 25486@hnd6 6<br />
dscronin 14897@hnd18 6<br />
dscronin 14971@hnd18 6<br />
dscronin 15046@hnd18 6<br />
dscronin 31237@hnd16 6<br />
dscronin 31313@hnd16 6<br />
dscronin 6396@hnd15 6<br />
csharson 28890@saw175.saw.sharcnet 6<br />
LICENSE GROUP 58 966 1024 | 0<br />
<br />
PROGRAM EXPIRATION CPUS USED FREE MAX | QUEUE<br />
---------------- ---------- ----- ------ ------ | -----<br />
LS-OPT 12/31/2013 0 1024 1024 | 0<br />
LICENSE GROUP 0 1024 1024 | 0<br />
</source><br />
<br />
===Killling A Program===<br />
<br />
The queue normally kills lsdyna jobs cleanly. However its possible that licences for a job (which is no longer running in the queue and therefore no longer has any processes running on the cluster) will continue to be tied up according to the lstc_qrun command. To kill such a program determine the pid@hostname from the lstc_qrun command then run the following kill command. The following example demonstrates the procedure for username roberpj:<br />
<br />
<pre><br />
[roberpj@orc-login2:~] lstc_qrun -R | grep roberpj<br />
roberpj 45312@orc329.orca.shar MPPDYNA Wed Dec 5 15:50 12<br />
<br />
[roberpj@orc-login2:~] lstc_qkill 45312@orc329.orca.sharcnet.ca<br />
Defaulting to server 1 specified by LSTC_LICENSE_SERVER variable<br />
Program queued for termination<br />
<br />
[roberpj@orc-login2:~] lstc_qrun -R | grep roberpj<br />
[roberpj@orc-login2:~] <br />
</pre><br />
<br />
==Legacy Instructions==<br />
<br />
The following binaries remain available on saw and orca for backward compatibility testing:<br />
<br />
<pre><br />
[roberpj@orc-login2:~] cd /opt/sharcnet/local/lsdyna<br />
[roberpj@orc129:/opt/sharcnet/local/lsdyna] ls ls971*<br />
ls971_d_R3_1 ls971_d_R4_2_1 ls971_s_R3_1 ls971_s_R4_2_1 ls971_s_R5_1_1<br />
ls971_d_R4_2_0 ls971_d_R5_0 ls971_s_R4_2_0 ls971_s_R5_0<br />
</pre><br />
<br />
There are currently no sharcnet modules for these versions, hence jobs should be submitted as follows:<br />
<br />
<pre><br />
module load lsdyna<br />
export PATH=/opt/sharcnet/local/lsdyna:$PATH<br />
export LSTC_LICENSE_SERVER=XXXXX@license3.uwo.sharcnet<br />
cp /opt/sharcnet/local/lsdyna/examples/airbag.deploy.k airbag.deploy.k<br />
SERIAL JOB: sqsub -q serial -r 1d -o ofile.%J ls971_d_R4_2_1 i=airbag.deploy.k<br />
THREADED JOB: sqsub -q threaded -n 4 -r 1d -o ofile.%J ls971_d_R4_2_1 ncpu=4 para=2 i=airbag.deploy.k<br />
</pre><br />
<br />
Note! the <i>ls971_s_R3_1</i> and <i>ls971_d_R3_1</i> binaries do not work, a fix is being looked for.<br />
<br />
=References=<br />
<br />
o LSTC LS-DYNA Homepage<br><br />
http://www.lstc.com/products/ls-dyna<br />
<br />
o LSTC LS-DYNA Support (Tutorials, HowTos, Faq, Manuals, Release Notes, News, Links)<br><br />
http://www.dynasupport.com/<br />
<br />
o LS-DYNA Release Notes<br><br />
http://www.dynasupport.com/release-notes<br />
<br />
o LS-PrePost Online Documentation FAQ<br><br />
http://www.lstc.com/lspp/content/faq.shtml<br />
<br />
o LSTC Download/Install Overview Page<br><br />
Provides links to binaries for LS-DYNA SMP/MPP, LS-OPT, LS-PREPOST, LS-TASC.<br><br />
http://www.lstc.com/download<br />
<br />
==Memory Links==<br />
<br />
o Implicit: Memory notes<br> <br />
http://www.dynasupport.com/howtos/implicit/implicit-memory-notes<br />
<br />
o LS-DYNA Support Environment variables<br><br />
http://www.dynasupport.com/howtos/general/environment-variables<br />
<br />
o LS-DYNA and d3VIEW Blog (a few words on memory settings)<br><br />
http://blog2.d3view.com/a-few-words-on-memory-settings-in-ls-dyna/<br />
<br />
o Convert Words to GB ie) memory=500MW (3.73GB)<br><br />
http://deviceanalytics.com/memcalc.php</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=GEANT4&diff=17943GEANT42019-06-06T14:11:46Z<p>Edward: </p>
<hr />
<div>{{Template:CCMissing}}<br />
{{Software<br />
|package_name=GEANT4<br />
|package_description=Suite of programs for the simulation of the passage of particles through matter<br />
|package_idnumber=59<br />
}}<br />
<br />
=Introduction=<br />
<br />
Geant4 is a toolkit for the simulation of the passage of particles through matter. Its areas of application include high energy, nuclear and accelerator physics, as well as studies in medical and space science. <br />
<br />
=Usage on graham and cedar=<br />
<br />
Geant4 is installed as a module on cedar and graham.<br />
<br />
To see which versions are available, execute:<br />
<br />
module avail geant4<br />
<br />
To then use a particular version, execute for example:<br />
<br />
module load geant4/10.02.p03<br />
<br />
After you load the module, you can see where the software is installed by executing:<br />
<br />
echo $EBROOTGEANT4<br />
<br />
To see the useful examples included with the version we have loaded, execute:<br />
<br />
ls $EBROOTGEANT4/share/Geant4-10.2.3/examples<br />
<br />
Consider the AnaEx01 example as follows:<br />
<br />
mkdir -p ~/samples/geant4/test-100202<br />
cd ~/samples/geant4/test-100202<br />
module load geant4/10.02.p03<br />
cp -r $EBROOTGEANT4/share/Geant4-10.2.3/examples/extended/analysis/AnaEx01 .<br />
source $EBROOTGEANT4/share/Geant4-10.2.3/geant4make/geant4make.sh<br />
cd AnaEx01<br />
mkdir build<br />
cd build<br />
cmake ..<br />
make<br />
<br />
This creates the executable ''AnaEx01'' in the current (build) folder. You can run it by:<br />
<br />
./AnaEx01<br />
<br />
You run this executable and enter "help" for the available actions.<br />
<br />
Once the binary runs interactively as desired you can submit it as a job. (see [https://docs.computecanada.ca/wiki/Running_jobs Running jobs] page).<br />
<br />
=Usage on older SHARCNET systems=<br />
<br />
SHARCNET provides a build of geant4 on sharcnet CentOS clusters .<br />
<br />
Geant4 comes with many examples useful for testing purposes:<br />
<br />
[roberpj@vdi-centos6:/opt/sharcnet/geant4/10.02.02/share/Geant4-10.2.2/examples/extended/analysis] ls <br />
AnaEx01 AnaEx02 AnaEx03 B1Con CMakeLists.txt History README shared<br />
<br />
==Interactive==<br />
<br />
Consider the AnaEx01 example as follows:<br />
<br />
mkdir -p ~/samples/geant4/test-100202<br />
cd ~/samples/geant4/test-100202<br />
module unload intel<br />
module load ldwrapper gcc/4.9.3 geant4/10.02.02<br />
cp -r /opt/sharcnet/geant4/10.02.02/share/Geant4-10.2.2/examples/extended/analysis/AnaEx01 .<br />
source geant4make.sh<br />
cd AnaEx01<br />
gmake clean_setup<br />
gmake setup<br />
gmake<br />
<br />
where ...<br />
[roberpj@vdi-centos6:~/samples/geant4/test-100202/AnaEx01] which AnaEx01 <br />
~/geant4_workdir/bin/Linux-g++/AnaEx01<br />
<br />
To run the binary do:<br />
<br />
[roberpj@vdi-centos6:~/samples/geant4/test-100202/AnaEx01] AnaEx01<br />
Idle> help<br />
Command directory path : /<br />
Sub-directories : <br />
1) /control/ UI control commands.<br />
2) /units/ Available units.<br />
3) /process/ Process Table control commands.<br />
4) /analysis/ ...Title not available...<br />
5) /particle/ Particle control commands.<br />
6) /geometry/ Geometry control commands.<br />
7) /tracking/ TrackingManager and SteppingManager control commands.<br />
8) /event/ EventManager control commands.<br />
9) /cuts/ Commands for G4VUserPhysicsList.<br />
10) /run/ Run control commands.<br />
11) /random/ Random number status control commands.<br />
12) /material/ Commands for materials<br />
13) /N03/ UI commands of this example<br />
14) /physics_lists/ commands related to the physics simulation engine.<br />
15) /gun/ Particle Gun control commands.<br />
16) /heptst/ Controls for the hadronic energy/momentum test<br />
17) /physics_engine/ ...Title not available...<br />
18) /vis/ Visualization commands.<br />
Commands : <br />
Type the number ( 0:end, -n:n level back ) :<br />
<br />
==Cluster==<br />
<br />
Once the binary runs interactively as desired submit it to the serial queue:<br />
<br />
sqsub -q serial -r 5m -o file.%J ~/geant4_workdir/bin/Linux-g++/AnaEx01<br />
<br />
==Graphical==<br />
<br />
None at this time.<br />
<br />
=Notes=<br />
<br />
None at this time.<br />
<br />
=References=<br />
<br />
o Geant4 Homepage<br><br />
http://geant4.web.cern.ch/geant4/<br />
<br />
o Online User Documentation<br><br />
http://geant4.web.cern.ch/geant4/support/userdocuments.shtml<br />
<br />
<!--checked2015--><br />
<!--checked by looking at the directory /opt/sharcnet/geant4 it's still 9.6.1 (on orca)--></div>Edwardhttps://www.sharcnet.ca/help/index.php?title=BOOST&diff=17936BOOST2019-06-06T14:05:42Z<p>Edward: </p>
<hr />
<div>{{Template:CCMissing}}<br />
{{Software<br />
|package_name=BOOST<br />
|package_description=free peer-reviewed portable C++ source libraries<br />
|package_idnumber=137<br />
}}<br />
<br />
{{Template:GrahamUpdate}}<br />
<br />
== Which Versions of Boost are Available? ==<br />
<br />
SHARCNET has a number of Boost versions available as special modules. The available versions can be viewed by running:<br />
<pre><br />
module avail boost<br />
</pre><br />
and/or by viewing the [https://www.sharcnet.ca/my/software/show/137 BOOST software page in the web portal].<br />
<br />
When you look up the available boost modules you will see output similar tossh-:<br />
<pre><br />
$ module avail boost<br />
<br />
------------------------------------------------ /opt/sharcnet/modules -------------------------------------------------<br />
boost/gcc482-openmpi183/1.53.0 boost/gcc510-openmpi187/1.59.0 boost/intel1503-openmpi187/1.59.0<br />
boost/gcc482-openmpi183/1.55.0 boost/gcc510-openmpi187debug/1.59.0 boost/intel1503-openmpi187debug/1.59.0<br />
boost/gcc492-openmpi187/1.59.0 boost/intel12-openmpi162/1.53.0<br />
boost/gcc492-openmpi187debug/1.59.0 boost/intel12-openmpi162/1.55.0<br />
$<br />
</pre><br />
<br />
i.e., the following versions of Boost are available:<br />
# versions 1.53 and 1.55 built with GCC v4.8.2 and OpenMPI v1.8.3<br />
#* i.e., boost/gcc482-openmpi183/1.53.0 and boost/gcc482-openmpi183/1.55.0<br />
# version 1.59 built with GCC v4.9.2 and non-debug and debug versions of OpenMPI v1.8.7<br />
#* i.e.,, boost/gcc492-openmpi187/1.59.0 and boost/gcc492-openmpi187debug/1.59.0<br />
# version 1.59 built with GCC v5.1.0 and non-debug and debug versions of OpenMPI v1.8.7<br />
#* i.e., boost/gcc510-openmpi187/1.59.0 and boost/gcc510-openmpi187debug/1.59.0<br />
# versions 1.53 and 1.55 built with Intel v12 and OpenMPI v1.6.2<br />
#* i.e., boost/intel12-openmpi162/1.53.0 and boost/intel12-openmpi162/1.55.0<br />
# version 1.59 built with Intel v15.0.3 and non-debug and debug versions of OpenMPI v1.8.7<br />
#* i.e., boost/intel1503-openmpi187/1.59.0 and boost/intel1503-openmpi187debug/1.59.0<br />
<br />
'''NOTE:''' You may experience problems compiling Boost codes with the Intel v12 compiler (e.g., some incompatible headers). If you do, try the GCC compiler to see if that works.<br />
<br />
All of the aforementioned versions of Boost have been built with the operating system's installed Python. You can see which version of Python it will use by running:<br />
<pre><br />
rpm -qi python<br />
</pre><br />
<br />
'''NOTE:''' If you need a specific version of Boost compiled for a specific C++ compiler, version of OpenMPI, and/or a specific version of Python, then submit a ticket requesting such. Be sure to mention all required software packages and which versions are needed.<br />
<br />
=== Loading a module ===<br />
<br />
To load a specific module of Boost simply use "module load" followed by the module name, e.g.,<br />
<br />
<pre>module load boost/intel1503-openmpi187debug/1.59.0</pre><br />
<br />
If an error occurs, ensure that you load all pre-requisite modules first. Often it is easier to remove all modules, load the required ldwrapper module (which is needed for your programs to work when submitted using sqsub), and the the modules you need to use, e.g.,<br />
<br />
<pre>$ module purge<br />
$ module load ldwrapper<br />
$ module load gcc/5.1.0<br />
$ module load openmpi/gcc-5.1.0/std/1.8.7<br />
$ module load boost/gcc510-openmpi187/1.59.0</pre><br />
<br />
== Compiling Programs Using CMake With Boost ==<br />
<br />
With CMake you will want code similar to the following to properly load Boost:<br />
<br />
<pre>find_package(<br />
Boost 1.56.0 REQUIRED<br />
COMPONENTS<br />
filesystem<br />
system<br />
)</pre><br />
<br />
where:<br />
<br />
# 1.56.0 is the MINIMUM version that can be used by your program. You should replace this with the version you are using, e.g., 1.59.0. (Know that omitting the minimum version setting can cause issues with CMake not finding Boost.)<br />
# REQUIRED means that if the minimum version is not about to be used, an error will occur and compilation will be aborted. Generally it is a good idea to use REQUIRED to avoid cryptic compilation error messages when an incompatible version of Boost is used.<br />
# If you use any Boost components that have library code that must be linked into your program, you will need to specify COMPONENTS followed by the names of those components. <br />
<br />
If CMake finds the wrong version of boost. Try telling CMake to not use the built in boost system path:<br />
<br />
<pre><br />
set(Boost_NO_SYSTEM_PATHS ON)<br />
</pre><br />
<br />
This will cause CMake to look at the BOOST_ROOT environment variable, instead of the default locations.<br />
<br />
You can view the names of all required-to-be-linked-to-a-library components for the version of Boost you are using by running the following command (after using "module load" to load the version of Boost you want to use):<br />
<br />
<pre>ls -1 $BOOST_ROOT/lib/libboost*{so,a}* | xargs -n 1 basename | sed -e 's/^libboost_//' -e 's/\.so.*$//' -e 's/\.a.*$//' | sort -u</pre><br />
<br />
which will output something similar to this:<br />
<br />
<pre>$ ls -1 $BOOST_ROOT/lib/libboost*{so,a}* | xargs -n 1 basename | sed -e 's/^libboost_//' -e 's/\.so.*$//' -e 's/\.a.*$//' | sort -u<br />
atomic<br />
chrono<br />
context<br />
coroutine<br />
date_time<br />
exception<br />
filesystem<br />
graph<br />
graph_parallel<br />
iostreams<br />
locale<br />
log<br />
log_setup<br />
math_c99<br />
math_c99f<br />
math_c99l<br />
math_tr1<br />
math_tr1f<br />
math_tr1l<br />
mpi<br />
prg_exec_monitor<br />
program_options<br />
python<br />
random<br />
regex<br />
serialization<br />
signals<br />
system<br />
test_exec_monitor<br />
thread<br />
timer<br />
unit_test_framework<br />
wave<br />
wserialization<br />
$</pre><br />
<br />
Some of the library components depend on other components. For example, the "filesystem" component depends on the "system" component --so both of these must be specified in the find_package() CMake function call.<br />
<br />
You also will need lines similar to the follwing in your CMakeLists.txt:<br />
<br />
<pre>add_executable(<br />
program.exe<br />
src/somecode.cxx<br />
)<br />
<br />
target_link_libraries(<br />
program.exe<br />
${Boost_LIBRARIES}<br />
)</pre><br />
<br />
where:<br />
<br />
# program.exe is the executable being produced.<br />
# ${Boost_LIBRARIES} is set by the find_package() function and contains all of the libraries it discovered (for the listed COMPONENTS section above).<br />
<br />
'''NOTE:''' With the target_link_libraries() set, you do not need or want to set LD_LIBRARY_PATH at all for Boost to work, i.e., if you are loading the SHARCNET module, everything is properly set. (With CMake you must tell it which libraries must be linked in to your executable --it is not automatic.)<br />
<br />
== Using a Specific Version of Boost ==<br />
<br />
To use a specific version of Boost, do the following:<br />
* Unload any loaded OpenMPI modules that will conflict.<br />
* Unload any loaded C++ compiler modules that will conflict.<br />
* Load any required C++ and OpenMPI modules.<br />
* Load the required version of Boost.<br />
e.g.,<br />
<pre><br />
$ module list # View which modules are currently loaded<br />
$ module unload openmpi/intel/1.6.2 # Unload the OpenMPI (for Intel)<br />
$ module unload intel/12.1.3 # Unload the Intel C++ compiler<br />
$ module load gcc/4.8.2 # Load GCC v4.8.2<br />
$ module load openmpi/gcc/1.8.3 # Load OpenMPI (for GCC)<br />
$ module load boost/gcc482-openmpi183/1.57.0 # Load Boost v1.57 for GCC 4.8.2 and OpenMPI v1.8.3<br />
</pre><br />
<br />
Once this is done, you may use Boost (e.g., write #include <boost/filesystem.hpp> in your code, etc.). No special compiler options will be required.<br />
<br />
If you want to start with NO modules loaded, then you can do the same by purging first, loading ldwrapper, followed by what you need, e.g.,<br />
<pre><br />
$ module purge<br />
$ module load ldwrapper<br />
$ module load gcc/4.8.2 # Load GCC v4.8.2<br />
$ module load openmpi/gcc/1.8.3 # Load OpenMPI (for GCC)<br />
$ module load boost/gcc482-openmpi183/1.57.0 # Load Boost v1.57 for GCC 4.8.2 and OpenMPI v1.8.3<br />
</pre><br />
<br />
== If Special Compiler Options are Required ==<br />
<br />
If you are not having success building your code using Boost (e.g., in a Makefile), you might need to manually specify the INCLUDE and LIB directories. You can get these paths by running the following command (replacing the full Boost module name with the one you've loaded), e.g.,<br />
<pre><br />
$ module show boost/gcc482-openmpi183/1.57.0<br />
-------------------------------------------------------------------<br />
/opt/sharcnet/modules/boost/gcc482-openmpi183/1.57.0:<br />
<br />
module-whatis Provides boost 1.57.0 (flavor gcc482-openmpi183) built using gcc/4.8.2, openmpi/gcc/1.8.3 on centos64.<br />
conflict intel<br />
conflict pgi<br />
conflict python<br />
prereq gcc/4.8.2<br />
prereq openmpi/gcc/1.8.3<br />
prepend-path --delim CPPFLAGS -I/opt/sharcnet/boost/1.57.0/gcc482-openmpi183/include<br />
prepend-path --delim LDFLAGS -L/opt/sharcnet/boost/1.57.0/gcc482-openmpi183/lib<br />
prepend-path LD_RUN_PATH /opt/sharcnet/boost/1.57.0/gcc482-openmpi183/lib<br />
-------------------------------------------------------------------<br />
$<br />
</pre><br />
i.e., for Boost 1.57.0 (flavour gcc482-openmpi183) you could explicitly invoke the compiler with:<br />
<pre><br />
g++ $CPPFLAGS $CXXFLAGS $LDFLAGS yoursourcecode.cxx<br />
</pre><br />
<br />
Also be aware that you must specify any Boost-required libraries for linking purposes using the -l option. You can see the actual available library names for any given Boost installation by running:<br />
<br />
<pre><br />
$ for a in `ls $BLIBS` ; do basename $a ; done<br />
</pre><br />
<br />
where $BLIBS is the directory you see with LDFLAGS in the module show output, thus, to see all libraries available with the gcc482-openmpi183/lib module, you would write:<br />
<br />
<pre><br />
$ for a in `ls /opt/sharcnet/boost/1.57.0/gcc482-openmpi183/lib/libboost_*` ; do basename $a ; done<br />
</pre><br />
<br />
'''NOTE:''' Using the -l option is important when using any non-header-only Boost library, e.g., MPI, serialization, program_options, etc.<br />
<br />
<br />
== Building an Example Boost Program ==<br />
<br />
First create/write/copy your program somewhere suitable (this example is from the Boost.MPI documentation):<br />
<pre><br />
$ cd<br />
$ mkdir boost-test<br />
$ cat >boost-mpi-eg.cxx<br />
#include <iostream><br />
#include <string><br />
#include <boost/mpi.hpp><br />
#include <boost/serialization/string.hpp><br />
<br />
namespace mpi = boost::mpi;<br />
<br />
int main(int argc, char *argv[])<br />
{<br />
mpi::environment env(argc, argv);<br />
mpi::communicator world;<br />
<br />
if (world.rank() == 0) {<br />
world.send(1, 0, std::string("Hello"));<br />
std::string msg;<br />
world.recv(1, 1, msg);<br />
std::cout << msg << "!" << std::endl;<br />
} else {<br />
std::string msg;<br />
world.recv(0, 0, msg);<br />
std::cout << msg << ", ";<br />
std::cout.flush();<br />
world.send(0, 1, std::string("world"));<br />
}<br />
<br />
return 0;<br />
}<br />
$<br />
</pre><br />
compile it:<br />
<pre><br />
$ mpic++ $CPPFLAGS $CXXFLAGS $LDFLAGS -lboost_mpi -lboost_serialization boost-mpi-eg.cxx<br />
</pre><br />
and run it (as it is very fast and short, sqsub is not used here):<br />
<pre><br />
$ mpirun -np 2 ./a.out<br />
</pre><br />
<br />
== A Header-Only Example ==<br />
<br />
Many libraries in Boost are header-only (i.e., the program does not need to be linked to a library). With header-only libraries one does not specify any linking options when compiling/linking the code.<br />
<br />
Consider the following Boost.Multiprecision program:<br />
<br />
<pre><br />
// This program's code combines code from the following Boost.Multiprecision documentation's pages:<br />
// 1) http://www.boost.org/doc/libs/1_63_0/libs/multiprecision/doc/html/boost_multiprecision/tut/ints/cpp_int.html<br />
// 2) http://www.boost.org/doc/libs/1_63_0/libs/multiprecision/doc/html/boost_multiprecision/tut/floats/cpp_bin_float.html<br />
// 3) http://www.boost.org/doc/libs/1_63_0/libs/multiprecision/doc/html/boost_multiprecision/tut/floats/cpp_dec_float.html<br />
<br />
#include &lt;boost/multiprecision/cpp_int.hpp&gt; <br />
#include &lt;boost/multiprecision/cpp_bin_float.hpp&gt;<br />
#include &lt;boost/multiprecision/cpp_dec_float.hpp&gt; <br />
#include &lt;boost/math/special_functions/gamma.hpp&gt;<br />
#include &lt;iostream&gt;<br />
<br />
void mp_int()<br />
{<br />
// For integer type docs see:<br />
// http://www.boost.org/doc/libs/1_63_0/libs/multiprecision/doc/html/boost_multiprecision/tut/ints.html<br />
<br />
// Example Boost docs code follows...<br />
<br />
using namespace boost::multiprecision;<br />
<br />
// Do some fixed precision arithmetic:<br />
int128_t v = 1;<br />
for (unsigned i = 1; i &lt;= 20; ++i)<br />
v *= i;<br />
std::cout &lt;&lt; v &lt;&lt; std::endl; // prints 20<br />
<br />
// Repeat at arbitrary precision:<br />
cpp_int u = 1;<br />
for (unsigned i = 1; i &lt;= 100; ++i)<br />
u *= i;<br />
std::cout &lt;&lt; u &lt;&lt; std::endl; // prints 100<br />
}<br />
<br />
void mp_bin_float()<br />
{<br />
// For floating-point type docs see:<br />
// http://www.boost.org/doc/libs/1_63_0/libs/multiprecision/doc/html/boost_multiprecision/tut/floats.html<br />
<br />
// Example Boost docs code follows...<br />
<br />
using namespace boost::multiprecision;<br />
<br />
// Operations at fixed precision and full numeric_limits support:<br />
cpp_bin_float_100 b = 2;<br />
std::cout &lt;&lt; std::numeric_limits&lt;cpp_bin_float_100&gt;::digits &lt;&lt; std::endl;<br />
std::cout &lt;&lt; std::numeric_limits&lt;cpp_bin_float_100&gt;::digits10 &lt;&lt; std::endl;<br />
<br />
// We can use any C++ std lib function, lets print all the digits as well:<br />
std::cout &lt;&lt; std::setprecision(std::numeric_limits&lt;cpp_bin_float_100&gt;::max_digits10)<br />
&lt;&lt; log(b) &lt;&lt; std::endl; // print log(2)<br />
<br />
// We can also use any function from Boost.Math:<br />
std::cout &lt;&lt; boost::math::tgamma(b) &lt;&lt; std::endl;<br />
<br />
// These even work when the argument is an expression template:<br />
std::cout &lt;&lt; boost::math::tgamma(b * b) &lt;&lt; std::endl;<br />
<br />
// And since we have an extended exponent range we can generate some really large <br />
// numbers here (4.0238726007709377354370243e+2564):<br />
std::cout &lt;&lt; boost::math::tgamma(cpp_bin_float_100(1000)) &lt;&lt; std::endl;<br />
}<br />
<br />
void mp_dec_float()<br />
{<br />
using namespace boost::multiprecision;<br />
<br />
// Operations at fixed precision and full numeric_limits support:<br />
cpp_dec_float_100 b = 2;<br />
std::cout &lt;&lt; std::numeric_limits&lt;cpp_dec_float_100&gt;::digits &lt;&lt; std::endl;<br />
<br />
// Note that digits10 is the same as digits, since we&apos;re base 10! :<br />
std::cout &lt;&lt; std::numeric_limits&lt;cpp_dec_float_100&gt;::digits10 &lt;&lt; std::endl;<br />
<br />
// We can use any C++ std lib function, lets print all the digits as well:<br />
std::cout &lt;&lt; std::setprecision(std::numeric_limits&lt;cpp_dec_float_100&gt;::max_digits10)<br />
&lt;&lt; log(b) &lt;&lt; std::endl; // print log(2)<br />
<br />
// We can also use any function from Boost.Math:<br />
std::cout &lt;&lt; boost::math::tgamma(b) &lt;&lt; std::endl;<br />
<br />
// These even work when the argument is an expression template:<br />
std::cout &lt;&lt; boost::math::tgamma(b * b) &lt;&lt; std::endl;<br />
<br />
// And since we have an extended exponent range we can generate some really large <br />
// numbers here (4.0238726007709377354370243e+2564):<br />
std::cout &lt;&lt; boost::math::tgamma(cpp_dec_float_100(1000)) &lt;&lt; std::endl;<br />
}<br />
<br />
<br />
int main()<br />
{<br />
mp_int();<br />
mp_bin_float();<br />
mp_dec_float();<br />
}<br />
</pre><br />
<br />
It be compiled and run using GCC v6.3.0 as follows:<br />
<br />
<pre><br />
$ module unload gcc intel openmpi mkl boost<br />
$ module load gcc/6.3.0 boost/gcc630/1.63.0<br />
$ g++ -I$BOOST_ROOT/include boost-multiprecision.cxx<br />
$ ./a.out<br />
</pre><br />
<br />
or with Intel v17.0.1:<br />
<br />
<pre><br />
$ module unload gcc intel openmpi mkl boost<br />
$ module load intel/17.0.1 boost/intel1701/1.63.0<br />
$ icpc -I$BOOST_ROOT/include boost-multiprecision.cxx<br />
$ ./a.out<br />
</pre><br />
<br />
== Operating System Installed Boost ==<br />
<br />
If you don't load a SHARCNET boost module, your program will most likely link to the operating system's installed version of Boost. To determine which version(s) of boost and boost-devel are installed run the following commands:<br />
<pre><br />
rpm -qi boost<br />
rpm -qi boost-devel<br />
</pre><br />
<br />
'''NOTE:''' System versions are rather old and won't be upgraded or maintained by SHARCNET staff unless something in the operating system requires it to be updated. Therefore we '''strongly encourag''' using the SHARCNET provided Boost modules discussed above on this page.<br />
<br />
==References==<br />
<br />
o Boost Homepage<br><br />
http://www.boost.org/</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=Template:CCDelete&diff=17933Template:CCDelete2019-06-06T14:02:20Z<p>Edward: </p>
<hr />
<div>{| cellpadding=5 style="border:4px solid #BF4513"<br />
|- bgcolor="#FFF8DC"<br />
| This page is scheduled for deletion because it is either redundant with information available on the CC wiki, or the software is no longer supported.<br />
|}<br />
__TOC__</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=Template:CCMissing&diff=17931Template:CCMissing2019-06-06T13:59:11Z<p>Edward: Created page with "{| cellpadding=5 style="border:4px solid #BF4513" |- bgcolor="#FFF8DC" | This page is will be deleted pending it's creation on the CC wiki. |} __TOC__"</p>
<hr />
<div>{| cellpadding=5 style="border:4px solid #BF4513"<br />
|- bgcolor="#FFF8DC"<br />
| This page is will be deleted pending it's creation on the CC wiki.<br />
|}<br />
__TOC__</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=Template:CCDelete&diff=17930Template:CCDelete2019-06-06T13:57:13Z<p>Edward: Created page with "{| cellpadding=5 style="border:4px solid #BF4513" |- bgcolor="#FFF8DC" | This page is scheduled for deletion because it is redundant with information available on the CC wiki...."</p>
<hr />
<div>{| cellpadding=5 style="border:4px solid #BF4513"<br />
|- bgcolor="#FFF8DC"<br />
| This page is scheduled for deletion because it is redundant with information available on the CC wiki.<br />
|}<br />
__TOC__</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=Template:GrahamMissing&diff=17928Template:GrahamMissing2019-06-06T13:55:17Z<p>Edward: Created page with "{| cellpadding=5 style="border:4px solid #BF4513" |- bgcolor="#FFF8DC" | This page is will be deleted pending it's creation on the CC wiki. |} __TOC__"</p>
<hr />
<div>{| cellpadding=5 style="border:4px solid #BF4513"<br />
|- bgcolor="#FFF8DC"<br />
| This page is will be deleted pending it's creation on the CC wiki.<br />
|}<br />
__TOC__</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=Code_Repositories&diff=17742Code Repositories2019-05-06T14:05:45Z<p>Edward: Created page with "=Code repositories= The code for (most) dedicated programming projects is stored in the SHARCNET [http://www.sharcnet.ca/git git repository]. What follows is some details re..."</p>
<hr />
<div>=Code repositories=<br />
<br />
The code for (most) dedicated programming projects is stored in the SHARCNET [http://www.sharcnet.ca/git git repository]. What follows is some details regarding using git with these repositories. Please refer to our [[Git|git page]] for general information on using git.<br />
<br />
==Certificate problems==<br />
<br />
If you are seeing error messages like ''error: server certificate verification failed ...'' while performing a clone, push, fetch, or pull, your system cannot verify the authenticity of ''www.sharcnet.ca''. This is caused by either not connecting directly to ''www.sharcnet.ca'' (as when using ssh port tunneling as described below) or by your SSL stack having dropped support for the [http://www.win.tue.nl/hashclash/rogue-ca deprecated MD5] signatures (as with GnuTLS for SSL under Debian).<br />
<br />
In either case, you will have to turn off git's SSL verification by setting the environment variable ''GIT_SSL_NO_VERIFY=1'' or running the command<br />
<br />
git config http.sslVerify false<br />
<br />
inside your repository (the former must be used for cloning as the repository won't exist yet).<br />
<br />
Our current certificate expires on March 10, 2012. If you have disabled SSL verification due to the MD5 issue, you should re-enable it at this point as we will have purchased a new certificate without this issue.<br />
<br />
==Dropped connections==<br />
<br />
If you are periodically seeing error messages like ''Cannot obtain needed blob ...'' while performing a clone, push, fetch, or pull, an intermediate network is occasionally dropping your connections. This has been a known problem on at least the Western campus.<br />
<br />
A solution is to use SSH to create a tunnel across the intermediate network to the SHARCNET network.<br />
<br />
ssh -N -L <port>:www.sharcnet.ca:443 <user>@<cluster><br />
<br />
where ''<port>'' is a free port (such as 4000), ''<user>'' is your SHARCNET user name, and ''<cluster>'' is any of the SHARCNET clusters. While this ssh session is running, connections to ''localhost:<port>'' will be forwarded ''www.sharcnet.ca:443'' via ''<cluster>''. For example<br />
<br />
GIT_SSL_NO_VERIFY=1 git clone https://localhost:<port>/<project><br />
<br />
will check out ''<project>'' over the tunnel.<br />
<br />
The ''GIT_SSL_NO_VERIFY'' environment variable is required as SSL verification will otherwise fail due to the port forwarding. Once you have checked out the project, you can permanently disable SSL verification by running<br />
<br />
git config http.sslVerify false<br />
<br />
in the project directory.<br />
<br />
Note that every time you access the SHARCNET git server (i.e., do a clone, push, fetch, or pull), you need to have the ssh tunnel running. If you try and access the remote without the ssh tunnel running, you will get an ''error: couldn't connect to host while accessing ...'' message.</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=Deploying_a_web_server&diff=16578Deploying a web server2018-05-29T18:48:49Z<p>Edward: /* Use self signed certificate */</p>
<hr />
<div>=Dependencies - Launch Cloud Instance=<br />
Before deploying a web server, first a cloud instance must be lauched. The instruction to do this can be found [[Carpet|here]].<br />
For the remainder of the instructions the ip address used is 199.241.164.95, this is for demonstration purposes and you should replace<br />
it with your assigned floating IP address.<br />
<br />
# login to cloud.sharcnet.ca<br />
# Don’t forget volume size<br />
# Choose Debian 9.2.2 (not required, but the remaining instructions are Debian centric).<br />
# Choose persistent (ephemeral is for shorter jobs) (4C-8GB)<br />
# Setup keypair<br />
# Associate floating ip<br />
<br />
==(Optional) Apply IP Address to your name server==<br />
If you have a registered domain name you should apply your floating IP address to it so that you can use the "let's encrypt" service to <br />
enable the secure socket layer without client side warnings. It is often best to do this at the beginning as there is typically a delay<br />
in updating the name service.<br />
<br />
= Server Setup =<br />
<br />
==Login to Server & perform preliminaries==<br />
At this point log into the server to ensure that the service is up and running. There are a number of steps you can perform that will make <br />
the remaining steps easier. First loging to the server. Then setup a user prompt to make navigation easier. Switch to root (or use sudo in<br />
from of the remaining commands). Update and upgrade the system. Last install the manual pages.<br />
$ ssh debian@199.241.164.95<br />
$ echo 'export PS1="\[\e[33m\]\w\[\e[0m\]\n\[\e[32m\]\u@\h$ \[\e[0m\]"' >> .bash_aliases<br />
$ sudo su root<br />
$ apt update<br />
$ apt upgrade<br />
$ apt install man<br />
<br />
==Apt error==<br />
If you receive the "apt error" mesage put "nameserver 8.8.8.8" in /etc/resolve.conf.<br />
$ echo 'nameserver 8.8.8.8 >>' /etc/resolve.conf<br />
<br />
==Secure the SSH login==<br />
The following changes will disable logging into the server by using a password on any account and prevent logging into root remotly. Root can<br />
still be accessed by logging into a ''sudo'' enabled account and using the command ''su root''. For more information on the sshd_config file<br />
options, go [https://man.openbsd.org/sshd_config here]. The unattended-upgrades package is used to keep they server up to date automatically.<br />
$ sudo vim /etc/ssh/sshd_config<br />
ChallengeResponseAuthentication no<br />
PasswordAuthentication no<br />
PermitRootLogin no<br />
$ service ssh reload<br />
$ apt install unattended-upgrades<br />
$ dpkg-reconfigure --priority=low unattended-upgrades<br />
$ sudo unattended-upgrade -d<br />
<br />
=Install webserver and suppporting packages=<br />
==MYSQL==<br />
$ apt install mysql-server<br />
$ mysql_secure_installation #password=password<br />
<br />
==Apache==<br />
Installing the Apache webserver will allow you to access a default page at your ip address. (ex http://199.241.164.95, or http://frar.ca).<br />
Note that https will not yet work. httpd is the Apache HyperText Transfer Protocol (HTTP) server program. It is designed to be run as a standalone daemon process. When used like this it will create a pool of child processes or threads to handle requests. In general, httpd should not be invoked directly, but rather should be invoked via apachectl on Unix-based systems or as a service on Windows NT, 2000 and XP and as a console application on Windows 9x and ME.<br />
<br />
$ apt install apache2<br />
$ service apache2 start #add 80, 443 to default security group rules<br />
$ a2enmod cgid.load #(optional) enable cgi scripting, a2dismod to remove<br />
<br />
===Basic information & settings===<br />
# Document root Directory: /var/www/html or /var/www<br />
# Main Configuration file:<br />
## /etc/httpd/conf/httpd.conf (RHEL/CentOS/Fedora) <br />
## /etc/apache2/apache2.conf (Debian/Ubuntu).<br />
# Default HTTP Port: 80 TCP<br />
# Default HTTPS Port: 443 TCP<br />
# Access Log files of Web Server: /var/log/apache2/access_log<br />
# Error Log files of Web Server: /var/log/apache2/error_log<br />
# service apache2 {start|stop|graceful-stop|restart|reload|force-reload}<br />
# apachectl -v <br />
## [https://httpd.apache.org/docs/2.4/programs/apachectl.html see]<br />
<br />
===Secure Apache with SSL certificates===<br />
Going to https is now possible but will trigger a warning.<br />
$ a2enmod ssl<br />
$ a2ensite default-ssl.conf<br />
$ service apache2 restart<br />
<br />
===Enable let's encrypt (requires a domain name)===<br />
$ apt install git<br />
$ sudo git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt<br />
$ cd /opt/letsencrypt<br />
$ ./certbot-auto --authenticator webroot --installer apache<br />
www.frar.ca<br />
/var/www/html<br />
<br />
===Use self signed certificate [https://wiki.debian.org/Self-Signed_Certificate]===<br />
$ mkdir /etc/ssl/localcerts<br />
$ openssl req -new -x509 -days 365 -nodes -out /etc/ssl/localcerts/apache.pem -keyout /etc/ssl/localcerts/apache.key<br />
$ chmod 600 /etc/ssl/localcerts/apache*<br />
$ cd /etc/apache2/sites-available/<br />
$ cp default-ssl.conf my-ssl.conf<br />
$ vim my-ssl.conf<br />
NameVirtualHost *:443<br />
<VirtualHost *:443><br />
SSLEngine On<br />
SSLCertificateFile /etc/ssl/localcerts/apache.pem<br />
SSLCertificateKeyFile /etc/ssl/localcerts/apache.key<br />
<br />
$ a2ensite my-ssl.conf<br />
<br />
===Add basic user-password verfication===<br />
$ mkdir /var/www/passwd<br />
$ htpasswd -c /var/www/passwd/passwords user<br />
$ vim /etc/apache2/apache2.conf<br />
<Directory /var/www/html><br />
Require valid-user<br />
AuthType basic<br />
AuthName "Restricted Files"<br />
AuthUserFile "/var/www/passwd/passwords"<br />
</Directory><br />
$ service apache2 restart<br />
<br />
==Helper packages==<br />
===Install PHP===<br />
$ apt -y install php7.0 libapache2-mod-php7.0 php7.0-mysql php7.0-gd php7.0-opcache<br />
$ echo ‘<?php phpinfo(); ?>’ > /var/www/html/test.php<br />
<br />
===Give www-data an ssh key and ownership of home directory===<br />
$ chmod 774 /var/www/html<br />
$ chown www-data:www-data -R /var/www<br />
$ mkdir /var/www/.ssh<br />
$ chown www-data:www-data /var/www/.ssh<br />
$ sudo -u www-data ssh-keygen -C www-data #no passphrase for now<br />
<br />
==SSH Fuse==<br />
Give “debian” user www-data group so that files can be uploaded to the web server<br />
$ sudo usermod -aG www-data debian<br />
<br />
====Copy the ssh key to graham====<br />
(lamp) $ sudo cp .ssh/id_rsa.pub ~<br />
(local) $ scp -3 cloud:~/id_rsa.pub graham:~ # the -3 disables host to host copying<br />
(graham)$ cat id_rsa.pub >> authorized_keys<br />
<br />
====Setup Graham (placeholder section)====<br />
$ ssh graham <br />
$ mkdir ~/project/edward/linked-cloud<br />
$ cd linked-cloud<br />
$ wget https://git.sharcnet.ca/edward/CSVSorterDemo/raw/master/CSVSorter.jar<br />
$ mkdir input<br />
$ mkdir output<br />
$ vim submitjob.sh<br />
#!/bin/bash<br />
#SBATCH -t 0-00:01<br />
#SBATCH --mem=4G<br />
#SBATCH -A def-edward<br />
java -Xmx10m -jar /home/edward/project/edward/linked-cloud/CSVSorter.jar \<br />
/home/edward/project/edward/linked-cloud/input/$1 \<br />
/home/edward/project/edward/linked-cloud/output/$1<br />
<br />
====Link LAMP server with Graham through sshfuse====<br />
$ ssh cloud<br />
$ apt install sshfs<br />
$ cd /var/www<br />
$ sudo -u www-data mkdir linked-graham<br />
$ sudo -u www-data sshfs edward@graham.sharcnet.ca:/home/edward/project/linked-cloud /var/www/linked-graham<br />
- (to unmount) fusermount -u ./project</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=Deploying_a_web_server&diff=16577Deploying a web server2018-05-29T18:34:24Z<p>Edward: /* Use self signed certificate */</p>
<hr />
<div>=Dependencies - Launch Cloud Instance=<br />
Before deploying a web server, first a cloud instance must be lauched. The instruction to do this can be found [[Carpet|here]].<br />
For the remainder of the instructions the ip address used is 199.241.164.95, this is for demonstration purposes and you should replace<br />
it with your assigned floating IP address.<br />
<br />
# login to cloud.sharcnet.ca<br />
# Don’t forget volume size<br />
# Choose Debian 9.2.2 (not required, but the remaining instructions are Debian centric).<br />
# Choose persistent (ephemeral is for shorter jobs) (4C-8GB)<br />
# Setup keypair<br />
# Associate floating ip<br />
<br />
==(Optional) Apply IP Address to your name server==<br />
If you have a registered domain name you should apply your floating IP address to it so that you can use the "let's encrypt" service to <br />
enable the secure socket layer without client side warnings. It is often best to do this at the beginning as there is typically a delay<br />
in updating the name service.<br />
<br />
= Server Setup =<br />
<br />
==Login to Server & perform preliminaries==<br />
At this point log into the server to ensure that the service is up and running. There are a number of steps you can perform that will make <br />
the remaining steps easier. First loging to the server. Then setup a user prompt to make navigation easier. Switch to root (or use sudo in<br />
from of the remaining commands). Update and upgrade the system. Last install the manual pages.<br />
$ ssh debian@199.241.164.95<br />
$ echo 'export PS1="\[\e[33m\]\w\[\e[0m\]\n\[\e[32m\]\u@\h$ \[\e[0m\]"' >> .bash_aliases<br />
$ sudo su root<br />
$ apt update<br />
$ apt upgrade<br />
$ apt install man<br />
<br />
==Apt error==<br />
If you receive the "apt error" mesage put "nameserver 8.8.8.8" in /etc/resolve.conf.<br />
$ echo 'nameserver 8.8.8.8 >>' /etc/resolve.conf<br />
<br />
==Secure the SSH login==<br />
The following changes will disable logging into the server by using a password on any account and prevent logging into root remotly. Root can<br />
still be accessed by logging into a ''sudo'' enabled account and using the command ''su root''. For more information on the sshd_config file<br />
options, go [https://man.openbsd.org/sshd_config here]. The unattended-upgrades package is used to keep they server up to date automatically.<br />
$ sudo vim /etc/ssh/sshd_config<br />
ChallengeResponseAuthentication no<br />
PasswordAuthentication no<br />
PermitRootLogin no<br />
$ service ssh reload<br />
$ apt install unattended-upgrades<br />
$ dpkg-reconfigure --priority=low unattended-upgrades<br />
$ sudo unattended-upgrade -d<br />
<br />
=Install webserver and suppporting packages=<br />
==MYSQL==<br />
$ apt install mysql-server<br />
$ mysql_secure_installation #password=password<br />
<br />
==Apache==<br />
Installing the Apache webserver will allow you to access a default page at your ip address. (ex http://199.241.164.95, or http://frar.ca).<br />
Note that https will not yet work. httpd is the Apache HyperText Transfer Protocol (HTTP) server program. It is designed to be run as a standalone daemon process. When used like this it will create a pool of child processes or threads to handle requests. In general, httpd should not be invoked directly, but rather should be invoked via apachectl on Unix-based systems or as a service on Windows NT, 2000 and XP and as a console application on Windows 9x and ME.<br />
<br />
$ apt install apache2<br />
$ service apache2 start #add 80, 443 to default security group rules<br />
$ a2enmod cgid.load #(optional) enable cgi scripting, a2dismod to remove<br />
<br />
===Basic information & settings===<br />
# Document root Directory: /var/www/html or /var/www<br />
# Main Configuration file:<br />
## /etc/httpd/conf/httpd.conf (RHEL/CentOS/Fedora) <br />
## /etc/apache2/apache2.conf (Debian/Ubuntu).<br />
# Default HTTP Port: 80 TCP<br />
# Default HTTPS Port: 443 TCP<br />
# Access Log files of Web Server: /var/log/apache2/access_log<br />
# Error Log files of Web Server: /var/log/apache2/error_log<br />
# service apache2 {start|stop|graceful-stop|restart|reload|force-reload}<br />
# apachectl -v <br />
## [https://httpd.apache.org/docs/2.4/programs/apachectl.html see]<br />
<br />
===Secure Apache with SSL certificates===<br />
Going to https is now possible but will trigger a warning.<br />
$ a2enmod ssl<br />
$ a2ensite default-ssl.conf<br />
$ service apache2 restart<br />
<br />
===Enable let's encrypt (requires a domain name)===<br />
$ apt install git<br />
$ sudo git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt<br />
$ cd /opt/letsencrypt<br />
$ ./certbot-auto --authenticator webroot --installer apache<br />
www.frar.ca<br />
/var/www/html<br />
<br />
===Use self signed certificate===<br />
$ mkdir /etc/ssl/localcerts<br />
$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/apache-selfsigned.key -out /etc/ssl/certs/apache-selfsigned.crt<br />
$ vim /etc/apache2/sites-enabled/default-ssl.conf<br />
SSLCertificateFile /etc/ssl/certs/apache-selfsigned.crt<br />
SSLCertificateKeyFile /etc/ssl/private/apache-selfsigned.key<br />
<br />
===Add basic user-password verfication===<br />
$ mkdir /var/www/passwd<br />
$ htpasswd -c /var/www/passwd/passwords user<br />
$ vim /etc/apache2/apache2.conf<br />
<Directory /var/www/html><br />
Require valid-user<br />
AuthType basic<br />
AuthName "Restricted Files"<br />
AuthUserFile "/var/www/passwd/passwords"<br />
</Directory><br />
$ service apache2 restart<br />
<br />
==Helper packages==<br />
===Install PHP===<br />
$ apt -y install php7.0 libapache2-mod-php7.0 php7.0-mysql php7.0-gd php7.0-opcache<br />
$ echo ‘<?php phpinfo(); ?>’ > /var/www/html/test.php<br />
<br />
===Give www-data an ssh key and ownership of home directory===<br />
$ chmod 774 /var/www/html<br />
$ chown www-data:www-data -R /var/www<br />
$ mkdir /var/www/.ssh<br />
$ chown www-data:www-data /var/www/.ssh<br />
$ sudo -u www-data ssh-keygen -C www-data #no passphrase for now<br />
<br />
==SSH Fuse==<br />
Give “debian” user www-data group so that files can be uploaded to the web server<br />
$ sudo usermod -aG www-data debian<br />
<br />
====Copy the ssh key to graham====<br />
(lamp) $ sudo cp .ssh/id_rsa.pub ~<br />
(local) $ scp -3 cloud:~/id_rsa.pub graham:~ # the -3 disables host to host copying<br />
(graham)$ cat id_rsa.pub >> authorized_keys<br />
<br />
====Setup Graham (placeholder section)====<br />
$ ssh graham <br />
$ mkdir ~/project/edward/linked-cloud<br />
$ cd linked-cloud<br />
$ wget https://git.sharcnet.ca/edward/CSVSorterDemo/raw/master/CSVSorter.jar<br />
$ mkdir input<br />
$ mkdir output<br />
$ vim submitjob.sh<br />
#!/bin/bash<br />
#SBATCH -t 0-00:01<br />
#SBATCH --mem=4G<br />
#SBATCH -A def-edward<br />
java -Xmx10m -jar /home/edward/project/edward/linked-cloud/CSVSorter.jar \<br />
/home/edward/project/edward/linked-cloud/input/$1 \<br />
/home/edward/project/edward/linked-cloud/output/$1<br />
<br />
====Link LAMP server with Graham through sshfuse====<br />
$ ssh cloud<br />
$ apt install sshfs<br />
$ cd /var/www<br />
$ sudo -u www-data mkdir linked-graham<br />
$ sudo -u www-data sshfs edward@graham.sharcnet.ca:/home/edward/project/linked-cloud /var/www/linked-graham<br />
- (to unmount) fusermount -u ./project</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=Deploying_a_web_server&diff=16576Deploying a web server2018-05-29T18:28:17Z<p>Edward: /* Use self signed certificate */</p>
<hr />
<div>=Dependencies - Launch Cloud Instance=<br />
Before deploying a web server, first a cloud instance must be lauched. The instruction to do this can be found [[Carpet|here]].<br />
For the remainder of the instructions the ip address used is 199.241.164.95, this is for demonstration purposes and you should replace<br />
it with your assigned floating IP address.<br />
<br />
# login to cloud.sharcnet.ca<br />
# Don’t forget volume size<br />
# Choose Debian 9.2.2 (not required, but the remaining instructions are Debian centric).<br />
# Choose persistent (ephemeral is for shorter jobs) (4C-8GB)<br />
# Setup keypair<br />
# Associate floating ip<br />
<br />
==(Optional) Apply IP Address to your name server==<br />
If you have a registered domain name you should apply your floating IP address to it so that you can use the "let's encrypt" service to <br />
enable the secure socket layer without client side warnings. It is often best to do this at the beginning as there is typically a delay<br />
in updating the name service.<br />
<br />
= Server Setup =<br />
<br />
==Login to Server & perform preliminaries==<br />
At this point log into the server to ensure that the service is up and running. There are a number of steps you can perform that will make <br />
the remaining steps easier. First loging to the server. Then setup a user prompt to make navigation easier. Switch to root (or use sudo in<br />
from of the remaining commands). Update and upgrade the system. Last install the manual pages.<br />
$ ssh debian@199.241.164.95<br />
$ echo 'export PS1="\[\e[33m\]\w\[\e[0m\]\n\[\e[32m\]\u@\h$ \[\e[0m\]"' >> .bash_aliases<br />
$ sudo su root<br />
$ apt update<br />
$ apt upgrade<br />
$ apt install man<br />
<br />
==Apt error==<br />
If you receive the "apt error" mesage put "nameserver 8.8.8.8" in /etc/resolve.conf.<br />
$ echo 'nameserver 8.8.8.8 >>' /etc/resolve.conf<br />
<br />
==Secure the SSH login==<br />
The following changes will disable logging into the server by using a password on any account and prevent logging into root remotly. Root can<br />
still be accessed by logging into a ''sudo'' enabled account and using the command ''su root''. For more information on the sshd_config file<br />
options, go [https://man.openbsd.org/sshd_config here]. The unattended-upgrades package is used to keep they server up to date automatically.<br />
$ sudo vim /etc/ssh/sshd_config<br />
ChallengeResponseAuthentication no<br />
PasswordAuthentication no<br />
PermitRootLogin no<br />
$ service ssh reload<br />
$ apt install unattended-upgrades<br />
$ dpkg-reconfigure --priority=low unattended-upgrades<br />
$ sudo unattended-upgrade -d<br />
<br />
=Install webserver and suppporting packages=<br />
==MYSQL==<br />
$ apt install mysql-server<br />
$ mysql_secure_installation #password=password<br />
<br />
==Apache==<br />
Installing the Apache webserver will allow you to access a default page at your ip address. (ex http://199.241.164.95, or http://frar.ca).<br />
Note that https will not yet work. httpd is the Apache HyperText Transfer Protocol (HTTP) server program. It is designed to be run as a standalone daemon process. When used like this it will create a pool of child processes or threads to handle requests. In general, httpd should not be invoked directly, but rather should be invoked via apachectl on Unix-based systems or as a service on Windows NT, 2000 and XP and as a console application on Windows 9x and ME.<br />
<br />
$ apt install apache2<br />
$ service apache2 start #add 80, 443 to default security group rules<br />
$ a2enmod cgid.load #(optional) enable cgi scripting, a2dismod to remove<br />
<br />
===Basic information & settings===<br />
# Document root Directory: /var/www/html or /var/www<br />
# Main Configuration file:<br />
## /etc/httpd/conf/httpd.conf (RHEL/CentOS/Fedora) <br />
## /etc/apache2/apache2.conf (Debian/Ubuntu).<br />
# Default HTTP Port: 80 TCP<br />
# Default HTTPS Port: 443 TCP<br />
# Access Log files of Web Server: /var/log/apache2/access_log<br />
# Error Log files of Web Server: /var/log/apache2/error_log<br />
# service apache2 {start|stop|graceful-stop|restart|reload|force-reload}<br />
# apachectl -v <br />
## [https://httpd.apache.org/docs/2.4/programs/apachectl.html see]<br />
<br />
===Secure Apache with SSL certificates===<br />
Going to https is now possible but will trigger a warning.<br />
$ a2enmod ssl<br />
$ a2ensite default-ssl.conf<br />
$ service apache2 restart<br />
<br />
===Enable let's encrypt (requires a domain name)===<br />
$ apt install git<br />
$ sudo git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt<br />
$ cd /opt/letsencrypt<br />
$ ./certbot-auto --authenticator webroot --installer apache<br />
www.frar.ca<br />
/var/www/html<br />
<br />
===Use self signed certificate===<br />
$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/apache-selfsigned.key -out /etc/ssl/certs/apache-selfsigned.crt<br />
$ vim /etc/apache2/sites-enabled/default-ssl.conf<br />
SSLCertificateFile /etc/ssl/certs/apache-selfsigned.crt<br />
SSLCertificateKeyFile /etc/ssl/private/apache-selfsigned.key<br />
<br />
===Add basic user-password verfication===<br />
$ mkdir /var/www/passwd<br />
$ htpasswd -c /var/www/passwd/passwords user<br />
$ vim /etc/apache2/apache2.conf<br />
<Directory /var/www/html><br />
Require valid-user<br />
AuthType basic<br />
AuthName "Restricted Files"<br />
AuthUserFile "/var/www/passwd/passwords"<br />
</Directory><br />
$ service apache2 restart<br />
<br />
==Helper packages==<br />
===Install PHP===<br />
$ apt -y install php7.0 libapache2-mod-php7.0 php7.0-mysql php7.0-gd php7.0-opcache<br />
$ echo ‘<?php phpinfo(); ?>’ > /var/www/html/test.php<br />
<br />
===Give www-data an ssh key and ownership of home directory===<br />
$ chmod 774 /var/www/html<br />
$ chown www-data:www-data -R /var/www<br />
$ mkdir /var/www/.ssh<br />
$ chown www-data:www-data /var/www/.ssh<br />
$ sudo -u www-data ssh-keygen -C www-data #no passphrase for now<br />
<br />
==SSH Fuse==<br />
Give “debian” user www-data group so that files can be uploaded to the web server<br />
$ sudo usermod -aG www-data debian<br />
<br />
====Copy the ssh key to graham====<br />
(lamp) $ sudo cp .ssh/id_rsa.pub ~<br />
(local) $ scp -3 cloud:~/id_rsa.pub graham:~ # the -3 disables host to host copying<br />
(graham)$ cat id_rsa.pub >> authorized_keys<br />
<br />
====Setup Graham (placeholder section)====<br />
$ ssh graham <br />
$ mkdir ~/project/edward/linked-cloud<br />
$ cd linked-cloud<br />
$ wget https://git.sharcnet.ca/edward/CSVSorterDemo/raw/master/CSVSorter.jar<br />
$ mkdir input<br />
$ mkdir output<br />
$ vim submitjob.sh<br />
#!/bin/bash<br />
#SBATCH -t 0-00:01<br />
#SBATCH --mem=4G<br />
#SBATCH -A def-edward<br />
java -Xmx10m -jar /home/edward/project/edward/linked-cloud/CSVSorter.jar \<br />
/home/edward/project/edward/linked-cloud/input/$1 \<br />
/home/edward/project/edward/linked-cloud/output/$1<br />
<br />
====Link LAMP server with Graham through sshfuse====<br />
$ ssh cloud<br />
$ apt install sshfs<br />
$ cd /var/www<br />
$ sudo -u www-data mkdir linked-graham<br />
$ sudo -u www-data sshfs edward@graham.sharcnet.ca:/home/edward/project/linked-cloud /var/www/linked-graham<br />
- (to unmount) fusermount -u ./project</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=Deploying_a_web_server&diff=16575Deploying a web server2018-05-29T18:23:01Z<p>Edward: /* Use self signed certificate */</p>
<hr />
<div>=Dependencies - Launch Cloud Instance=<br />
Before deploying a web server, first a cloud instance must be lauched. The instruction to do this can be found [[Carpet|here]].<br />
For the remainder of the instructions the ip address used is 199.241.164.95, this is for demonstration purposes and you should replace<br />
it with your assigned floating IP address.<br />
<br />
# login to cloud.sharcnet.ca<br />
# Don’t forget volume size<br />
# Choose Debian 9.2.2 (not required, but the remaining instructions are Debian centric).<br />
# Choose persistent (ephemeral is for shorter jobs) (4C-8GB)<br />
# Setup keypair<br />
# Associate floating ip<br />
<br />
==(Optional) Apply IP Address to your name server==<br />
If you have a registered domain name you should apply your floating IP address to it so that you can use the "let's encrypt" service to <br />
enable the secure socket layer without client side warnings. It is often best to do this at the beginning as there is typically a delay<br />
in updating the name service.<br />
<br />
= Server Setup =<br />
<br />
==Login to Server & perform preliminaries==<br />
At this point log into the server to ensure that the service is up and running. There are a number of steps you can perform that will make <br />
the remaining steps easier. First loging to the server. Then setup a user prompt to make navigation easier. Switch to root (or use sudo in<br />
from of the remaining commands). Update and upgrade the system. Last install the manual pages.<br />
$ ssh debian@199.241.164.95<br />
$ echo 'export PS1="\[\e[33m\]\w\[\e[0m\]\n\[\e[32m\]\u@\h$ \[\e[0m\]"' >> .bash_aliases<br />
$ sudo su root<br />
$ apt update<br />
$ apt upgrade<br />
$ apt install man<br />
<br />
==Apt error==<br />
If you receive the "apt error" mesage put "nameserver 8.8.8.8" in /etc/resolve.conf.<br />
$ echo 'nameserver 8.8.8.8 >>' /etc/resolve.conf<br />
<br />
==Secure the SSH login==<br />
The following changes will disable logging into the server by using a password on any account and prevent logging into root remotly. Root can<br />
still be accessed by logging into a ''sudo'' enabled account and using the command ''su root''. For more information on the sshd_config file<br />
options, go [https://man.openbsd.org/sshd_config here]. The unattended-upgrades package is used to keep they server up to date automatically.<br />
$ sudo vim /etc/ssh/sshd_config<br />
ChallengeResponseAuthentication no<br />
PasswordAuthentication no<br />
PermitRootLogin no<br />
$ service ssh reload<br />
$ apt install unattended-upgrades<br />
$ dpkg-reconfigure --priority=low unattended-upgrades<br />
$ sudo unattended-upgrade -d<br />
<br />
=Install webserver and suppporting packages=<br />
==MYSQL==<br />
$ apt install mysql-server<br />
$ mysql_secure_installation #password=password<br />
<br />
==Apache==<br />
Installing the Apache webserver will allow you to access a default page at your ip address. (ex http://199.241.164.95, or http://frar.ca).<br />
Note that https will not yet work. httpd is the Apache HyperText Transfer Protocol (HTTP) server program. It is designed to be run as a standalone daemon process. When used like this it will create a pool of child processes or threads to handle requests. In general, httpd should not be invoked directly, but rather should be invoked via apachectl on Unix-based systems or as a service on Windows NT, 2000 and XP and as a console application on Windows 9x and ME.<br />
<br />
$ apt install apache2<br />
$ service apache2 start #add 80, 443 to default security group rules<br />
$ a2enmod cgid.load #(optional) enable cgi scripting, a2dismod to remove<br />
<br />
===Basic information & settings===<br />
# Document root Directory: /var/www/html or /var/www<br />
# Main Configuration file:<br />
## /etc/httpd/conf/httpd.conf (RHEL/CentOS/Fedora) <br />
## /etc/apache2/apache2.conf (Debian/Ubuntu).<br />
# Default HTTP Port: 80 TCP<br />
# Default HTTPS Port: 443 TCP<br />
# Access Log files of Web Server: /var/log/apache2/access_log<br />
# Error Log files of Web Server: /var/log/apache2/error_log<br />
# service apache2 {start|stop|graceful-stop|restart|reload|force-reload}<br />
# apachectl -v <br />
## [https://httpd.apache.org/docs/2.4/programs/apachectl.html see]<br />
<br />
===Secure Apache with SSL certificates===<br />
Going to https is now possible but will trigger a warning.<br />
$ a2enmod ssl<br />
$ a2ensite default-ssl.conf<br />
$ service apache2 restart<br />
<br />
===Enable let's encrypt (requires a domain name)===<br />
$ apt install git<br />
$ sudo git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt<br />
$ cd /opt/letsencrypt<br />
$ ./certbot-auto --authenticator webroot --installer apache<br />
www.frar.ca<br />
/var/www/html<br />
<br />
===Use self signed certificate===<br />
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/apache-selfsigned.key -out /etc/ssl/certs/apache-selfsigned.crt<br />
<br />
vim<br />
<br />
===Add basic user-password verfication===<br />
$ mkdir /var/www/passwd<br />
$ htpasswd -c /var/www/passwd/passwords user<br />
$ vim /etc/apache2/apache2.conf<br />
<Directory /var/www/html><br />
Require valid-user<br />
AuthType basic<br />
AuthName "Restricted Files"<br />
AuthUserFile "/var/www/passwd/passwords"<br />
</Directory><br />
$ service apache2 restart<br />
<br />
==Helper packages==<br />
===Install PHP===<br />
$ apt -y install php7.0 libapache2-mod-php7.0 php7.0-mysql php7.0-gd php7.0-opcache<br />
$ echo ‘<?php phpinfo(); ?>’ > /var/www/html/test.php<br />
<br />
===Give www-data an ssh key and ownership of home directory===<br />
$ chmod 774 /var/www/html<br />
$ chown www-data:www-data -R /var/www<br />
$ mkdir /var/www/.ssh<br />
$ chown www-data:www-data /var/www/.ssh<br />
$ sudo -u www-data ssh-keygen -C www-data #no passphrase for now<br />
<br />
==SSH Fuse==<br />
Give “debian” user www-data group so that files can be uploaded to the web server<br />
$ sudo usermod -aG www-data debian<br />
<br />
====Copy the ssh key to graham====<br />
(lamp) $ sudo cp .ssh/id_rsa.pub ~<br />
(local) $ scp -3 cloud:~/id_rsa.pub graham:~ # the -3 disables host to host copying<br />
(graham)$ cat id_rsa.pub >> authorized_keys<br />
<br />
====Setup Graham (placeholder section)====<br />
$ ssh graham <br />
$ mkdir ~/project/edward/linked-cloud<br />
$ cd linked-cloud<br />
$ wget https://git.sharcnet.ca/edward/CSVSorterDemo/raw/master/CSVSorter.jar<br />
$ mkdir input<br />
$ mkdir output<br />
$ vim submitjob.sh<br />
#!/bin/bash<br />
#SBATCH -t 0-00:01<br />
#SBATCH --mem=4G<br />
#SBATCH -A def-edward<br />
java -Xmx10m -jar /home/edward/project/edward/linked-cloud/CSVSorter.jar \<br />
/home/edward/project/edward/linked-cloud/input/$1 \<br />
/home/edward/project/edward/linked-cloud/output/$1<br />
<br />
====Link LAMP server with Graham through sshfuse====<br />
$ ssh cloud<br />
$ apt install sshfs<br />
$ cd /var/www<br />
$ sudo -u www-data mkdir linked-graham<br />
$ sudo -u www-data sshfs edward@graham.sharcnet.ca:/home/edward/project/linked-cloud /var/www/linked-graham<br />
- (to unmount) fusermount -u ./project</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=Deploying_a_web_server&diff=16574Deploying a web server2018-05-29T18:17:55Z<p>Edward: /* Apache */</p>
<hr />
<div>=Dependencies - Launch Cloud Instance=<br />
Before deploying a web server, first a cloud instance must be lauched. The instruction to do this can be found [[Carpet|here]].<br />
For the remainder of the instructions the ip address used is 199.241.164.95, this is for demonstration purposes and you should replace<br />
it with your assigned floating IP address.<br />
<br />
# login to cloud.sharcnet.ca<br />
# Don’t forget volume size<br />
# Choose Debian 9.2.2 (not required, but the remaining instructions are Debian centric).<br />
# Choose persistent (ephemeral is for shorter jobs) (4C-8GB)<br />
# Setup keypair<br />
# Associate floating ip<br />
<br />
==(Optional) Apply IP Address to your name server==<br />
If you have a registered domain name you should apply your floating IP address to it so that you can use the "let's encrypt" service to <br />
enable the secure socket layer without client side warnings. It is often best to do this at the beginning as there is typically a delay<br />
in updating the name service.<br />
<br />
= Server Setup =<br />
<br />
==Login to Server & perform preliminaries==<br />
At this point log into the server to ensure that the service is up and running. There are a number of steps you can perform that will make <br />
the remaining steps easier. First loging to the server. Then setup a user prompt to make navigation easier. Switch to root (or use sudo in<br />
from of the remaining commands). Update and upgrade the system. Last install the manual pages.<br />
$ ssh debian@199.241.164.95<br />
$ echo 'export PS1="\[\e[33m\]\w\[\e[0m\]\n\[\e[32m\]\u@\h$ \[\e[0m\]"' >> .bash_aliases<br />
$ sudo su root<br />
$ apt update<br />
$ apt upgrade<br />
$ apt install man<br />
<br />
==Apt error==<br />
If you receive the "apt error" mesage put "nameserver 8.8.8.8" in /etc/resolve.conf.<br />
$ echo 'nameserver 8.8.8.8 >>' /etc/resolve.conf<br />
<br />
==Secure the SSH login==<br />
The following changes will disable logging into the server by using a password on any account and prevent logging into root remotly. Root can<br />
still be accessed by logging into a ''sudo'' enabled account and using the command ''su root''. For more information on the sshd_config file<br />
options, go [https://man.openbsd.org/sshd_config here]. The unattended-upgrades package is used to keep they server up to date automatically.<br />
$ sudo vim /etc/ssh/sshd_config<br />
ChallengeResponseAuthentication no<br />
PasswordAuthentication no<br />
PermitRootLogin no<br />
$ service ssh reload<br />
$ apt install unattended-upgrades<br />
$ dpkg-reconfigure --priority=low unattended-upgrades<br />
$ sudo unattended-upgrade -d<br />
<br />
=Install webserver and suppporting packages=<br />
==MYSQL==<br />
$ apt install mysql-server<br />
$ mysql_secure_installation #password=password<br />
<br />
==Apache==<br />
Installing the Apache webserver will allow you to access a default page at your ip address. (ex http://199.241.164.95, or http://frar.ca).<br />
Note that https will not yet work. httpd is the Apache HyperText Transfer Protocol (HTTP) server program. It is designed to be run as a standalone daemon process. When used like this it will create a pool of child processes or threads to handle requests. In general, httpd should not be invoked directly, but rather should be invoked via apachectl on Unix-based systems or as a service on Windows NT, 2000 and XP and as a console application on Windows 9x and ME.<br />
<br />
$ apt install apache2<br />
$ service apache2 start #add 80, 443 to default security group rules<br />
$ a2enmod cgid.load #(optional) enable cgi scripting, a2dismod to remove<br />
<br />
===Basic information & settings===<br />
# Document root Directory: /var/www/html or /var/www<br />
# Main Configuration file:<br />
## /etc/httpd/conf/httpd.conf (RHEL/CentOS/Fedora) <br />
## /etc/apache2/apache2.conf (Debian/Ubuntu).<br />
# Default HTTP Port: 80 TCP<br />
# Default HTTPS Port: 443 TCP<br />
# Access Log files of Web Server: /var/log/apache2/access_log<br />
# Error Log files of Web Server: /var/log/apache2/error_log<br />
# service apache2 {start|stop|graceful-stop|restart|reload|force-reload}<br />
# apachectl -v <br />
## [https://httpd.apache.org/docs/2.4/programs/apachectl.html see]<br />
<br />
===Secure Apache with SSL certificates===<br />
Going to https is now possible but will trigger a warning.<br />
$ a2enmod ssl<br />
$ a2ensite default-ssl.conf<br />
$ service apache2 restart<br />
<br />
===Enable let's encrypt (requires a domain name)===<br />
$ apt install git<br />
$ sudo git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt<br />
$ cd /opt/letsencrypt<br />
$ ./certbot-auto --authenticator webroot --installer apache<br />
www.frar.ca<br />
/var/www/html<br />
<br />
===Use self signed certificate===<br />
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/apache-selfsigned.key -out /etc/ssl/certs/apache-selfsigned.crt<br />
<br />
<br />
===Add basic user-password verfication===<br />
$ mkdir /var/www/passwd<br />
$ htpasswd -c /var/www/passwd/passwords user<br />
$ vim /etc/apache2/apache2.conf<br />
<Directory /var/www/html><br />
Require valid-user<br />
AuthType basic<br />
AuthName "Restricted Files"<br />
AuthUserFile "/var/www/passwd/passwords"<br />
</Directory><br />
$ service apache2 restart<br />
<br />
==Helper packages==<br />
===Install PHP===<br />
$ apt -y install php7.0 libapache2-mod-php7.0 php7.0-mysql php7.0-gd php7.0-opcache<br />
$ echo ‘<?php phpinfo(); ?>’ > /var/www/html/test.php<br />
<br />
===Give www-data an ssh key and ownership of home directory===<br />
$ chmod 774 /var/www/html<br />
$ chown www-data:www-data -R /var/www<br />
$ mkdir /var/www/.ssh<br />
$ chown www-data:www-data /var/www/.ssh<br />
$ sudo -u www-data ssh-keygen -C www-data #no passphrase for now<br />
<br />
==SSH Fuse==<br />
Give “debian” user www-data group so that files can be uploaded to the web server<br />
$ sudo usermod -aG www-data debian<br />
<br />
====Copy the ssh key to graham====<br />
(lamp) $ sudo cp .ssh/id_rsa.pub ~<br />
(local) $ scp -3 cloud:~/id_rsa.pub graham:~ # the -3 disables host to host copying<br />
(graham)$ cat id_rsa.pub >> authorized_keys<br />
<br />
====Setup Graham (placeholder section)====<br />
$ ssh graham <br />
$ mkdir ~/project/edward/linked-cloud<br />
$ cd linked-cloud<br />
$ wget https://git.sharcnet.ca/edward/CSVSorterDemo/raw/master/CSVSorter.jar<br />
$ mkdir input<br />
$ mkdir output<br />
$ vim submitjob.sh<br />
#!/bin/bash<br />
#SBATCH -t 0-00:01<br />
#SBATCH --mem=4G<br />
#SBATCH -A def-edward<br />
java -Xmx10m -jar /home/edward/project/edward/linked-cloud/CSVSorter.jar \<br />
/home/edward/project/edward/linked-cloud/input/$1 \<br />
/home/edward/project/edward/linked-cloud/output/$1<br />
<br />
====Link LAMP server with Graham through sshfuse====<br />
$ ssh cloud<br />
$ apt install sshfs<br />
$ cd /var/www<br />
$ sudo -u www-data mkdir linked-graham<br />
$ sudo -u www-data sshfs edward@graham.sharcnet.ca:/home/edward/project/linked-cloud /var/www/linked-graham<br />
- (to unmount) fusermount -u ./project</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=Deploying_a_web_server&diff=16573Deploying a web server2018-05-29T17:11:26Z<p>Edward: /* Helper packages */</p>
<hr />
<div>=Dependencies - Launch Cloud Instance=<br />
Before deploying a web server, first a cloud instance must be lauched. The instruction to do this can be found [[Carpet|here]].<br />
For the remainder of the instructions the ip address used is 199.241.164.95, this is for demonstration purposes and you should replace<br />
it with your assigned floating IP address.<br />
<br />
# login to cloud.sharcnet.ca<br />
# Don’t forget volume size<br />
# Choose Debian 9.2.2 (not required, but the remaining instructions are Debian centric).<br />
# Choose persistent (ephemeral is for shorter jobs) (4C-8GB)<br />
# Setup keypair<br />
# Associate floating ip<br />
<br />
==(Optional) Apply IP Address to your name server==<br />
If you have a registered domain name you should apply your floating IP address to it so that you can use the "let's encrypt" service to <br />
enable the secure socket layer without client side warnings. It is often best to do this at the beginning as there is typically a delay<br />
in updating the name service.<br />
<br />
= Server Setup =<br />
<br />
==Login to Server & perform preliminaries==<br />
At this point log into the server to ensure that the service is up and running. There are a number of steps you can perform that will make <br />
the remaining steps easier. First loging to the server. Then setup a user prompt to make navigation easier. Switch to root (or use sudo in<br />
from of the remaining commands). Update and upgrade the system. Last install the manual pages.<br />
$ ssh debian@199.241.164.95<br />
$ echo 'export PS1="\[\e[33m\]\w\[\e[0m\]\n\[\e[32m\]\u@\h$ \[\e[0m\]"' >> .bash_aliases<br />
$ sudo su root<br />
$ apt update<br />
$ apt upgrade<br />
$ apt install man<br />
<br />
==Apt error==<br />
If you receive the "apt error" mesage put "nameserver 8.8.8.8" in /etc/resolve.conf.<br />
$ echo 'nameserver 8.8.8.8 >>' /etc/resolve.conf<br />
<br />
==Secure the SSH login==<br />
The following changes will disable logging into the server by using a password on any account and prevent logging into root remotly. Root can<br />
still be accessed by logging into a ''sudo'' enabled account and using the command ''su root''. For more information on the sshd_config file<br />
options, go [https://man.openbsd.org/sshd_config here]. The unattended-upgrades package is used to keep they server up to date automatically.<br />
$ sudo vim /etc/ssh/sshd_config<br />
ChallengeResponseAuthentication no<br />
PasswordAuthentication no<br />
PermitRootLogin no<br />
$ service ssh reload<br />
$ apt install unattended-upgrades<br />
$ dpkg-reconfigure --priority=low unattended-upgrades<br />
$ sudo unattended-upgrade -d<br />
<br />
=Install webserver and suppporting packages=<br />
==MYSQL==<br />
$ apt install mysql-server<br />
$ mysql_secure_installation #password=password<br />
<br />
==Apache==<br />
Installing the Apache webserver will allow you to access a default page at your ip address. (ex http://199.241.164.95, or http://frar.ca).<br />
Note that https will not yet work. httpd is the Apache HyperText Transfer Protocol (HTTP) server program. It is designed to be run as a standalone daemon process. When used like this it will create a pool of child processes or threads to handle requests. In general, httpd should not be invoked directly, but rather should be invoked via apachectl on Unix-based systems or as a service on Windows NT, 2000 and XP and as a console application on Windows 9x and ME.<br />
<br />
$ apt install apache2<br />
$ service apache2 start #add 80, 443 to default security group rules<br />
$ a2enmod cgid.load #(optional) enable cgi scripting, a2dismod to remove<br />
<br />
===Basic information & settings===<br />
# Document root Directory: /var/www/html or /var/www<br />
# Main Configuration file:<br />
## /etc/httpd/conf/httpd.conf (RHEL/CentOS/Fedora) <br />
## /etc/apache2/apache2.conf (Debian/Ubuntu).<br />
# Default HTTP Port: 80 TCP<br />
# Default HTTPS Port: 443 TCP<br />
# Access Log files of Web Server: /var/log/apache2/access_log<br />
# Error Log files of Web Server: /var/log/apache2/error_log<br />
# service apache2 {start|stop|graceful-stop|restart|reload|force-reload}<br />
# apachectl -v <br />
## [https://httpd.apache.org/docs/2.4/programs/apachectl.html see]<br />
<br />
===Secure Apache with SSL certificates===<br />
Going to https is now possible but will trigger a warning.<br />
$ a2enmod ssl<br />
$ a2ensite default-ssl.conf<br />
$ service apache2 restart<br />
<br />
===Enable let's encrypt (requires a domain name)===<br />
$ apt install git<br />
$ sudo git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt<br />
$ cd /opt/letsencrypt<br />
$ ./certbot-auto --authenticator webroot --installer apache<br />
www.frar.ca<br />
/var/www/html<br />
<br />
===Add basic user-password verfication===<br />
$ mkdir /var/www/passwd<br />
$ htpasswd -c /var/www/passwd/passwords user<br />
$ vim /etc/apache2/apache2.conf<br />
<Directory /var/www/html><br />
Require valid-user<br />
AuthType basic<br />
AuthName "Restricted Files"<br />
AuthUserFile "/var/www/passwd/passwords"<br />
</Directory><br />
$ service apache2 restart<br />
<br />
==Helper packages==<br />
===Install PHP===<br />
$ apt -y install php7.0 libapache2-mod-php7.0 php7.0-mysql php7.0-gd php7.0-opcache<br />
$ echo ‘<?php phpinfo(); ?>’ > /var/www/html/test.php<br />
<br />
===Give www-data an ssh key and ownership of home directory===<br />
$ chmod 774 /var/www/html<br />
$ chown www-data:www-data -R /var/www<br />
$ mkdir /var/www/.ssh<br />
$ chown www-data:www-data /var/www/.ssh<br />
$ sudo -u www-data ssh-keygen -C www-data #no passphrase for now<br />
<br />
==SSH Fuse==<br />
Give “debian” user www-data group so that files can be uploaded to the web server<br />
$ sudo usermod -aG www-data debian<br />
<br />
====Copy the ssh key to graham====<br />
(lamp) $ sudo cp .ssh/id_rsa.pub ~<br />
(local) $ scp -3 cloud:~/id_rsa.pub graham:~ # the -3 disables host to host copying<br />
(graham)$ cat id_rsa.pub >> authorized_keys<br />
<br />
====Setup Graham (placeholder section)====<br />
$ ssh graham <br />
$ mkdir ~/project/edward/linked-cloud<br />
$ cd linked-cloud<br />
$ wget https://git.sharcnet.ca/edward/CSVSorterDemo/raw/master/CSVSorter.jar<br />
$ mkdir input<br />
$ mkdir output<br />
$ vim submitjob.sh<br />
#!/bin/bash<br />
#SBATCH -t 0-00:01<br />
#SBATCH --mem=4G<br />
#SBATCH -A def-edward<br />
java -Xmx10m -jar /home/edward/project/edward/linked-cloud/CSVSorter.jar \<br />
/home/edward/project/edward/linked-cloud/input/$1 \<br />
/home/edward/project/edward/linked-cloud/output/$1<br />
<br />
====Link LAMP server with Graham through sshfuse====<br />
$ ssh cloud<br />
$ apt install sshfs<br />
$ cd /var/www<br />
$ sudo -u www-data mkdir linked-graham<br />
$ sudo -u www-data sshfs edward@graham.sharcnet.ca:/home/edward/project/linked-cloud /var/www/linked-graham<br />
- (to unmount) fusermount -u ./project</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=Deploying_a_web_server&diff=16572Deploying a web server2018-05-29T17:10:41Z<p>Edward: </p>
<hr />
<div>=Dependencies - Launch Cloud Instance=<br />
Before deploying a web server, first a cloud instance must be lauched. The instruction to do this can be found [[Carpet|here]].<br />
For the remainder of the instructions the ip address used is 199.241.164.95, this is for demonstration purposes and you should replace<br />
it with your assigned floating IP address.<br />
<br />
# login to cloud.sharcnet.ca<br />
# Don’t forget volume size<br />
# Choose Debian 9.2.2 (not required, but the remaining instructions are Debian centric).<br />
# Choose persistent (ephemeral is for shorter jobs) (4C-8GB)<br />
# Setup keypair<br />
# Associate floating ip<br />
<br />
==(Optional) Apply IP Address to your name server==<br />
If you have a registered domain name you should apply your floating IP address to it so that you can use the "let's encrypt" service to <br />
enable the secure socket layer without client side warnings. It is often best to do this at the beginning as there is typically a delay<br />
in updating the name service.<br />
<br />
= Server Setup =<br />
<br />
==Login to Server & perform preliminaries==<br />
At this point log into the server to ensure that the service is up and running. There are a number of steps you can perform that will make <br />
the remaining steps easier. First loging to the server. Then setup a user prompt to make navigation easier. Switch to root (or use sudo in<br />
from of the remaining commands). Update and upgrade the system. Last install the manual pages.<br />
$ ssh debian@199.241.164.95<br />
$ echo 'export PS1="\[\e[33m\]\w\[\e[0m\]\n\[\e[32m\]\u@\h$ \[\e[0m\]"' >> .bash_aliases<br />
$ sudo su root<br />
$ apt update<br />
$ apt upgrade<br />
$ apt install man<br />
<br />
==Apt error==<br />
If you receive the "apt error" mesage put "nameserver 8.8.8.8" in /etc/resolve.conf.<br />
$ echo 'nameserver 8.8.8.8 >>' /etc/resolve.conf<br />
<br />
==Secure the SSH login==<br />
The following changes will disable logging into the server by using a password on any account and prevent logging into root remotly. Root can<br />
still be accessed by logging into a ''sudo'' enabled account and using the command ''su root''. For more information on the sshd_config file<br />
options, go [https://man.openbsd.org/sshd_config here]. The unattended-upgrades package is used to keep they server up to date automatically.<br />
$ sudo vim /etc/ssh/sshd_config<br />
ChallengeResponseAuthentication no<br />
PasswordAuthentication no<br />
PermitRootLogin no<br />
$ service ssh reload<br />
$ apt install unattended-upgrades<br />
$ dpkg-reconfigure --priority=low unattended-upgrades<br />
$ sudo unattended-upgrade -d<br />
<br />
=Install webserver and suppporting packages=<br />
==MYSQL==<br />
$ apt install mysql-server<br />
$ mysql_secure_installation #password=password<br />
<br />
==Apache==<br />
Installing the Apache webserver will allow you to access a default page at your ip address. (ex http://199.241.164.95, or http://frar.ca).<br />
Note that https will not yet work. httpd is the Apache HyperText Transfer Protocol (HTTP) server program. It is designed to be run as a standalone daemon process. When used like this it will create a pool of child processes or threads to handle requests. In general, httpd should not be invoked directly, but rather should be invoked via apachectl on Unix-based systems or as a service on Windows NT, 2000 and XP and as a console application on Windows 9x and ME.<br />
<br />
$ apt install apache2<br />
$ service apache2 start #add 80, 443 to default security group rules<br />
$ a2enmod cgid.load #(optional) enable cgi scripting, a2dismod to remove<br />
<br />
===Basic information & settings===<br />
# Document root Directory: /var/www/html or /var/www<br />
# Main Configuration file:<br />
## /etc/httpd/conf/httpd.conf (RHEL/CentOS/Fedora) <br />
## /etc/apache2/apache2.conf (Debian/Ubuntu).<br />
# Default HTTP Port: 80 TCP<br />
# Default HTTPS Port: 443 TCP<br />
# Access Log files of Web Server: /var/log/apache2/access_log<br />
# Error Log files of Web Server: /var/log/apache2/error_log<br />
# service apache2 {start|stop|graceful-stop|restart|reload|force-reload}<br />
# apachectl -v <br />
## [https://httpd.apache.org/docs/2.4/programs/apachectl.html see]<br />
<br />
===Secure Apache with SSL certificates===<br />
Going to https is now possible but will trigger a warning.<br />
$ a2enmod ssl<br />
$ a2ensite default-ssl.conf<br />
$ service apache2 restart<br />
<br />
===Enable let's encrypt (requires a domain name)===<br />
$ apt install git<br />
$ sudo git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt<br />
$ cd /opt/letsencrypt<br />
$ ./certbot-auto --authenticator webroot --installer apache<br />
www.frar.ca<br />
/var/www/html<br />
<br />
===Add basic user-password verfication===<br />
$ mkdir /var/www/passwd<br />
$ htpasswd -c /var/www/passwd/passwords user<br />
$ vim /etc/apache2/apache2.conf<br />
<Directory /var/www/html><br />
Require valid-user<br />
AuthType basic<br />
AuthName "Restricted Files"<br />
AuthUserFile "/var/www/passwd/passwords"<br />
</Directory><br />
$ service apache2 restart<br />
<br />
==Helper packages==<br />
===Install PHP===<br />
$ apt -y install php7.0 libapache2-mod-php7.0 php7.0-mysql php7.0-gd php7.0-opcache<br />
$ echo ‘<?php phpinfo(); ?>’ > /var/www/html/test.php<br />
<br />
===Give www-data an ssh key and ownership of home directory===<br />
$ chmod 774 /var/www/html<br />
$ chown www-data:www-data -R /var/www<br />
$ mkdir /var/www/.ssh<br />
$ chown www-data:www-data /var/www/.ssh<br />
$ sudo -u www-data ssh-keygen -C www-data #no passphrase for now<br />
<br />
===SSH Fuse===<br />
Give “debian” user www-data group so that files can be uploaded to the web server<br />
$ sudo usermod -aG www-data debian<br />
<br />
====Copy the ssh key to graham====<br />
(lamp) $ sudo cp .ssh/id_rsa.pub ~<br />
(local) $ scp -3 cloud:~/id_rsa.pub graham:~ # the -3 disables host to host copying<br />
(graham)$ cat id_rsa.pub >> authorized_keys<br />
<br />
====Setup Graham====<br />
$ ssh graham <br />
$ mkdir ~/project/edward/linked-cloud<br />
$ cd linked-cloud<br />
$ wget https://git.sharcnet.ca/edward/CSVSorterDemo/raw/master/CSVSorter.jar<br />
$ mkdir input<br />
$ mkdir output<br />
$ vim submitjob.sh<br />
#!/bin/bash<br />
#SBATCH -t 0-00:01<br />
#SBATCH --mem=4G<br />
#SBATCH -A def-edward<br />
java -Xmx10m -jar /home/edward/project/edward/linked-cloud/CSVSorter.jar \<br />
/home/edward/project/edward/linked-cloud/input/$1 \<br />
/home/edward/project/edward/linked-cloud/output/$1<br />
<br />
====Link LAMP server with Graham through sshfuse====<br />
$ ssh cloud<br />
$ apt install sshfs<br />
$ cd /var/www<br />
$ sudo -u www-data mkdir linked-graham<br />
$ sudo -u www-data sshfs edward@graham.sharcnet.ca:/home/edward/project/linked-cloud /var/www/linked-graham<br />
- (to unmount) fusermount -u ./project</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=Deploying_a_web_server&diff=16566Deploying a web server2018-05-29T15:20:57Z<p>Edward: </p>
<hr />
<div>=Dependencies - Launch Cloud Instance=<br />
Before deploying a web server, first a cloud instance must be lauched. The instruction to do this can be found [[Carpet|here]].<br />
For the remainder of the instructions the ip address used is 199.241.164.95, this is for demonstration purposes and you should replace<br />
it with your assigned floating IP address.<br />
<br />
# login to cloud.sharcnet.ca<br />
# Don’t forget volume size<br />
# Choose Debian 9.2.2 (not required, but the remaining instructions are Debian centric).<br />
# Choose persistent (ephemeral is for shorter jobs) (4C-8GB)<br />
# Setup keypair<br />
# Associate floating ip<br />
<br />
==(Optional) Apply IP Address to your name server==<br />
If you have a registered domain name you should apply your floating IP address to it so that you can use the "let's encrypt" service to <br />
enable the secure socket layer without client side warnings. It is often best to do this at the beginning as there is typically a delay<br />
in updating the name service.<br />
<br />
= Server Setup =<br />
<br />
==Login to Server & perform preliminaries==<br />
At this point log into the server to ensure that the service is up and running. There are a number of steps you can perform that will make <br />
the remaining steps easier. First loging to the server. Then setup a user prompt to make navigation easier. Switch to root (or use sudo in<br />
from of the remaining commands). Update and upgrade the system. Last install the manual pages.<br />
$ ssh debian@199.241.164.95<br />
$ echo 'export PS1="\[\e[33m\]\w\[\e[0m\]\n\[\e[32m\]\u@\h$ \[\e[0m\]"' >> .bash_aliases<br />
$ sudo su root<br />
$ apt update<br />
$ apt upgrade<br />
$ apt install man<br />
<br />
==Apt error==<br />
If you receive the "apt error" mesage put "nameserver 8.8.8.8" in /etc/resolve.conf.<br />
$ echo 'nameserver 8.8.8.8 >>' /etc/resolve.conf<br />
<br />
==Secure the SSH login==<br />
The following changes will disable logging into the server by using a password on any account and prevent logging into root remotly. Root can<br />
still be accessed by logging into a ''sudo'' enabled account and using the command ''su root''. For more information on the sshd_config file<br />
options, go [https://man.openbsd.org/sshd_config here]. The unattended-upgrades package is used to keep they server up to date automatically.<br />
$ sudo vim /etc/ssh/sshd_config<br />
ChallengeResponseAuthentication no<br />
PasswordAuthentication no<br />
PermitRootLogin no<br />
$ service ssh reload<br />
$ apt install unattended-upgrades<br />
$ dpkg-reconfigure --priority=low unattended-upgrades<br />
$ sudo unattended-upgrade -d<br />
<br />
=Install webserver and suppporting packages=<br />
==MYSQL==<br />
$ apt install mysql-server<br />
$ mysql_secure_installation #password=password<br />
<br />
==Apache==<br />
Installing the Apache webserver will allow you to access a default page at your ip address. (ex http://199.241.164.95, or http://frar.ca).<br />
Note that https will not yet work. httpd is the Apache HyperText Transfer Protocol (HTTP) server program. It is designed to be run as a standalone daemon process. When used like this it will create a pool of child processes or threads to handle requests. In general, httpd should not be invoked directly, but rather should be invoked via apachectl on Unix-based systems or as a service on Windows NT, 2000 and XP and as a console application on Windows 9x and ME.<br />
<br />
$ apt install apache2<br />
$ service apache2 start #add 80, 443 to default security group rules<br />
$ a2enmod cgid.load #(optional) enable cgi scripting, a2dismod to remove<br />
<br />
===Basic information & settings===<br />
# Document root Directory: /var/www/html or /var/www<br />
# Main Configuration file:<br />
## /etc/httpd/conf/httpd.conf (RHEL/CentOS/Fedora) <br />
## /etc/apache2/apache2.conf (Debian/Ubuntu).<br />
# Default HTTP Port: 80 TCP<br />
# Default HTTPS Port: 443 TCP<br />
# Access Log files of Web Server: /var/log/apache2/access_log<br />
# Error Log files of Web Server: /var/log/apache2/error_log<br />
# service apache2 {start|stop|graceful-stop|restart|reload|force-reload}<br />
# apachectl -v <br />
## [https://httpd.apache.org/docs/2.4/programs/apachectl.html see]<br />
<br />
===Secure Apache with SSL certificates===<br />
Going to https is now possible but will trigger a warning.<br />
$ a2enmod ssl<br />
$ a2ensite default-ssl.conf<br />
$ service apache2 restart<br />
<br />
===Enable let's encrypt (requires a domain name)===<br />
$ apt install git<br />
$ sudo git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt<br />
$ cd /opt/letsencrypt<br />
$ ./certbot-auto --authenticator webroot --installer apache<br />
www.frar.ca<br />
/var/www/html<br />
<br />
===Add basic user-password verfication===<br />
$ mkdir /var/www/passwd<br />
$ htpasswd -c /var/www/passwd/passwords user<br />
$ vim /etc/apache2/apache2.conf<br />
<Directory /var/www/html><br />
Require valid-user<br />
AuthType basic<br />
AuthName "Restricted Files"<br />
AuthUserFile "/var/www/passwd/passwords"<br />
</Directory><br />
$ service apache2 restart<br />
<br />
==Helper packages==<br />
===Install PHP===<br />
$ apt -y install php7.0 libapache2-mod-php7.0 php7.0-mysql php7.0-gd php7.0-opcache<br />
$ echo ‘<?php phpinfo(); ?>’ > /var/www/html/test.php<br />
<br />
===Give www-data an ssh key and ownership of home directory===<br />
$ chmod 774 /var/www/html<br />
$ chown www-data:www-data -R /var/www<br />
$ mkdir /var/www/.ssh<br />
$ chown www-data:www-data /var/www/.ssh<br />
$ sudo -u www-data ssh-keygen -C www-data #no passphrase for now</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=Deploying_a_web_server&diff=16565Deploying a web server2018-05-28T16:58:27Z<p>Edward: </p>
<hr />
<div>Deploying a Web Server<br />
<br />
=Dependencies - Launch Cloud Instance=<br />
Before deploying a web server, first a cloud instance must be lauched. The instruction to do this can be found [[Carpet|here]].<br />
For the remainder of the instructions the ip address used is 199.241.164.95, this is for demonstration purposes and you should replace<br />
it with your assigned floating IP address.<br />
<br />
# login to cloud.sharcnet.ca<br />
# Don’t forget volume size<br />
# Choose Debian 9.2.2 (not required, but the remaining instructions are Debian centric).<br />
# Choose persistent (ephemeral is for shorter jobs) (4C-8GB)<br />
# Setup keypair<br />
# Associate floating ip<br />
<br />
==(Optional) Apply IP Address to your name server==<br />
If you have a registered domain name you should apply your floating IP address to it so that you can use the "let's encrypt" service to <br />
enable the secure socket layer without client side warnings. It is often best to do this at the beginning as there is typically a delay<br />
in updating the name service.<br />
<br />
= Server Setup =<br />
<br />
==Login to Server & perform preliminaries==<br />
At this point log into the server to ensure that the service is up and running. There are a number of steps you can perform that will make <br />
the remaining steps easier. First loging to the server. Then setup a user prompt to make navigation easier. Switch to root (or use sudo in<br />
from of the remaining commands). Update and upgrade the system. Last install the manual pages.<br />
$ ssh debian@199.241.164.95<br />
$ echo 'export PS1="\[\e[33m\]\w\[\e[0m\]\n\[\e[32m\]\u@\h$ \[\e[0m\]"' >> .bash_aliases<br />
$ sudo su root<br />
$ apt update<br />
$ apt upgrade<br />
$ apt install man<br />
<br />
==Apt error==<br />
If you receive the "apt error" mesage put "nameserver 8.8.8.8" in /etc/resolve.conf.<br />
$ echo 'nameserver 8.8.8.8 >>' /etc/resolve.conf<br />
<br />
==Secure the SSH login==<br />
The following changes will disable logging into the server by using a password on any account and prevent logging into root remotly. Root can<br />
still be accessed by logging into a ''sudo'' enabled account and using the command ''su root''. For more information on the sshd_config file<br />
options, go [https://man.openbsd.org/sshd_config here]. The unattended-upgrades package is used to keep they server up to date automatically.<br />
$ sudo vim /etc/ssh/sshd_config<br />
ChallengeResponseAuthentication no<br />
PasswordAuthentication no<br />
PermitRootLogin no<br />
$ service ssh reload<br />
$ apt install unattended-upgrades<br />
$ dpkg-reconfigure --priority=low unattended-upgrades<br />
$ sudo unattended-upgrade -d<br />
<br />
=Install webserver and suppporting packages=<br />
==MYSQL==<br />
$ apt install mysql-server<br />
$ mysql_secure_installation #password=password<br />
<br />
==Apache==<br />
Installing the Apache webserver will allow you to access a default page at your ip address. (ex http://199.241.164.95, or http://frar.ca).<br />
Note that https will not yet work. httpd is the Apache HyperText Transfer Protocol (HTTP) server program. It is designed to be run as a standalone daemon process. When used like this it will create a pool of child processes or threads to handle requests. In general, httpd should not be invoked directly, but rather should be invoked via apachectl on Unix-based systems or as a service on Windows NT, 2000 and XP and as a console application on Windows 9x and ME.<br />
<br />
$ apt install apache2<br />
$ service apache2 start #add 80, 443 to default security group rules<br />
$ a2enmod cgid.load #(optional) enable cgi scripting, a2dismod to remove<br />
<br />
===Basic information & settings===<br />
# Document root Directory: /var/www/html or /var/www<br />
# Main Configuration file:<br />
## /etc/httpd/conf/httpd.conf (RHEL/CentOS/Fedora) <br />
## /etc/apache2/apache2.conf (Debian/Ubuntu).<br />
# Default HTTP Port: 80 TCP<br />
# Default HTTPS Port: 443 TCP<br />
# Access Log files of Web Server: /var/log/apache2/access_log<br />
# Error Log files of Web Server: /var/log/apache2/error_log<br />
# service apache2 {start|stop|graceful-stop|restart|reload|force-reload}<br />
# apachectl -v <br />
## [https://httpd.apache.org/docs/2.4/programs/apachectl.html see]<br />
<br />
===Secure Apache with SSL certificates===<br />
Going to https is now possible but will trigger a warning.<br />
$ a2enmod ssl<br />
$ a2ensite default-ssl.conf<br />
$ service apache2 restart<br />
<br />
===Enable let's encrypt (requires a domain name)===<br />
$ apt install git<br />
$ sudo git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt<br />
$ cd /opt/letsencrypt<br />
$ ./certbot-auto --authenticator webroot --installer apache<br />
www.frar.ca<br />
/var/www/html<br />
<br />
===Add basic user-password verfication===<br />
$ mkdir /var/www/passwd<br />
$ htpasswd -c /var/www/passwd/passwords user<br />
$ vim /etc/apache2/apache2.conf<br />
<Directory /var/www/html><br />
Require valid-user<br />
AuthType basic<br />
AuthName "Restricted Files"<br />
AuthUserFile "/var/www/passwd/passwords"<br />
</Directory><br />
$ service apache2 restart<br />
<br />
==Helper packages==<br />
===Install PHP===<br />
$ apt -y install php7.0 libapache2-mod-php7.0 php7.0-mysql php7.0-gd php7.0-opcache<br />
$ echo ‘<?php phpinfo(); ?>’ > /var/www/html/test.php<br />
<br />
===Give www-data an ssh key and ownership of home directory===<br />
$ chmod 774 /var/www/html<br />
$ chown www-data:www-data -R /var/www<br />
$ mkdir /var/www/.ssh<br />
$ chown www-data:www-data /var/www/.ssh<br />
$ sudo -u www-data ssh-keygen -C www-data #no passphrase for now</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=Deploying_a_web_server&diff=16564Deploying a web server2018-05-28T16:57:42Z<p>Edward: </p>
<hr />
<div>Deploying a Web Server<br />
<br />
=Dependencies - Launch Cloud Instance=<br />
Before deploying a web server, first a cloud instance must be lauched. The instruction to do this can be found [[Carpet|here]].<br />
For the remainder of the instructions the ip address used is 199.241.164.95, this is for demonstration purposes and you should replace<br />
it with your assigned floating IP address.<br />
<br />
# login to cloud.sharcnet.ca<br />
# Don’t forget volume size<br />
# Choose Debian 9.2.2 (not required, but the remaining instructions are Debian centric).<br />
# Choose persistent (ephemeral is for shorter jobs) (4C-8GB)<br />
# Setup keypair<br />
# Associate floating ip<br />
<br />
==(Optional) Apply IP Address to your name server==<br />
If you have a registered domain name you should apply your floating IP address to it so that you can use the "let's encrypt" service to <br />
enable the secure socket layer without client side warnings. It is often best to do this at the beginning as there is typically a delay<br />
in updating the name service.<br />
<br />
= Server Setup =<br />
<br />
==Login to Server & perform preliminaries==<br />
At this point log into the server to ensure that the service is up and running. There are a number of steps you can perform that will make <br />
the remaining steps easier. First loging to the server. Then setup a user prompt to make navigation easier. Switch to root (or use sudo in<br />
from of the remaining commands). Update and upgrade the system. Last install the manual pages.<br />
$ ssh debian@199.241.164.95<br />
$ echo 'export PS1="\[\e[33m\]\w\[\e[0m\]\n\[\e[32m\]\u@\h$ \[\e[0m\]"' >> .bash_aliases<br />
$ sudo su root<br />
$ apt update<br />
$ apt upgrade<br />
$ apt install man<br />
<br />
==Apt error==<br />
If you receive the "apt error" mesage put "nameserver 8.8.8.8" in /etc/resolve.conf.<br />
$ echo 'nameserver 8.8.8.8 >>' /etc/resolve.conf<br />
<br />
==Secure the SSH login==<br />
The following changes will disable logging into the server by using a password on any account and prevent logging into root remotly. Root can<br />
still be accessed by logging into a ''sudo'' enabled account and using the command ''su root''. For more information on the sshd_config file<br />
options, go [https://man.openbsd.org/sshd_config here]. The unattended-upgrades package is used to keep they server up to date automatically.<br />
$ sudo vim /etc/ssh/sshd_config<br />
ChallengeResponseAuthentication no<br />
PasswordAuthentication no<br />
PermitRootLogin no<br />
$ service ssh reload<br />
$ apt install unattended-upgrades<br />
$ dpkg-reconfigure --priority=low unattended-upgrades<br />
$ sudo unattended-upgrade -d<br />
<br />
=Install webserver and suppporting packages=<br />
==MYSQL==<br />
$ apt install mysql-server<br />
$ mysql_secure_installation #password=password<br />
<br />
==Apache==<br />
Installing the Apache webserver will allow you to access a default page at your ip address. (ex http://199.241.164.95, or http://frar.ca).<br />
Note that https will not yet work. httpd is the Apache HyperText Transfer Protocol (HTTP) server program. It is designed to be run as a standalone daemon process. When used like this it will create a pool of child processes or threads to handle requests. In general, httpd should not be invoked directly, but rather should be invoked via apachectl on Unix-based systems or as a service on Windows NT, 2000 and XP and as a console application on Windows 9x and ME.<br />
<br />
$ apt install apache2<br />
$ service apache2 start #add 80, 443 to default security group rules<br />
$ a2enmod cgid.load #(optional) enable cgi scripting, a2dismod to remove<br />
<br />
===Basic information & settings===<br />
# Document root Directory: /var/www/html or /var/www<br />
# Main Configuration file:<br />
## /etc/httpd/conf/httpd.conf (RHEL/CentOS/Fedora) <br />
## /etc/apache2/apache2.conf (Debian/Ubuntu).<br />
# Default HTTP Port: 80 TCP<br />
# Default HTTPS Port: 443 TCP<br />
# Access Log files of Web Server: /var/log/apache2/access_log<br />
# Error Log files of Web Server: /var/log/apache2/error_log<br />
# service apache2 {start|stop|graceful-stop|restart|reload|force-reload}<br />
# apachectl -v <br />
## [https://httpd.apache.org/docs/2.4/programs/apachectl.html see]<br />
<br />
===Secure Apache with SSL certificates===<br />
Going to https is now possible but will trigger a warning.<br />
$ a2enmod ssl<br />
$ a2ensite default-ssl.conf<br />
$ service apache2 restart<br />
<br />
===Enable let's encrypt (requires a domain name)===<br />
$ apt install git<br />
$ sudo git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt<br />
$ cd /opt/letsencrypt<br />
$ ./certbot-auto --authenticator webroot --installer apache<br />
www.frar.ca<br />
/var/www/html<br />
<br />
===Add basic user-password verfication===<br />
$ mkdir /var/www/passwd<br />
$ htpasswd -c /var/www/passwd/passwords user<br />
$ vim /etc/apache2/apache2.conf<br />
<Directory /var/www/html><br />
Require valid-user<br />
AuthType basic<br />
AuthName "Restricted Files"<br />
AuthUserFile "/var/www/passwd/passwords"<br />
</Directory><br />
$ service apache2 restart<br />
<br />
===Install PHP===<br />
$ apt -y install php7.0 libapache2-mod-php7.0 php7.0-mysql php7.0-gd php7.0-opcache<br />
$ echo ‘<?php phpinfo(); ?>’ > /var/www/html/test.php</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=Deploying_a_web_server&diff=16563Deploying a web server2018-05-28T14:46:58Z<p>Edward: /* Install webserver and suppporting packages */</p>
<hr />
<div>Deploying a Web Server<br />
<br />
=Dependencies - Launch Cloud Instance=<br />
Before deploying a web server, first a cloud instance must be lauched. The instruction to do this can be found [[Carpet|here]].<br />
For the remainder of the instructions the ip address used is 199.241.164.95, this is for demonstration purposes and you should replace<br />
it with your assigned floating IP address.<br />
<br />
# login to cloud.sharcnet.ca<br />
# Don’t forget volume size<br />
# Choose Debian 9.2.2 (not required, but the remaining instructions are Debian centric).<br />
# Choose persistent (ephemeral is for shorter jobs) (4C-8GB)<br />
# Setup keypair<br />
# Associate floating ip<br />
<br />
==(Optional) Apply IP Address to your name server==<br />
If you have a registered domain name you should apply your floating IP address to it so that you can use the "let's encrypt" service to <br />
enable the secure socket layer without client side warnings. It is often best to do this at the beginning as there is typically a delay<br />
in updating the name service.<br />
<br />
= Server Setup =<br />
<br />
==Login to Server & perform preliminaries==<br />
At this point log into the server to ensure that the service is up and running. There are a number of steps you can perform that will make <br />
the remaining steps easier. First loging to the server. Then setup a user prompt to make navigation easier. Switch to root (or use sudo in<br />
from of the remaining commands). Update and upgrade the system. Last install the manual pages.<br />
$ ssh debian@199.241.164.95<br />
$ echo 'export PS1="\[\e[33m\]\w\[\e[0m\]\n\[\e[32m\]\u@\h$ \[\e[0m\]"' >> .bash_aliases<br />
$ sudo su root<br />
$ apt update<br />
$ apt upgrade<br />
$ apt install man<br />
<br />
==Apt error==<br />
If you receive the "apt error" mesage put "nameserver 8.8.8.8" in /etc/resolve.conf.<br />
$ echo 'nameserver 8.8.8.8 >>' /etc/resolve.conf<br />
<br />
==Secure the SSH login==<br />
The following changes will disable logging into the server by using a password on any account and prevent logging into root remotly. Root can<br />
still be accessed by logging into a ''sudo'' enabled account and using the command ''su root''. For more information on the sshd_config file<br />
options, go [https://man.openbsd.org/sshd_config here]. The unattended-upgrades package is used to keep they server up to date automatically.<br />
$ sudo vim /etc/ssh/sshd_config<br />
ChallengeResponseAuthentication no<br />
PasswordAuthentication no<br />
PermitRootLogin no<br />
$ service ssh reload<br />
$ apt install unattended-upgrades<br />
$ dpkg-reconfigure --priority=low unattended-upgrades<br />
$ sudo unattended-upgrade -d<br />
<br />
=Install webserver and suppporting packages=<br />
==MYSQL==<br />
$ apt install mysql-server<br />
$ mysql_secure_installation #password=password<br />
<br />
==Apache==<br />
Installing the Apache webserver will allow you to access a default page at your ip address. (ex http://199.241.164.95, or http://frar.ca).<br />
Note that https will not yet work. httpd is the Apache HyperText Transfer Protocol (HTTP) server program. It is designed to be run as a standalone daemon process. When used like this it will create a pool of child processes or threads to handle requests. In general, httpd should not be invoked directly, but rather should be invoked via apachectl on Unix-based systems or as a service on Windows NT, 2000 and XP and as a console application on Windows 9x and ME.<br />
<br />
$ apt install apache2<br />
$ service apache2 start #add 80, 443 to default security group rules<br />
$ a2enmod cgid.load #(optional) enable cgi scripting, a2dismod to remove<br />
<br />
===Basic information & settings===<br />
# Document root Directory: /var/www/html or /var/www<br />
# Main Configuration file:<br />
## /etc/httpd/conf/httpd.conf (RHEL/CentOS/Fedora) <br />
## /etc/apache2/apache2.conf (Debian/Ubuntu).<br />
# Default HTTP Port: 80 TCP<br />
# Default HTTPS Port: 443 TCP<br />
# Access Log files of Web Server: /var/log/apache2/access_log<br />
# Error Log files of Web Server: /var/log/apache2/error_log<br />
# service apache2 {start|stop|graceful-stop|restart|reload|force-reload}<br />
# apachectl -v <br />
## [https://httpd.apache.org/docs/2.4/programs/apachectl.html see]<br />
<br />
===Secure Apache with SSL certificates===<br />
Going to https is now possible but will trigger a warning.<br />
$ a2enmod ssl<br />
$ a2ensite default-ssl.conf<br />
$ service apache2 restart<br />
<br />
===Enable let's encrypt (requires a domain name)===<br />
$ apt install git<br />
$ sudo git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt<br />
$ cd /opt/letsencrypt<br />
$ ./certbot-auto --authenticator webroot --installer apache<br />
www.frar.ca<br />
/var/www/html</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=Deploying_a_web_server&diff=16562Deploying a web server2018-05-28T14:44:12Z<p>Edward: /* Apache */</p>
<hr />
<div>Deploying a Web Server<br />
<br />
=Dependencies - Launch Cloud Instance=<br />
Before deploying a web server, first a cloud instance must be lauched. The instruction to do this can be found [[Carpet|here]].<br />
For the remainder of the instructions the ip address used is 199.241.164.95, this is for demonstration purposes and you should replace<br />
it with your assigned floating IP address.<br />
<br />
# login to cloud.sharcnet.ca<br />
# Don’t forget volume size<br />
# Choose Debian 9.2.2 (not required, but the remaining instructions are Debian centric).<br />
# Choose persistent (ephemeral is for shorter jobs) (4C-8GB)<br />
# Setup keypair<br />
# Associate floating ip<br />
<br />
==(Optional) Apply IP Address to your name server==<br />
If you have a registered domain name you should apply your floating IP address to it so that you can use the "let's encrypt" service to <br />
enable the secure socket layer without client side warnings. It is often best to do this at the beginning as there is typically a delay<br />
in updating the name service.<br />
<br />
= Server Setup =<br />
<br />
==Login to Server & perform preliminaries==<br />
At this point log into the server to ensure that the service is up and running. There are a number of steps you can perform that will make <br />
the remaining steps easier. First loging to the server. Then setup a user prompt to make navigation easier. Switch to root (or use sudo in<br />
from of the remaining commands). Update and upgrade the system. Last install the manual pages.<br />
$ ssh debian@199.241.164.95<br />
$ echo 'export PS1="\[\e[33m\]\w\[\e[0m\]\n\[\e[32m\]\u@\h$ \[\e[0m\]"' >> .bash_aliases<br />
$ sudo su root<br />
$ apt update<br />
$ apt upgrade<br />
$ apt install man<br />
<br />
==Apt error==<br />
If you receive the "apt error" mesage put "nameserver 8.8.8.8" in /etc/resolve.conf.<br />
$ echo 'nameserver 8.8.8.8 >>' /etc/resolve.conf<br />
<br />
==Secure the SSH login==<br />
The following changes will disable logging into the server by using a password on any account and prevent logging into root remotly. Root can<br />
still be accessed by logging into a ''sudo'' enabled account and using the command ''su root''. For more information on the sshd_config file<br />
options, go [https://man.openbsd.org/sshd_config here]. The unattended-upgrades package is used to keep they server up to date automatically.<br />
$ sudo vim /etc/ssh/sshd_config<br />
ChallengeResponseAuthentication no<br />
PasswordAuthentication no<br />
PermitRootLogin no<br />
$ service ssh reload<br />
$ apt install unattended-upgrades<br />
$ dpkg-reconfigure --priority=low unattended-upgrades<br />
$ sudo unattended-upgrade -d<br />
<br />
=Install webserver and suppporting packages=<br />
==MYSQL==<br />
$ apt install mysql-server<br />
$ mysql_secure_installation #password=password<br />
<br />
==Apache==<br />
Installing the Apache webserver will allow you to access a default page at your ip address. (ex http://199.241.164.95, or http://frar.ca).<br />
Note that https will not yet work. httpd is the Apache HyperText Transfer Protocol (HTTP) server program. It is designed to be run as a standalone daemon process. When used like this it will create a pool of child processes or threads to handle requests. In general, httpd should not be invoked directly, but rather should be invoked via apachectl on Unix-based systems or as a service on Windows NT, 2000 and XP and as a console application on Windows 9x and ME.<br />
<br />
$ apt install apache2<br />
$ service apache2 start #add 80, 443 to default security group rules<br />
$ a2enmod cgid.load #(optional) enable cgi scripting, a2dismod to remove<br />
<br />
===Basic information & settings===<br />
# Document root Directory: /var/www/html or /var/www<br />
# Main Configuration file:<br />
## /etc/httpd/conf/httpd.conf (RHEL/CentOS/Fedora) <br />
## /etc/apache2/apache2.conf (Debian/Ubuntu).<br />
# Default HTTP Port: 80 TCP<br />
# Default HTTPS Port: 443 TCP<br />
# Access Log files of Web Server: /var/log/apache2/access_log<br />
# Error Log files of Web Server: /var/log/apache2/error_log<br />
# service apache2 {start|stop|graceful-stop|restart|reload|force-reload}<br />
# apachectl -v <br />
## [https://httpd.apache.org/docs/2.4/programs/apachectl.html see]</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=Deploying_a_web_server&diff=16561Deploying a web server2018-05-28T14:41:39Z<p>Edward: /* Basic information & settings */</p>
<hr />
<div>Deploying a Web Server<br />
<br />
=Dependencies - Launch Cloud Instance=<br />
Before deploying a web server, first a cloud instance must be lauched. The instruction to do this can be found [[Carpet|here]].<br />
For the remainder of the instructions the ip address used is 199.241.164.95, this is for demonstration purposes and you should replace<br />
it with your assigned floating IP address.<br />
<br />
# login to cloud.sharcnet.ca<br />
# Don’t forget volume size<br />
# Choose Debian 9.2.2 (not required, but the remaining instructions are Debian centric).<br />
# Choose persistent (ephemeral is for shorter jobs) (4C-8GB)<br />
# Setup keypair<br />
# Associate floating ip<br />
<br />
==(Optional) Apply IP Address to your name server==<br />
If you have a registered domain name you should apply your floating IP address to it so that you can use the "let's encrypt" service to <br />
enable the secure socket layer without client side warnings. It is often best to do this at the beginning as there is typically a delay<br />
in updating the name service.<br />
<br />
= Server Setup =<br />
<br />
==Login to Server & perform preliminaries==<br />
At this point log into the server to ensure that the service is up and running. There are a number of steps you can perform that will make <br />
the remaining steps easier. First loging to the server. Then setup a user prompt to make navigation easier. Switch to root (or use sudo in<br />
from of the remaining commands). Update and upgrade the system. Last install the manual pages.<br />
$ ssh debian@199.241.164.95<br />
$ echo 'export PS1="\[\e[33m\]\w\[\e[0m\]\n\[\e[32m\]\u@\h$ \[\e[0m\]"' >> .bash_aliases<br />
$ sudo su root<br />
$ apt update<br />
$ apt upgrade<br />
$ apt install man<br />
<br />
==Apt error==<br />
If you receive the "apt error" mesage put "nameserver 8.8.8.8" in /etc/resolve.conf.<br />
$ echo 'nameserver 8.8.8.8 >>' /etc/resolve.conf<br />
<br />
==Secure the SSH login==<br />
The following changes will disable logging into the server by using a password on any account and prevent logging into root remotly. Root can<br />
still be accessed by logging into a ''sudo'' enabled account and using the command ''su root''. For more information on the sshd_config file<br />
options, go [https://man.openbsd.org/sshd_config here]. The unattended-upgrades package is used to keep they server up to date automatically.<br />
$ sudo vim /etc/ssh/sshd_config<br />
ChallengeResponseAuthentication no<br />
PasswordAuthentication no<br />
PermitRootLogin no<br />
$ service ssh reload<br />
$ apt install unattended-upgrades<br />
$ dpkg-reconfigure --priority=low unattended-upgrades<br />
$ sudo unattended-upgrade -d<br />
<br />
=Install webserver and suppporting packages=<br />
==MYSQL==<br />
$ apt install mysql-server<br />
$ mysql_secure_installation #password=password<br />
<br />
==Apache==<br />
Installing the Apache webserver will allow you to access a default page at your ip address. (ex http://199.241.164.95, or http://frar.ca).<br />
Note that https will not yet work.<br />
$ apt install apache2<br />
$ service apache2 start #add 80, 443 to default security group rules<br />
$ a2enmod cgid.load #(optional) enable cgi scripting, a2dismod to remove<br />
===Basic information & settings===<br />
# Document root Directory: /var/www/html or /var/www<br />
# Main Configuration file:<br />
## /etc/httpd/conf/httpd.conf (RHEL/CentOS/Fedora) <br />
## /etc/apache2/apache2.conf (Debian/Ubuntu).<br />
# Default HTTP Port: 80 TCP<br />
# Default HTTPS Port: 443 TCP<br />
# Access Log files of Web Server: /var/log/apache2/access_log<br />
# Error Log files of Web Server: /var/log/apache2/error_log<br />
# service apache2 {start|stop|graceful-stop|restart|reload|force-reload}<br />
# apachectl -v <br />
## [https://httpd.apache.org/docs/2.4/programs/apachectl.html see]</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=Deploying_a_web_server&diff=16560Deploying a web server2018-05-28T14:40:29Z<p>Edward: /* Apache */</p>
<hr />
<div>Deploying a Web Server<br />
<br />
=Dependencies - Launch Cloud Instance=<br />
Before deploying a web server, first a cloud instance must be lauched. The instruction to do this can be found [[Carpet|here]].<br />
For the remainder of the instructions the ip address used is 199.241.164.95, this is for demonstration purposes and you should replace<br />
it with your assigned floating IP address.<br />
<br />
# login to cloud.sharcnet.ca<br />
# Don’t forget volume size<br />
# Choose Debian 9.2.2 (not required, but the remaining instructions are Debian centric).<br />
# Choose persistent (ephemeral is for shorter jobs) (4C-8GB)<br />
# Setup keypair<br />
# Associate floating ip<br />
<br />
==(Optional) Apply IP Address to your name server==<br />
If you have a registered domain name you should apply your floating IP address to it so that you can use the "let's encrypt" service to <br />
enable the secure socket layer without client side warnings. It is often best to do this at the beginning as there is typically a delay<br />
in updating the name service.<br />
<br />
= Server Setup =<br />
<br />
==Login to Server & perform preliminaries==<br />
At this point log into the server to ensure that the service is up and running. There are a number of steps you can perform that will make <br />
the remaining steps easier. First loging to the server. Then setup a user prompt to make navigation easier. Switch to root (or use sudo in<br />
from of the remaining commands). Update and upgrade the system. Last install the manual pages.<br />
$ ssh debian@199.241.164.95<br />
$ echo 'export PS1="\[\e[33m\]\w\[\e[0m\]\n\[\e[32m\]\u@\h$ \[\e[0m\]"' >> .bash_aliases<br />
$ sudo su root<br />
$ apt update<br />
$ apt upgrade<br />
$ apt install man<br />
<br />
==Apt error==<br />
If you receive the "apt error" mesage put "nameserver 8.8.8.8" in /etc/resolve.conf.<br />
$ echo 'nameserver 8.8.8.8 >>' /etc/resolve.conf<br />
<br />
==Secure the SSH login==<br />
The following changes will disable logging into the server by using a password on any account and prevent logging into root remotly. Root can<br />
still be accessed by logging into a ''sudo'' enabled account and using the command ''su root''. For more information on the sshd_config file<br />
options, go [https://man.openbsd.org/sshd_config here]. The unattended-upgrades package is used to keep they server up to date automatically.<br />
$ sudo vim /etc/ssh/sshd_config<br />
ChallengeResponseAuthentication no<br />
PasswordAuthentication no<br />
PermitRootLogin no<br />
$ service ssh reload<br />
$ apt install unattended-upgrades<br />
$ dpkg-reconfigure --priority=low unattended-upgrades<br />
$ sudo unattended-upgrade -d<br />
<br />
=Install webserver and suppporting packages=<br />
==MYSQL==<br />
$ apt install mysql-server<br />
$ mysql_secure_installation #password=password<br />
<br />
==Apache==<br />
Installing the Apache webserver will allow you to access a default page at your ip address. (ex http://199.241.164.95, or http://frar.ca).<br />
Note that https will not yet work.<br />
$ apt install apache2<br />
$ service apache2 start #add 80, 443 to default security group rules<br />
$ a2enmod cgid.load #(optional) enable cgi scripting, a2dismod to remove<br />
===Basic information & settings===<br />
# Document root Directory: /var/www/html or /var/www<br />
# Main Configuration file:<br />
## /etc/httpd/conf/httpd.conf (RHEL/CentOS/Fedora) <br />
## /etc/apache2/apache2.conf (Debian/Ubuntu).<br />
# Default HTTP Port: 80 TCP<br />
# Default HTTPS Port: 443 TCP<br />
# Access Log files of Web Server: /var/log/apache2/access_log<br />
# Error Log files of Web Server: /var/log/apache2/error_log<br />
# service apache2 {start|stop|graceful-stop|restart|reload|force-reload}<br />
# apachectl -v</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=Deploying_a_web_server&diff=16559Deploying a web server2018-05-28T14:39:17Z<p>Edward: </p>
<hr />
<div>Deploying a Web Server<br />
<br />
=Dependencies - Launch Cloud Instance=<br />
Before deploying a web server, first a cloud instance must be lauched. The instruction to do this can be found [[Carpet|here]].<br />
For the remainder of the instructions the ip address used is 199.241.164.95, this is for demonstration purposes and you should replace<br />
it with your assigned floating IP address.<br />
<br />
# login to cloud.sharcnet.ca<br />
# Don’t forget volume size<br />
# Choose Debian 9.2.2 (not required, but the remaining instructions are Debian centric).<br />
# Choose persistent (ephemeral is for shorter jobs) (4C-8GB)<br />
# Setup keypair<br />
# Associate floating ip<br />
<br />
==(Optional) Apply IP Address to your name server==<br />
If you have a registered domain name you should apply your floating IP address to it so that you can use the "let's encrypt" service to <br />
enable the secure socket layer without client side warnings. It is often best to do this at the beginning as there is typically a delay<br />
in updating the name service.<br />
<br />
= Server Setup =<br />
<br />
==Login to Server & perform preliminaries==<br />
At this point log into the server to ensure that the service is up and running. There are a number of steps you can perform that will make <br />
the remaining steps easier. First loging to the server. Then setup a user prompt to make navigation easier. Switch to root (or use sudo in<br />
from of the remaining commands). Update and upgrade the system. Last install the manual pages.<br />
$ ssh debian@199.241.164.95<br />
$ echo 'export PS1="\[\e[33m\]\w\[\e[0m\]\n\[\e[32m\]\u@\h$ \[\e[0m\]"' >> .bash_aliases<br />
$ sudo su root<br />
$ apt update<br />
$ apt upgrade<br />
$ apt install man<br />
<br />
==Apt error==<br />
If you receive the "apt error" mesage put "nameserver 8.8.8.8" in /etc/resolve.conf.<br />
$ echo 'nameserver 8.8.8.8 >>' /etc/resolve.conf<br />
<br />
==Secure the SSH login==<br />
The following changes will disable logging into the server by using a password on any account and prevent logging into root remotly. Root can<br />
still be accessed by logging into a ''sudo'' enabled account and using the command ''su root''. For more information on the sshd_config file<br />
options, go [https://man.openbsd.org/sshd_config here]. The unattended-upgrades package is used to keep they server up to date automatically.<br />
$ sudo vim /etc/ssh/sshd_config<br />
ChallengeResponseAuthentication no<br />
PasswordAuthentication no<br />
PermitRootLogin no<br />
$ service ssh reload<br />
$ apt install unattended-upgrades<br />
$ dpkg-reconfigure --priority=low unattended-upgrades<br />
$ sudo unattended-upgrade -d<br />
<br />
=Install webserver and suppporting packages=<br />
==MYSQL==<br />
$ apt install mysql-server<br />
$ mysql_secure_installation #password=password<br />
<br />
==Apache==<br />
Installing the Apache webserver will allow you to access a default page at your ip address. (ex http://199.241.164.95, or http://frar.ca).<br />
Note that https will not yet work.<br />
$ apt install apache2<br />
$ service apache2 start #add 80, 443 to default security group rules<br />
$ a2enmod cgid.load #(optional) enable cgi scripting, a2dismod to remove</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=Deploying_a_web_server&diff=16558Deploying a web server2018-05-28T14:32:43Z<p>Edward: </p>
<hr />
<div>Deploying a Web Server<br />
<br />
=Dependencies - Launch Cloud Instance=<br />
Before deploying a web server, first a cloud instance must be lauched. The instruction to do this can be found [[Carpet|here]].<br />
For the remainder of the instructions the ip address used is 199.241.164.95, this is for demonstration purposes and you should replace<br />
it with your assigned floating IP address.<br />
<br />
# login to cloud.sharcnet.ca<br />
# Don’t forget volume size<br />
# Choose Debian 9.2.2 (not required, but the remaining instructions are Debian centric).<br />
# Choose persistent (ephemeral is for shorter jobs) (4C-8GB)<br />
# Setup keypair<br />
# Associate floating ip<br />
<br />
==(Optional) Apply IP Address to your name server==<br />
If you have a registered domain name you should apply your floating IP address to it so that you can use the "let's encrypt" service to <br />
enable the secure socket layer without client side warnings. It is often best to do this at the beginning as there is typically a delay<br />
in updating the name service.<br />
<br />
= Server Setup =<br />
<br />
==Login to Server & perform preliminaries==<br />
At this point log into the server to ensure that the service is up and running. There are a number of steps you can perform that will make <br />
the remaining steps easier. First loging to the server. Then setup a user prompt to make navigation easier. Switch to root (or use sudo in<br />
from of the remaining commands). Update and upgrade the system. Last install the manual pages.<br />
$ ssh debian@199.241.164.95<br />
$ echo 'export PS1="\[\e[33m\]\w\[\e[0m\]\n\[\e[32m\]\u@\h$ \[\e[0m\]"' >> .bash_aliases<br />
$ sudo su root<br />
$ apt update<br />
$ apt upgrade<br />
$ apt install man<br />
<br />
==Apt error==<br />
If you receive the "apt error" mesage put "nameserver 8.8.8.8" in /etc/resolve.conf.<br />
$ echo 'nameserver 8.8.8.8 >>' /etc/resolve.conf<br />
<br />
==Secure the SSH login==<br />
The following changes will disable logging into the server by using a password on any account and prevent logging into root remotly. Root can<br />
still be accessed by logging into a ''sudo'' enabled account and using the command ''su root''. For more information on the sshd_config file<br />
options, go [https://man.openbsd.org/sshd_config here]. The unattended-upgrades package is used to keep they server up to date automatically.<br />
$ sudo vim /etc/ssh/sshd_config<br />
ChallengeResponseAuthentication no<br />
PasswordAuthentication no<br />
PermitRootLogin no<br />
$ service ssh reload<br />
$ apt install unattended-upgrades<br />
$ dpkg-reconfigure --priority=low unattended-upgrades<br />
$ sudo unattended-upgrade -d<br />
<br />
=Install webserer and suppporting packages=<br />
==MYSQL==<br />
$ apt install mysql-server<br />
$ mysql_secure_installation #password=password<br />
<br />
==Apache==<br />
Installing the Apache webserver will allow you to access a default page at your ip address. (ex http://199.241.164.95, or http://frar.ca).<br />
<br />
$ apt install apache2<br />
$ service apache2 start #add 80, 443 to default security group rules<br />
$ a2enmod cgid.load #(optional) enable cgi scripting, a2dismod to remove</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=Deploying_a_web_server&diff=16557Deploying a web server2018-05-28T14:27:35Z<p>Edward: </p>
<hr />
<div>Deploying a Web Server<br />
<br />
=Dependencies - Launch Cloud Instance=<br />
Before deploying a web server, first a cloud instance must be lauched. The instruction to do this can be found [[Carpet|here]].<br />
For the remainder of the instructions the ip address used is 199.241.164.95, this is for demonstration purposes and you should replace<br />
it with your assigned floating IP address.<br />
<br />
# login to cloud.sharcnet.ca<br />
# Don’t forget volume size<br />
# Choose Debian 9.2.2 (not required, but the remaining instructions are Debian centric).<br />
# Choose persistent (ephemeral is for shorter jobs) (4C-8GB)<br />
# Setup keypair<br />
# Associate floating ip<br />
<br />
==(Optional) Apply IP Address to your name server==<br />
If you have a registered domain name you should apply your floating IP address to it so that you can use the "let's encrypt" service to <br />
enable the secure socket layer without client side warnings. It is often best to do this at the beginning as there is typically a delay<br />
in updating the name service.<br />
<br />
= Server Setup =<br />
<br />
==Login to Server & perform preliminaries==<br />
At this point log into the server to ensure that the service is up and running. There are a number of steps you can perform that will make <br />
the remaining steps easier. First loging to the server. Then setup a user prompt to make navigation easier. Switch to root (or use sudo in<br />
from of the remaining commands). Update and upgrade the system. Last install the manual pages.<br />
$ ssh debian@199.241.164.95<br />
$ echo 'export PS1="\[\e[33m\]\w\[\e[0m\]\n\[\e[32m\]\u@\h$ \[\e[0m\]"' >> .bash_aliases<br />
$ sudo su root<br />
$ apt update<br />
$ apt upgrade<br />
$ apt install man<br />
<br />
==Apt error==<br />
If you receive the "apt error" mesage put "nameserver 8.8.8.8" in /etc/resolve.conf.<br />
$ echo 'nameserver 8.8.8.8 >>' /etc/resolve.conf<br />
<br />
==Secure the SSH login==<br />
The following changes will disable logging into the server by using a password on any account and prevent logging into root remotly. Root can<br />
still be accessed by logging into a ''sudo'' enabled account and using the command ''su root''. For more information on the sshd_config file<br />
options, go [https://man.openbsd.org/sshd_config here]. The unattended-upgrades package is used to keep they server up to date automatically.<br />
$ sudo vim /etc/ssh/sshd_config<br />
ChallengeResponseAuthentication no<br />
PasswordAuthentication no<br />
PermitRootLogin no<br />
$ service ssh reload<br />
$ apt install unattended-upgrades<br />
$ dpkg-reconfigure --priority=low unattended-upgrades<br />
$ sudo unattended-upgrade -d</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=Deploying_a_web_server&diff=16556Deploying a web server2018-05-28T14:04:05Z<p>Edward: </p>
<hr />
<div>Deploying a Web Server<br />
<br />
=Dependencies - Launch Cloud Instance=<br />
Before deploying a web server, first a cloud instance must be lauched. The instruction to do this can be found [[Carpet|here]].<br />
For the remainder of the instructions the ip address used is 199.241.164.95, this is for demonstration purposes and you should replace<br />
it with your assigned floating IP address.<br />
<br />
# login to cloud.sharcnet.ca<br />
# Don’t forget volume size<br />
# Choose Debian 9.2.2 (not required, but the remaining instructions are Debian centric).<br />
# Choose persistent (ephemeral is for shorter jobs) (4C-8GB)<br />
# Setup keypair<br />
# Associate floating ip<br />
<br />
==(Optional) Apply IP Address to your name server==<br />
If you have a registered domain name you should apply your floating IP address to it so that you can use the "let's encrypt" service to <br />
enable the secure socket layer without client side warnings. It is often best to do this at the beginning as there is typically a delay<br />
in updating the name service.<br />
<br />
=Login to Server & perform preliminaries=<br />
At this point log into the server to ensure that the service is up and running. There are a number of steps you can perform that will make <br />
the remaining steps easier. First loging to the server. Then setup a user prompt to make navigation easier. Switch to root (or use sudo in<br />
from of the remaining commands). Update and upgrade the system. Last install the manual pages.<br />
$ ssh debian@199.241.164.95<br />
$ echo 'export PS1="\[\e[33m\]\w\[\e[0m\]\n\[\e[32m\]\u@\h$ \[\e[0m\]"' >> .bash_aliases<br />
$ sudo su root<br />
$ apt update<br />
$ apt upgrade<br />
$ apt install man</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=Deploying_a_web_server&diff=16555Deploying a web server2018-05-28T13:44:29Z<p>Edward: </p>
<hr />
<div>Deploying a Web Server<br />
<br />
=Dependencies - Launch Cloud Instance=<br />
Before deploying a web server, first a cloud instance must be lauched. The instruction to do this can be found [[Carpet|here]].<br />
# login to cloud.sharcnet.ca<br />
# Don’t forget volume size<br />
# Choose Debian 9.2.2 (not required, but the remaining instructions are Debian centric).<br />
# Choose persistent (ephemeral is for shorter jobs) (4C-8GB)<br />
# Setup keypair<br />
# Associate floating ip<br />
<br />
==(Optional) Apply IP Address to your name server==<br />
If you have a registered domain name you should apply your floating IP address to it so that you can use the "let's encrypt" service to <br />
enable the secure socket layer without client side warnings. It is often best to do this at the beginning as there is typically a delay<br />
in updating the name service.<br />
<br />
=Login to Server & perform preliminaries=</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=Deploying_a_web_server&diff=16554Deploying a web server2018-05-28T13:38:48Z<p>Edward: </p>
<hr />
<div>Deploying a Web Server<br />
<br />
=Dependencies=<br />
Before deploying a web server, first a cloud instance must be lauched. [[See page|Carpet]].</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=Deploying_a_web_server&diff=16553Deploying a web server2018-05-28T13:37:01Z<p>Edward: Created page with "Deploying a Web Server"</p>
<hr />
<div>Deploying a Web Server</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=File:Winsshfs.png&diff=15561File:Winsshfs.png2017-08-11T14:48:50Z<p>Edward: Edward uploaded a new version of File:Winsshfs.png</p>
<hr />
<div></div>Edwardhttps://www.sharcnet.ca/help/index.php?title=File:Winsshfs.png&diff=15560File:Winsshfs.png2017-08-11T14:48:04Z<p>Edward: Edward uploaded a new version of File:Winsshfs.png</p>
<hr />
<div></div>Edwardhttps://www.sharcnet.ca/help/index.php?title=File:Winsshfs.png&diff=15559File:Winsshfs.png2017-08-11T14:47:43Z<p>Edward: Edward uploaded a new version of File:Winsshfs.png</p>
<hr />
<div></div>Edwardhttps://www.sharcnet.ca/help/index.php?title=File:Winsshfs.png&diff=15558File:Winsshfs.png2017-08-11T14:46:57Z<p>Edward: Edward uploaded a new version of File:Winsshfs.png</p>
<hr />
<div></div>Edwardhttps://www.sharcnet.ca/help/index.php?title=File:Winsshfs.png&diff=15552File:Winsshfs.png2017-08-11T14:27:25Z<p>Edward: </p>
<hr />
<div></div>Edwardhttps://www.sharcnet.ca/help/index.php?title=OCTAVE&diff=15528OCTAVE2017-08-01T18:04:02Z<p>Edward: </p>
<hr />
<div>{{GrahamUpdate}}<br />
<!--checked2016--><br />
{{Software<br />
|package_name=OCTAVE<br />
|package_description=Mostly compatible language with Matlab primarily intended for numerical computations<br />
|package_idnumber=28<br />
}}<br />
<br />
=Introduction=<br />
<br />
Octave is provided on SHARCNET clusters to allow serial or threaded jobs to be run in the queue as described below.<br />
<br />
=Version Selection=<br />
<br />
To see what versions of Octave are available on sharcnet clusters consult the Availability table on the sharcnet OCTAVE web portal software page OR run the following command directly on a cluster:<br />
<br />
<pre>module avail</pre><br />
<br />
Starting with version 3.6.3 its necessary to first unload the intel compiler module (or any other compiler that might be loaded) before loading the Octave module. This is done to ensure any octave-forge packages that are to be downloaded and installed into user space are built with the native gcc compiler which is version 4.4.6 at the time of this writing. Therefore to load octave/3.6.3 one would do:<br />
<br />
<pre><br />
module unload intel<br />
module load octave/3.6.3<br />
</pre><br />
<br />
Similarly, to load the default Octave module (currently version 3.8.1) one would do:<br />
<br />
module unload intel<br />
module load octave<br />
<br />
==On the Graham national system==<br />
The module setup on the new national systems general purpose systems Graham and Cedar are slightly different from other SHARCNET systems. The module loading procedures introduce more automation of dependency interactions.<br />
<br />
On the Graham system the user simply needs to call the "module load" command without manually unloading conflicting packages.<br />
<br />
$ module load octave/4.2.1<br />
<br />
To obtain information about the package's dependencies the "spider" action can be included in the call to "module":<br />
<br />
<pre><br />
$ module spider octave<br />
<br />
---------------------------------------------------------------------------------------------------------------------------------------------------------<br />
octave:<br />
---------------------------------------------------------------------------------------------------------------------------------------------------------<br />
Description:<br />
GNU Octave is a high-level interpreted language, primarily intended for numerical computations. - Homepage: http://www.gnu.org/software/octave/<br />
<br />
Versions:<br />
octave/4.2.1<br />
<br />
---------------------------------------------------------------------------------------------------------------------------------------------------------<br />
For detailed information about a specific "octave" module (including how to load the modules) use the module's full name.<br />
For example:<br />
<br />
$ module spider octave/4.2.1<br />
---------------------------------------------------------------------------------------------------------------------------------------------------------<br />
<br />
<br />
<br />
[jdesjard@gra-login1 ~]$ module spider octave/4.2.1<br />
<br />
---------------------------------------------------------------------------------------------------------------------------------------------------------<br />
octave: octave/4.2.1<br />
---------------------------------------------------------------------------------------------------------------------------------------------------------<br />
Description:<br />
GNU Octave is a high-level interpreted language, primarily intended for numerical computations. - Homepage: http://www.gnu.org/software/octave/<br />
<br />
Properties:<br />
Tools for development / Outils de développement<br />
<br />
You will need to load all module(s) on any one of the lines below before the "octave/4.2.1" module is available to load.<br />
<br />
nixpkgs/16.09 gcc/5.4.0<br />
nixpkgs/16.09 intel/2016.4<br />
nixpkgs/16.09 intel/2017.1<br />
<br />
Help:<br />
GNU Octave is a high-level interpreted language, primarily intended for numerical computations. - Homepage: http://www.gnu.org/software/octave/<br />
</pre><br />
<br />
=Job Submission=<br />
<br />
On SHARCNET clusters for production work Octave should only be run via the queuing system. Octave serial jobs can be submitted to the serial queue using following sqsub command:<br />
<br />
<pre>sqsub -r 60m -o ofile.%J octave mycode.m</pre><br />
<br />
As of version 3.4.3 the sharcnet octave installation supports multi-threading which based on initial testing lapack/blas intensive octave jobs run in the threaded queue achieve an order of magnitude speedup compared to running single core jobs in the serial queue, without making any changes to your code. Once the optimal number of processors is determined by scaling tests submit the job to the threaded queue for example:<br />
<br />
<pre>sqsub -r 60m -n 8 -q threaded --mpp=1G -o ofile.%J time octave mycode.m</pre><br />
<br />
<br />
==On the Graham national system==<br />
<br />
The Graham national system uses the Slurm scheduler which is different from the Torque Moab scheduler that is typical to the majority of other SHARCNET systems. In order to submit a simple "Hellow World!" process to Graham the user would first need to create a submit script as follows.<br />
<br />
$ cat oct_serial.sh <br />
#!/bin/bash<br />
#SBATCH -t 0-00:01<br />
#SBATCH --mem=400<br />
octave --eval 'disp("Hello World! from Octave")'<br />
<br />
Then pass this submit script to "sbatch" to be entered into the queue:<br />
<br />
$ sbatch --account=myusername oct_serial.sh<br />
<br />
=Example Job=<br />
<br />
This section shows howto submit a [http://www.gnu.org/software/octave/doc/interpreter/Executable-Octave-Programs.html sample.m] file to the serial queue that accepts [http://www.gnu.org/software/octave/doc/interpreter/Command-Line-Options.html#Command-Line-Options command line] arguments. <br />
<br />
<pre><br />
[roberpj@hnd20:~/samples/octave/args] cat sample.m <br />
#! /bin/octave -qf<br />
printf ("%s", program_name ());<br />
arg_list = argv ();<br />
for i = 1:nargin<br />
printf (" %s", arg_list{i});<br />
endfor<br />
printf ("\n");<br />
</pre><br />
<br />
To eliminate exatraneous verbosity in the output file two switches are passed:<br />
<br />
<pre><br />
[roberpj@hnd20:~/samples/octave/args] sqsub -r 60m -o ofile.%J octave -qf --no-window-system sample.m arg1 arg2 arg3 etc<br />
WARNING: no memory requirement defined; assuming 2GB per process.<br />
submitted as jobid 6937872<br />
</pre><br />
<br />
The output file from the job appears as:<br />
<br />
<pre><br />
[roberpj@hnd20:~/samples/octave/args] cat ofile.6937872.hnd50<br />
sample.m arg1 arg2 arg3 etc<br />
</pre><br />
<br />
=Running in the development nodes=<br />
<br />
Besides running production work in Octave via the scheduler on SharcNET systems it is also possible to use Octave interactively on systems' development nodes.<br />
<br />
To start Octave from a development node one can login to a system with trusted X11 forwarding enabled:<br />
<br />
ssh -Y username@kraken.sharcnet.ca<br />
<br />
Then login to one of the development nodes with trusted X11 forwarding enabled:<br />
<br />
ssh -Y kraken-devel1<br />
<br />
Once logged into the development node one can load the Octave modules and launch the program:<br />
<br />
module unload intel<br />
module load octave<br />
octave<br />
<br />
<pre><br />
GNU Octave, version 3.8.1<br />
Copyright (C) 2014 John W. Eaton and others.<br />
This is free software; see the source code for copying conditions.<br />
There is ABSOLUTELY NO WARRANTY; not even for MERCHANTABILITY or<br />
FITNESS FOR A PARTICULAR PURPOSE. For details, type 'warranty'.<br />
<br />
Octave was configured for "x86_64-unknown-linux-gnu".<br />
<br />
Additional information about Octave is available at http://www.octave.org.<br />
<br />
Please contribute if you find this software useful.<br />
For more information, visit http://www.octave.org/get-involved.html<br />
<br />
Read http://www.octave.org/bugs.html to learn how to submit bug reports.<br />
For information about changes from previous versions, type 'news'.<br />
<br />
octave:1> <br />
<br />
</pre><br />
<br />
With X11 forwarding enabled in the development node interactive work with Octave can include figure generation.<br />
<br />
==One the Graham nation system==<br />
<br />
Rather than dedicated development as it typical on some of the other SHARCNET systems the national systems have nodes that prioritize interactive jobs allocated by call to "salloc" of the Slurm scheduler.<br />
<br />
To use Octave interactively on Graham you can do the following:<br />
<br />
Use salloc to request a single task for one hour:<br />
<br />
$ salloc --account=def-roxa88 --time=1:0:0 --ntasks=1<br />
<br />
Then you will receive notification when the reservation is granted (including jobid), for<br />
example:<br />
<br />
salloc: Granted job allocation 12345<br />
<br />
Then use srun to start a terminal to work in on the reservation:<br />
<br />
$ srun --wait 0 --pty bash<br />
<br />
Once running the terminal in the allocation you can start your work with Octave by loading<br />
the module and starting the program:<br />
<br />
<pre><br />
$ module load octave<br />
$ octave<br />
octave: X11 DISPLAY environment variable not set<br />
octave: disabling GUI features<br />
GNU Octave, version 4.2.1<br />
Copyright (C) 2017 John W. Eaton and others.<br />
This is free software; see the source code for copying conditions.<br />
There is ABSOLUTELY NO WARRANTY; not even for MERCHANTABILITY or<br />
FITNESS FOR A PARTICULAR PURPOSE. For details, type 'warranty'.<br />
<br />
Octave was configured for "x86_64-pc-linux-gnu".<br />
<br />
Additional information about Octave is available at http://www.octave.org.<br />
<br />
Please contribute if you find this software useful.<br />
For more information, visit http://www.octave.org/get-involved.html<br />
<br />
Read http://www.octave.org/bugs.html to learn how to submit bug reports.<br />
For information about changes from previous versions, type 'news'.<br />
<br />
octave:1><br />
<br />
</pre><br />
<br />
... Then once you have the "octave:#>" prompt the terminal will behave like the Matlab<br />
command line.<br />
<br />
Once you are finished with Octave use "quit" to exit the program, then "exit" to logout of<br />
the compute node (terminate srun), then "exit" again to terminate the allocation<br />
(relinquishing the allocate resources).<br />
<br />
=Running on a visualization system=<br />
<br />
Once logged in to a visualization system an instance of Octave can be launched from Applications Menu > Eduction > GNU Octave.<br />
<br />
=General notes=<br />
<br />
==Matlab Compatibility==<br />
<br />
The online wiki resources "Octave Wiki":<br />
http://wiki.octave.org/ <br />
or "Wikibook":<br />
http://en.wikibooks.org/wiki/MATLAB_Programming/Differences_between_Octave_and_MATLAB <br />
provide good introductory explanations of code compatibility between Octave and Matlab.<br />
<br />
In order to run Octave so that it interprets scripts as closely as possible to Matlab, use the --traditional flag. Taking from the above sqsub example the maximum Matlab compatible submission would be:<br />
<br />
sqsub -r 60m -o ofile.%J octave --traditional mycode.m<br />
<br />
==Reading/Writing Files==<br />
<br />
There are two strategies for handling file <i>input and output</i> described in the "Octave manual" viz ...<br><br />
http://www.gnu.org/software/octave/doc/interpreter/Input-and-Output.html#Input-and-Output<br> http://www.gnu.org/software/octave/docs.html.<br />
<br />
The following stanza demonstrates the "simple file I/O" approach:<br />
http://www.gnu.org/software/octave/doc/interpreter/Simple-File-I_002fO.html#Simple-File-I_002fO <br />
<br />
<pre><br />
save myiofile.dat A B C<br />
save ("-text", "myiofile.dat", "A", "B", "C")<br />
save ("-binary", "myiofile.dat", "A", "B", "C")<br />
load myiofile.dat<br />
load ("-text", "myiofile.dat", "A", "B", "C")<br />
load ("-binary", "myiofile.dat", "A", "B", "C")<br />
</pre><br />
<br />
where A, B and C are a potential mix of entities such as scalars, vectors or matrices. Note that for large files the binary format is strongly recommended to both minimize disk space and file read/write wallclock times.<br />
<br />
==Octave-Forge==<br />
<br />
Octave-forge packages are not pre-installed on SHARCNET clusters since none are available at this time for the operating system. Therefore any required packages must be downloaded, manually installed from source and then managed in local user accounts (as described below).<br />
<br />
However 16 of approximately 95 octave-forge packages are installed on some sharcnet visualization workstations including viz1,2,3-uwaterloo, viz2-uwo, viz6-uoguelph under /usr/share/octave/packages. As a word of caution however, all package versions will likely be significantly old since they are tied to the operating system fedora /etc/redhat-release major version and hence not contain recent critical bug fixes (which could be numerical in nature). Users are therefore strongly recommended to likewise download and install the latest version similarly as would be done on the clusters.<br />
<br />
===Sharing Packages===<br />
<br />
Note that packages installed by a single sharcnet user in their home account can be shared out to other research group members or even among general SHARCNET users. To do this simply set access permissions for group or global read access accordingly. For details on how to change <br />
the permissions see: https://www.sharcnet.ca/help/index.php/Knowledge_Base#How_do_I_give_other_users_access_to_my_files_.3F<br />
<br />
===Managing Packages===<br />
<br />
The "Octave Forge" http://octave.sourceforge.net/ project provides extra packages for use with octave that can be downloaded into a directory such as <i>~/my_octave_sources</i> then manually installed into your own sharcnet account as will be shown in the follow example. A current list of packages available for download that notable are generally only compatible with the latest major release of octave cat be found here http://octave.sourceforge.net/packages.php. <br />
<br />
In the event where the major version of octave on sharcnet you are using (for instance 3.4.x) is not the same as latest major version (at the time of writing 3.6.x) and there were programming api changes between the two versions that effect the package you want to use, then you probably will need to download an older version of a package from http://sourceforge.net/projects/octave/files/Octave%20Forge%20Packages/Individual%20Package%20Releases/. for it to work.<br />
<br />
To help estimate the package version required as a starting point, consider the major release dates of recent which are octave-3.4.3 (10/10/11), octave-3.6.0 (01/15/12), octave-3.6.1 (02/22/12) and finally octave-3.6.2 (05/31/12). Next consider you want to download the latest geometry package that was compatible with octave-3.4.3 at the time of its release, then simply display all the archived versions as shown in the following stanza and pick the last available geometry release date before the next major release octave-3.6.0 (01/15/12). To show a list of all archived geometry versions, do the following steps:<br />
<br />
<pre><br />
http://octave.sourceforge.net/<br />
Click Packages on top menu menu bar<br />
Scroll down to the miscellaneous package row and click details<br />
Click (older versions) located below "Download Package"<br />
Click Octave Forge Packages<br />
Click Individual Package Releases<br />
Please wait for the page to load ...<br />
Click "Name" at the top of first colum to sort packages alphabetically<br />
Scroll down you will find all available archived geometry packages:<br />
geometry-1.0.1.tar.gz 2011-09-27<br />
geometry-1.1.1.tar.gz 2011-10-06<br />
geometry-1.1.2.tar.gz 2011-10-09<br />
geometry-1.1.3.tar.gz 2011-10-13<br />
geometry-1.1.tar.gz 2011-10-04<br />
geometry-1.2.0.tar.gz 2011-10-22<br />
geometry-1.2.1.tar.gz 2011-11-02<br />
geometry-1.2.2.tar.gz 2011-11-04<br />
geometry-1.4.0.tar.gz 2012-01-25<br />
geometry-1.4.1.tar.gz 2012-03-24<br />
geometry-1.5.0.tar.gz 2012-06-05<br />
</pre><br />
<br />
Therefore you will download geometry-1.2.2.tar.gz (2011-11-04) since the next<br />
release geometry-1.4.0.tar.gz (2012-01-25) and then install it into octave<br />
3.4.3 as follows:<br />
<br />
<pre><br />
[roberpj@tope:~/my_octave_sources] octave<br />
GNU Octave, version 3.4.3<br />
octave:1> pkg install geometry-1.2.2.tar.gz<br />
octave:2> pkg list<br />
Package Name | Version | Installation directory<br />
---------------+---------+-----------------------<br />
geometry | 1.2.2 | /home/roberpj/octave/geometry-1.2.2<br />
octave:15> pkg load geometry<br />
octave:16> pkg describe geometry<br />
---<br />
Package name:<br />
geometry<br />
Version:<br />
1.2.2<br />
Short description:<br />
Library for geometric computing extending MatGeom functions. Useful to create,<br />
transform, manipulate and display geometric primitives.<br />
Status:<br />
Loaded<br />
</pre><br />
<br />
<b>Note:</b> Any questions regarding package version compatibility with major or minor octave release should be referred to the developers.<br />
<br />
===Package Commands===<br />
<br />
Additional examples of package commands are shown in this section. For demonstration purpose linear-algebra will first be downloaded:<br />
<br />
<pre><br />
[roberpj@iqaluk:~] mkdir my_octave_sources<br />
[roberpj@iqaluk:~] cd my_octave_sources<br />
wget http://downloads.sourceforge.net/octave/general-1.3.2.tar.gz<br />
wget http://downloads.sourceforge.net/octave/linear-algebra-2.2.0.tar.gz<br />
</pre><br />
<br />
<pre><br />
[myusername@orc-login1:~]octave<br />
octave:1> help pkg<br />
</pre><br />
<br />
To install a package such as linear-algebra do:<br />
<br />
<pre><br />
octave:2> cd ~/my_octave_sources<br />
octave:3> pkg install general-1.3.2.tar.gz<br />
octave:4> pkg install linear-algebra-2.2.0.tar.gz<br />
</pre><br />
<br />
To remove a package, from a terminal first do:<br />
<pre>cd ~/octave<br />
rm -rf linear-algebra-2.2.0</pre><br />
<br />
... then from within octave do:<br />
<pre>octave:5> pkg uninstall linear-algebra</pre><br />
<br />
To list all installed packages do:<br />
<br />
<pre>octave:6> pkg list</pre><br />
<br />
To add a named packages to your path:<br />
<br />
<pre>octave:7> pkg load name</pre><br />
<br />
To remove a named package from your path:<br />
<br />
<pre>octave:8> pkg unload</pre><br />
<br />
To list all functions provided by a package:<br />
<br />
<pre>octave:9> pkg describe -verbose all</pre><br />
<br />
To list functions provide by extra odepkg package:<br />
<br />
<pre>octave:10> pkg describe odepkg</pre><br />
<br />
To list functions details for extra financial package:<br />
<br />
<pre>octave:11> pkg describe financial -verbose</pre><br />
<br />
To get documentation for a topic such as sin do:<br />
<br />
<pre>octave:12> doc sin</pre><br />
<br />
To get help using the doc command do:<br />
<br />
<pre>octave:13> help doc</pre><br />
<br />
To get documentation for a topic such as matrix do:<br />
<br />
<pre>octave:14> doc matrix</pre><br />
<br />
To get documentation for any topic run the doc command alone:<br />
<br />
<pre>octave:15> doc</pre><br />
<br />
===Package Example===<br />
<br />
This example assumes you first create two directories in your home account one being named<br />
~/my_octave_packages <br />
and the other <br />
~/my_octave_sources <br />
then download the required three tar.gz files into the latter from <br />
http://octave.sourceforge.net/packages.php ...<br />
<br />
[roberpj@orc-login1:~] module unload octave<br />
[roberpj@orc-login1:~] module load octave<br />
[roberpj@orc-login1:~] octave<br />
GNU Octave, version 3.4.3<br />
octave:1> pkg prefix ~/my_octave_packages<br />
ans = /home/roberpj/my_octave_packages<br />
octave:2> cd my_octave_sources<br />
octave:3> ls<br />
miscellaneous-1.0.11.tar.gz optim-1.0.17.tar.gz struct-1.0.9.tar.gz<br />
octave:4> pkg install miscellaneous-1.0.11.tar.gz optim-1.0.17.tar.gz struct-1.0.9.tar.gz<br />
octave:5> pkg list<br />
Package Name | Version | Installation directory<br />
---------------|---------|-----------------------<br />
miscellaneous *| 1.0.11 | /home/roberpj/my_octave_packages/miscellaneous-1.0.11<br />
optim *| 1.0.17 | /home/roberpj/my_octave_packages/optim-1.0.17<br />
struct *| 1.0.9 | /home/roberpj/my_octave_packages/struct-1.0.9</PRE><br />
<br />
=References=<br />
<br />
o Octave Homepage<br><br />
http://www.gnu.org/software/octave/<br />
<br />
o Octave 725 Page Manual (Version 3.4.0)<br><br />
http://www.gnu.org/software/octave/doc/interpreter/<br />
<br />
o Statistic Package Function Reference<br><br />
http://octave.sourceforge.net/doc/funref_statistics.html<br />
<br />
o GNU Octave Wiki<br><br />
http://wiki.octave.org/<br />
<br />
o Matlab-Like Tools for HPC (article)<br><br />
http://www.admin-magazine.com/HPC/Articles/Matlab-Like-Tools-for-HPC</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=VALGRIND&diff=15523VALGRIND2017-07-27T12:52:26Z<p>Edward: </p>
<hr />
<div>{{GrahamUpdate}}<br />
{{Software<br />
|package_name=VALGRIND<br />
|package_description=Memory debugger<br />
|package_idnumber=132<br />
}}<br />
'''Valgrind''' is a powerful tool for analyzing programs, memory debugging, memory leak detection and profiling. It is freely available under GNU license. Version 3.5.0 is available on most SHARCNET systems.<br />
<br />
'''NOTE''' To avoid spurious warnings it is important to not use too new of a version of GCC or OpenMPI. We recommend<br />
<br />
* gcc/4.8.2<br />
* openmpi/gcc-debug/1.8.3 modules<br />
<br />
= Overview =<br />
<br />
Valgrind is a dynamic binary instrumentation framework that dynamically translates executables to add instrumentation and track all memory and register usages by a program. The advantages of this approach are that<br />
<br />
* it can be directly run on any executable, and<br />
* dynamic translation allows ultimate instrumentation<br />
<br />
while the disadvantages are<br />
<br />
* 5-100 x slow down depending on tool,<br />
* 12-18 x increase in size of translated code, and<br />
* corner cases may exist between translated code and original.<br />
<br />
Several tools have been built upon this framework. These include<br />
<br />
* ''memcheck'' - memory error detector<br />
* ''cachegrind'' - cache and branch-prediction profiler<br />
* ''callgrind'' - call-graph generating cache and branch prediction profiler<br />
* ''helgrind'' - thread error detector<br />
* ''DRD'' - thread error detector<br />
* ''Massif'' - heap profiler<br />
* ''DHAT'' - dynamic heap analysis tool<br />
* ''SGCheck'' - experimental stack and global array overrun detector<br />
* ''BBV'' - experimental basic block vector generation tool<br />
<br />
You are welcome to use any or all of these, but we have only used ''memcheck'' and ''cachegrind'' and only support ''memcheck''.<br />
<br />
= Usage =<br />
<br />
The primary used tool is ''memcheck''. This is the default tool and the only one we discuss here. Documentation for other tools can be found on the [http://www.valgrind.org valgrind website]. The memcheck tool detects several common memory errors<br />
<br />
* overrunning and underrunning heap blocks,<br />
* overrunning top of stack,<br />
* continuing to access released memory,<br />
* using uninitialized values,<br />
* incorrectly using memory copying routines,<br />
* incorrectly paired allocation/release calls,<br />
* relasing unallocated memory, and<br />
* not releasing memory.<br />
<br />
We recommend running all new code under valgrind on small test cases (small due to the aforementioned ~10x slowdown). This can save hours and hours of debugging. Running the program under valgrind can be as simple as compiling with debugging information (adding the <tt>-g</tt> flag) and running as <tt>valgrind <program> <arguements></tt>.<br />
<br />
== Serial code ==<br />
<br />
Consider the following ''bug.c'' code<br />
<br />
<source lang="C"><br />
#include <stdio.h><br />
<br />
int main() {<br />
double array[10];<br />
<br />
// Execution depends on uninitialized value<br />
if (array[4] < 0.0)<br />
printf("the results is negative\n");<br />
else if (array[4] == 0.0)<br />
printf("the results is zero\n");<br />
else if (array[4] > 0.0)<br />
printf("the results is positive\n");<br />
else<br />
printf("the results are special\n");<br />
<br />
return 0;<br />
}<br />
</source><br />
<br />
It has an uninitialized value bug. Running this under valgrind<br />
<br />
<source lang="bash"><br />
cc -Wall -g bug.c -o bug<br />
valgrind ./bug<br />
</source><br />
<br />
reports this<br />
<br />
==5955== Conditional jump or move depends on uninitialised value(s)<br />
==5955== at 0x400511: main (bug.c:7)<br />
==5955== <br />
==5955== Conditional jump or move depends on uninitialised value(s)<br />
==5955== at 0x40052C: main (bug.c:9)<br />
==5955== <br />
==5955== Conditional jump or move depends on uninitialised value(s)<br />
==5955== at 0x400536: main (bug.c:9)<br />
==5955== <br />
==5955== Conditional jump or move depends on uninitialised value(s)<br />
==5955== at 0x400551: main (bug.c:11)<br />
<br />
Note that valgrind only reports uninitialized values usage once they lead to non-determinism (i.e., when the program encounters a branch whose choice depends on the result of an uninitiated value). This means you the first report you get is frequently not in the calculation done on the uninitialized values but rather the convergence test or print statement at the end of calculations (print statements contains a bunch of branches in order to handling printing of each different digit).<br />
<br />
=== Tracking down uninitialized values ===<br />
<br />
More typically a program will be composed of multiple routines that all work on the data. To this end, consider the following ''bug.c'' code<br />
<br />
<source lang="C"><br />
#include <stdio.h><br />
<br />
void initialize_sequence(double* array, const int array_length) {<br />
int i;<br />
<br />
for (i=1; i<array_length+1; ++i)<br />
array[i] = i;<br />
}<br />
<br />
double sum(const double* array, const int array_length) {<br />
double array_sum;<br />
int i;<br />
<br />
for (i=0; i<array_length; ++i)<br />
array_sum += array[i];<br />
<br />
return array_sum;<br />
}<br />
<br />
int main() {<br />
double array[10];<br />
const int array_length = sizeof(array)/sizeof(array[0]);<br />
<br />
double array_sum;<br />
<br />
initialize_sequence(array,array_length);<br />
array_sum = sum(array,array_length);<br />
<br />
printf("the sum of 0..%d is %f\n", array_length-1, array_sum);<br />
<br />
return 0;<br />
}<br />
</source><br />
<br />
It has both indexing and uninitialized value bugs. Despite this, directly running the code<br />
<br />
<source lang="bash"><br />
cc -Wall -g bug.c -o bug<br />
./bug<br />
</source><br />
<br />
will mostly likely produce the correct answer on most machines most of the time<br />
<br />
the sum of 0..9 is 45.000000<br />
<br />
Running under valgrind<br />
<br />
<source lang="bash"><br />
valgrind ./bug<br />
</source><br />
<br />
reliably gives many warnings of the following form<br />
<br />
==15930== Conditional jump or move depends on uninitialised value(s)<br />
==15930== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)<br />
==15930== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)<br />
==15930== by 0x5318189: printf (in /lib64/libc-2.12.so)<br />
==15930== by 0x400722: main (bug.c:29)<br />
<br />
This is a typical numeric code example were the warnings first occur at a print statement because this is the first time the uninitialized value leads to non-determinism (i.e., the program's behaviour is random as it is taking or does not taking a branch based on a result computed from something that was not set by the programmer).<br />
<br />
We now know the problem is that something unset went into computing the value of ''array_sum'', so we should trace the ''array_sum'' calculation backwards through the program. This quickly gets difficult as we then find ourselves also tracing back all variables that went into the ''array_sum'' calculation, and then all variables that went into those variables, and so on.<br />
<br />
Fortunately Valgrind provides a <tt>--track-origins=yes</tt> flag to ease our search by telling us which variables are the source of the problem. Re-running with this flag<br />
<br />
<source lang="bash"><br />
valgrind --track-origins=yes ./bug<br />
</source><br />
<br />
gives warning messages augmented to include the source of the uninitialized value that went into the computation of ''array_sum''<br />
<br />
==17589== Conditional jump or move depends on uninitialised value(s)<br />
==17589== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)<br />
==17589== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)<br />
==17589== by 0x5318189: printf (in /lib64/libc-2.12.so)<br />
==17589== by 0x400722: main (bug.c:29)<br />
==17589== Uninitialised value was created by a stack allocation<br />
==17589== at 0x400632: sum (bug.c:10)<br />
<br />
Now we know the source of the uninitialized value in the ''array_sum'' computation was a local variable in the ''sum'' routine. Looking into ''sum'' we hopefully figure out without too much difficulty that we forgot to initialize ''array_sum'' to zero. If our program is producing the correct answer, it is only because we are getting lucky and ''array_sum'' happens to be allocated from memory that is initially zero.<br />
<br />
Correcting the ''sum'' routine<br />
<br />
<source lang="bash"><br />
double sum(const double* array, const int array_length) {<br />
double array_sum;<br />
int i;<br />
<br />
array_sum = 0.0;<br />
for (i=0; i<array_length; ++i)<br />
array_sum += array[i];<br />
<br />
return array_sum;<br />
}<br />
</source><br />
<br />
and re-running under valgrind reveals we are still getting uninitialized values warnings associated with the ''array_sum'' computation. Now (using the <tt>--track-origins=yes</tt>) they are of the form<br />
<br />
==18613== Conditional jump or move depends on uninitialised value(s)<br />
==18613== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)<br />
==18613== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)<br />
==18613== by 0x5318189: printf (in /lib64/libc-2.12.so)<br />
==18613== by 0x40072C: main (bug.c:30)<br />
==18613== Uninitialised value was created by a stack allocation<br />
==18613== at 0x400690: main (bug.c:21)<br />
<br />
Valgrind is now telling us that a local variable inside ''main'' went into the computation of ''array_sum'' despite never being set. This must be ''array'' itself as ''array_length'' is clearly set to the compiler computed <tt>sizeof(array)/sizeof(array[0])</tt> value.<br />
<br />
Looking at ''main'' it is clear ''array'' should have been fully initialized by ''initialize_sequence''. Careful examination of this routine reveals the final error. The initialization loop was done using Fortran indices (1...''array_length'') instead of C (0...''array_length-1''). Correcting this<br />
<br />
<source lang="bash"><br />
void initialize_sequence(double* array, const int array_length) {<br />
int i;<br />
<br />
for (i=0; i<array_length; ++i)<br />
array[i] = i;<br />
}<br />
</source><br />
<br />
finally produces code the runs under valgrind without any warnings.<br />
<br />
'''NOTE''' The Valgrind warnings were being generated because the ''array[0]'' value was not being initialized. The code was also incorrect in that it was initializing ''array[array_length]'' which is one past the end of ''array''. If you put this later error back into the program and run under Valgrind you will discover it does not produce any warnings. This shows that even codes that successfully under Valgrind can still contain errors.<br />
<br />
=== Calling Valgrind from your program ===<br />
<br />
In some cases it can be helpful to call Valgrind directly from your program in order to check the status of various bits of memory. This is easily done by including the ''valgrind/memcheck.h'' header file<br />
<br />
<source lang="C"><br />
#include <valgrind/memcheck.h><br />
</source><br />
<br />
and then adding calls to the appropriate [http://valgrind.org/docs/manual/mc-manual.html#mc-manual.clientreqs client check routines]. For example, here is a modified ''sum'' routine for the above program that prints a warning if it is called with an ''array'' that is not fully initialized<br />
<br />
<source lang="C"><br />
double sum(const double* array, const int array_length) {<br />
double array_sum;<br />
int i;<br />
<br />
if (VALGRIND_CHECK_MEM_IS_DEFINED(array,sizeof(double)*array_length)) <br />
fprintf(stderr,"sum called with an array that is not fully defined...\n"); <br />
<br />
for (i=0; i<array_length; ++i)<br />
array_sum += array[i];<br />
<br />
return array_sum;<br />
}<br />
</source><br />
<br />
Running the ''bug.c'' code under Valgrind with this modification produces the following output<br />
<br />
==29765== Uninitialised byte(s) found during client check request<br />
==29765== at 0x400745: sum (bug.c:15)<br />
==29765== by 0x400832: main (bug.c:31)<br />
==29765== Address 0x7fefff8e8 is on thread 1's stack<br />
==29765== <br />
sum called with an array that is not fully defined...<br />
<br />
== MPI code ==<br />
<br />
Consider the following ''bug.c'' MPI code<br />
<br />
<source lang="C"><br />
#include <mpi.h><br />
<br />
int main(int argc,char *argv[]){<br />
int rank, size;<br />
int value;<br />
<br />
MPI_Init(&argc, &argv);<br />
<br />
MPI_Comm_rank(MPI_COMM_WORLD, &rank);<br />
MPI_Comm_size(MPI_COMM_WORLD, &size);<br />
<br />
if (rank == 0 && size > 1) {<br />
MPI_Request request;<br />
<br />
MPI_Irecv(&value, 1, MPI_INT, 1, 0, MPI_COMM_WORLD, &request);<br />
value = 0;<br />
MPI_Wait (&request, MPI_STATUS_IGNORE);<br />
}<br />
else if (rank == 1) {<br />
MPI_Request request;<br />
<br />
MPI_Isend(&value, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &request);<br />
value = 1;<br />
MPI_Wait(&request, MPI_STATUS_IGNORE);<br />
} <br />
<br />
MPI_Finalize();<br />
<br />
return 0;<br />
}<br />
</source><br />
<br />
It has an uninitialized value bug and two race condition bugs around the use of value.<br />
<br />
# The first race condition is that rank==0 sets value=0 while at the same time doing a non-blocking receive into value (bug:16).<br />
# The uninitialized value problem is that rank==1 starts a send of value to rank==0 without ever setting value (bug:22).<br />
# The second race condition is that rank==1 sets value=1 while at the same time doing a non-blocking send of value (bug:23).<br />
<br />
=== Basic Functionality ===<br />
<br />
If we just straight up compile and run this<br />
<br />
<source lang="bash"><br />
module unload intel openmpi<br />
module load gcc/4.8.2 openmpi/gcc/1.8.3<br />
mpicc -Wall -g bug.c -o bug<br />
mpirun -np 2 valgrind ./bug<br />
</source><br />
<br />
we presumably get a report about the uninitialized value, but it is buried in tens of thousands of other bogus error messages.<br />
<br />
This solution is to link against the valgrind openmpi debug wrapper library too<br />
<br />
<source lang="bash"><br />
mpicc -Wall -g bug.c -L/usr/lib64/valgrind -lmpiwrap-amd64-linux -Xlinker -rpath=/usr/lib64/valgrind -o bug<br />
mpirun -np 2 valgrind ./bug<br />
</source><br />
<br />
Now there are only a few bogus errors reported of the form<br />
<br />
==12598== Syscall param write(buf) points to uninitialised byte(s)<br />
==12598== at 0x53916FD: ??? (in /lib64/libpthread-2.12.so)<br />
==12598== by 0x8AC8E40: send_bytes (oob_tcp_sendrecv.c:84)<br />
==12598== by 0x8AC9471: mca_oob_tcp_send_handler (oob_tcp_sendrecv.c:205)<br />
==12598== by 0x616FA23: opal_libevent2021_event_base_loop (in /opt/sharcnet/openmpi/1.8.1/gcc-debug/lib/libopen-pal.so.6.1.1)<br />
==12598== by 0x5B7E78D: orte_progress_thread_engine (ess_base_std_app.c:456)<br />
==12598== by 0x538A9D0: start_thread (in /lib64/libpthread-2.12.so)<br />
==12598== by 0x8AB86FF: ???<br />
<br />
We know this isn't an issue with ''bug.c'' as the backtrace shows it is occurring in a pure OpenMPI thread (the backtrace starts with ''start_thread'' and all subsequent routines are not from ''bug.c''). Skipping over these we now see Valgrind is picking up on sending the uninitialized value<br />
<br />
==12599== Uninitialised byte(s) found during client check request<br />
==12599== at 0x4E3641D: check_mem_is_defined_untyped (libmpiwrap.c:952)<br />
==12599== by 0x4E5BBC5: generic_Isend (libmpiwrap.c:908)<br />
==12599== by 0x4E5BEE9: PMPI_Isend (libmpiwrap.c:1393)<br />
==12599== by 0x400B02: main (bug.c:22)<br />
==12599== Address 0x7fefff5c4 is on thread 1's stack<br />
<br />
'''NOTE''' If we want to avoid recompiling our program, we can also preload the ''mpiwrap-amd64-linux'' library instead of linking against it<br />
<br />
<source lang="bash"><br />
mpicc -Wall -g bug.c -o bug<br />
LD_LIBRARY_PATH="$(which mpirun | sed -e 's|/bin/.*$|/lib|')${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" \<br />
LD_PRELOAD=/usr/lib64/valgrind/libmpiwrap-amd64-linux.so mpirun -np 2 valgrind ./bug<br />
</source><br />
<br />
=== Breaking out output by rank ===<br />
<br />
By default the Valgrind output from all the MPI ranks gets intermingled together on stderr. Although they can be distinguished for the most part as each line is prefixed with the process number, it is frequently nice to write them out to individual files. This can be done by using the ''mpirun'' <tt>--output-filename <filename></tt> option. It captures the output of each rank to ''<filename>'' suffixed with the ''MPI_COMM_WORLD'' rank<br />
<br />
<source lang="bash"><br />
mpirun --output-filename bug.log -np 2 valgrind ./bug<br />
</source><br />
<br />
To redirect just the Valgrind output we need to use the Valgrind <tt>--log-file=<filename></tt> option with the special <tt>%q{<environment-variable>}</tt> syntax<br />
<br />
<source lang="bash"><br />
mpirun -np 2 valgrind --log-file=bug.log-%q{OMPI_COMM_WORLD_RANK} ./bug<br />
</source><br />
<br />
Both of the above can also be done by wrapping the Valgrind call with Bash to perform redirection and/or environment variable expansion<br />
<br />
<source lang="bash"><br />
mpirun -np 2 /bin/bash -c 'exec valgrind ./bug > bug.log-$OMPI_COMM_WORLD_RANK 2>&1'<br />
</source><br />
<br />
and<br />
<br />
<source lang="bash"><br />
mpirun -np 2 /bin/bash -c 'exec valgrind --log-file=bug.log-$OMPI_COMM_WORLD_RANK ./bug'<br />
</source><br />
<br />
=== Advanced Functionality ===<br />
<br />
Now, if on top of this, we also bring in the valgrind enabled openmpi debug library (i.e., the ''debug'' version of our ''openmpi'' module), things get really sweet<br />
<br />
<source lang="bash"><br />
module unload gcc openmpi<br />
module load gcc/4.8.2 openmpi/gcc-debug/1.8.3<br />
mpicc -Wall -g bug.c -L/usr/lib64/valgrind -lmpiwrap-amd64-linux -Xlinker -rpath=/usr/lib64/valgrind -o bug<br />
</source><br />
<br />
Now all the bugs in the code are detected and reported<br />
<br />
==27774== Uninitialised byte(s) found during client check request<br />
==27774== at 0x4E3641D: check_mem_is_defined_untyped (libmpiwrap.c:952)<br />
==27774== by 0x4E5BBC5: generic_Isend (libmpiwrap.c:908)<br />
==27774== by 0x4E5BEE9: PMPI_Isend (libmpiwrap.c:1393)<br />
==27774== by 0x402713: main (bug.c:22)<br />
==27774== Address 0x7fefff6f4 is on thread 1's stack<br />
<br />
==27773== Invalid write of size 4<br />
==27773== at 0x4026A0: main (bug.c:16)<br />
==27773== Address 0x7fefff6f4 is on thread 1's stack<br />
<br />
==27774== Invalid write of size 4<br />
==27774== at 0x40271B: main (bug.c:23)<br />
==27774== Address 0x7fefff6f4 is on thread 1's stack<br />
<br />
=== Suppression File ===<br />
<br />
There is also a valgrind suppression option <tt>--suppressions="$(which mpirun | sed -e 's|/bin/.*|/share/openmpi/openmpi-valgrind.supp|')"</tt>. We have not observed any cases where this makes a difference yet though.<br />
<br />
=References=<br />
<br />
o Valgrind Homepage<br><br />
http://www.valgrind.org<br />
<br />
o Valgrind's Tool Suite<br><br />
http://valgrind.org/info/tools.html<br />
<br />
o kcachegrind (sharcnet does not have)<br><br />
http://kcachegrind.sourceforge.net/html/Download.html<br />
<br />
[[Category:Software packages]]</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=File:Volume.right_menu.where_is_manage_attachments.PNG&diff=15303File:Volume.right menu.where is manage attachments.PNG2017-06-27T15:41:34Z<p>Edward: </p>
<hr />
<div></div>Edwardhttps://www.sharcnet.ca/help/index.php?title=File:Manage_volume_dialog.PNG&diff=15301File:Manage volume dialog.PNG2017-06-27T15:28:11Z<p>Edward: </p>
<hr />
<div></div>Edwardhttps://www.sharcnet.ca/help/index.php?title=File:Volume.where.PNG&diff=15299File:Volume.where.PNG2017-06-27T15:22:55Z<p>Edward: </p>
<hr />
<div></div>Edwardhttps://www.sharcnet.ca/help/index.php?title=File:Volume.right_menu.PNG&diff=15294File:Volume.right menu.PNG2017-06-27T15:11:06Z<p>Edward: The right hand side menu of the volumes page.</p>
<hr />
<div>The right hand side menu of the volumes page.</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=OCTAVE&diff=14173OCTAVE2016-08-30T14:17:55Z<p>Edward: </p>
<hr />
<div><!--checked2016--><br />
{{Software<br />
|package_name=OCTAVE<br />
|package_description=Mostly compatible language with Matlab primarily intended for numerical computations<br />
|package_idnumber=28<br />
}}<br />
<br />
=Introduction=<br />
<br />
Octave is provided on SHARCNET clusters to allow serial or threaded jobs to be run in the queue as described below.<br />
<br />
=Version Selection=<br />
<br />
To see what versions of Octave are available on sharcnet clusters consult the Availability table on the sharcnet OCTAVE web portal software page OR run the following command directly on a cluster:<br />
<br />
<pre>module avail</pre><br />
<br />
Starting with version 3.6.3 its necessary to first unload the intel compiler module (or any other compiler that might be loaded) before loading the Octave module. This is done to ensure any octave-forge packages that are to be downloaded and installed into user space are built with the native gcc compiler which is version 4.4.6 at the time of this writing. Therefore to load octave/3.6.3 one would do:<br />
<br />
<pre><br />
module unload intel<br />
module load octave/3.6.3<br />
</pre><br />
<br />
Similarly, to load the default Octave module (currently version 3.8.1) one would do:<br />
<br />
module unload intel<br />
module load octave<br />
<br />
=Job Submission=<br />
<br />
On SHARCNET clusters for production work Octave should only be run via the queuing system. Octave serial jobs can be submitted to the serial queue using following sqsub command:<br />
<br />
<pre>sqsub -r 60m -o ofile.%J octave mycode.m</pre><br />
<br />
As of version 3.4.3 the sharcnet octave installation supports multi-threading which based on initial testing lapack/blas intensive octave jobs run in the threaded queue achieve an order of magnitude speedup compared to running single core jobs in the serial queue, without making any changes to your code. Once the optimal number of processors is determined by scaling tests submit the job to the threaded queue for example:<br />
<br />
<pre>sqsub -r 60m -n 8 -q threaded --mpp=1G -o ofile.%J time octave mycode.m</pre><br />
<br />
=Example Job=<br />
<br />
This section shows howto submit a [http://www.gnu.org/software/octave/doc/interpreter/Executable-Octave-Programs.html sample.m] file to the serial queue that accepts [http://www.gnu.org/software/octave/doc/interpreter/Command-Line-Options.html#Command-Line-Options command line] arguments. <br />
<br />
<pre><br />
[roberpj@hnd20:~/samples/octave/args] cat sample.m <br />
#! /bin/octave -qf<br />
printf ("%s", program_name ());<br />
arg_list = argv ();<br />
for i = 1:nargin<br />
printf (" %s", arg_list{i});<br />
endfor<br />
printf ("\n");<br />
</pre><br />
<br />
To eliminate exatraneous verbosity in the output file two switches are passed:<br />
<br />
<pre><br />
[roberpj@hnd20:~/samples/octave/args] sqsub -r 60m -o ofile.%J octave -qf --no-window-system sample.m arg1 arg2 arg3 etc<br />
WARNING: no memory requirement defined; assuming 2GB per process.<br />
submitted as jobid 6937872<br />
</pre><br />
<br />
The output file from the job appears as:<br />
<br />
<pre><br />
[roberpj@hnd20:~/samples/octave/args] cat ofile.6937872.hnd50<br />
sample.m arg1 arg2 arg3 etc<br />
</pre><br />
<br />
=Running in the development nodes=<br />
<br />
Besides running production work in Octave via the scheduler on SharcNET systems it is also possible to use Octave interactively on systems' development nodes.<br />
<br />
To start Octave from a development node one can login to a system with trusted X11 forwarding enabled:<br />
<br />
ssh -Y username@kraken.sharcnet.ca<br />
<br />
Then login to one of the development nodes with trusted X11 forwarding enabled:<br />
<br />
ssh -Y kraken-devel1<br />
<br />
Once logged into the development node one can load the Octave modules and launch the program:<br />
<br />
module unload intel<br />
module load octave<br />
octave<br />
<br />
<pre><br />
GNU Octave, version 3.8.1<br />
Copyright (C) 2014 John W. Eaton and others.<br />
This is free software; see the source code for copying conditions.<br />
There is ABSOLUTELY NO WARRANTY; not even for MERCHANTABILITY or<br />
FITNESS FOR A PARTICULAR PURPOSE. For details, type 'warranty'.<br />
<br />
Octave was configured for "x86_64-unknown-linux-gnu".<br />
<br />
Additional information about Octave is available at http://www.octave.org.<br />
<br />
Please contribute if you find this software useful.<br />
For more information, visit http://www.octave.org/get-involved.html<br />
<br />
Read http://www.octave.org/bugs.html to learn how to submit bug reports.<br />
For information about changes from previous versions, type 'news'.<br />
<br />
octave:1> <br />
<br />
</pre><br />
<br />
With X11 forwarding enabled in the development node interactive work with Octave can include figure generation.<br />
<br />
=Running on a visualization system=<br />
<br />
Once logged in to a visualization system an instance of Octave can be launched from Applications Menu > Eduction > GNU Octave.<br />
<br />
=General notes=<br />
<br />
==Matlab Compatibility==<br />
<br />
The online wiki resources "Octave Wiki":<br />
http://wiki.octave.org/ <br />
or "Wikibook":<br />
http://en.wikibooks.org/wiki/MATLAB_Programming/Differences_between_Octave_and_MATLAB <br />
provide good introductory explanations of code compatibility between Octave and Matlab.<br />
<br />
In order to run Octave so that it interprets scripts as closely as possible to Matlab, use the --traditional flag. Taking from the above sqsub example the maximum Matlab compatible submission would be:<br />
<br />
sqsub -r 60m -o ofile.%J octave --traditional mycode.m<br />
<br />
==Reading/Writing Files==<br />
<br />
There are two strategies for handling file <i>input and output</i> described in the "Octave manual" viz ...<br><br />
http://www.gnu.org/software/octave/doc/interpreter/Input-and-Output.html#Input-and-Output<br> http://www.gnu.org/software/octave/docs.html.<br />
<br />
The following stanza demonstrates the "simple file I/O" approach:<br />
http://www.gnu.org/software/octave/doc/interpreter/Simple-File-I_002fO.html#Simple-File-I_002fO <br />
<br />
<pre><br />
save myiofile.dat A B C<br />
save ("-text", "myiofile.dat", "A", "B", "C")<br />
save ("-binary", "myiofile.dat", "A", "B", "C")<br />
load myiofile.dat<br />
load ("-text", "myiofile.dat", "A", "B", "C")<br />
load ("-binary", "myiofile.dat", "A", "B", "C")<br />
</pre><br />
<br />
where A, B and C are a potential mix of entities such as scalars, vectors or matrices. Note that for large files the binary format is strongly recommended to both minimize disk space and file read/write wallclock times.<br />
<br />
==Octave-Forge==<br />
<br />
Octave-forge packages are not pre-installed on SHARCNET clusters since none are available at this time for the operating system. Therefore any required packages must be downloaded, manually installed from source and then managed in local user accounts (as described below).<br />
<br />
However 16 of approximately 95 octave-forge packages are installed on some sharcnet visualization workstations including viz1,2,3-uwaterloo, viz2-uwo, viz6-uoguelph under /usr/share/octave/packages. As a word of caution however, all package versions will likely be significantly old since they are tied to the operating system fedora /etc/redhat-release major version and hence not contain recent critical bug fixes (which could be numerical in nature). Users are therefore strongly recommended to likewise download and install the latest version similarly as would be done on the clusters.<br />
<br />
===Sharing Packages===<br />
<br />
Note that packages installed by a single sharcnet user in their home account can be shared out to other research group members or even among general SHARCNET users. To do this simply set access permissions for group or global read access accordingly. For details on how to change <br />
the permissions see: https://www.sharcnet.ca/help/index.php/Knowledge_Base#How_do_I_give_other_users_access_to_my_files_.3F<br />
<br />
===Managing Packages===<br />
<br />
The "Octave Forge" http://octave.sourceforge.net/ project provides extra packages for use with octave that can be downloaded into a directory such as <i>~/my_octave_sources</i> then manually installed into your own sharcnet account as will be shown in the follow example. A current list of packages available for download that notable are generally only compatible with the latest major release of octave cat be found here http://octave.sourceforge.net/packages.php. <br />
<br />
In the event where the major version of octave on sharcnet you are using (for instance 3.4.x) is not the same as latest major version (at the time of writing 3.6.x) and there were programming api changes between the two versions that effect the package you want to use, then you probably will need to download an older version of a package from http://sourceforge.net/projects/octave/files/Octave%20Forge%20Packages/Individual%20Package%20Releases/. for it to work.<br />
<br />
To help estimate the package version required as a starting point, consider the major release dates of recent which are octave-3.4.3 (10/10/11), octave-3.6.0 (01/15/12), octave-3.6.1 (02/22/12) and finally octave-3.6.2 (05/31/12). Next consider you want to download the latest geometry package that was compatible with octave-3.4.3 at the time of its release, then simply display all the archived versions as shown in the following stanza and pick the last available geometry release date before the next major release octave-3.6.0 (01/15/12). To show a list of all archived geometry versions, do the following steps:<br />
<br />
<pre><br />
http://octave.sourceforge.net/<br />
Click Packages on top menu menu bar<br />
Scroll down to the miscellaneous package row and click details<br />
Click (older versions) located below "Download Package"<br />
Click Octave Forge Packages<br />
Click Individual Package Releases<br />
Please wait for the page to load ...<br />
Click "Name" at the top of first colum to sort packages alphabetically<br />
Scroll down you will find all available archived geometry packages:<br />
geometry-1.0.1.tar.gz 2011-09-27<br />
geometry-1.1.1.tar.gz 2011-10-06<br />
geometry-1.1.2.tar.gz 2011-10-09<br />
geometry-1.1.3.tar.gz 2011-10-13<br />
geometry-1.1.tar.gz 2011-10-04<br />
geometry-1.2.0.tar.gz 2011-10-22<br />
geometry-1.2.1.tar.gz 2011-11-02<br />
geometry-1.2.2.tar.gz 2011-11-04<br />
geometry-1.4.0.tar.gz 2012-01-25<br />
geometry-1.4.1.tar.gz 2012-03-24<br />
geometry-1.5.0.tar.gz 2012-06-05<br />
</pre><br />
<br />
Therefore you will download geometry-1.2.2.tar.gz (2011-11-04) since the next<br />
release geometry-1.4.0.tar.gz (2012-01-25) and then install it into octave<br />
3.4.3 as follows:<br />
<br />
<pre><br />
[roberpj@tope:~/my_octave_sources] octave<br />
GNU Octave, version 3.4.3<br />
octave:1> pkg install geometry-1.2.2.tar.gz<br />
octave:2> pkg list<br />
Package Name | Version | Installation directory<br />
---------------+---------+-----------------------<br />
geometry | 1.2.2 | /home/roberpj/octave/geometry-1.2.2<br />
octave:15> pkg load geometry<br />
octave:16> pkg describe geometry<br />
---<br />
Package name:<br />
geometry<br />
Version:<br />
1.2.2<br />
Short description:<br />
Library for geometric computing extending MatGeom functions. Useful to create,<br />
transform, manipulate and display geometric primitives.<br />
Status:<br />
Loaded<br />
</pre><br />
<br />
<b>Note:</b> Any questions regarding package version compatibility with major or minor octave release should be referred to the developers.<br />
<br />
===Package Commands===<br />
<br />
Additional examples of package commands are shown in this section. For demonstration purpose linear-algebra will first be downloaded:<br />
<br />
<pre><br />
[roberpj@iqaluk:~] mkdir my_octave_sources<br />
[roberpj@iqaluk:~] cd my_octave_sources<br />
wget http://downloads.sourceforge.net/octave/general-1.3.2.tar.gz<br />
wget http://downloads.sourceforge.net/octave/linear-algebra-2.2.0.tar.gz<br />
</pre><br />
<br />
<pre><br />
[myusername@orc-login1:~]octave<br />
octave:1> help pkg<br />
</pre><br />
<br />
To install a package such as linear-algebra do:<br />
<br />
<pre><br />
octave:2> cd ~/my_octave_sources<br />
octave:3> pkg install general-1.3.2.tar.gz<br />
octave:4> pkg install linear-algebra-2.2.0.tar.gz<br />
</pre><br />
<br />
To remove a package, from a terminal first do:<br />
<pre>cd ~/octave<br />
rm -rf linear-algebra-2.2.0</pre><br />
<br />
... then from within octave do:<br />
<pre>octave:5> pkg uninstall linear-algebra</pre><br />
<br />
To list all installed packages do:<br />
<br />
<pre>octave:6> pkg list</pre><br />
<br />
To add a named packages to your path:<br />
<br />
<pre>octave:7> pkg load name</pre><br />
<br />
To remove a named package from your path:<br />
<br />
<pre>octave:8> pkg unload</pre><br />
<br />
To list all functions provided by a package:<br />
<br />
<pre>octave:9> pkg describe -verbose all</pre><br />
<br />
To list functions provide by extra odepkg package:<br />
<br />
<pre>octave:10> pkg describe odepkg</pre><br />
<br />
To list functions details for extra financial package:<br />
<br />
<pre>octave:11> pkg describe financial -verbose</pre><br />
<br />
To get documentation for a topic such as sin do:<br />
<br />
<pre>octave:12> doc sin</pre><br />
<br />
To get help using the doc command do:<br />
<br />
<pre>octave:13> help doc</pre><br />
<br />
To get documentation for a topic such as matrix do:<br />
<br />
<pre>octave:14> doc matrix</pre><br />
<br />
To get documentation for any topic run the doc command alone:<br />
<br />
<pre>octave:15> doc</pre><br />
<br />
===Package Example===<br />
<br />
This example assumes you first create two directories in your home account one being named<br />
~/my_octave_packages <br />
and the other <br />
~/my_octave_sources <br />
then download the required three tar.gz files into the latter from <br />
http://octave.sourceforge.net/packages.php ...<br />
<br />
[roberpj@orc-login1:~] module unload octave<br />
[roberpj@orc-login1:~] module load octave<br />
[roberpj@orc-login1:~] octave<br />
GNU Octave, version 3.4.3<br />
octave:1> pkg prefix ~/my_octave_packages<br />
ans = /home/roberpj/my_octave_packages<br />
octave:2> cd my_octave_sources<br />
octave:3> ls<br />
miscellaneous-1.0.11.tar.gz optim-1.0.17.tar.gz struct-1.0.9.tar.gz<br />
octave:4> pkg install miscellaneous-1.0.11.tar.gz optim-1.0.17.tar.gz struct-1.0.9.tar.gz<br />
octave:5> pkg list<br />
Package Name | Version | Installation directory<br />
---------------|---------|-----------------------<br />
miscellaneous *| 1.0.11 | /home/roberpj/my_octave_packages/miscellaneous-1.0.11<br />
optim *| 1.0.17 | /home/roberpj/my_octave_packages/optim-1.0.17<br />
struct *| 1.0.9 | /home/roberpj/my_octave_packages/struct-1.0.9</PRE><br />
<br />
=References=<br />
<br />
o Octave Homepage<br><br />
http://www.gnu.org/software/octave/<br />
<br />
o Octave 725 Page Manual (Version 3.4.0)<br><br />
http://www.gnu.org/software/octave/doc/interpreter/<br />
<br />
o Statistic Package Function Reference<br><br />
http://octave.sourceforge.net/doc/funref_statistics.html<br />
<br />
o GNU Octave Wiki<br><br />
http://wiki.octave.org/<br />
<br />
o Matlab-Like Tools for HPC (article)<br><br />
http://www.admin-magazine.com/HPC/Articles/Matlab-Like-Tools-for-HPC</div>Edwardhttps://www.sharcnet.ca/help/index.php?title=MPIBLAST&diff=13263MPIBLAST2016-03-01T16:03:48Z<p>Edward: </p>
<hr />
<div>{{Software<br />
|package_name=MPIBLAST<br />
|package_description=Parallel implementation of NCBI BLAST<br />
|package_idnumber=55}}<br />
<br />
=Introduction=<br />
<br />
The mpiblast module must be manually loaded before submitting any mpiblast jobs. Two examples below <br />
demonstrate how to setup and submit jobs to the mpi queue.<br />
<br />
=Version Selection=<br />
<br />
===All Clusters (except Guppy)===<br />
<br />
module unload openmpi intel<br />
module load intel/12.1.3 openmpi/intel/1.6.2 mpiblast/1.6.0<br />
<br />
===Guppy Only===<br />
<br />
module unload openmpi intel<br />
module load intel/11.0.083 openmpi/intel/1.4.2 mpiblast/1.6.0<br />
<br />
=Job Submission=<br />
<br />
sqsub -r 15m -n 24 -q mpi --mpp=3g -o ofile%J mpiblast etc<br />
<br />
where the number of cores n=24 is arbitrary.<br />
<br />
=Example Jobs=<br />
<br />
==DROSOPH==<br />
<br />
Copy sample problem files (fasta database and input) from the /opt/sharcnet <i>examples</i> directory a directory under work as shown here. The fasta database used in this example can be obtained as a guest from NCBI here http://www.ncbi.nlm.nih.gov/guide/all/#downloads_ then clicking "FTP: FASTA BLAST Databases".<br />
<br />
<pre><br />
mkdir -p /work/$USER/samples/mpiblast/test1<br />
rm /work/$USER/samples/mpiblast/test1/*<br />
cd /work/$USER/samples/mpiblast/test1<br />
cp /opt/sharcnet/mpiblast/1.6.0/examples/drosoph.in drosoph.in<br />
gunzip -c /opt/sharcnet/mpiblast/1.6.0/examples/drosoph.nt.gz > drosoph.nt.$USER<br />
</pre><br />
<br />
Create a hidden configuration file using a text editor (such as vi) to define a Shared storage location between nodes and a Local storage directory available on each compute node as follows:<br />
<br />
<pre><br />
[roberpj@hnd20:/work/$USER/samples/mpiblast/test1] vi .ncbirc<br />
[NCBI]<br />
Data=/opt/sharcnet/mpiblast/1.6.0/ncbi/data<br />
[BLAST]<br />
BLASTDB=/scratch/YourSharcnetUsername/mpiblasttest1<br />
BLASTMAT=/work/YourSharcnetUsername/samples/mpiblast/test1<br />
[mpiBLAST]<br />
Shared=/scratch/YourSharcnetUsername/mpiblasttest1<br />
Local=/tmp<br />
</pre><br />
<br />
Partition the database into 8 fragments into <i>/scratch/$USER/mpiblasttest1</i> where they will be searched for when running a job in queue according to the <i>.ncbirc</i> file:<br />
<br />
<pre><br />
module load mpiblast/1.6.0<br />
mkdir /scratch/roberpj/mpiblasttest1<br />
rm -f /scratch/$USER/mpiblasttest1/*<br />
cd /work/$USER/samples/mpiblast/test1<br />
<br />
[roberpj@hnd20:/work/$USER/samples/mpiblast/test1] mpiformatdb -N 8 -i drosoph.nt.$USER -o T -p F -n /scratch/$USER/mpiblasttest1<br />
Reading input file<br />
Done, read 1534943 lines<br />
Breaking drosoph.nt into 8 fragments<br />
Executing: formatdb -p F -i drosoph.nt -N 8 -n /scratch/roberpj/mpiblasttest1/drosoph.nt -o T <br />
Created 8 fragments.<br />
<<< Please make sure the formatted database fragments are placed in /scratch/roberpj/mpiblasttest1/ before executing mpiblast. >>><br />
<br />
[roberpj@hnd20:/scratch/roberpj/mpiblasttest1] ls<br />
drosoph.nt.000.nhr drosoph.nt.002.nin drosoph.nt.004.nnd drosoph.nt.006.nni<br />
drosoph.nt.000.nin drosoph.nt.002.nnd drosoph.nt.004.nni drosoph.nt.006.nsd<br />
drosoph.nt.000.nnd drosoph.nt.002.nni drosoph.nt.004.nsd drosoph.nt.006.nsi<br />
drosoph.nt.000.nni drosoph.nt.002.nsd drosoph.nt.004.nsi drosoph.nt.006.nsq<br />
drosoph.nt.000.nsd drosoph.nt.002.nsi drosoph.nt.004.nsq drosoph.nt.007.nhr<br />
drosoph.nt.000.nsi drosoph.nt.002.nsq drosoph.nt.005.nhr drosoph.nt.007.nin<br />
drosoph.nt.000.nsq drosoph.nt.003.nhr drosoph.nt.005.nin drosoph.nt.007.nnd<br />
drosoph.nt.001.nhr drosoph.nt.003.nin drosoph.nt.005.nnd drosoph.nt.007.nni<br />
drosoph.nt.001.nin drosoph.nt.003.nnd drosoph.nt.005.nni drosoph.nt.007.nsd<br />
drosoph.nt.001.nnd drosoph.nt.003.nni drosoph.nt.005.nsd drosoph.nt.007.nsi<br />
drosoph.nt.001.nni drosoph.nt.003.nsd drosoph.nt.005.nsi drosoph.nt.007.nsq<br />
drosoph.nt.001.nsd drosoph.nt.003.nsi drosoph.nt.005.nsq drosoph.nt.dbs<br />
drosoph.nt.001.nsi drosoph.nt.003.nsq drosoph.nt.006.nhr drosoph.nt.mbf<br />
drosoph.nt.001.nsq drosoph.nt.004.nhr drosoph.nt.006.nin drosoph.nt.nal<br />
drosoph.nt.002.nhr drosoph.nt.004.nin drosoph.nt.006.nnd<br />
</pre><br />
<br />
Submit a short job with a 15m time limit on n=16 cores calculate by taking N=8 fragments + 8. If all goes well output results will be written to <i>drosoph.out</i> and the execution time will appear in ofile%J where %J is the job number:<br />
<br />
<pre><br />
[roberpj@hnd20:/work/$USER/samples/mpiblast/test1]<br />
sqsub -r 15m -n 106 -q mpi --mpp=1G -o ofile%J mpiblast -d drosoph.nt.$USER -i drosoph.in<br />
-p blastn -o drosoph.out --use-parallel-write --use-virtual-frags<br />
submitted as jobid 6966896<br />
<br />
[roberpj@hnd20:/work/roberpj/samples/mpiblast/test1] cat ofile6966896.hnd50 <br />
Total Execution Time: 1.80031<br />
</pre><br />
<br />
When submitting a mpiblast job on a cluster such as goblin that doesnt have an inifiniband interconnect better performance (at least double speedup) will be achieved running the mpi job on one compute node. For regular users of non-contributed hardware typically specify "-n 8" to reflect the max number of cores on a single node:<br />
<br />
<pre><br />
sqsub -r 15m -n 8 -N 1 -q mpi --mpp=4G -o ofile%J mpiblast -d drosoph.nt -i drosoph.in<br />
-p blastn -o drosoph.out --use-parallel-write --use-virtual-frags<br />
</pre><br />
<br />
Sample output results computed previously with BLASTN 2.2.15 [Oct-15-2006] are included in <i>/opt/sharcnet/mpiblast/1.6.0/examples/ROSOPH.out</i> to compare your newly generated <i>drosoph.out</i> file with.<br />
<br />
==UNIGENE==<br />
<br />
The main purpose of this example is to illustrate some additional options and switchs that maybe useful for debugging and for dealing with larger databases as described in official detail at http://www.mpiblast.org/Docs/Guide. The fasta database used in this example can also be downloaded from http://www.ncbi.nlm.nih.gov/guide/all/#downloads_ as a guest by clicking "FTP: UniGene" then entering the "Homo_sapiens" sub-directory. More information about UniGene alignments can be found at https://cgwb.nci.nih.gov/cgi-bin/hgTrackUi?hgsid=95443&c=chr1&g=uniGene_3 . As with Example1 above, for convenience all required files can simply be copied from the /opt/sharcnet <i>examples</i> subdirectory to work as shown here:<br />
<br />
<pre><br />
mkdir /work/$USER/samples/mpiblast/test2<br />
rm -f /work/$USER/samples/mpiblast/test2/*<br />
cd /work/$USER/samples/mpiblast/test2<br />
cp /opt/sharcnet/mpiblast/1.6.0/examples/il2ra.in il2ra.in<br />
gunzip -c /opt/sharcnet/mpiblast/1.6.0/examples/Hs.seq.uniq.gz > Hs.seq.$USER<br />
<br />
[roberpj@orc-login2:/work/roberpj/samples/mpiblast/test2] ls<br />
Hs.seq.roberpj il2ra.in<br />
</pre><br />
<br />
Create a hidden configuration file using a text editor (such as vi) to define a Shared storage location between nodes and a Local storage directory available on each compute node as followw. Note that the ncbi/data directory is not used in this example and hence can be omitted. If the Local and Shared directories are the same replace <b>--copy-via=mpi</b> with <b>--copy-via=none</b> as will be demonstrated in the below sqsub commands.<br />
<br />
<pre><br />
[username@orc-login1:/work/$USER/samples/mpiblast/test2] vi .ncbirc<br />
[NCBI]<br />
Data=/opt/sharcnet/mpiblast/1.6.0/ncbi/data<br />
[BLAST]<br />
BLASTDB=/work/YourSharcnetUsername/mpiblasttest2<br />
BLASTMAT=/work/YourSharcnetUsername/samples/mpiblast/test2<br />
[mpiBLAST]<br />
Shared=/work/YourSharcnetUsername/mpiblasttest2<br />
Local=/tmp<br />
</pre><br />
<br />
Again partition the database into 8 fragments into <i>/work/$USER/mpiblasttest2</i> where they will be searched for when a job runs in the queue according to the <i>.ncbirc</i> file, first removing previous partitioning if any such files exist. For this example assume $USER=roberpj however be sure to replace with your username!<br />
<br />
<pre><br />
module load mpiblast/1.6.0<br />
mkdir -p /work/$USER/mpiblasttest2<br />
rm -f /work/$USER/mpiblasttest2/*<br />
cd /work/$USER/samples/mpiblast/test2<br />
<br />
[roberpj@orc-login1:/work/roberpj/samples/mpiblast/test2] mpiformatdb -N 8 -i Hs.seq.$USER -o T -p F -n /work/roberpj/mpiblasttest2<br />
Reading input file<br />
Done, read 2348651 lines<br />
Breaking Hs.seq.roberpj into 8 fragments<br />
Executing: formatdb -p F -i Hs.seq.roberpj -N 8 -n /work/roberpj/mpiblasttest2/Hs.seq.roberpj -o T <br />
Created 8 fragments.<br />
<<< Please make sure the formatted database fragments are placed in /work/roberpj/mpiblasttest2/ before executing mpiblast. >>><br />
<br />
[roberpj@orc-login1:/work/roberpj/samples/mpiblast/test1] ls<br />
formatdb.log Hs.seq.roberpj il2ra.in<br />
<br />
[roberpj@orc-login1:/work/roberpj/mpiblasttest2] ls<br />
Hs.seq.roberpj.000.nhr Hs.seq.roberpj.002.nin Hs.seq.roberpj.004.nnd Hs.seq.roberpj.006.nni<br />
Hs.seq.roberpj.000.nin Hs.seq.roberpj.002.nnd Hs.seq.roberpj.004.nni Hs.seq.roberpj.006.nsd<br />
Hs.seq.roberpj.000.nnd Hs.seq.roberpj.002.nni Hs.seq.roberpj.004.nsd Hs.seq.roberpj.006.nsi<br />
Hs.seq.roberpj.000.nni Hs.seq.roberpj.002.nsd Hs.seq.roberpj.004.nsi Hs.seq.roberpj.006.nsq<br />
Hs.seq.roberpj.000.nsd Hs.seq.roberpj.002.nsi Hs.seq.roberpj.004.nsq Hs.seq.roberpj.007.nhr<br />
Hs.seq.roberpj.000.nsi Hs.seq.roberpj.002.nsq Hs.seq.roberpj.005.nhr Hs.seq.roberpj.007.nin<br />
Hs.seq.roberpj.000.nsq Hs.seq.roberpj.003.nhr Hs.seq.roberpj.005.nin Hs.seq.roberpj.007.nnd<br />
Hs.seq.roberpj.001.nhr Hs.seq.roberpj.003.nin Hs.seq.roberpj.005.nnd Hs.seq.roberpj.007.nni<br />
Hs.seq.roberpj.001.nin Hs.seq.roberpj.003.nnd Hs.seq.roberpj.005.nni Hs.seq.roberpj.007.nsd<br />
Hs.seq.roberpj.001.nnd Hs.seq.roberpj.003.nni Hs.seq.roberpj.005.nsd Hs.seq.roberpj.007.nsi<br />
Hs.seq.roberpj.001.nni Hs.seq.roberpj.003.nsd Hs.seq.roberpj.005.nsi Hs.seq.roberpj.007.nsq<br />
Hs.seq.roberpj.001.nsd Hs.seq.roberpj.003.nsi Hs.seq.roberpj.005.nsq Hs.seq.roberpj.dbs<br />
Hs.seq.roberpj.001.nsi Hs.seq.roberpj.003.nsq Hs.seq.roberpj.006.nhr Hs.seq.roberpj.mbf<br />
Hs.seq.roberpj.001.nsq Hs.seq.roberpj.004.nhr Hs.seq.roberpj.006.nin Hs.seq.roberpj.nal<br />
Hs.seq.roberpj.002.nhr Hs.seq.roberpj.004.nin Hs.seq.roberpj.006.nnd <br />
</pre><br />
<br />
Submit a couple of short jobs 15m time limit. If all goes well output results will be written to <i>biobrewA.out</i> and <i>biobrewB.out</i> and the execution time appear in corresponding ofile%J's where %J is the job number as per usual. In these examples we will submit the job from a saw specific directory which can be copied from test2 as follows:<br />
<br />
<pre><br />
[roberpj@saw-login1:~] module load mpiblast/1.6.0<br />
[roberpj@saw-login1:~] cd /work/roberpj/samples/mpiblast<br />
[roberpj@saw-login1:/work/roberpj/samples/mpiblast] cp -a test2 test2.saw<br />
<br />
[roberpj@saw-login1:/work/roberpj/samples/mpiblast/test2.saw] vi .ncbirc<br />
[roberpj@saw-login1:/work/roberpj/samples/mpiblast/test2.saw] cat .ncbirc | grep BLASTMAT<br />
BLASTMAT=/work/roberpj/samples/mpiblast/test2.saw<br />
</pre><br />
<br />
A) Submit job with profile time option choosing n=24 (8 fragments + 8 or greater):<br />
<br />
<pre><br />
[roberpj@saw-login1:/work/roberpj/samples/mpiblast/test2.saw] rm -f oTime*;<br />
sqsub -r 15m -n 24 -q mpi -o ofile%J mpiblast --use-parallel-write --copy-via=mpi<br />
-d Hs.seq.roberpj -i il2ra.in -p blastn -o biobrew.out --time-profile=oTime<br />
</pre><br />
<br />
B) Usage of the debug option again choosing n=16 (8 fragments + 8 or greater):<br />
<br />
<pre><br />
[roberpj@saw-login1:/work/roberpj/samples/mpiblast/test2.saw] rm -f oLog*;<br />
sqsub -r 15m -n 16 -q mpi -o ofile%J mpiblast --use-parallel-write --copy-via=none<br />
-d Hs.seq.$USER -i il2ra.in -p blastn -o biobrew.out --debug=oLog<br />
</pre><br />
<br />
Finally compare <i>/opt/sharcnet/mpiblast/1.6.0/examples/BIOBREW.out</i> computed previously with BLASTN 2.2.15 [Oct-15-2006] with your newly generated <i>biobrew.out</i> output file to verify the results and submit a ticket if there are any problems!<br />
<br />
<u>SUPPORTED PROGRAMS IN MPIBLAST</U><br />
<br />
As described in http://www.mpiblast.org/Docs/FAQ mpiblast supports the standard blast programs http://www.ncbi.nlm.nih.gov/BLAST/blast_program.shtml which are reproduced here for reference:<br />
<br />
<pre><br />
blastp: Compares an amino acid query sequence against a protein sequence database.<br />
blastn: Compares a nucleotide query sequence against a nucleotide sequence database.<br />
blastx: Compares a nucleotide query sequence translated in all reading frames against<br />
a protein sequence database.<br />
tblastn: Compares a protein query sequence against a nucleotide sequence database<br />
dynamically translated in all reading frames.<br />
tblastx: Compares the six-frame translations of a nucleotide query sequence against<br />
the six-frame translations of a nucleotide sequence database.<br />
</pre><br />
<br />
<u>MPIBLAST BINARIES OPTIONS</u><br />
<br />
<pre><br />
[roberpj@orc-login1:/opt/sharcnet/mpiblast/1.6.0/bin] ./mpiblast -help<br />
mpiBLAST requires the following options: -d [database] -i [query file] -p [blast program name]<br />
</pre><br />
<br />
<pre><br />
[roberpj@orc-login1:/opt/sharcnet/mpiblast/1.6.0/bin] ./mpiformatdb --help<br />
Executing: formatdb - <br />
<br />
formatdb 2.2.20 arguments:<br />
<br />
-t Title for database file [String] Optional<br />
-i Input file(s) for formatting [File In] Optional<br />
-l Logfile name: [File Out] Optional<br />
default = formatdb.log<br />
-p Type of file<br />
T - protein <br />
F - nucleotide [T/F] Optional<br />
default = T<br />
-o Parse options<br />
T - True: Parse SeqId and create indexes.<br />
F - False: Do not parse SeqId. Do not create indexes.<br />
[T/F] Optional<br />
default = F<br />
-a Input file is database in ASN.1 format (otherwise FASTA is expected)<br />
T - True, <br />
F - False.<br />
[T/F] Optional<br />
default = F<br />
-b ASN.1 database in binary mode<br />
T - binary, <br />
F - text mode.<br />
[T/F] Optional<br />
default = F<br />
-e Input is a Seq-entry [T/F] Optional<br />
default = F<br />
-n Base name for BLAST files [String] Optional<br />
-v Database volume size in millions of letters [Integer] Optional<br />
default = 4000<br />
-s Create indexes limited only to accessions - sparse [T/F] Optional<br />
default = F<br />
-V Verbose: check for non-unique string ids in the database [T/F] Optional<br />
default = F<br />
-L Create an alias file with this name<br />
use the gifile arg (below) if set to calculate db size<br />
use the BLAST db specified with -i (above) [File Out] Optional<br />
-F Gifile (file containing list of gi's) [File In] Optional<br />
-B Binary Gifile produced from the Gifile specified above [File Out] Optional<br />
-T Taxid file to set the taxonomy ids in ASN.1 deflines [File In] Optional<br />
-N Number of database volumes [Integer] Optional<br />
default = 0<br />
</pre><br />
<br />
=General Notes=<br />
<br />
==Issue1: Jobs Fail to Start==<br />
<br />
If the following error message occurs it maybe necessary to stop all mpiblast jobs you are running on the cluster and clear<br />
out all the files from /tmp then submit your job again. In the case of Example2 above the following steps work to resolve the problem, where the nodes used from the last run job were red[7-14]. Should there be any questions please open a problem ticket.<br />
<br />
<pre><br />
Purging Job Node Subset (redfin)<br />
pdsh -w red[7-14] ls /tmp/Hs* (confirm existence of file from previous runs)<br />
pdsh -w red[7-14] rm -f /tmp/Hs*<br />
pdsh -w red[7-14] ls /tmp/Hs* (confirm all files from previous runs removed)<br />
<br />
Purging Full Cluster (hound)<br />
pdsh -w hnd[1-18] -f 4 rm -f /tmp/Hs*<br />
pdsh -w hnd[1-18] -f 4 ls /tmp/Hs*<br />
</pre><br />
<br />
<pre><br />
[roberpj@red-admin:/work/roberpj/samples/mpiblast/test2.red4] cat ofile489214.red-admin.redfin.sharcnet<br />
8 0.577904 Bailing out with signal 11<br />
--------------------------------------------------------------------------<br />
MPI_ABORT was invoked on rank 8 in communicator MPI_COMM_WORLD<br />
with errorcode 0.<br />
<br />
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.<br />
You may or may not see output from other processes, depending on<br />
exactly when Open MPI kills them.<br />
--------------------------------------------------------------------------<br />
20 0.577719 Bailing out with signal 11<br />
0 0.591128 Bailing out with signal 15<br />
[red14:09194] [[36424,0],0]-[[36424,1],0] mca_oob_tcp_msg_recv: readv failed: Connection reset by peer (104)<br />
1 0.591327 Bailing out with signal 15<br />
etc<br />
</pre><br />
<br />
==Issue2: Jobs Run for a While then Die==<br />
<br />
The solution here is to filter the input sequence file. For reasons yet understood the presence of repeat sections results in many thousands of WARNING and ERROR messages rapidly written to the "sqsub -o ofile" output file presumably as mpiblast ignores sequences before eventually diing after several hours, or possibly days.<br />
<br />
<pre><br />
# cat ofile1635556.saw-admin.saw.sharcnet | grep "WARNING\|ERROR" | wc -l<br />
10560<br />
</pre><br />
<br />
<pre><br />
# cat ofile1635556.saw-admin.saw.sharcnet<br />
Selenocysteine (U) at position 60 replaced by X<br />
Selenocysteine (U) at position 42 replaced by X<br />
[blastall] WARNING: [000.000] NODE_84_length_162_cov_46.259258_1_192_-: SetUpBlastSearch failed.<br />
[blastall] ERROR: [000.000] NODE_84_length_162_cov_46.259258_1_192_-: BLASTSetUpSearch: Unable to calculate Karlin-Altschul params, check query sequence<br />
<<<< snipped out ~10000 similar WARNING and ERROR messages from this example >>>><br />
[blastall] WARNING: [000.000] NODE_65409_length_87_cov_2.367816_1_77_+: SetUpBlastSearch failed.<br />
[blastall] ERROR: [000.000] NODE_65409_length_87_cov_2.367816_1_77_+: BLASTSetUpSearch: Unable to calculate Karlin-Altschul params, check query sequence<br />
Selenocysteine (U) at position 61 replaced by X<br />
Selenocysteine (U) at position 62 replaced by X<br />
Selenocysteine (U) at position 34 replaced by X<br />
Selenocysteine (U) at position 1058 replaced by X<br />
--------------------------------------------------------------------------<br />
mpirun noticed that process rank 52 with PID 30067 on node saw214 exited on signal 9 (Killed).<br />
--------------------------------------------------------------------------<br />
1618 17 651 62909.3 Bailing out with signal 15<br />
14 3 62909.3 Bailing out with signal 15<br />
19 15 62909.3 Bailing out with signal 1536 5 7 62909.312 62909.310 62909.3 Bailing out with signal 1547 21 62909.3 Bailing out with signal 159 62909.3 Bailing out with signal 15<br />
45 62909.3 Bailing out with signal 1525 62909.3 Bailing out with signal 158 62909.3 Bailing out with signal 15<br />
50 62909.3 Bailing out with signal 1522 62909.3 Bailing out with signal 15<br />
11 62909.3 Bailing out with signal 15<br />
48 62909.323 62909.3 Bailing out with signal 15<br />
Bailing out with signal 15<br />
46 62909.3 Bailing out with signal 15<br />
etc<br />
</pre><br />
<br />
=References=<br />
<br />
o MPIBLAST Homepage<br><br />
http://www.mpiblast.org/<br />
<br />
o MPIBLAST Version History<br><br />
http://www.mpiblast.org/Downloads/Version-History<br />
<br />
[[Category:Bioinformatics]]<br />
<br />
<!--checked2016--></div>Edwardhttps://www.sharcnet.ca/help/index.php?title=Category:Decommissioned_Systems&diff=13261Category:Decommissioned Systems2016-03-01T16:00:19Z<p>Edward: Blanked the page</p>
<hr />
<div></div>Edward