MediaWiki API result

{
    "query": {
        "pages": {
            "792": {
                "pageid": 792,
                "ns": 0,
                "title": "Using OpenBUGS with R",
                "revisions": [
                    {
                        "contentformat": "text/x-wiki",
                        "contentmodel": "wikitext",
                        "*": "=Instructions for usage on new Compute Canada clusters (cedar and graham)=\n\n==Installation of OpenBUGS==\nDownload the source code from the [http://www.openbugs.net/w/Downloads Downloads] page and then copy it to graham or cedar.\n\n tar xvfz OpenBUGS-3.2.3.tar.gz\n cd OpenBUGS-3.2.3\n ./configure --prefix=/home/$USER/lib/openbugs-3.2.3\n make\n make install\n\nIf this fails because of missing gnu/stubs-32.h file, you must contact the system administrator to install that file.\n\n==Installation of R2OpenBUGS==\nFirst, load the R module.  You can find out which are available by running:\n\n module spider r\n\nIn this case we choose the latest (as of November, 2017):\n\n module load r/3.4.0\n\nThen start R:\n\n R\n\nand inside it run:\n\n install.packages(\"R2OpenBUGS\")\n\nTo make it able to find OpenBUGS at runtime, you need to set the variable that point it to the location where the binary is installed.\n\n export OpenBUGS_PATH=$HOME/bin/OpenBUGS\n\n\n\n=Instructions for usage on older SHARCNET clusters (eg. orca)=\n\n==Obtaining OpenBUGS==\n\nOpenBUGS is an open source project. It can be downloaded from [http://www.openbugs.net/w/FrontPage here]. Look for the package for Linux, it comes as gzipped Unix tar file, named as '''OpenBUGS-''version''.tar.gz''', e.g. OpenBUGS-3.2.3.tar.gz.\n\n==Installing OpenBUGS for Linux==\n\nFirst, save it and copy it to your account, say, in folder Downloads. Then unpack it with command\n\n tar zxvf OpenBUGS-3.2.3.tar.gz\n\nThis will extract the content from the tar file and put in folder OpenBUGS-3.2.3. Then go into that folder\n\n cd OpenBUGS\n\nand run configuration first\n\n ./configure --prefix=$HOME/lib/openbugs-3.2.3\n\nThis sets where to install the package. In this case, it will be installed in /home/you/lib/openbugs-3.2.3. You may choose another location you like. By default the installation directory is /usr/local, which is not what you want.\n\nThen compile and install it with commands\n\n make\n make install\n\nThe compilation shouldn't take long to complete. It is likely, however, that the compilation may fail due to missing system files. specifically, gnu/stubs-32.h. If this is the case, you will need to ask the system administrators to install the missing system files.\n\nUpon success, you should have three folders created in /home/your/lib/openbugs-3.2.3\n\n bin   lib   share\n\nIn bin, there are two binaries: '''OpenBUGS''' and '''OpenBUGSCli'''. '''OpenBUGS''' is what you will need when you run R calling bugs.\n\n==Installing R2OpenBUGS for R==\n\nIn order to user OpenBUGS in R, you need an R interface to BUGS. The package R2OpenBUGS is recommended. The package can be installed in user space. To install it, first, load R module. You might need to unload intel module on SHARCNET systems to avoid possible conflict\n\n module unload intel\n\nTo see what releases of R are available, use command\n\n module avail r\n\nIf you prefer a specific version, e.g. 3.1, then use command\n\n module load r/3.1\n\nOtherwise, the command\n\n module load r\n\nwould just load the latest version by default.\n\nThen run R from command line. At the R prompt, use R command '''install.packages()''' to install R2OpenBUGS\n\n >install.packages(\"R2OpenBUGS\")\n\nand follow the instructions.\n\n==Using OpenBUGS in R==\n\nThe R2OpenBUGS is an interface to OpenBUGS. In order to use OpenBUGS in R, you need to tell where OpenBUGS resides. According to the help info, as you would get from R command\n\n >library(R2OpenBUGS)\n >... ...\n >help(bugs)\n\nthere are two ways of telling '''bugs()''' where OpenBUGS is. First you may specify the path to OpenBUGS in the call to bugs() in your R code. Assume that OpenBUGS executable is installed in /home/you/bin (you need to change it to where you have put the binary OpenBUGS), the call to bugs() would like something like this\n\n >schools.sim <- bugs(data, inits, parameters, OpenBUGS.pgm=\"/home/you/bin/OpenBUGS\", model.file=\"/home/youschools.txt\", n.chains = 3, n.iter = 1000, working.directory = NULL)\n\nNote, this line has the paths to the OpenBUGS and the model file specific to the system you are using. To make your R code portable, you may simply set environment variable at the command line or put the following line in '''.bash_profile''' in your home directory\n\n export OpenBUGS_PATH=$HOME/bin/OpenBUGS\n\nand also, set the working directory to the current folder (the last argument) as follows\n\n >schools.sim <- bugs(data, inits, parameters, model.file=\"schools.txt\", n.chains = 3, n.iter = 1000, working.directory = \".\")\n\nWhen working.directory is where the input and output are stored. By default, it is set to NULL and the location is unexpected from one system to another."
                    }
                ]
            },
            "453": {
                "pageid": 453,
                "ns": 0,
                "title": "VALGRIND",
                "revisions": [
                    {
                        "contentformat": "text/x-wiki",
                        "contentmodel": "wikitext",
                        "*": "{{Template:CCDelete}}\n{{GrahamUpdate}}\n{{Software\n|package_name=VALGRIND\n|package_description=Memory debugger\n|package_idnumber=132\n}}\n'''Valgrind''' is a powerful tool for analyzing programs, memory debugging, memory leak detection and profiling.  It is freely available under GNU license.  Version 3.5.0 is available on most SHARCNET systems.\n\n'''NOTE''' To avoid spurious warnings it is important to not use too new of a version of GCC or OpenMPI.  We recommend\n\n* gcc/4.8.2\n* openmpi/gcc-debug/1.8.3 modules\n\n= Overview =\n\nValgrind is a dynamic binary instrumentation framework that dynamically translates executables to add instrumentation and track all memory and register usages by a program.  The advantages of this approach are that\n\n* it can be directly run on any executable, and\n* dynamic translation allows ultimate instrumentation\n\nwhile the disadvantages are\n\n* 5-100 x slow down depending on tool,\n* 12-18 x increase in size of translated code, and\n* corner cases may exist between translated code and original.\n\nSeveral tools have been built upon this framework.  These include\n\n* ''memcheck'' -  memory error detector\n* ''cachegrind'' - cache and branch-prediction profiler\n* ''callgrind'' - call-graph generating cache and branch prediction profiler\n* ''helgrind'' - thread error detector\n* ''DRD'' - thread error detector\n* ''Massif'' - heap profiler\n* ''DHAT'' - dynamic heap analysis tool\n* ''SGCheck'' - experimental stack and global array overrun detector\n* ''BBV'' - experimental basic block vector generation tool\n\nYou are welcome to use any or all of these, but we have only used ''memcheck'' and ''cachegrind'' and only support ''memcheck''.\n\n= Usage =\n\nThe primary used tool is ''memcheck''.  This is the default tool and the only one we discuss here.  Documentation for other tools can be found on the [http://www.valgrind.org valgrind website].  The memcheck tool detects several common memory errors\n\n* overrunning and underrunning heap blocks,\n* overrunning top of stack,\n* continuing to access released memory,\n* using uninitialized values,\n* incorrectly using memory copying routines,\n* incorrectly paired allocation/release calls,\n* relasing unallocated memory, and\n* not releasing memory.\n\nWe recommend running all new code under valgrind on small test cases (small due to the aforementioned ~10x slowdown).  This can save hours and hours of debugging.  Running the program under valgrind can be as simple as compiling with debugging information (adding the <tt>-g</tt> flag) and running as <tt>valgrind <program> <arguements></tt>.\n\n== Serial code ==\n\nConsider the following ''bug.c'' code\n\n<source lang=\"C\">\n#include <stdio.h>\n\nint main() {\n  double array[10];\n\n  // Execution depends on uninitialized value\n  if (array[4] < 0.0)\n    printf(\"the results is negative\\n\");\n  else if (array[4] == 0.0)\n    printf(\"the results is zero\\n\");\n  else if (array[4] > 0.0)\n    printf(\"the results is positive\\n\");\n  else\n    printf(\"the results are special\\n\");\n\n  return 0;\n}\n</source>\n\nIt has an uninitialized value bug.  Running this under valgrind\n\n<source lang=\"bash\">\ncc -Wall -g bug.c -o bug\nvalgrind ./bug\n</source>\n\nreports this\n\n ==5955== Conditional jump or move depends on uninitialised value(s)\n ==5955==    at 0x400511: main (bug.c:7)\n ==5955== \n ==5955== Conditional jump or move depends on uninitialised value(s)\n ==5955==    at 0x40052C: main (bug.c:9)\n ==5955== \n ==5955== Conditional jump or move depends on uninitialised value(s)\n ==5955==    at 0x400536: main (bug.c:9)\n ==5955== \n ==5955== Conditional jump or move depends on uninitialised value(s)\n ==5955==    at 0x400551: main (bug.c:11)\n\nNote that valgrind only reports uninitialized values usage once they lead to non-determinism (i.e., when the program encounters a branch whose choice depends on the result of an uninitiated value).  This means you the first report you get is frequently not in the calculation done on the uninitialized values but rather the convergence test or print statement at the end of calculations (print statements contains a bunch of branches in order to handling printing of each different digit).\n\n=== Tracking down uninitialized values ===\n\nMore typically a program will be composed of multiple routines that all work on the data.  To this end, consider the following ''bug.c'' code\n\n<source lang=\"C\">\n#include <stdio.h>\n\nvoid initialize_sequence(double* array, const int array_length) {\n  int i;\n\n  for (i=1; i<array_length+1; ++i)\n    array[i] = i;\n}\n\ndouble sum(const double* array, const int array_length) {\n  double array_sum;\n  int i;\n\n  for (i=0; i<array_length; ++i)\n    array_sum += array[i];\n\n  return array_sum;\n}\n\nint main() {\n  double array[10];\n  const int array_length = sizeof(array)/sizeof(array[0]);\n\n  double array_sum;\n\n  initialize_sequence(array,array_length);\n  array_sum = sum(array,array_length);\n\n  printf(\"the sum of 0..%d is %f\\n\", array_length-1, array_sum);\n\n  return 0;\n}\n</source>\n\nIt has both indexing and uninitialized value bugs.  Despite this, directly running the code\n\n<source lang=\"bash\">\ncc -Wall -g bug.c -o bug\n./bug\n</source>\n\nwill mostly likely produce the correct answer on most machines most of the time\n\n the sum of 0..9 is 45.000000\n\nRunning under valgrind\n\n<source lang=\"bash\">\nvalgrind ./bug\n</source>\n\nreliably gives many warnings of the following form\n\n ==15930== Conditional jump or move depends on uninitialised value(s)\n ==15930==    at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)\n ==15930==    by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)\n ==15930==    by 0x5318189: printf (in /lib64/libc-2.12.so)\n ==15930==    by 0x400722: main (bug.c:29)\n\nThis is a typical numeric code example were the warnings first occur at a print statement because this is the first time the uninitialized value leads to non-determinism (i.e., the program's behaviour is random as it is taking or does not taking a branch based on a result computed from something that was not set by the programmer).\n\nWe now know the problem is that something unset went into computing the value of ''array_sum'', so we should trace the ''array_sum'' calculation backwards through the program.  This quickly gets difficult as we then find ourselves also tracing back all variables that went into the ''array_sum'' calculation, and then all variables that went into those variables, and so on.\n\nFortunately Valgrind provides a <tt>--track-origins=yes</tt> flag to ease our search by telling us which variables are the source of the problem.  Re-running with this flag\n\n<source lang=\"bash\">\nvalgrind --track-origins=yes ./bug\n</source>\n\ngives warning messages augmented to include the source of the uninitialized value that went into the computation of ''array_sum''\n\n ==17589== Conditional jump or move depends on uninitialised value(s)\n ==17589==    at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)\n ==17589==    by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)\n ==17589==    by 0x5318189: printf (in /lib64/libc-2.12.so)\n ==17589==    by 0x400722: main (bug.c:29)\n ==17589==  Uninitialised value was created by a stack allocation\n ==17589==    at 0x400632: sum (bug.c:10)\n\nNow we know the source of the uninitialized value in the ''array_sum'' computation was a local variable in the ''sum'' routine.  Looking into ''sum'' we hopefully figure out without too much difficulty that we forgot to initialize ''array_sum'' to zero.  If our program is producing the correct answer, it is only because we are getting lucky and ''array_sum'' happens to be allocated from memory that is initially zero.\n\nCorrecting the ''sum'' routine\n\n<source lang=\"bash\">\ndouble sum(const double* array, const int array_length) {\n  double array_sum;\n  int i;\n\n  array_sum = 0.0;\n  for (i=0; i<array_length; ++i)\n    array_sum += array[i];\n\n  return array_sum;\n}\n</source>\n\nand re-running under valgrind reveals we are still getting uninitialized values warnings associated with the ''array_sum'' computation.  Now (using the <tt>--track-origins=yes</tt>) they are of the form\n\n ==18613== Conditional jump or move depends on uninitialised value(s)\n ==18613==    at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)\n ==18613==    by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)\n ==18613==    by 0x5318189: printf (in /lib64/libc-2.12.so)\n ==18613==    by 0x40072C: main (bug.c:30)\n ==18613==  Uninitialised value was created by a stack allocation\n ==18613==    at 0x400690: main (bug.c:21)\n\nValgrind is now telling us that a local variable inside ''main'' went into the computation of ''array_sum'' despite never being set.  This must be ''array'' itself as ''array_length'' is clearly set to the compiler computed <tt>sizeof(array)/sizeof(array[0])</tt> value.\n\nLooking at ''main'' it is clear ''array'' should have been fully initialized by ''initialize_sequence''.  Careful examination of this routine reveals the final error.  The initialization loop was done using Fortran indices (1...''array_length'')  instead of C (0...''array_length-1'').  Correcting this\n\n<source lang=\"bash\">\nvoid initialize_sequence(double* array, const int array_length) {\n  int i;\n\n  for (i=0; i<array_length; ++i)\n    array[i] = i;\n}\n</source>\n\nfinally produces code the runs under valgrind without any warnings.\n\n'''NOTE''' The Valgrind warnings were being generated because the ''array[0]'' value was not being initialized.  The code was also incorrect in that it was initializing ''array[array_length]'' which is one past the end of ''array''.  If you put this later error back into the program and run under Valgrind you will discover it does not produce any warnings.  This shows that even codes that successfully under Valgrind can still contain errors.\n\n=== Calling Valgrind from your program ===\n\nIn some cases it can be helpful to call Valgrind directly from your program in order to check the status of various bits of memory.  This is easily done by including the ''valgrind/memcheck.h'' header file\n\n<source lang=\"C\">\n#include <valgrind/memcheck.h>\n</source>\n\nand then adding calls to the appropriate [http://valgrind.org/docs/manual/mc-manual.html#mc-manual.clientreqs client check routines].  For example, here is a modified ''sum'' routine for the above program that prints a warning if it is called with an ''array'' that is not fully initialized\n\n<source lang=\"C\">\ndouble sum(const double* array, const int array_length) {\n  double array_sum;\n  int i;\n\n  if (VALGRIND_CHECK_MEM_IS_DEFINED(array,sizeof(double)*array_length)) \n    fprintf(stderr,\"sum called with an array that is not fully defined...\\n\"); \n\n  for (i=0; i<array_length; ++i)\n    array_sum += array[i];\n\n  return array_sum;\n}\n</source>\n\nRunning the ''bug.c'' code under Valgrind with this modification produces the following output\n\n ==29765== Uninitialised byte(s) found during client check request\n ==29765==    at 0x400745: sum (bug.c:15)\n ==29765==    by 0x400832: main (bug.c:31)\n ==29765==  Address 0x7fefff8e8 is on thread 1's stack\n ==29765== \n sum called with an array that is not fully defined...\n\n== MPI code ==\n\nConsider the following ''bug.c'' MPI code\n\n<source lang=\"C\">\n#include <mpi.h>\n\nint main(int argc,char *argv[]){\n  int rank, size;\n  int value;\n\n  MPI_Init(&argc, &argv);\n\n  MPI_Comm_rank(MPI_COMM_WORLD, &rank);\n  MPI_Comm_size(MPI_COMM_WORLD, &size);\n\n  if (rank == 0 && size > 1) {\n    MPI_Request request;\n\n    MPI_Irecv(&value, 1, MPI_INT, 1, 0, MPI_COMM_WORLD, &request);\n    value = 0;\n    MPI_Wait (&request, MPI_STATUS_IGNORE);\n  }\n  else if (rank == 1) {\n    MPI_Request request;\n\n    MPI_Isend(&value, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &request);\n    value = 1;\n    MPI_Wait(&request, MPI_STATUS_IGNORE);\n  } \n\n  MPI_Finalize();\n\n  return 0;\n}\n</source>\n\nIt has an uninitialized value bug and two race condition bugs around the use of value.\n\n# The first race condition is that rank==0 sets value=0 while at the same time doing a non-blocking receive into value (bug:16).\n# The uninitialized value problem is that rank==1 starts a send of value to rank==0 without ever setting value (bug:22).\n# The second race condition is that rank==1 sets value=1 while at the same time doing a non-blocking send of value (bug:23).\n\n=== Basic Functionality ===\n\nIf we just straight up compile and run this\n\n<source lang=\"bash\">\nmodule unload intel openmpi\nmodule load gcc/4.8.2 openmpi/gcc/1.8.3\nmpicc -Wall -g bug.c -o bug\nmpirun -np 2 valgrind ./bug\n</source>\n\nwe presumably get a report about the uninitialized value, but it is buried in tens of thousands of other bogus error messages.\n\nThis solution is to link against the valgrind openmpi debug wrapper library too\n\n<source lang=\"bash\">\nmpicc -Wall -g bug.c -L/usr/lib64/valgrind -lmpiwrap-amd64-linux -Xlinker -rpath=/usr/lib64/valgrind -o bug\nmpirun -np 2 valgrind ./bug\n</source>\n\nNow there are only a few bogus errors reported of the form\n\n ==12598== Syscall param write(buf) points to uninitialised byte(s)\n ==12598==    at 0x53916FD: ??? (in /lib64/libpthread-2.12.so)\n ==12598==    by 0x8AC8E40: send_bytes (oob_tcp_sendrecv.c:84)\n ==12598==    by 0x8AC9471: mca_oob_tcp_send_handler (oob_tcp_sendrecv.c:205)\n ==12598==    by 0x616FA23: opal_libevent2021_event_base_loop (in /opt/sharcnet/openmpi/1.8.1/gcc-debug/lib/libopen-pal.so.6.1.1)\n ==12598==    by 0x5B7E78D: orte_progress_thread_engine (ess_base_std_app.c:456)\n ==12598==    by 0x538A9D0: start_thread (in /lib64/libpthread-2.12.so)\n ==12598==    by 0x8AB86FF: ???\n\nWe know this isn't an issue with ''bug.c'' as the backtrace shows it is occurring in a pure OpenMPI thread (the backtrace starts with ''start_thread'' and all subsequent routines are not from ''bug.c'').  Skipping over these we now see Valgrind is picking up on sending the uninitialized value\n\n ==12599== Uninitialised byte(s) found during client check request\n ==12599==    at 0x4E3641D: check_mem_is_defined_untyped (libmpiwrap.c:952)\n ==12599==    by 0x4E5BBC5: generic_Isend (libmpiwrap.c:908)\n ==12599==    by 0x4E5BEE9: PMPI_Isend (libmpiwrap.c:1393)\n ==12599==    by 0x400B02: main (bug.c:22)\n ==12599==  Address 0x7fefff5c4 is on thread 1's stack\n\n'''NOTE''' If we want to avoid recompiling our program, we can also preload the ''mpiwrap-amd64-linux'' library instead of linking against it\n\n<source lang=\"bash\">\nmpicc -Wall -g bug.c -o bug\nLD_LIBRARY_PATH=\"$(which mpirun | sed -e 's|/bin/.*$|/lib|')${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}\" \\\nLD_PRELOAD=/usr/lib64/valgrind/libmpiwrap-amd64-linux.so mpirun -np 2 valgrind ./bug\n</source>\n\n=== Breaking out output by rank ===\n\nBy default the Valgrind output from all the MPI ranks gets intermingled together on stderr.  Although they can be distinguished for the most part as each line is prefixed with the process number, it is frequently nice to write them out to individual files.  This can be done by using the ''mpirun'' <tt>--output-filename <filename></tt> option.  It captures the output of each rank to ''<filename>'' suffixed with the ''MPI_COMM_WORLD'' rank\n\n<source lang=\"bash\">\nmpirun --output-filename bug.log -np 2 valgrind ./bug\n</source>\n\nTo redirect just the Valgrind output we need to use the Valgrind <tt>--log-file=<filename></tt> option with the special <tt>%q{<environment-variable>}</tt> syntax\n\n<source lang=\"bash\">\nmpirun -np 2 valgrind --log-file=bug.log-%q{OMPI_COMM_WORLD_RANK} ./bug\n</source>\n\nBoth of the above can also be done by wrapping the Valgrind call with Bash to perform redirection and/or environment variable expansion\n\n<source lang=\"bash\">\nmpirun -np 2 /bin/bash -c 'exec valgrind ./bug > bug.log-$OMPI_COMM_WORLD_RANK 2>&1'\n</source>\n\nand\n\n<source lang=\"bash\">\nmpirun -np 2 /bin/bash -c 'exec valgrind --log-file=bug.log-$OMPI_COMM_WORLD_RANK ./bug'\n</source>\n\n=== Advanced Functionality ===\n\nNow, if on top of this, we also bring in the valgrind enabled openmpi debug library (i.e., the ''debug'' version of our ''openmpi'' module), things get really sweet\n\n<source lang=\"bash\">\nmodule unload gcc openmpi\nmodule load gcc/4.8.2 openmpi/gcc-debug/1.8.3\nmpicc -Wall -g bug.c -L/usr/lib64/valgrind -lmpiwrap-amd64-linux -Xlinker -rpath=/usr/lib64/valgrind -o bug\n</source>\n\nNow all the bugs in the code are detected and reported\n\n ==27774== Uninitialised byte(s) found during client check request\n ==27774==    at 0x4E3641D: check_mem_is_defined_untyped (libmpiwrap.c:952)\n ==27774==    by 0x4E5BBC5: generic_Isend (libmpiwrap.c:908)\n ==27774==    by 0x4E5BEE9: PMPI_Isend (libmpiwrap.c:1393)\n ==27774==    by 0x402713: main (bug.c:22)\n ==27774==  Address 0x7fefff6f4 is on thread 1's stack\n \n ==27773== Invalid write of size 4\n ==27773==    at 0x4026A0: main (bug.c:16)\n ==27773==  Address 0x7fefff6f4 is on thread 1's stack\n \n ==27774== Invalid write of size 4\n ==27774==    at 0x40271B: main (bug.c:23)\n ==27774==  Address 0x7fefff6f4 is on thread 1's stack\n\n=== Suppression File ===\n\nThere is also a valgrind suppression option <tt>--suppressions=\"$(which mpirun | sed -e 's|/bin/.*|/share/openmpi/openmpi-valgrind.supp|')\"</tt>.  We have not observed any cases where this makes a difference yet though.\n\n=References=\n\no Valgrind Homepage<br>\nhttp://www.valgrind.org\n\no Valgrind's Tool Suite<br>\nhttp://valgrind.org/info/tools.html\n\no kcachegrind (sharcnet does not have)<br>\nhttp://kcachegrind.sourceforge.net/html/Download.html\n\n[[Category:Software packages]]"
                    }
                ]
            }
        }
    }
}