VALGRIND - Revision history

Edward at 14:41, 6 June 2019

2019-06-06T14:41:49Z

Edward at 12:52, 27 July 2017

2017-07-27T12:52:26Z

Roberpj: /* References */

2015-10-07T23:35:43Z

‎References

Tyson: Clarify what is the valgrind enabled openmpi debug library

2015-09-22T15:25:21Z

Clarify what is the valgrind enabled openmpi debug library

Tyson: Add details about splitting out output by ranks

2015-09-08T20:32:46Z

Add details about splitting out output by ranks

Tyson: Missed version number change and various formatting cleanups

2015-09-08T19:38:23Z

Missed version number change and various formatting cleanups

Tyson: Update MPI section for recommended GCC and OpenMPI versions

2015-09-08T19:34:50Z

Update MPI section for recommended GCC and OpenMPI versions

Tyson: Minor correction

2015-09-08T18:03:09Z

Minor correction

Tyson: Add section about using client API

2015-09-08T18:02:44Z

Add section about using client API

Tyson: Example of tracking down uninitialized value warnings

2015-09-08T16:20:54Z

Example of tracking down uninitialized value warnings

← Older revision		Revision as of 14:41, 6 June 2019
Line 1:		Line 1:
		+	{{Template:CCDelete}}
	{{GrahamUpdate}}		{{GrahamUpdate}}
	{{Software		{{Software

← Older revision		Revision as of 12:52, 27 July 2017
Line 1:		Line 1:
		+	{{GrahamUpdate}}
	{{Software		{{Software
	\|package_name=VALGRIND		\|package_name=VALGRIND

@@ Line 412: / Line 412: @@
 =References=
-o Package website<br>
+o Valgrind Homepage<br>
 http://www.valgrind.org
 [[Category:Software packages]]

@@ Line 381: / Line 381: @@
 === Advanced Functionality ===
-Now, if on top of this, we also bring in the valgrind enabled openmpi debug library, things get really sweet
+Now, if on top of this, we also bring in the valgrind enabled openmpi debug library (i.e., the ''debug'' version of our ''openmpi'' module), things get really sweet
 <source lang="bash">

@@ Line 351: / Line 351: @@
 LD_LIBRARY_PATH="$(which mpirun | sed -e 's|/bin/.*$|/lib|')${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" \
 LD_PRELOAD=/usr/lib64/valgrind/libmpiwrap-amd64-linux.so mpirun -np 2 valgrind ./bug
 </source>

@@ Line 9: / Line 9: @@
 * gcc/4.8.2
-* openmpi/gcc-debug/1.8.1 modules
+* openmpi/gcc-debug/1.8.3 modules
 = Overview =
@@ Line 168: / Line 168: @@
 We now know the problem is that something unset went into computing the value of ''array_sum'', so we should trace the ''array_sum'' calculation backwards through the program.  This quickly gets difficult as we then find ourselves also tracing back all variables that went into the ''array_sum'' calculation, and then all variables that went into those variables, and so on.
-Fortunately Valgrind provides a ''--track-origins=yes'' flag to ease our search by telling us which variables are the source of the problem.  Re-running with this flag
+Fortunately Valgrind provides a <tt>--track-origins=yes</tt> flag to ease our search by telling us which variables are the source of the problem.  Re-running with this flag
 <source lang="bash">
@@ Line 201: / Line 201: @@
 </source>
-and re-running under valgrind reveals we are still getting uninitialized values warnings associated with the ''array_sum'' computation.  Now (using the ''--track-origins=yes'') they are of the form
+and re-running under valgrind reveals we are still getting uninitialized values warnings associated with the ''array_sum'' computation.  Now (using the <tt>--track-origins=yes</tt>) they are of the form
   ==18613== Conditional jump or move depends on uninitialised value(s)
@@ Line 211: / Line 211: @@
   ==18613==    at 0x400690: main (bug.c:21)
-Valgrind is now telling us that a local variable inside ''main'' went into the computation of ''array_sum'' despite never being set.  This must be ''array'' itself as ''array_length'' is clearly set to the compiler computed ''sizeof(array)/sizeof(array[0])'' value.
+Valgrind is now telling us that a local variable inside ''main'' went into the computation of ''array_sum'' despite never being set.  This must be ''array'' itself as ''array_length'' is clearly set to the compiler computed <tt>sizeof(array)/sizeof(array[0])</tt> value.
 Looking at ''main'' it is clear ''array'' should have been fully initialized by ''initialize_sequence''.  Careful examination of this routine reveals the final error.  The initialization loop was done using Fortran indices (1...''array_length'')  instead of C (0...''array_length-1'').  Correcting this
@@ Line 382: / Line 382: @@
 === Suppression File ===
-There is also a valgrind suppression option <tt>--suppressions="$(which mpirun | sed -e 's|/bin/.*|/share/openmpi/openmpi-valgrind.supp|')"</tt>, however we have not observed any cases where this makes a difference yet.
+There is also a valgrind suppression option <tt>--suppressions="$(which mpirun | sed -e 's|/bin/.*|/share/openmpi/openmpi-valgrind.supp|')"</tt>.  We have not observed any cases where this makes a difference yet though.
 =References=

@@ Line 307: / Line 307: @@
 === Basic Functionality ===
-If you run this using the non-debug/valgrind openmpi valgrind
+If we just straight up compile and run this
 <source lang="bash">
 module unload intel openmpi
-module load intel/12.1.3 openmpi/intel/1.6.2
+module load gcc/4.8.2 openmpi/gcc/1.8.3
 mpicc -Wall -g bug.c -o bug
 mpirun -np 2 valgrind ./bug
 </source>
-you are presumably getting a report about the uninitialized value, but it is burried in 55517 other bogus error messages.
+we presumably get a report about the uninitialized value, but it is buried in tens of thousands of other bogus error messages.
-This solution to this is to link against the valgrind openmpi debug wrapper library too.
+This solution is to link against the valgrind openmpi debug wrapper library too
 <source lang="bash">
@@ Line 325: / Line 325: @@
 </source>
-This now only reports one bogus error (a free on exit) and picks up the sending of the uninitialized value
+Now there are only a few bogus errors reported of the form
-  ==27638== Uninitialised byte(s) found during client check request
+  ==12598== Syscall param write(buf) points to uninitialised byte(s)
-  ==27638==    at 0x4E3641D: check_mem_is_defined_untyped (libmpiwrap.c:952)
+  ==12598==    at 0x53916FD: ??? (in /lib64/libpthread-2.12.so)
-  ==27638==    by 0x4E5BBC5: generic_Isend (libmpiwrap.c:908)
+  ==12598==    by 0x8AC8E40: send_bytes (oob_tcp_sendrecv.c:84)
-  ==27638==    by 0x4E5BEE9: PMPI_Isend (libmpiwrap.c:1393)
+  ==12598==    by 0x8AC9471: mca_oob_tcp_send_handler (oob_tcp_sendrecv.c:205)
-  ==27638==    by 0x402713: main (bug.c:22)
+  ==12598==    by 0x616FA23: opal_libevent2021_event_base_loop (in /opt/sharcnet/openmpi/1.8.1/gcc-debug/lib/libopen-pal.so.6.1.1)
-  ==27638==  Address 0x7fefff734 is on thread 1's stack
+ ==12598==    by 0x5B7E78D: orte_progress_thread_engine (ess_base_std_app.c:456)
-=== Advanced Functionality ===
+We know this isn't an issue with ''bug.c'' as the backtrace shows it is occurring in a pure OpenMPI thread (the backtrace starts with ''start_thread'' and all subsequent routines are not from ''bug.c'').  Skipping over these we now see Valgrind is picking up on sending the uninitialized value
-Now, if on top of this, you also bring in the valgrind enabled openmpi debug library, things get really sweet
+ ==12599== Uninitialised byte(s) found during client check request
 <source lang="bash">
 mpicc -Wall -g bug.c -o bug
 LD_PRELOAD=/usr/lib64/valgrind/libmpiwrap-amd64-linux.so mpirun -np 2 valgrind ./bug
 </source>
-You still only get one bogus error (a free on exit) and all the bugs in the code are detected and reported
+=== Advanced Functionality ===
   ==27774== Uninitialised byte(s) found during client check request
@@ Line 364: / Line 382: @@
 === Suppression File ===
-There is also a valgrind suppression option <tt>--suppressions=/opt/sharcnet/openmpi/1.6.2/intel-debug/share/openmpi/openmpi-valgrind.supp</tt>, however we have not observed any cases where this makes a difference yet.
+There is also a valgrind suppression option <tt>--suppressions="$(which mpirun | sed -e 's|/bin/.*|/share/openmpi/openmpi-valgrind.supp|')"</tt>, however we have not observed any cases where this makes a difference yet.
 =References=

@@ Line 228: / Line 228: @@
 '''NOTE''' The Valgrind warnings were being generated because the ''array[0]'' value was not being initialized.  The code was also incorrect in that it was initializing ''array[array_length]'' which is one past the end of ''array''.  If you put this later error back into the program and run under Valgrind you will discover it does not produce any warnings.  This shows that even codes that successfully under Valgrind can still contain errors.
-=== Call Valgrind from your program ===
+=== Calling Valgrind from your program ===
 In some cases it can be helpful to call Valgrind directly from your program in order to check the status of various bits of memory.  This is easily done by including the ''valgrind/memcheck.h'' header file

← Older revision		Revision as of 18:02, 8 September 2015
Line 227:		Line 227:

	'''NOTE''' The Valgrind warnings were being generated because the ''array[0]'' value was not being initialized. The code was also incorrect in that it was initializing ''array[array_length]'' which is one past the end of ''array''. If you put this later error back into the program and run under Valgrind you will discover it does not produce any warnings. This shows that even codes that successfully under Valgrind can still contain errors.		'''NOTE''' The Valgrind warnings were being generated because the ''array[0]'' value was not being initialized. The code was also incorrect in that it was initializing ''array[array_length]'' which is one past the end of ''array''. If you put this later error back into the program and run under Valgrind you will discover it does not produce any warnings. This shows that even codes that successfully under Valgrind can still contain errors.
		+
		+	=== Call Valgrind from your program ===
		+
		+	In some cases it can be helpful to call Valgrind directly from your program in order to check the status of various bits of memory. This is easily done by including the ''valgrind/memcheck.h'' header file
		+
		+	<source lang="C">
		+	#include <valgrind/memcheck.h>
		+	</source>
		+
		+	and then adding calls to the appropriate [http://valgrind.org/docs/manual/mc-manual.html#mc-manual.clientreqs client check routines]. For example, here is a modified ''sum'' routine for the above program that prints a warning if it is called with an ''array'' that is not fully initialized
		+
		+	<source lang="C">
		+	double sum(const double* array, const int array_length) {
		+	double array_sum;
		+	int i;
		+
		+	if (VALGRIND_CHECK_MEM_IS_DEFINED(array,sizeof(double)*array_length))
		+	fprintf(stderr,"sum called with an array that is not fully defined...\n");
		+
		+	for (i=0; i<array_length; ++i)
		+	array_sum += array[i];
		+
		+	return array_sum;
		+	}
		+	</source>
		+
		+	Running the ''bug.c'' code under Valgrind with this modification produces the following output
		+
		+	==29765== Uninitialised byte(s) found during client check request
		+	==29765== at 0x400745: sum (bug.c:15)
		+	==29765== by 0x400832: main (bug.c:31)
		+	==29765== Address 0x7fefff8e8 is on thread 1's stack
		+	==29765==
		+	sum called with an array that is not fully defined...

	== MPI code ==		== MPI code ==

@@ Line 156: / Line 156: @@
 </source>
-however, reliably gives many warnings of the following form
+reliably gives many warnings of the following form
   ==15930== Conditional jump or move depends on uninitialised value(s)
@@ Line 164: / Line 164: @@
   ==15930==    by 0x400722: main (bug.c:29)
-This is a typical numeric code example were the warnings first occur at a print statement because this is the first time the uninitialized value leads to non-determinism (i.e., the program's behaviour is altered as it takes or does not take a branch based on a result computed from something that was not set by the programmer).
+This is a typical numeric code example were the warnings first occur at a print statement because this is the first time the uninitialized value leads to non-determinism (i.e., the program's behaviour is random as it is taking or does not taking a branch based on a result computed from something that was not set by the programmer).
-We now know the problem is that something unset went into computing the value of ''array_sum''.  We can then trace each of the ''array_sum'' calculation backwards.  This quickly gets difficult though as we then find ourselves also tracing back all variables that went into the ''array_sum'' calculation, and then all variables that went into those variables and so on.
+We now know the problem is that something unset went into computing the value of ''array_sum'', so we should trace the ''array_sum'' calculation backwards through the program.  This quickly gets difficult as we then find ourselves also tracing back all variables that went into the ''array_sum'' calculation, and then all variables that went into those variables, and so on.
-Fortunately Valgrind provides a ''--track-origins=yes'' flag to ease our search by telling us which ones we can ignore.  Re-running with this flag
+Fortunately Valgrind provides a ''--track-origins=yes'' flag to ease our search by telling us which variables are the source of the problem.  Re-running with this flag
 <source lang="bash">
@@ Line 174: / Line 174: @@
 </source>
-our warnings messages are augmented to include information on the source of the uninitialized value that went into the computation of ''array_sum''
+gives warning messages augmented to include the source of the uninitialized value that went into the computation of ''array_sum''
   ==17589== Conditional jump or move depends on uninitialised value(s)
@@ Line 184: / Line 184: @@
   ==17589==    at 0x400632: sum (bug.c:10)
-Now we know the source of our uninitialized value in the ''array_sum'' computation is a local variable in the ''sum'' routine.  Looking into ''sum'' we hopefully figure out without too much difficulty that we forgot to initialize ''array_sum'' to zero.  If our program is producing the correct answer, it is only because we are getting lucky and ''array_sum'' happens to be allocated from memory that is initially zero.
+Now we know the source of the uninitialized value in the ''array_sum'' computation was a local variable in the ''sum'' routine.  Looking into ''sum'' we hopefully figure out without too much difficulty that we forgot to initialize ''array_sum'' to zero.  If our program is producing the correct answer, it is only because we are getting lucky and ''array_sum'' happens to be allocated from memory that is initially zero.
 Correcting the ''sum'' routine
@@ Line 211: / Line 211: @@
   ==18613==    at 0x400690: main (bug.c:21)
-Valgrind is telling us that a local variable inside the ''main'' went into the computation of ''array_sum'' despite never being set.  This means it must be ''array'' itself or ''array_length''.  It cannot be ''array_length'' as the former is clearly set to the compiler computed ''sizeof(array)/sizeof(array[0])'' so it must be ''array'' itself.
+Valgrind is now telling us that a local variable inside ''main'' went into the computation of ''array_sum'' despite never being set.  This must be ''array'' itself as ''array_length'' is clearly set to the compiler computed ''sizeof(array)/sizeof(array[0])'' value.
-It is clear the intent of the code is that ''array'' is initialized by the ''initialize_sequence'' routine.  Careful examination of this routine reveals the final error.  The initialization loop was done using Fortran indices (1...''array_length'')  instead of C (0...''array_length-1'').  Correcting this routine
+Looking at ''main'' it is clear ''array'' should have been fully initialized by ''initialize_sequence''.  Careful examination of this routine reveals the final error.  The initialization loop was done using Fortran indices (1...''array_length'')  instead of C (0...''array_length-1'').  Correcting this
 <source lang="bash">