From Documentation
Revision as of 23:03, 26 September 2017 by Isaac (Talk | contribs)

Jump to: navigation, search
Note: Some of the information on this page is for our legacy systems only. The page is scheduled for an update to make it applicable to Graham.

Overview of DDT


The Distributed Debugging Tool (DDT) is a powerful commercial debugger with a graphical user interface (GUI). It’s main intended use is for debugging parallel MPI codes, though it can also be used with serial, threaded (OpenMP / pthreads), and GPU (CUDA; also mixed MPI and CUDA) programs. The product was developed by Allinea (U.K.). It is installed on many SHARCNET clusters. The DDT software page can be found here.

DDT supports C, C++, and Fortran 77 / 90 / 95 / 2003. Detailed documentation (the User Guide) is available as a PDF file on clusters where DDT is installed (currently these are orca, monk, kraken, and requin), in


Preparing your program

The code has to be compiled with the switch –g, which tells the compiler to generate symbolic information required by any debugger. Normally, all optimizations have to be turned off. For example,

f77 -g -o program program.f
mpicc -g -o code code.c

For CUDA code (on monk), you can first compile the CUDA part (*.cu files) using nvcc,

nvcc -g -G -c

and then link with the non-CUDA part of the code, using cc for serial

cc -g main.c cuda_code.o -lcudart

or mpicc for mixed MPI/CUDA code:

mpicc -g main.c cuda_code.o -lcudart

Launching DDT

Simply type

ddt program [optional arguments]

After a couple of seconds, you will be presented with the following window:

Ddt wiki1.png

User interface

DDT uses a tabbed-document interface. Each component of DDT is a dockable window, which may be dragged around by a handle. Components can also be double-clicked, or dragged outside of DDT, to form a new window. Some of the user-modified parameters and windows are saved by right-clicking and selecting a save option in the corresponding window (Groups; Evaluations; Breakpoints etc).

DDT also has the ability to load and save all these options concurrently to minimize the inconvenience in restarting sessions. Saving the session stores such things as process groups, the contents of the Evaluate window and more. When DDT begins a session, source code is automatically found from the information compiled in the executable.

The "Find" and "Find In Files" dialogs are found from the "Search" menu. The "Find" dialog will find occurrences of an expression in the currently visible source file. The "Find In Files" dialog searches all source and header files associated with your program and lists the matches in a result box. Click on a match to display the file in the main Source Code window and highlight the matching line.

DDT has a "Goto line" function which enables the user to go directly to a line of code (in the "Search" menu, or Ctrl-G).

Controlling Program Execution

  • Process Control And Process Groups:
    • Ability to group many process together, with a one-click access to the whole group
    • Three predefined groups: All, Root, Workers. (Newest DDT versions only have one group - All).
    • Groups can be created, deleted, or modified by the user at any time

Ddt wiki2.png

  • Starting, Stopping And Restarting A Program
    • Session Control dialog
  • Stepping Through A Program
    • Play/Continue, Pause, Step Into, Step Over, Step Out
  • Setting Breakpoints
    • All breakpoints are listed under the breakpoints tab.
    • You can suspend, jump to, delete, load or save breakpoints.
    • Breakpoints can easily be made conditional:

Ddt wiki3.png

Ddt wiki4.png

  • Synchronizing Processes in a Group
    • "Run to here" command (right mouse click)
  • Can work with individual processes or threads (by changing “Focus”)
  • Setting A Watch (for the current process)
  • Working with stacks
    • Moving (Down / Up / Bottom Stack Frame)
    • Aligning of stacks for parallel codes (Ctrl-A)
  • Viewing Stacks in Parallel
    • Stacks tab.
    • Shows a tree of functions, merged from every process in the group
    • Click on any branch to see its location in the Source Code viewer
    • Hover the mouse to see the list of the process ranks at this location in the code
    • Can automatically gather the processes at a function together in a new group

Ddt wiki5.png

  • Browsing Source Code
    • Highlights the current source line
    • Different color coding for synchronous and asynchronous state in the group
  • Simultaneously Viewing Multiple Files
    • Right click to split the source window into two or more, with each one having its own set of tabs.
  • Signal Handling: will stop on the following signals:
    • SIGSEGV (segmentation fault)
    • SIGFPE (Floating Point Exception)
    • SIGPIPE (Broken Pipe)
    • SIGILL (Illegal Instruction)
  • To enable FPE handling, extra compiler options are required:
    • Intel: -fp0

Variables and data

  • Current Line tab
    • Viewing all the variable for the current line(s) (click and drag for multiple lines)
  • Locals tab
    • Shows all local variables for the process

Ddt wiki6.png

  • Evaluate window
    • Can be used to view values for arbitrary expressions and global variables
  • Support for Fortran modules
    • Fortran Modules tab in the Project Navigator window
  • Changing Data Values
    • Right-click and select "Edit value" (in Evaluate window)
  • Examining Pointers
    • Drag a pointer into the Evaluate window
    • Can be viewed as Vector, Reference, or Dereference (right-click to choose).
  • Multi-Dimensional Arrays
    • Can be viewed in the Variable View
    • Multi-Dimensional Array viewer – visualization of a 2-D slice of an array using OpenGL

Ddt wiki7.png

Ddt wiki8.png

  • Cross-Process Comparison
    • Right-click on a variable name, or
    • From View menu: Cross-Process/Thread Comparison (type in any valid expression)
    • Three modes – Compare, Statistics, and Visualize

Ddt wiki9.png

Ddt wiki10.png

Ddt wiki13.png

Message queues

  • View -> Message Queues (older versions), or Tools -> Message Queues in newer versions, in the control panel.
  • Produces both a graphical view and a table for all active communications.
  • Helps to debug such MPI problems as deadlocks etc.

Ddt wiki11.png

Ddt wiki12.png

Memory Debugging

  • Can intercept memory allocation and de-allocation calls and perform lots of complex heap- and bounds- checking.
  • Off by default, can be turned on before starting a debugging session (in Advanced settings).
  • Different levels of memory debugging – from minimal to high.

Ddt wiki14.png

  • On error, stops with a message like

Ddt wiki15.png

  • Check Validity
    • In Evaluate window, right-click to "Check pointer is valid"
    • Useful for checking function arguments
  • Detecting memory leaks
    • "View->Current Memory Usage" window (Tools->Current Memory Usage in newer versions)
    • Shows current memory usage across processes in the group
    • Click on a color bar to get allocation details
    • For more details, choose "Table View"

Ddt wiki16.png

Ddt wiki17.png

  • Memory Statistics
    • Menu option "View->Overall Memory Stats" ("Tools->Overall Memory Stats" in newer versions).

Ddt wiki18.png

Ddt wiki19.png

Ddt wiki20.png

Debugging OpenMP programs

  • The current version of DDT has almost the same OpenMP functionality as for MPI:
    • Single click access to threads
    • Viewing stacks in parallel
    • Setting thread-specific breakpoints
    • Compare expressions across threads, etc

Debugging GPU programs

  • Only on monk cluster.
  • Compilation instructions were given earlier.
  • Can be a mixed MPI/CUDA code, but can only use one CPU and one GPU per node, up to 8 nodes.
  • New instructions for monk (different from the way it used to be on angel).
  • There is no test queue on monk (because GPU codes cannot be suspended), so DDT should be run directly, on development nodes (currently only one - mon-devel1).
  • One has to first ssh to the development node, and then load the ddt module:
ssh mon-devel1
module load ddt
  • After that DDT can be launched as usual: ddt ./code . When debugging CUDA codes, DDT allows to choose a specific GPU block and thread, and specific MPI rank for mixed codes:

Ddt wiki21.jpg

Using DDT


  • The environment has to be set properly to run X-windows (graphical) applications on our clusters.
    • Microsoft Windows: mobaxterm, cygwin (not easy to set up) or Xming.
    • Linux / Unix: add “ForwardX11 yes” – in
/etc/ssh/ssh_config, or
    • To test, ssh to a cluster and type xterm
  • On SHARCNET clusters, DDT can be used:
    • On development nodes (orca, saw, kraken, monk, ...)
    • Using "-t" switch of sqsub, - test queue (only on requin).