TotalView

From ScorecWiki

Jump to: navigation, search

Contents

Debugging an MPI program with TotalView interactively on Hydra or Borg

From a VNC session

These instructions are for running Totalview on the Hydra compute nodes when you are working within a VNC session on portalN where N=1, 2, or 3. Variables in an instruction are preceeded with VAR__ and must be substituted before execution.

  1. Disable X access control:
    xhost +
    • Note that access control should be re-enabled when the debugging session is complete.
      xhost -
  2. Determine the local port number. This number will be referred to as VAR__PORT in subsequent instructions.
    echo $DISPLAY
  3. Connect to Hydra from another terminal and request an interactive allocation on M processes.
    ssh hydra; salloc -I -p normal -N VAR__M
  4. Edit the TotalView preferences file to cache runtime libraries in your fasttmp directory. Open
    ~/.totalview/preferences6.tvd
    in a text editor. Right above the end of file notation (roughly the second to last or third to last line in the file) add a line:
    dset -version 13 TV::library_cache_directory /fasttmp/VAR__SCOREC_USER_NAME/TVcache
  5. Manually SSH to 'each' of the allocated nodes:
    ssh nodeVAR__NODE#.hydra.scorec.rpi.edu hostname
    . A bash for loop can simplify this. Suppose the allocation included nodes 1,2,4 and 11 then the following command would ssh to each node and run the 'hostname' command:
    for nodeId in {1,2,4,11}; do ssh node${nodeId}.hydra.scorec.rpi.edu hostname; done
    • This step can be skipped if you have previously ssh'd to these nodes.
  6. Launch the MPI job within TotalView. VAR__MPI_INSTALL_DIR should be the path to the directory where MPI is installed. mpirun is assumed to be in the PATH.:
    DISPLAY='portalN.scorec.rpi.edu:VAR__PORT' /usr/local/toolworks/totalview.8.11.0-0/bin/totalview mpirun -a -x DISPLAY -np VAR__M VAR__YOUR_EXECUTABLE VAR__ARGUMENTS_TO_YOUR_EXECUTABLE

From a workstation

These instructions are for running Totalview on the Hydra compute nodes when you are working at a SCOREC workstation. Variables in an instruction are preceeded with VAR__ and must be substituted before execution.

  1. Connect to Hydra and request an interactive allocation on M processes.
    ssh hydra; salloc -I -p normal -N VAR__M
  2. Edit the TotalView preferences file to cache runtime libraries in your fasttmp directory. Open
    ~/.totalview/preferences6.tvd
    in a text editor. Right above the end of file notation (roughly the second to last or third to last line in the file) add a line:
    dset -version 13 TV::library_cache_directory /fasttmp/VAR__SCOREC_USER_NAME/TVcache
  3. Manually SSH to 'each' of the allocated nodes:
    ssh nodeVAR__NODE#.hydra.scorec.rpi.edu hostname
    . A bash for loop can simplify this. Suppose the allocation included nodes 1,2,4 and 11 then the following command would ssh to each node and run the 'hostname' command:
    for nodeId in {1,2,4,11}; do ssh node${nodeId}.hydra.scorec.rpi.edu hostname; done
    • This step can be skipped if you have previously ssh'd to these nodes.
  4. Launch the MPI job within TotalView. VAR__MPI_INSTALL_DIR should be the path to the directory where MPI is installed. mpirun is assumed to be in the PATH.:
    /usr/local/toolworks/totalview.8.11.0-0/bin/totalview mpirun -a -np VAR__M VAR__YOUR_EXECUTABLE VAR__ARGUMENTS_TO_YOUR_EXECUTABLE

Viewing source code in Totalview

If you have compiled your code with debug symbols but Totalview does not display your code you will need to tell Totalview where your source files are by settings the path(s) to your source directories in 'File->Search Paths'.

Personal tools