MPI Debugging with VS Code

In order to debug a program running under MPI, a debugger must be able to attach to multiple running processes. VS Code, and other IDEs based on VS Code, like Cursor for example, can do this using a launch.json file that specifies a unique debug configuration for each process. Setting this up manually isn’t much fun, so OpenMDAO has automated the process.

OpenMDAO injects new configurations into launch.json and attempts not to overwrite any of your manual changes to it

Activating the Debugger

The automated MPI debugging process uses the debugpy package, so before you start, install that into your python environment using your preferred method, e.g., pip, conda, pixi, etc.

Run Your Program

The first step is to run your program. This can be done using various commands depending on whether you’re debugging a test or a standalone python script. The key point is that in order to allow MPI debugging to happen, you must set the VSCODE_DBG environment variable.

For example, let’s say you are running a simple python script using 2 MPI processes. You would normally run it like this:

    mpirun -n 2 python myscript.py

In order to activate the automated MPI debugging setup, just do the following to set the environment variable:

    VSCODE_DBG=1 mpirun -n 2 python myscript.py

If, on the other hand, you want to run an MPI test on two processes, you could use the run_om_test command to do it like this:

    mpirun -n 2 run_om_test test_distribcomp.py:MPITests.test_overlapping_inputs_idxs

To activate debugging in this case, just add VSCODE_DBG=1 before the command as before:

    VSCODE_DBG=1 mpirun -n 2 run_om_test test_distribcomp.py:MPITests.test_overlapping_inputs_idxs

Note that the VSCODE_DBG environment variable processing occurs in the openmdao.api file, so if that file is not included in your script or test file, the MPI debugging will not activate. If you don’t want to include the entire openmdao.api file, you can import the setup_dbg function directly from openmdao.utils.general_utils and call it directly, for example:

    from openmdao.utils.general_utils import setup_dbg
    setup_dbg()

When your program runs, assuming you have imported openmdao.api or called setup_dbg directly, you should see a message from debugpy on each rank, for example:

    Rank 0: Debugger listening on port 51111
    Rank 1: Debugger listening on port 51112

Your program will block until you attach to the running processes with (VS Code / Cursor)

Set Your Breakpoint(s)

Set up your breakpoints as you normally would when debugging in your IDE, typically by clicking on the red dot in the far left column of the editor window containing the file where you want to set the breakpoint.

Attach to Your Running MPI Processes

In your (VS Code / Cursor) IDE, select Run and Debug. The location for this differs slightly depending on which IDE your’re using, but it’s generally on the left side in the same place as Explorer and Search.

Once you’ve selected Run and Debug, you’ll see a dropdown menu near the top on the left. Select MPI Debug. Finally, under the top level Run menu, select Start Debugging, or just hit the F5 key.

Debugging

Once you’ve attached to the running processes and have stopped at a breakpoint, you can debug as you normally would with a single process. The main difference is that now the call stack will show separate call trees for each MPI rank configuration. The configurations are named _rank_?_config, where ? is the rank number. Also, near the top of the editor window, the little toolbar with choices like Continue, Step Over, Step Into, etc. will have a dropdown menu next to it that allows you to choose which rank you want to step through. It also has a Disconnect option to disconnect from the current rank’s process.

The Debug Console tab of the terminal below the editor window also has a dropdown menu to allow you to switch between rank processes so you can interact with each specific process.