Unit Testing#

Overview#

Generally, every sub-package in the OpenMDAO hierarchy has a tests directory with test modules for the various features of that package. If your change impacts one of those features, then you should add your test(s) to the relevant module. Optionally, you can create a new test module under the appropriate sub-package.

testflo#

While you can use python -m unittest to run many of the OpenMDAO unit tests, it is recommended that you use testflo, which is a testing framework that was designed specifically to facilitate OpenMDAO testing.

Per the README, testflo is a python testing framework that uses a pipeline of iterators to process test specifications, run the tests, and process the results. Some OpenMDAO features require execution under MPI while some others don’t, so testflo has the basic functionality of other test frameworks but with the additional ability to run MPI unit tests using unittest.TestCase in the same way as all of our other tests.

In addition to MPI unit testing, there are a number of other convenient features that testflo provides over and above other testing frameworks. Some of these are:

  • concurrent testing

  • can be given a directory, a file, a module path, file:testcase.method, module:testcase.method, or a file containing a list of any of these

  • pre_announce option to print test name before running in order to quickly identify hanging MPI tests

  • end of testing summary including failed tests and optionally skipped tests, test coverage, all tests that run for longer than a certain time threshold, and tests that generate deprecation warnings

You can install testflo with several other optional testing dependencies by appending [test] to the pip install target. For example:

pip install openmdao[test] or

pip install -e .[test]

Test Suite#

The openmdao.test_suite package provides a number of test models, scripts and utilities that can be useful when writing unit tests. Here you will find multiple variations on the simple Paraboloid and Sellar models, for example, as well as more complex examples like a circuit analysis and a multipoint beam optimization. Using components and models from the test suite can expedite the task of writing new tests and reduce the occurrence of repetitive code in unit tests. When creating new tests, consider whether there is an existing component or model in the test suite that can be used to exercise the feature under test, and if not then adding one for others to use.

Testing Utilities#

The openmdao.utils package provides a couple of modules to assist with common needs of unit testing.

testing_utils#

The testing_utils module has decorators and context managers to control if and how a test is run. Here are a few examples:

  • The @use_tempdirs decorator will run test functions in a temporary directory, which is useful when a test generates outputs files that need to be cleaned up or can cause collisions with other tests.

  • pyOptSparse is an optional dependency for OpenMDAO, but it provides some of the most powerful optimizers (IPOPT and SNOPT in particular). If your test makes use of one of these, you should use the @require_pyoptsparse decorator so that the test is skipped for users that have not installed pyOptSparse and the specified optimizer.

  • For tests that depend on environment variables, the set_env_vars context manager is provided to temporarily modify the environment to suit a specific test.

  • Normally, the check_partials function will not check the partial derivatives of Components that are part of OpenMDAO itself. The force_check_partials function provided here will override that behavior if needed in a particular test.

assert_utils#

The assert_utils module provides helper functions and context managers that are useful for asserting the outcome of tests. Some of these are:

  • assert_near_equal is a convenient function for comparing values, including a variety of supported data structures, to the values expected by a test. It is more flexible and can be used in place of various other assert functions provided in the Python unittest and numpy.testing packages.

  • assert_check_partials, assert_check_totals, assert_no_approx_partials and assert_no_dict_jacobians are beneficial for checking that derivatives are calculated correctly for the model under test.

  • The assert_warning, assert_warnings and assert_no_warning context managers can be used to check that warnings are issued (or not) as expected.