OpenMDAO V3.13.0 is live. You can read the release notes for complete details, but there are a few key changes worth highlighting:

There are some minor improvements to the way OpenMDAO computes relative step sizes. Full details in POEM 051, but in short you can now ask for step sizes for array variables on a per element basis or from an avg array value. The old method has been kept as well for backwards compatibility, though it is deprecated and will be removed in a later version.

POEMS 053 and 054 summarize a change to the way src_indices work. It’s a bit subtle, so I encourage you to read the POEMS. The key take away though is that src_indices behavior now matches what numpy does with array indexing. This is a backwards incompatible change, though you can can check for its impact by using OpenMDAO V3.12 which will give you a clear deprecation warning.

Another small tidibit: We’re starting to work on on speeding up the interpolation routines. We’re not done yet, but we have some early progress. Check out the akima1d and trilinear interpolation methods if you’re looking for speedier table lookups.

We revamped our docs based on jupyter notebooks, so you can run all our docs code on Google Collab. Just look for the rocket-ship icon in the upper right corner of a docs page!

That will take you to a live notebook of the same docs page so you can try things out in your browser without installing anything locally. Check out the paraboliod optimization example running in the cloud.

New features and APIs

V3.10 has a lot of new features, APIs, and some important deprecations. You can get all the details in the release notes, but here are the highlights.

• Major redesign of the APIs for distributed components and variables. See POEM_046 for a lot of details. You now specify distributed independently on each variable.
• We now use val everywhere (before there was a mix of val and value. The older keyword has been deprecated, which gives you a chance to update before the 4.0 release. See POEM_050.
• An error is raised if you run check_partials and the same settings are used for both the approximated derivatives and the check.
  Note: We found a surprising number of cases where this was happening. The check is useless in this case. So You might get an new error, but you should be glad that you’re finding what is effectively a bug in your code.
• You can now use “fancy” indices (i.e. multi-dimensional slices) for constraints
• There is a new flag called under_finite_difference in components to tell you when you’re being finite-differenced (mirrors the existing under_complex_step flag)

New Logo

Have you noticed out new logo? Feel free to stamp it on any thing you like!

OpenMDAO V3 is a little inconsistent with it use of val and value as keyword arguments to different API methods. We are going to fix this in V4 by standardizing on val for everything. This change is proposed in POEM 050.

Unfortunately, that is going to cause some “minor” backwards incompatibility for any user code that specifically called out the value keyword. So in V3.11 and higher, we’ll keep backwards compatibility with the old keyword and give you a deprecation warning. This should provide a smooth upgrade path, since you can try out the new code and work to remove the deprecations.

We recognize that these kind of API changes are modestly annoying, but we feel they are a net positive in the long run because don’t have to guess which keyword to use where. If you don’t agree with us, or have an alternate proposal then feel free to chime in with comments or PRs on POEM 050.

Introducing two new proposals for OpenMDAO enhancement (POEMs), for your consideration. Both are fairly low impact, though POEM 049 does propose removal of some unused APIs.

POEM 048: MetaModelSemiStructured Component

OpenMDAO Already has both structured and unstructured metamodel components. You might think that those two are the all you would ever need, but there is a semi-structured data format that is fairly common (at least in the aircraft design world). This kind of data is useful for things like performance tables where you can’t necessarily get valid data on a full structured grid. The data looks like this:

x = 1
    y = 1, 2, 3, 4, 5, 10, 11
    z = 10, 20, 30, 40, 50, 100, 110
x = 2 
    y = 1, 2, 3, 4, 5, 10, 11
    Z = 60, 80, 100, 200, 220
x = 3
    y = 1, 2, 3, 4, 5, 10, 11
    z = 300, 330, 120, 150, 300, 330

The independent values are always monotonic, but the dependent data does not necessarily have to be. Despite being non-structured, the monotonic inputs allow for the same kind of recursive interpolation that OpenMDAO’s MetaModelStructuredComp uses.

POEM 049: Removal of the matrix-matrix derivative APIs.

While this POEM does propose a backwards incompatible change, to the best of our knowledge no one is actually using these APIs. If you are using them and want to object to their removal, now is your chance! Please comment on the POEM PR and provide us an example use case where you really need to have it.

These APIs date back to early OpenMDAO V1 days when performance for serial components that provided a dense partial derivative Jacobian were very poor due to some slower internal data formats and the lack of a DirectSolver. At that time, our solution involved augmenting the matrix-vector product APIs with a matrix-matrix product API that would skip a particularly slow OpenMDAO for-loop. It worked to an extend, but was not a very general solution.

Today we have both sparse and dense assembled Jacobians that provide better overall performance in a wider range of use cases. We believe these older matrix-matrix features are not needed, and removing them will simplify several places in the code and make our linear solvers more easily understandable.

If you think otherwise, please speak up. We’d appreciate you providing a use case that shows how the matrix-matrix APIs are significantly faster than using the an Assembled Jacobian and LinearRunOnce or DirectSolver.

Version 3.9’s biggest change is the new serial/distributed api for variables. We now allow you to label individual variables as serial or distributed, which ultimately clears up a lot of confusion when working with distributed memory computations. All of the details are covered in POEM 046. There are lots of other changes in this release, including performance improvements and a few other new features. You can read the release notes for a full accounting, but here are the highlights.

ExecComp Improvements

A small but useful update was made to ExecComp. You can now call the add_expr method on that Component to create new outputs. This change makes the ExecComp API look more similar to the BalanceComp and EQConstraintComps. It also makes for slightly cleaner inputs when you’re using a lot of expressions in your model.

Quieter Warnings

Were you annoyed by the warnings about missing MPI4py or petsc4py? You were not alone! We’ve specifically quieted those two, and you’ll only see them if you try to use those features without the proper packages in stalled. More generally though, we reworked our whole warnings system.

OpenMDAO gives you a lot of freedom, and sometimes that lets you get into trouble. We’ve tried hard to add clear error/warning messages to cover these situations. However, the net result is that sometimes your runs can get kind of noisy. So we re-built our warning system to be more organized, and gave users the ability to limit which ones actually get reported. Check out the docs for the new warning system.

Try to remember, with great power comes great responsibility 🙂 When you suppress warnings you might be missing helpful information. We recommend you leave most warnings on during development and turn them off only for production runs.

OpenMDAO V3.8.0 is out. This is a modest release, but does have some new features. Check out all the release notes for complete details, but here are some highlights:

• Overhaul of the internal storage vectors (relevant if you’re doing cool stuff inside the guts of OpenMDAO like these folks from DLR)
• Ability to save/load views of an N2, so you can keep that specific view you spent a few minutes setting up 🙂
• added some preliminary support for ipython notebooks to some of the visualization and variable listing methods.
• Bug fixes for exec-comp has_diag_partials argument
• Bug fix for the residual based filtering of list_outputs
• Bug fix for Nan values breaking the N2

If you do any work with OpenMDAO distributed components, you should check out POEM_046. Ongoing collaborative work between University of Michigan, NASA Glenn, NASA Langley, and Georgia Tech exposed some weaknesses in the current (i.e. as of V3.8.0) apis. So we did a very deep dive to figure out the “right” way. POEM_046 is currently just a proposal, but it should be accepted and implemented in time for V3.10.0 in late May or early June 2021. We put a lot of detail into this POEM, so it could serve as a reference for anyone who wants to understand how OpenMDAO does its stuff. The final APIs are pretty simple, but the majority of the POEM deals with why we decided to go this way. Feedback is more than welcome from anyone!

So what about this 99% speedup? Sounds pretty great right? A few weeks ago we noticed this in one of our internal benchmarks:

On April 1st something pretty spiffy happened there right? Please excuse the obtuse benchmark test names. We have a lot of internal benchmarks (we run nightly tests on OpenMDAO core, pyCycle, and Dymos) and we use rather verbose names for them to help make it easy to figure out whats going on. In this case, the benchmarks were from Dymos, and (as the long names suggest) they delt with partial derivative coloring algorithms.

We recently completed a major overhaul of the internal FD/CS partial derivative approximations and the associated partial derivative coloring. Things are now fully sparse internally and work on a column by column basis that is a lot more memory efficient. That massive drop in benchmark times came from this upgrade. If you haven’t tried out the partial derivative coloring, you should give it a look. That goes triple if you’re a dymos user. It lets you use complex-step derivatives and still get a lot of the benefits of sparse partials!

These benchmarks literally got 99% faster, or 100x speedup. Props go to Bret Naylor on the dev team for doing the hard implementation work and to Sandy Mader — the user who suggested the improvement. Bret is pretty humble about the speed up though. He says that the secret was to make the initial code as slow as possible, so there was lots of room for improvement.
The new, faster coloring will be released with Version 3.9.0.

Here is the next monthly installment of your favorite show: OpenMDAO Gets New Features

On this episode:

• Release notes for V3.7.0
• Register your own functions to be used with ExecComp
• debugging methods give nicely formatted output in Jupyter notebooks
• ExecComp now shows the equations in the info panel for the N2
• Bug fixes for parallel solvers

User Defined ExecComp Functions

You can now add register own functions to be used in ExecComp. They can be complex-stepped for derivatives just like normal (or you can mark them as requiring finite-difference instead of you prefer). One of the nice things this will let you do is quickly compose a component that chains several methods together. You can read our thinking on details in POEM 39. Here are the docs for user function registration. If you try this out and have any feedback, don’t hesitate to let the dev team know!

N2 viewer Improvements

If you haven’t looked closely at the N2 for a while you might have missed a number of small improvements over the last few releases. The info tool can be used on components and variables to see values, units, and other metadata. New in V3.7, if you get the info for an exec-comp, you can see the equations that were used to define it.

You can also show/hide the solver hierarchy and highlight the design variables/constraints/objective as well.

Version 3.6.0 marks a huge milestone for OpenMDAO. This is the second release in a row where there are no backwards incompatible API changes! Hurray! To celebrate we released our new logo!

Lest you think we just made a release without changing anything … check out the release notes. Here are the highlights

Scaling Reports

In our internal use cases, particularly with Dymos based problems, we’ve been struggling with optimization scaling. We’re starting to look into some more advanced methods for scaling, but the first step to improving things is understanding what you have in the first place. We built a scaling report tool that helps you get that information in one place. To run it, you can call the OpenMDAO command line openmdao scaling <your run script

N2 Improvements

• Include data about which surrogate is used in metamodels
• Can now hide the solver hierarchy in the N2 (useful if you want to make slightly lower-aspect ratio movies of the N2)
• Included a button to show/hide the design variables, objective, and constraints

There is a new POEM (Proposal for OpenMDAO EnhanceMent) on the street. POEMS are the way that any new feature is vetted before being implemented. Developers and users are all welcome to propose POEMS! Also, we value any user input on POEMS before they are implemented. So please hop over to the proposal for POEM 039 and feel free to leave any comments you like (or submit suggested modifications as a PR of your own).

POEM 039 would enable users to register their own functions to the ExecComp, so they can use them in a simplified and streamlined manner without the boilerplate of writing their own component wrapper.

Here is an example of how it could work:

def aero_forces(rho, v, CD, CL, S)
    q = 0.5 * rho * v**2
    lift = q * CL * S
    drag = q * CD * S
    return lift, drag

om.ExecComp.register('aero_forces', aero_forces)

om.ExecComp('L,D = aero_forces(rho, v, CD, CL, S)', 
             rho={'units': 'kg/m**3'},
             v={'units': 'm/s'},
             S={'units': 'm**2'},
             lift={'units': 'N'},
             drag={'units': 'N'}, 
             vectorized=True, shape_by_conn=True, has_diag_partials=True
            )

Also included is a small addition to make it easier to set up ExecComps to have I/O sized by connection. This will reduce the amount of boiler plate you need to set up ExecComps with vectorized I/O.

While the Dev team is putting the finishing touches on OpenMDAO RevHack2020, I wanted give a wrap-up summary.

First, let say thank you to everyone who submitted problems for us to work on. Thanks to Remi Lafage, Adam Chase, John Jasa, Anil Yildirim. Extra special thanks for Shamsheer Chauhan for submitting 4 different problems! Our goal for RevHack 2020 was to develop a better understanding of your perspective and we simply couldn’t have done that without your help.

Check out the cool stuff we did:

• Wrapped OpenVSP in OpenMDAO and integrated it with OpenAerostruct.
• Got OpenMDAO + OpenVSP working in google collab (run OpenMDAO in the cloud!)
• Sped up the optimization of an eVTOL take-off trajectory optimization by an order of magnitude.
• Wrote a driver for the CMA-ES optimization library (serial and parallel versions)
• Updated an unsteady aerodynamic simulation from V0 to V3, and experimented with a sub-problem based solution structure
• Played with nested optimizations, trying out sub-problems and solver-based alternatives.

A new perspective on the “right way” to use OpenMDAO

We used RevHack2020 as a chance to step outside the normal development process and examine things from a new perspective. That started with trying to really understand what was the core of a particular user struggle, but we took that further, asking the following:

Is there a better way to do this?
What is the value of OpenMDAO for this problem?
Are we really making this easier than it would be without OpenMDAO?

Here are some of the semi-philosophical answers that I came up with.

If you don’t want to use analytic derivatives, is OpenMDAO worth it?

As the leader of the OpenMDAO project, it is a little scary to ask myself this question. I know that it is extremely powerful and effective for our work at NASA. We couldn’t write tools like pyCycle or Dymos without it but we leverage the derivatives capabilities heavily.

OpenMDAO enforces some specific coding structures on you, and those are derived from the goals of modularity and support for analytic derivatives. Sometimes those are getting users way and if you are not getting the benefits of the derivatives, I can see how frustrating that would be.

So I have come to two conclusions:

1) If you are not planning to ever use analytic derivatives, then the case for OpenMDAO is much weaker. If you do want to have a path to using them, OpenMDAO is worth the effort.

2) We need to better job of making OpenMDAO easier to use. That means making bigger components, supporting algorithmic differentiation, and giving clear examples of how to leverage sub-problems when you want to do things like for-loops, use derivatives as outputs in a model, and sub-optimizations.

Maybe we don’t need drivers at all

I was personally shocked when thinking about CMA-ES problem brought me to the conclusion that the whole Driver interface in OpenMDAO is non critical. I still think it has a lot of value, but its non-critical because you could get all of the value from OpenMDAO without it just using the apis on the problem itself.

There is a lot of code in our Driver interfaces, and I can see that users find it intimidating to figure out how to navigate that. That code is there because it allows the developers to switch between the multiple optimization libraries that we use. If you want to use a single specific library though, you don’t need any of that.

I am not going to rip the Driver interface out of OpenMDAO, because its useful to the development team and I don’t want to break models of anyone who uses it. However, I am explicitly saying that you should feel free to disregard it and roll your own interface to the optimizer. Here is a more in depth discussion of this topic

If you build a massively complex optimization, expect it to be hard to solve!

One thing that OpenMDAO has done is make it easier to build bigger, more complex optimization problems. I can see that a lot of users are still struggling, and I acknowledge that we have more work to do. However, I have also seen users build models that are intensely complex, and set up massive optimization problems that are fundamentally hard to solve.

So even if it’s hard to do, I argue that there are definitely classes of problems that you would struggle to solve without OpenMDAO. Ironically, since OpenMDAO is making it easier to build these hard problems, more users seem to be running into the fact that hard problems are … hard to solve (shocking, I know!)

In the end, this is a good problem to have. If we really are to the point where you can build a problem that is so complex you can’t figure out how to optimize it then OpenMDAO has contributed something to the collective community. Now one of our next challenges is to give you some tools that make these hard problems a little less hard.