There is a new online journal called distill.pub that has an interesting new format. It places a particular emphasis on clear explanations of concepts and high-quality infographics that help to further understanding. This is so obviously a good idea, that it may seem somewhat unclear why a journal would need to explicitly include it as a metric for judging publications at all. The reality is that a lot of really great papers are not written with accessibility as a primary goal of the work. Researchers often focus on developing new theory, new applications, or new implementations to advance the state-of-the-art. There often isn’t time to develop clear and concise explanations that will help specialists and non-specialists alike understand the work, but the lack there of does limit the impact the work on the broader field. Still confused? Here, we made this super-fancy animation to help you understand it better!

The term “research debt” has been coined to describe this phenomenon, where some members of a particular community have made large advances but the broader group has yet to catch up.  distil.pub has a great, in-depth discussion of research debt. They talk about different ways it can occur, what its fundamental costs really amount to, and when you should consider investing the time and effort to reduce it. Distill.pub is looking to publish papers that help to reduce research debt! As our helps to demonstrate, good explanation helps others in the field to better understand and build off your work and thus increases its impact.

When reading about the idea of research debt, it became clear that OpenMDAO could be viewed as project that reduces the research debt in the gradient-based optimization community. The most advanced gradient-based optimization techniques, with analytic derivatives, offer the potential for 10-to-10,000-times faster optimizations. Unfortunately these methods are usually implemented in an ad-hoc and problem-specific manner that makes it tough to generalize to a wide range of applications. What OpenMDAO does is implement these methods in a way that makes it possible to use analytic derivatives without being an expert in them, and without having to use a custom implementation for each problem.

Since OpenMDAO version 2.3, we’ve been continuously introducing new features that offer dramatic performance improvements. If you haven’t tried out assembled Jacobians (added in v2.3), you are missing out! There is also a new graph-coloring approach that works miracles for some problems (added in v2.4).  But along the way we’ve also been trying to limit our own research debt by keeping the docs up-to-date and working to improve their clarity and usefulness. Writing more docs keeps the new methods more accessible to OpenMDAO users. More importantly, it also forces us to think about ways to explain the concepts more clearly and concisely which often leads to simpler APIs and more general implementations.

So if you’ve ever gotten really deep down into a literature review on a subject and thought, “Hey, there is a better way to explain all this!”, now you have a great place to publish peer-reviewed journal articles that will help everyone in the broader community. The exact same concept goes for OpenMDAO docs too. If you’re working with our tools and you think you see a way to explain features in a simpler and more general manner, please feel free to submit a pull request. We’ll be happy to add an acknowledgment in the docs you contributed, so you get credit for helping to pay down the research debt!

OpenMDAO 2.5.0 is out, containing lots of new features, including some backwards-incompatible changes.  Here are the release notes:

Release Notes for OpenMDAO 2.5.0

October 31, 2018

New Features:
--------------
- list_outputs() method now includes a `prom_name` argument to include the promoted name in the printed output (Thanks John Jasa!).
- N2 viewer now includes tool tip (hover over) showing the promoted name of any variable.
- Improved error msg when building sparse partials with duplicate col/row entries.
- You can now build the docs without MPI/PETSc installed (you will get warnings, but no longer errors).
- Major internal refactor to reduce overhead on compute_totals calls (very noticeable on large design spaces but still cheap problems).
- Components now have a `under_complex_step` attribute you can check to see if complex-step is currently active.
- Components `distributed` attribute has been moved to an option (old attribute has been deprecated).
- MetaModelUnstructured will now use FD for partial derivatives of surrogates that don't provide analytic derivatives (user can override the default settings if they wish to use CS or different FD config).
- Improvements to SimpleGA to make it more stable, and added support constraints via penalty functions (Thanks madsmpedersen and onodip).
- Parallel FD and CS at the component and group level is now supported.
- Can turn off the analytic derivative sub-system if it is not needed via an argument to setup().
- Derivative coloring now works for problems that run under MPI.

- New Components in the standard library:
    - Mux and Demux components.

- New CaseRecording/CaseReading Features:
    - DesVar AND output variable bounds are both reordered now.
    - Improved error msg if you try to load a non-existent file with CaseReader.

- **Experimental Feature**: Discrete data passing is now supported as an experimental feature... we're still testing it, and may change the API!

Backwards Incompatible API Changes:
------------------------------------
- `get_objectives` method on CaseReader now returns a dict-like object.
- Output vector is now locked (read-only) when inside the `apply_nonlinear` method (you shouldn't have been changing it then anyway!).
- default step size for complex-step has been changed to 1e-40.
- Moderate refactor of the CaseReader API. It is now self-consistent.

Release Notes for OpenMDAO 2.4.0

August 1, 2018

New Features:
————–
– Better error message when upper and lower have the wrong shape for add_design_var and add_constraint methods.
– pyOptSparseDriver now runs the initial condition for ALPSO and NSGA-II optimizers.
– Normalization in EQConstraintComp and BalanceComp is now optional.

– New Components in the standard library:
– VectorMagnitudeComp

– New Solvers in the standard library:
– BroydenSolver

– New CaseRecording/CaseReading Features:
– .load_case() method lets you pull values from a case back into the problem.
– Updated the Case recording format for better performance.
– User can call .record_iteration() to save specific cases.
Recording options for this method are separate from driver/solver options.
This is so you can record one (or more) cases manually with *ALL* the variables in it if you want to.

Backwards-Incompatible API Changes:
————————————
– The input and output vectors are now put into read-only modes within certain component methods.
NOTE: This REALLY REALLY should not break anyone’s models, but it is technically a backwards-incompatible change…
If you were changing values in these arrays when you shouldn’t have been, you’re going to get an error now.
Fix it… the new error is enforcing correct behavior.
If it was working before, you got lucky.

Bug Fixes:
————
– Various small bug fixes to check_partials and check_totals.
– ArmijoGoldstein linesearch iteration counter works correctly now.
– The record_derivatives recorder_option now actually does something!

We’re proud to announce the release of OpenMDAO 2.3.0. This is a substantial release that comes with many new features. Some of the new features are small, but there are are few major ones. You can read all the details in the release notes, but here are the highlights:

More Speed:
OpenMDAO now has an in-memory assembled Jacobian structure that can significantly reduce the computational cost for computing derivatives and solving for Newton updates when using certain kinds of linear solvers.  If you have models with components that provide derivatives via the compute_partials() or linearize() methods, or which use FD to approximate component partials, then you’ll very likely want to use this feature. If you have any models that use the DirectSolver, then you definitely want to use this feature.

Check out the feature doc for assembled Jacobians to learn how to use this new feature. You can also read more generally about how to know if you should be using assembled Jacobians in this Theory Manual article.

If you are doing gradient-based optimization with analytic derivatives, or using our NewtonSolver, then you will really want to check out these docs and test out this new feature. We are seeing better than a 10X speed up for our models!

New docs about how to more efficiently compute derivatives:
We’ve done a massive re-write of the docs relating to how you work with derivatives.  There are a bunch of advanced features for speeding up derivative computations that you might not have realized were there because the old docs were not organized that well (sorry!). So, even if you’ve been using OpenMDAO for a while now, we suggest you give the new feature docs a read. If you want to dive a bit deeper into the theory behind the advanced algorithms, we’ve also added some new Theory Manual sections. If you’ve looked over our feature docs on derivatives and are now wondering if any of the more advanced features are useful for your model then we suggest you check out that Theory Manual section.

Not Just Gradient-Based Optimization:
Although our major focus is on gradient-based optimization, we know that there are problems where gradient-free methods are useful. We (finally) ported the DOEDriver from version 1, including support for parallel execution. We also added support for parallel execution to the SimpleGADriver.

Unit conversions:
Sometimes, you want to set or get a value with different units than those with which the value was initially defined. The Problem class now has get_val() and set_val() methods that accept a units argument. As long as the unit is compatible, OpenMDAO will handle the conversion for you when you use these methods.

OpenMDAO 2.2.1 came out today. Here are the release notes:

Release Notes for OpenMDAO 2.2.1

April 2, 2018

New Features:
————–
– check_partials() improvements to formatting and clarity. Report of potentially-bad derivatives summarized at bottom of output.
– check_partials() only compares fwd analytic to FD for any components that provide derivatives directly through the Jacobian
argument to compute_partials or linearize. (significantly less output to view now).
– Docs for UnstructuredMetaModel improved.
– pyoptsparse wrapper only calls run_model before optimization if LinearConstraints are included.
– ScipyOptimizerDriver is now smarter about how it handles linear constraints. It caches the derivatives and doesn’t recompute them anymore.
– Docs for ExternalCode improved to show how to handle derivatives.
– cache_linear_solution argument to add_design_var, add_constraint, add_objective, allows iterative linear solves to use previous solution as initial guess.
– New solver debugging tool via the `debug_print` option: writes out initial state values, so failed cases can be more easily replicated.
– Added generic KS component.
– Added generic Bspline component.
– Improved error msg when class is passed into add_subsystem.
– Automated Jacobian coloring algorithm now works across all variables (previously, it was just local within a variable).
– Major refactor of the `compute_totals` method to clean up and simplify.

Backwards-Compatible API Changes:
———————————–
N/A

Backwards-Incompatible API changes:
———————————–
N/A

Bug Fixes:
———–
– compute_totals works without any arguments now (just uses the default set of des_vars, objectives, and constraints).
– UnstructuredMetaModel can now be sub-classed.
– Deprecated ScipyOptimizer class wasn’t working correctly, but can now actually be used.

OpenMDAO 2.2.0 was released today. Documentation for this version can be found here. If you have any trouble with the release, feel free to contact us for support via our Stack Overflow tag, “openmdao.”

Release Notes for OpenMDAO 2.2.0

February 9, 2018

New Features:
————–
– `DirectSolver` now tells you which row or column is singular when it gets a singluar matrix error.
– `ScipyOptimizeDriver` now handles linear constraints more efficiently by only computing them one time.
– Added the `openmdao` command line script to allow for model checking, visualization, and profiling without making modifications to the run script.
– Added a `SimpleGADriver` with a basic genetic algorithm implementation.
– Added a `MetaModelStructured` component with a interpolative method.
– New option for derivative calculations: Simultaneous derivatives, useful when you have totally disjoint Jacobians (e.g. diagonal Jacobians).
– Automatic coloring algorithm added to compute the valid coloring scheme for simultaneous derivatives.
– `list_outputs` method updated with new display options, ability to filter variables by residual value, ability to change sorting scheme, and ability to display unit details.
– openmdao citation helper added to the `openmdao` command line script, making it easy to generate the correct bibtex citations based on which classes are being used.
– `NewtonSolver` modified so that maxiter=0 case now will compute residuals, but not do a linear solve (useful for debugging nonlinear errors).

Backwards-Compatible API Changes:
———————————–
– Changed `ScipyOptimizer` to `ScipyOptimizeDriver` for consistency (deprecated older class).
– Renamed `MetaModel` to `MetaModelUnstructured` to allow for new structured interpolant (deprecated old class).
– Renamed `PetscKSP` to `PETScKrylov` for consistency. (deprecated old class).
– Renamed `ScipyIterativeSolver` to `ScipyKrylov` for consistency. (deprecated old class).

Backwards-Incompatible API changes:
———————————–
– CaseRecorder now uses variables’ promoted names for storing and accessing data.
– Removed `DeprecatedComp` from codebase.
– `list_residuals` method on Groups and Components removed.

Bug Fixes:
———–
– Fixed error check for duplicate connections to a single input from multiple levels of the hierarchy

OpenMDAO 2.1.0 was released today. Documentation for this version can be found here. If you have any trouble with the release, feel free to contact us for support via our Stack Overflow tag, “openmdao.”

Here are the release notes for 2.1.0:

New Features:
————-
– Configure setup hook allowing changing of solver settings after hierarchy tree is instantiated.
– Component metadata system for specifying init_args with error checking.
– Parallel Groups were added.
– Units Reference added to the docs.
– Case recording now records all variables by default.
– `openmdao` console script that can activate useful debugging features
(e.g. view_model) without editing the run script.
– Scipy COBYLA optimizer converts des var bounds to constraints (the algorithm doesn’t natively handle bounds)
– StructuredMetaModel component offers a simple spline interpolation routine for structured data.

Backwards-Compatible API Changes:
———————————–
– `NonlinearRunOnce` changed `NonLinearRunOnce` for consistency (old class deprecated).
– `types_` argument to `self.metadata.declare` changed to `types`. (old argument deprecated).
– `types` and `values` arguments to `self.metadata.declare`
– `BalanceComp` has a `use_mult` argument to control if it has a `mult` input, defaulting to false
(the mult input isn’t used most of the time)
– Renamed `MetaModel` to `UnstructuredMetaModel` and `MultiFiMetaModel` to `UnStructuredMultiFiMetaModel`

Backwards-Incompatible API Changes:
———————————–
– Case Recording options API updated with `.recording_options` attribute on Driver, Solver, and System classes
– `get_subsystem` changed to a private method, removed from public API of System.
– `check_partials` now has a `method` argument that controls which type of check.

Bug Fixes:
———–
– Improved error msg on a corner case for when user doesn’t declare a partial derivative.
– Fixed docs embedding bug when `<>` included in the output text.

OpenMDAO 2 represents a total re-write and significant update of the framework with much cleaner APIs and performance improvements. It is also a backwards-incompatible release, and does not yet (as of November 2017) include all of the features that were in version 1. Given that, when and how should you go about updating your code? While each group will have to make this choice for itself, we want to give our perspective on the issue in the hope that it will help make the decision easier for you.

Why a total rewrite?

OpenMDAO 1.0 offered two key features that made it unique:

1. Tight integration with high-fidelity, parallel-analysis tools.
2. Automatic analytic derivatives.

The high fidelity HPC capability was achieved by integrating a novel computational architecture that supported distributed memory parallelism via MPI. The analytic derivatives features were enabled by the development of a sparse, matrix-free linear solver sub-system. The first feature worked extremely well, but the second was only partially successful because it suffered some performance limitations.

The challenge we faced was that we had designed a system that was efficient for large sparse analyses (e.g. problems including CFD models), but that had come at the cost of performance for smaller more dense analyses (e.g. low fidelity serial propulsion models).

Once we identified the performance issue, the fix was clear. We needed to use a dense linear solver system for smaller problems. The challenge was that 1.0 was designed around the sparse, matrix-free assumption, and it wasn’t clear that we could easily refactor the code around that. So we decided that a re-write was necessary.

So version 2 includes a hybrid linear solver architecture that supports a mixture of sparse and dense operations. Our testing on our own internal applications has shown an 10x reduction in compute cost for smaller problems while maintaining efficiency for larger scale problems.

These massive performance gains make a compelling reason to upgrade to  v2.0, and we do expect that the user base will all make the switch eventually! The time may not be quite right for everyone to upgrade, because we haven’t just re-written the underlying code. We’ve also made several backwards-incompatible changes to the user-facing APIs.

Why make it backwards incompatible?

In making the transition to the hybrid linear solver architecture, it became clear that some new component level APIs were going to be needed, and that it would not be easy to make them backwards incompatible. Also, in a number of spots we felt that the version 1 APIs were both a little inconsistent and somewhat confusing. So we decided to free ourselves to make backwards-incompatible changes to the API. We feel that these updates are for the best in the long term.

However, writing a clean and fully self-consistent API is easier said than done. As we’ve developed version 2, we’ve had to repeatedly re-evaluate our API choices as we implement new features and build new applications that stress the framework. As such, we’re still making tweaks to the APIs and likely will be for through summer 2018. At this point (November 2017), the API changes are fairly small and mostly consist of “find and replace” type operations, or represent expansions via new API methods, but changes are still being made.

This API-changing issue was the primary reason we decided to release version 2 when we did. We decided that it was important to let the community know about the new APIs and start getting some outside feedback. It also gave users the ability to fix their work to a specific version and guarantee a stable API for themselves. To that end, our version numbering system uses a three-digit convention: 2.x.y. The ‘x’ is for releases that include API changes. All new APIs will be listed in the release notes, and any backwards incompatible changes will be called out.  The ‘y’ is for minor releases (e.g. bug fixes and docs updates) and won’t represent new APIs. In other words: changing from 2.1 vs 2.2 will mean an API change, changing from 2.2.0 to 2.2.1 will not.

So which version should you use?

If you’re starting a new project from scratch, you should start from version 2! The only exception would be if you immediately need some of the features we have not yet implemented. As of November 2017, the most significant things on the list of missing features are the DOEDriver and Pass-By-Object variables. If you need either of those features, you’ll have to wait at least a few more months.

If you have a significant code base already implemented in 1.0, the decision is a little harder. Although the APIs are different, they are dauntingly so. As a point of reference to the difficulty of translating from 1 to 2:  an internal application built on OpenMDAO that included over 100 files and 10000 lines of code was updated from version 1 to version 2 by one person in less than 3 days. However, if your application is currently working fine in version 1 and you’d rather wait a few more months till we’ve finished porting all the major features and settled down on backwards incompatible API changes, thats a reasonable choice.

By August 2018, things will have settled down considerably, and at that point we’ll start recommending that everyone make the update. Again, new codes should definitely start out in version 2! For the time being, though, good arguments can be made for using either version 1 or version 2 with existing codes, and the choice is yours.

In October 2017, we released OpenMDAO 2.0 alpha. The 2.0 release of OpenMDAO has two important improvements to the code base:

1. Cleaner, more self-consistent API
2. Improved performance for models with lots of components and scalar variables

A new API?

The decision to make backwards incompatible changes to the OpenMDAO API isn’t one that we take lightly. We know it causes a lot of inconvenience to users, and presents a huge barrier to updating. However, the 1.7.3 API was built incrementally as we developed new applications, and we never had the chance to develop a self-consistent API design.

Many of the API changes that we wanted to make were superficial (e.g. changing `add_parameter` to `add_input`). However, there were some more significant API changes that were made for the sake of enabling improved performance (e.g. the new derivatives APIs). When we stood back and looked at all the changes as a whole, we decided that it was better to make them all at once in one big 2.0 release rather than make small changes as we iterated from 1.7 to 1.8 to 1.9 …

In the end, we feel that the new 2.0 API is much cleaner and more clear. Its also really not that big of a departure from the 1.7. We developed a translation guide that shows you how to update your models. It’s not long, which shows that the API changes are not massive.

We still have a few parts of the API that we’re not totally done with yet. Specifically the CaseRecording API is going to get a further update in the very near future. But the vast majority of the APIs are now in place and we wanted to get the new code out there for users to play with as a soon as possible.

“Simplify, then add lightness”

In 1.7.3, the derivatives system was designed primarily around use-cases that included high-fidelity analyses (e.g. CFD, FEA) in the model. It needed to scale well to distributed-memory computing environments. We succeeded in designing a system that worked on that scale, but we soon realized that our design was not very efficient on smaller-scale serial analyses. Things got especially inefficient when we tried to build models with hundreds or thousands of components in them.

We needed to re-work the guts of OpenMDAO to support both problem scales simultaneously. In the process, we’ve been able to get computational cost reductions of up to 10x on some of our trajectory analysis work, and 5x on some of our propulsion analysis work. We’re still working on getting more performance out of 2.0, but improvements like those have shown that we’re on the right track.

Why should I upgrade?

You shouldn’t feel the need to update right away. If 1.7.3 is working for you, and meets your needs, then by all means stay working with that code for now. Not all the 1.7.3 features have been implemented in 2.0 yet (for a current list, check out the README in the repository)

However, if you were having performance issues with 1.7.3 (memory or cpu performance), then 2.0 is definitely worth a look. If you’re developing new engineering applications natively in OpenMDAO (this is what the OpenMDAO development team does) then you’ll benefit greatly from the new speed improvements and the cleaner APIs.

We here at OpenMDAO have been hard at work writing a completely new version of OpenMDAO. After 1.7.3, we started from a clean slate, and OpenMDAO 2.0.0 is now an all new alpha. OpenMDAO 2.0 is all about performance, especially for smaller scale problems. On our in-house applications we’re seen up to a 10x speed up in performance.

Still Alpha?

We’re still tweaking the APIs a bit and we want you all to be aware of that. One of the reasons we’ve decided to make an official release of 2.0 is so that we could make it it clear when the APIs are changing. Any change in the version number at the second decimal (i.e. a change in the `x` from 2.x.0) will indicate an api change. The API change could be a modification to an existing API, or the addition of new APIs to extended functionality. Once we’re happy with the APIs, the alpha tag will come off.

And now for a bit of housekeeping:

The OpenMDAO 1.7.3 codebase repo has been renamed to OpenMDAO1, and it resides at https://github.com/OpenMDAO/OpenMDAO1. If you have a clone of this repo, you’ll need to update the URL on your remotes.

The OpenMDAO 2.0.x code has taken the name OpenMDAO,
and it resides at https://github.com/OpenMDAO/OpenMDAO.

Installation of 2.0.x code will now work with `pip install openmdao`.

Installation of 1.7.3 code will now only work with a version specifier: `pip install openmdao==1.7.3`

The release notes detail what is in 2.0.0, here’s an excerpt:

To get started, read our documentation at https://openmdao.org/twodocs