ExternalCodeComp

ExternalCodeComp is a component that runs an external program in a subprocess on your operating system.

If external programs do not have Python APIs, it is necessary to “file wrap” them. ExternalCodeComp is a utility component that makes file wrapping easier by taking care of the mundane tasks associated with executing the external application. These include:

• Making the system call using the Subprocess module

• Redirecting stdin, stdout, and stderr to the user’s specification

• Capturing error codes

• Defining environment variables

• Handling timeout and polling

• Running the code on a remote server if required

ExternalCodeComp Options

Option	Default	Acceptable Values	Acceptable Types	Description
allowed_return_codes	[0]	N/A	N/A	List of return codes that are considered successful.
always_opt	False	[True, False]	['bool']	If True, force nonlinear operations on this component to be included in the optimization loop even if this component is not relevant to the design variables and responses.
command	[]	N/A	['list', 'str']	Command to be executed. If it is a string, then this is the command line to execute and the 'shell' argument to 'subprocess.Popen()' is set to True. If it is a list; the first entry is the command to execute.
distributed	False	[True, False]	['bool']	If True, set all variables in this component as distributed across multiple processes
env_vars	{}	N/A	N/A	Environment variables required by the command.
external_input_files	[]	N/A	N/A	List of input files that must exist before execution, otherwise an Exception is raised.
external_output_files	[]	N/A	N/A	List of output files that must exist after execution, otherwise an Exception is raised.
fail_hard	True	[True, False]	['bool']	If True, external code errors raise a 'hard' exception (RuntimeError), otherwise errors raise a 'soft' exception (AnalysisError).
poll_delay	0.0	N/A	N/A	Delay between polling for command completion. A value of zero will use an internally computed default.
run_root_only	False	[True, False]	['bool']	If True, call compute, compute_partials, linearize, apply_linear, apply_nonlinear, and compute_jacvec_product only on rank 0 and broadcast the results to the other ranks.
timeout	0.0	N/A	N/A	Maximum time to wait for command completion. A value of zero implies an infinite wait.

ExternalCodeComp Constructor

The call signature for the ExternalCodeComp constructor is:

ExternalCodeComp.__init__(**kwargs)

Intialize the ExternalCodeComp component.

ExternalCodeComp Example

In this example we will give an example based on a common scenario of a code that takes its inputs from an input file, performs some computations, and then writes the results to an output file. ExternalCodeComp supports multiple input and output files but for simplicity, this example only uses one of each. Also, for the purposes of this example we have kept the input and output files as simple as possible. In practice, the data will likely be organized in some defined way and thus some care must be taken to read and write the data as dictated by the file format. OpenMDAO provides a set of File Wrapping tools to help with this.

Note

To make it easy for you to run our example external code in any operating system or environment, we built it as a Python script that evaluates the paraboloid equation. We’ll just call this script like any other executable, even though it is a Python script, and could be turned directly an OpenMDAO Component. Just keep in mind that any external code will work here, not just python scripts!

Here is the script for this external code. It simply reads its inputs, x and y, from an external file, does the same computation as the Paraboloid Tutorial and writes the output, f_xy, to an output file.

#!/usr/bin/env python
#
# usage: extcode_paraboloid.py input_filename output_filename
#
# Evaluates the equation f(x,y) = (x-3)^2 + xy + (y+4)^2 - 3.
#
# Read the values of `x` and `y` from input file
# and write the value of `f_xy` to output file.

if __name__ == '__main__':
    import sys

    input_filename = sys.argv[1]
    output_filename = sys.argv[2]

    with open(input_filename, 'r') as input_file:
        file_contents = input_file.readlines()

    x, y = [float(f) for f in file_contents]

    f_xy = (x-3.0)**2 + x*y + (y+4.0)**2 - 3.0

    with open(output_filename, 'w') as output_file:
        output_file.write('%.16f\n' % f_xy)

The following example demonstrates how to build an OpenMDAO component that makes use of this external code.

Note

If you pass a string as a command, OpenMDAO sets shell=True which can add overhead leading to a decrease in performance and a security loophole. Use list when possible.

import openmdao.api as om


class ParaboloidExternalCodeComp(om.ExternalCodeComp):
    def setup(self):
        self.add_input('x', val=0.0)
        self.add_input('y', val=0.0)

        self.add_output('f_xy', val=0.0)

        self.input_file = 'paraboloid_input.dat'
        self.output_file = 'paraboloid_output.dat'

        # providing these is optional; the component will verify that any input
        # files exist before execution and that the output files exist after.
        self.options['external_input_files'] = [self.input_file]
        self.options['external_output_files'] = [self.output_file]

        # If you want to write your command as a list, the code below will also work.
        # self.options['command'] = [
        #     sys.executable, 'extcode_paraboloid.py', self.input_file, self.output_file
        # ]

        self.options['command'] = ('python extcode_paraboloid.py {} {}').format(self.input_file, self.output_file)

    def compute(self, inputs, outputs):
        x = inputs['x']
        y = inputs['y']

        # generate the input file for the paraboloid external code
        with open(self.input_file, 'w') as input_file:
            input_file.write('%.16f\n%.16f\n' % (x, y))

        # the parent compute function actually runs the external code
        super().compute(inputs, outputs)

        # parse the output file from the external code and set the value of f_xy
        with open(self.output_file, 'r') as output_file:
            f_xy = float(output_file.read())

        outputs['f_xy'] = f_xy

We will go through each section and explain how this code works.

OpenMDAO provides a base class, ExternalCodeComp, which you should inherit from to build your wrapper components. Just like any other component, you will define the necessary inputs and outputs in the setup method. If you want the component to check to make sure any files exist before/after you run, then you can set the external_input_files and external_output_files, respectively. You’ll also define the command that should be called by the external code.

def setup(self):
    self.add_input('x', val=0.0)
    self.add_input('y', val=0.0)

    self.add_output('f_xy', val=0.0)

    self.input_file = 'paraboloid_input.dat'
    self.output_file = 'paraboloid_output.dat'

    # providing these is optional; the component will verify that any input
    # files exist before execution and that the output files exist after.
    self.options['external_input_files'] = [self.input_file]
    self.options['external_output_files'] = [self.output_file]

    # If you want to write your command as a list, the code below will also work.
    # self.options['command'] = [
    #     sys.executable, 'extcode_paraboloid.py', self.input_file, self.output_file
    # ]

    self.options['command'] = ('python extcode_paraboloid.py {} {}').format(self.input_file, self.output_file)

The compute method is responsible for calculating outputs for a given set of inputs. When running an external code, this means you have to take the parameter values and push them down into files, run your code, then pull the output values back up. So there is some Python code needed to do all that file writing, reading, and parsing.

def compute(self, inputs, outputs):
    x = inputs['x']
    y = inputs['y']

    # generate the input file for the paraboloid external code
    with open(self.input_file, 'w') as input_file:
        input_file.write('%.16f\n%.16f\n' % (x, y))

    # the parent compute function actually runs the external code
    super().compute(inputs, outputs)

    # parse the output file from the external code and set the value of f_xy
    with open(self.output_file, 'r') as output_file:
        f_xy = float(output_file.read())

    outputs['f_xy'] = f_xy

ParaboloidExternalCodeComp is now complete. All that is left is to actually use it in a model.

import sys

prob = om.Problem()
model = prob.model

model.add_subsystem('p', ParaboloidExternalCodeComp(), promotes_inputs=['x', 'y'])

# run the ExternalCodeComp Component
prob.setup()

# Set input values
prob.set_val('p.x', 3.0)
prob.set_val('p.y', -4.0)

prob.run_model()

# print the output
print(prob.get_val('p.f_xy'))

[-15.]

/tmp/ipykernel_18353/381977752.py:32: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
  input_file.write('%.16f\n%.16f\n' % (x, y))
/usr/share/miniconda/envs/test/lib/python3.11/subprocess.py:1883: RuntimeWarning: os.fork() was called. os.fork() is incompatible with multithreaded code, and JAX is multithreaded, so this will likely lead to a deadlock.
  self.pid = _fork_exec(

Using ExternalCodeComp in an Optimization

If you are going to use an ExternalCodeComp component in a gradient based optimization, you’ll need to get its partial derivatives somehow. One way would be just to use finite-difference approximations for the partials.

In the following example, the ParaboloidExternalCodeComp component has been modified to specify that partial derivatives are approximiated via finite difference.

class ParaboloidExternalCodeCompFD(om.ExternalCodeComp):
    def setup(self):
        self.add_input('x', val=0.0)
        self.add_input('y', val=0.0)

        self.add_output('f_xy', val=0.0)

        self.input_file = 'paraboloid_input.dat'
        self.output_file = 'paraboloid_output.dat'

        # providing these is optional; the component will verify that any input
        # files exist before execution and that the output files exist after.
        self.options['external_input_files'] = [self.input_file]
        self.options['external_output_files'] = [self.output_file]

        self.options['command'] = [
            sys.executable, 'extcode_paraboloid.py', self.input_file, self.output_file
        ]

    def setup_partials(self):
        # this external code does not provide derivatives, use finite difference
        self.declare_partials(of='*', wrt='*', method='fd')

    def compute(self, inputs, outputs):
        x = inputs['x']
        y = inputs['y']

        # generate the input file for the paraboloid external code
        with open(self.input_file, 'w') as input_file:
            input_file.write('%.16f\n%.16f\n' % (x, y))

        # the parent compute function actually runs the external code
        super().compute(inputs, outputs)

        # parse the output file from the external code and set the value of f_xy
        with open(self.output_file, 'r') as output_file:
            f_xy = float(output_file.read())

        outputs['f_xy'] = f_xy

Now we can perform an optimization using the external code, as shown here:

prob = om.Problem()
model = prob.model

model.add_subsystem('p', ParaboloidExternalCodeCompFD())

# find optimal solution with SciPy optimize
# solution (minimum): x = 6.6667; y = -7.3333
prob.driver = om.ScipyOptimizeDriver()
prob.driver.options['optimizer'] = 'SLSQP'

prob.model.add_design_var('p.x', lower=-50, upper=50)
prob.model.add_design_var('p.y', lower=-50, upper=50)

prob.model.add_objective('p.f_xy')

prob.driver.options['tol'] = 1e-9
prob.driver.options['disp'] = True

prob.setup()

# Set input values
prob.set_val('p.x', 3.0)
prob.set_val('p.y', -4.0)

prob.run_driver()

/tmp/ipykernel_18353/1682570986.py:30: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
  input_file.write('%.16f\n%.16f\n' % (x, y))

Optimization terminated successfully    (Exit mode 0)
            Current function value: -27.333333333333
            Iterations: 5
            Function evaluations: 6
            Gradient evaluations: 5
Optimization Complete
-----------------------------------

False

print(prob.get_val('p.x'))
print(prob.get_val('p.y'))

[6.66666633]
[-7.33333367]

Alternatively, if the code you are wrapping happens to provide analytic derivatives you could have those written out to a file and then parse that file in the compute_partials method.

Here is a version of our external script that writes its derivatives to a second output file:

#!/usr/bin/env python
#
# usage: extcode_paraboloid_derivs.py input_filename output_filename derivs_filename
#
# Evaluates the equation f(x,y) = (x-3)^2 + xy + (y+4)^2 - 3.
#
# Read the values of `x` and `y` from input file
# and write the value of `f_xy` to output file.
#
# Also write derivatives to another output file.

if __name__ == '__main__':
    import sys

    input_filename = sys.argv[1]
    output_filename = sys.argv[2]
    derivs_filename = sys.argv[3]

    with open(input_filename, 'r') as input_file:
        file_contents = input_file.readlines()

    x, y = [float(f) for f in file_contents]

    f_xy = (x-3.0)**2 + x*y + (y+4.0)**2 - 3.0

    with open(output_filename, 'w') as output_file:
        output_file.write('%.16f\n' % f_xy)

    with open(derivs_filename, 'w') as derivs_file:
        # partials['f_xy', 'x']
        derivs_file.write('%.16f\n' % (2.0*x - 6.0 + y))
        # partials['f_xy', 'y']
        derivs_file.write('%.16f\n' % (2.0*y + 8.0 + x))

And the corresponding ParaboloidExternalCodeCompDerivs component:

class ParaboloidExternalCodeCompDerivs(om.ExternalCodeComp):
    def setup(self):
        self.add_input('x', val=0.0)
        self.add_input('y', val=0.0)

        self.add_output('f_xy', val=0.0)

        self.input_file = 'paraboloid_input.dat'
        self.output_file = 'paraboloid_output.dat'
        self.derivs_file = 'paraboloid_derivs.dat'

        # providing these is optional; the component will verify that any input
        # files exist before execution and that the output files exist after.
        self.options['external_input_files'] = [self.input_file]
        self.options['external_output_files'] = [self.output_file, self.derivs_file]

        self.options['command'] = [
            sys.executable, 'extcode_paraboloid_derivs.py',
            self.input_file, self.output_file, self.derivs_file
        ]

    def setup_partials(self):
        # this external code does provide derivatives
        self.declare_partials(of='*', wrt='*')

    def compute(self, inputs, outputs):
        x = inputs['x']
        y = inputs['y']

        # generate the input file for the paraboloid external code
        with open(self.input_file, 'w') as input_file:
            input_file.write('%.16f\n%.16f\n' % (x, y))

        # the parent compute function actually runs the external code
        super().compute(inputs, outputs)

        # parse the output file from the external code and set the value of f_xy
        with open(self.output_file, 'r') as output_file:
            f_xy = float(output_file.read())

        outputs['f_xy'] = f_xy

    def compute_partials(self, inputs, partials):
        outputs = {}

        # the parent compute function actually runs the external code
        super().compute(inputs, outputs)

        # parse the derivs file from the external code and set partials
        with open(self.derivs_file, 'r') as derivs_file:
            partials['f_xy', 'x'] = float(derivs_file.readline())
            partials['f_xy', 'y'] = float(derivs_file.readline())

Again, we can perform an optimization using the external code with derivatives:

prob = om.Problem()
model = prob.model

model.add_subsystem('p', ParaboloidExternalCodeCompDerivs())

# find optimal solution with SciPy optimize
# solution (minimum): x = 6.6667; y = -7.3333
prob.driver = om.ScipyOptimizeDriver()
prob.driver.options['optimizer'] = 'SLSQP'

prob.model.add_design_var('p.x', lower=-50, upper=50)
prob.model.add_design_var('p.y', lower=-50, upper=50)

prob.model.add_objective('p.f_xy')

prob.driver.options['tol'] = 1e-9
prob.driver.options['disp'] = True

prob.setup()

# Set input values
prob.set_val('p.x', 3.0)
prob.set_val('p.y', -4.0)

prob.run_driver();

/tmp/ipykernel_18353/832503303.py:32: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
  input_file.write('%.16f\n%.16f\n' % (x, y))

Optimization terminated successfully    (Exit mode 0)
            Current function value: -27.333333333333336
            Iterations: 5
            Function evaluations: 6
            Gradient evaluations: 5
Optimization Complete
-----------------------------------

print(prob.get_val('p.x'))
print(prob.get_val('p.y'))

[6.66666667]
[-7.33333333]