I am running an optimization that uses provided gradients for most components. I am still getting the error "'component_name (2-component_name): does not provide analytical derivatives forVarName'" for some of these variables even though the component either does not provide any analytic derivatives (relies on FD), or uses the keywords deriv_ignore='true' or missing_deriv_policy='assume_zero'. I have duplicated the FD reliance problem in the following example. The error shows up in OpenMDAOv0.13 using the pyOptDriver with gradient based algorithms, but in my actual code the same problem occurs when using any gradient based optimizer driver. I'm sorry I could not do a better reproduction of the problem.

What causes this error and is there a good remedy for it?

from openmdao.main.api import Assembly, Component
from openmdao.lib.datatypes.api import Float, Array, List
from openmdao.lib.drivers.api import DOEdriver, SLSQPdriver, COBYLAdriver, CaseIteratorDriver
from pyopt_driver.pyopt_driver import pyOptDriver

import numpy as np

class component1(Component):

    x = Float(iotype='in')
    term1 = Float(iotype='out')
    a = Float(iotype='in', default_value=1)
    def execute(self):
        x = self.x
        a = self.a

        term1 = a*x**2
        self.term1 = term1

        print "In comp1", self.name, self.a, self.x, self.term1

    # def list_deriv_vars(self):
    #     return ('x',), ('term1',)
    #
    # def provideJ(self):
    #
    #     x = self.x
    #     a = self.a
    #     dterm1_dx = 2.*a*x
    #
    #     J = dterm1_dx
    #     print 'In comp1, J = %s' % J
    #     return J

class component2(Component):

    x = Float(iotype='in')
    y = Float(iotype='in')
    term1 = Float(iotype='in')
    f = Float(iotype='out')

    def execute(self):

        y = self.y
        x = self.x
        term1 = self.term1

        f = term1 + x + y**2

        self.f = f
        print "In comp2", self.name, self.x, self.y, self.term1, self.f

class summer(Component):

    total = Float(iotype='out', desc='sum of all f values')

    def __init__(self, size):
        super(summer, self).__init__()
        self.size = size

        self.add('fs', Array(np.zeros(size), iotype='in', desc='f values from all cases'))

    def execute(self):
        self.total = sum(self.fs)
        print 'In summer, fs = %s and total = %s' % (self.fs, self.total)

class assembly(Assembly):

    x = Float(iotype='in')
    y = Float(iotype='in')
    total = Float(iotype='out', default_value=100)

    def __init__(self, a_vals=[1, 1, 1, 1]):

        self.a_vals = a_vals

        super(assembly, self).__init__()

        print 'in init a_vals = %s' % self.a_vals

    def configure(self):

        # self.add('driver', SLSQPdriver())
        self.add('driver', pyOptDriver())
        self.driver.optimizer = 'SLSQP'
        # self.driver.pyopt_diff = True

        #create this first, so we can connect to it
        self.add('summer', summer(size=len(self.a_vals)))
        self.connect('summer.total', 'total')

        print 'in configure a_vals = %s' % self.a_vals

        # create instances of components
        for i, a in enumerate(self.a_vals):
            c1 = self.add('comp1_%d'%i, component1())
            c1.a = a
            c2 = self.add('comp2_%d'%i, component2())

            self.connect('x', ['comp1_%d.x'%i,'comp2_%d.x'%i])
            self.connect('y', 'comp2_%d.y'%i)
            self.connect( 'comp1_%d.term1'%i, 'comp2_%d.term1'%i)

            self.connect('comp2_%d.f'%i, 'summer.fs[%d]'%i)

            self.driver.workflow.add(['comp1_%d'%i, 'comp2_%d'%i])

        self.driver.workflow.add(['summer'])

        # set up main driver (optimizer)
        self.driver.iprint = 1
        self.driver.maxiter = 100
        self.driver.accuracy = 1.0e-6
        self.driver.add_parameter('x', low=-5., high=5.)
        self.driver.add_parameter('y', low=-5., high=5.)
        self.driver.add_objective('summer.total')

if __name__ == "__main__":
    """ the result should be -1 at (x, y) = (-0.5, 0) """

    import time
    from openmdao.main.api import set_as_top

    test = set_as_top(assembly([1, 1, 1, 1]))

    test.x = 2
    test.y = 4

    tt = time.time()
    test.run()

    print "Elapsed time: ", time.time()-tt, "seconds"

    print 'result = ', test.summer.total
    print '(x, y) = (%s, %s)' % (test.x, test.y)

asked 22 Jul '15, 15:40

jthomas's gravatar image

jthomas
372210


Hey, jthomas

I tried this out, and the problem was in the pyopt_driver (SLSQP worked fine). It was an easy fix, so I pushed up and tagged a new version of the pyopt_driver (ver 0.21). You can either pull down from the repo or just do another plugin install on it to pull down the new version.

https://github.com/OpenMDAO-Plugins/pyopt_driver/releases

link

answered 22 Jul '15, 16:28

Kenneth%20T%20Moore's gravatar image

Kenneth T Moore ♦
1.1k1212

Thank you for your response, and for fixing the pyopt_driver. That fixes the example case I posted, but I am still getting the same error in my real code indicating missing gradients when I have specified that deriv_ignore='true' or missing_deriv_policy='assume_zero' for the variables in question. I will continue to investigate and try to reproduce the problems in a simple case, but if you have any ideas about what might be going wrong I would really appreciate it.

(22 Jul '15, 17:12) jthomas jthomas's gravatar image

I have recreated the other errors in my simple case and posted a new question regarding them "The variable key words "ignore_deriv" and "missing_deriv_policy" don't seem to be working"

(22 Jul '15, 18:10) jthomas jthomas's gravatar image

Here is the output from running my code. I put it in an answer so it could be formatted properly as code. I have provided output for running with the SLSQPdriver and then running with pyopt_driver using SNOPT.


Running with SLSQPdriver:

   ---------------------------------------------------------------------------
     START OF THE SEQUENTIAL LEAST SQUARES PROGRAMMING ALGORIT  
   ---------------------------------------------------------------------------

     PARAMETERS:
        ACC =   0.1000D-11
        MAXITER =100
        IPRINT =   0
IOUT =   6

capi_return is NULL
Call-back cb_slgrad_in_slsqp__user__routines failed.
Traceback (most recent call last):
  File "/Users/user/Documents/project/test_call.py", line 112, in <module>
    myproject.run()
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/component.py", line 559, in run
    self.execute()
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/assembly.py", line 725, in execute
    self._system.run(self.itername, case_uuid=self._case_uuid)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/systems.py", line 1944, in run
    sub.run(iterbase, case_label=case_label, case_uuid=case_uuid)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/systems.py", line 1168, in run
    self._comp.run(case_uuid=case_uuid)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/driver.py", line 568, in run
    super(Driver, self).run(case_uuid)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/component.py", line 559, in run
    self.execute()
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/driver.py", line 596, in execute
    self.run_iteration()
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.lib-0.13.0-py2.7.egg/openmdao/lib/drivers/slsqpdriver.py", line 149, in run_iteration
    self._func, self._grad)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.lib-0.13.0-py2.7.egg/openmdao/lib/drivers/slsqpdriver.py", line 196, in _grad
    J = self._calc_gradient(self.inputs, self.obj + self.con)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/driver.py", line 889, in _calc_gradient
    options=options)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/workflow.py", line 174, in calc_gradient
    return_format=return_format)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/systems.py", line 861, in calc_gradient
    J = self.ln_solver.calc_gradient(inputs, outputs, return_format)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/linearsolver.py", line 167, in calc_gradient
    dx = self.solve(RHS)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/linearsolver.py", line 214, in solve
    maxiter=options.maxiter)
  File "<string>", line 2, in gmres
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/sparse/linalg/isolve/iterative.py", line 86, in non_reentrant
    return func(*a, **kw)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/sparse/linalg/isolve/iterative.py", line 479, in gmres
    work[slice2] += sclr1*matvec(work[slice1])
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/sparse/linalg/interface.py", line 134, in matvec
    y = self._matvec(x)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/linearsolver.py", line 243, in mult
    system.applyJ(vnames)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/systems.py", line 1810, in applyJ
    subsystem.applyJ(variables)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/systems.py", line 1241, in applyJ
    self._comp.applyJT(self, variables)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/component.py", line 519, in applyJT
    applyJT(system, variables)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/derivatives.py", line 407, in applyJT
    obj.raise_exception(msg, KeyError)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/container.py", line 1351, in raise_exception
    raise exception_class(full_msg)
KeyError: 'component_name (2-comonent_name): does not provide analytical derivatives forVarName'

Process finished with exit code 1

Running with pyopt_driver using SNOPT:

Exception: 'component_name (2-component_name): does not provide analytical derivatives forVarName'
======================================================================
======================================================================
Traceback (most recent call last):
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/pyopt_driver-0.20-py2.7.egg/pyopt_driver/pyopt_driver.py", line 302, in gradfunc
    J = self.workflow.calc_gradient(self.inputs, self.objs + self.cons)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/workflow.py", line 174, in calc_gradient
    return_format=return_format)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/systems.py", line 861, in calc_gradient
    J = self.ln_solver.calc_gradient(inputs, outputs, return_format)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/linearsolver.py", line 167, in calc_gradient
    dx = self.solve(RHS)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/linearsolver.py", line 214, in solve
    maxiter=options.maxiter)
  File "<string>", line 2, in gmres
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/sparse/linalg/isolve/iterative.py", line 86, in non_reentrant
    return func(*a, **kw)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/sparse/linalg/isolve/iterative.py", line 479, in gmres
    work[slice2] += sclr1*matvec(work[slice1])
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/sparse/linalg/interface.py", line 134, in matvec
    y = self._matvec(x)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/linearsolver.py", line 243, in mult
    system.applyJ(vnames)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/systems.py", line 1810, in applyJ
    subsystem.applyJ(variables)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/systems.py", line 1241, in applyJ
    self._comp.applyJT(self, variables)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/component.py", line 519, in applyJT
    applyJT(system, variables)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/derivatives.py", line 407, in applyJT
    obj.raise_exception(msg, KeyError)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/container.py", line 1351, in raise_exception
    raise exception_class(full_msg)
KeyError: 'component_name (2-component_name): does not provide analytical derivatives forVarName'
capi_return is NULL
Call-back cb_userfg_in_snoptc__user__routines failed.
Traceback (most recent call last):
  File "/Users/user/Documents/project/test_call.py", line 112, in <module>
    myproject.run()
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/component.py", line 559, in run
    self.execute()
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/assembly.py", line 725, in execute
    self._system.run(self.itername, case_uuid=self._case_uuid)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/systems.py", line 1944, in run
    sub.run(iterbase, case_label=case_label, case_uuid=case_uuid)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/systems.py", line 1168, in run
    self._comp.run(case_uuid=case_uuid)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/driver.py", line 568, in run
    super(Driver, self).run(case_uuid)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/openmdao.main-0.13.0-py2.7.egg/openmdao/main/component.py", line 559, in run
    self.execute()
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/pyopt_driver-0.20-py2.7.egg/pyopt_driver/pyopt_driver.py", line 170, in execute
    hot_start=self.hot_start)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/pyOpt/pyOpt_optimizer.py", line 160, in __call__
    return self.__solve__(opt_problem, *args, **kwargs)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/pyOpt/pySNOPT/pySNOPT.py", line 716, in __solve__
    iw, rw)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/pyOpt/pySNOPT/pySNOPT.py", line 444, in snfuncgrag
    dff,dgg = gradient.getGrad(x, group_ids, [f_obj], f_con, *args, **kwargs)
  File "/Users/user/OpenMDAO/openmdao-0.13.0/lib/python2.7/site-packages/pyOpt/pyOpt_gradient.py", line 269, in getGrad
    dfi[j,i] = df_user[j,i]
IndexError: index 0 is out of bounds for axis 1 with size 0

Process finished with exit code 1
link

answered 22 Jul '15, 17:30

jthomas's gravatar image

jthomas
372210

edited 22 Jul '15, 17:39

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Tags:

×43
×39
×22
×11
×9

Asked: 22 Jul '15, 15:40

Seen: 1,809 times

Last updated: 22 Jul '15, 18:10

powered by OSQA