Checking Total Derivatives

If you want to check the analytic derivatives of your model (or just part of it) against finite difference or complex-step approximations, you can use check_totals(). You should always converge your model before calling this method. By default, this method checks the derivatives of all of the driver responses (objectives, constraints) with respect to the des_vars, though you can also specify the variables you want to check. Derivatives are computed and compared in an unscaled form by default, but you can optionally request for them to be computed in scaled form using the ref and ref0 that were declared when adding the constraints, objectives, and des_vars.

Note

You should probably not use this method until you’ve used check_partials() to verify the partials for each component in your model. check_totals() is a blunt instrument, since it can only tell you that there is a problem, but will not give you much insight into which component or group is causing the problem.

Problem.check_totals(of=None, wrt=None, out_stream=<object object>, compact_print=False, driver_scaling=False, abs_err_tol=1e-06, rel_err_tol=1e-06, method='fd', step=1e-06, form='forward', step_calc='abs')[source]

Check total derivatives for the model vs. finite difference.

Parameters:
of : list of variable name strings or None

Variables whose derivatives will be computed. Default is None, which uses the driver’s objectives and constraints.

wrt : list of variable name strings or None

Variables with respect to which the derivatives will be computed. Default is None, which uses the driver’s desvars.

out_stream : file-like object

Where to send human readable output. By default it goes to stdout. Set to None to suppress.

compact_print : bool

Set to True to just print the essentials, one line per unknown-param pair.

driver_scaling : bool

Set to True to scale derivative values by the quantities specified when the desvars and responses were added. Default if False, which is unscaled.

abs_err_tol : float

Threshold value for absolute error. Errors about this value will have a ‘*’ displayed next to them in output, making them easy to search for. Default is 1.0E-6.

rel_err_tol : float

Threshold value for relative error. Errors about this value will have a ‘*’ displayed next to them in output, making them easy to search for. Note at times there may be a significant relative error due to a minor absolute error. Default is 1.0E-6.

method : str

Method, ‘fd’ for finite difference or ‘cs’ for complex step. Default is ‘fd’

step : float

Step size for approximation. Default is 1e-6.

form : string

Form for finite difference, can be ‘forward’, ‘backward’, or ‘central’. Default ‘forward’.

step_calc : string

Step type for finite difference, can be ‘abs’ for absolute’, or ‘rel’ for relative. Default is ‘abs’.

Returns:
Dict of Dicts of Tuples of Floats
First key:

is the (output, input) tuple of strings;

Second key:

is one of [‘rel error’, ‘abs error’, ‘magnitude’, ‘fdstep’];

For ‘rel error’, ‘abs error’, ‘magnitude’ the value is: A tuple containing norms for

forward - fd, adjoint - fd, forward - adjoint.

Examples

You can check specific combinations of variables by specifying them manually:

from openmdao.api import Problem, NonlinearBlockGS
from openmdao.test_suite.components.sellar import SellarDerivatives

prob = Problem()
prob.model = SellarDerivatives()
prob.model.nonlinear_solver = NonlinearBlockGS()

prob.setup()
prob.run_model()
NL: NLBGS Converged in 7 iterations
# manually specify which derivatives to check
prob.check_totals(of=['obj', 'con1'], wrt=['x', 'z'])
NL: NLBGS Converged in 3 iterations
NL: NLBGS Converged in 4 iterations
NL: NLBGS Converged in 3 iterations
-------------------------------------
Group: SellarDerivatives 'Full Model'
-------------------------------------
  Full Model: 'con1' wrt 'x'
    Forward Magnitude : 9.806145e-01
         Fd Magnitude : 9.804927e-01 (fd:forward)
    Absolute Error (Jfor  - Jfd) : 1.217530e-04 *

    Relative Error (Jfor  - Jfd) : 1.241753e-04 *

    Raw Forward Derivative (Jfor)
[[-0.98061448]]

    Raw FD Derivative (Jfd)
[[-0.98049272]]

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  Full Model: 'con1' wrt 'z'
    Forward Magnitude : 9.641989e+00
         Fd Magnitude : 9.641854e+00 (fd:forward)
    Absolute Error (Jfor  - Jfd) : 1.388696e-04 *

    Relative Error (Jfor  - Jfd) : 1.440279e-05 *

    Raw Forward Derivative (Jfor)
[[-9.61002186 -0.78449158]]

    Raw FD Derivative (Jfd)
[[-9.60989052 -0.78444647]]

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  Full Model: 'obj' wrt 'x'
    Forward Magnitude : 2.980614e+00
         Fd Magnitude : 2.980493e+00 (fd:forward)
    Absolute Error (Jfor  - Jfd) : 1.207479e-04 *

    Relative Error (Jfor  - Jfd) : 4.051274e-05 *

    Raw Forward Derivative (Jfor)
[[ 2.98061391]]

    Raw FD Derivative (Jfd)
[[ 2.98049317]]

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  Full Model: 'obj' wrt 'z'
    Forward Magnitude : 9.774287e+00
         Fd Magnitude : 9.774150e+00 (fd:forward)
    Absolute Error (Jfor  - Jfd) : 1.388623e-04 *

    Relative Error (Jfor  - Jfd) : 1.420710e-05 *

    Raw Forward Derivative (Jfor)
[[ 9.61001056  1.78448534]]

    Raw FD Derivative (Jfd)
[[ 9.60987922  1.78444024]]

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Check all the derivatives that the driver will need:

from openmdao.api import Problem, NonlinearBlockGS
from openmdao.test_suite.components.sellar import SellarDerivatives

prob = Problem()
prob.model = SellarDerivatives()
prob.model.nonlinear_solver = NonlinearBlockGS()

prob.model.add_design_var('x', lower=-100, upper=100)
prob.model.add_design_var('z', lower=-100, upper=100)
prob.model.add_objective('obj')
prob.model.add_constraint('con1', upper=0.0)
prob.model.add_constraint('con2', upper=0.0)

prob.setup()

# We don't call run_driver() here because we don't
# actually want the optimizer to run
prob.run_model()
NL: NLBGS Converged in 7 iterations
# check derivatives of all obj+constraints w.r.t all design variables
prob.check_totals()
NL: NLBGS Converged in 3 iterations
NL: NLBGS Converged in 4 iterations
NL: NLBGS Converged in 3 iterations
-------------------------------------
Group: SellarDerivatives 'Full Model'
-------------------------------------
  Full Model: 'con_cmp1.con1' wrt 'px.x'
    Forward Magnitude : 9.806145e-01
         Fd Magnitude : 9.804927e-01 (fd:forward)
    Absolute Error (Jfor  - Jfd) : 1.217530e-04 *

    Relative Error (Jfor  - Jfd) : 1.241753e-04 *

    Raw Forward Derivative (Jfor)
[[-0.98061448]]

    Raw FD Derivative (Jfd)
[[-0.98049272]]

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  Full Model: 'con_cmp1.con1' wrt 'pz.z'
    Forward Magnitude : 9.641989e+00
         Fd Magnitude : 9.641854e+00 (fd:forward)
    Absolute Error (Jfor  - Jfd) : 1.388696e-04 *

    Relative Error (Jfor  - Jfd) : 1.440279e-05 *

    Raw Forward Derivative (Jfor)
[[-9.61002186 -0.78449158]]

    Raw FD Derivative (Jfd)
[[-9.60989052 -0.78444647]]

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  Full Model: 'con_cmp2.con2' wrt 'px.x'
    Forward Magnitude : 9.692762e-02
         Fd Magnitude : 9.691559e-02 (fd:forward)
    Absolute Error (Jfor  - Jfd) : 1.203438e-05 *

    Relative Error (Jfor  - Jfd) : 1.241738e-04 *

    Raw Forward Derivative (Jfor)
[[ 0.09692762]]

    Raw FD Derivative (Jfd)
[[ 0.09691559]]

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  Full Model: 'con_cmp2.con2' wrt 'pz.z'
    Forward Magnitude : 2.227817e+00
         Fd Magnitude : 2.227804e+00 (fd:forward)
    Absolute Error (Jfor  - Jfd) : 1.381087e-05 *

    Relative Error (Jfor  - Jfd) : 6.199319e-06 *

    Raw Forward Derivative (Jfor)
[[ 1.94989072  1.0775421 ]]

    Raw FD Derivative (Jfd)
[[ 1.94987764  1.07753764]]

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  Full Model: 'obj_cmp.obj' wrt 'px.x'
    Forward Magnitude : 2.980614e+00
         Fd Magnitude : 2.980493e+00 (fd:forward)
    Absolute Error (Jfor  - Jfd) : 1.207479e-04 *

    Relative Error (Jfor  - Jfd) : 4.051274e-05 *

    Raw Forward Derivative (Jfor)
[[ 2.98061391]]

    Raw FD Derivative (Jfd)
[[ 2.98049317]]

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  Full Model: 'obj_cmp.obj' wrt 'pz.z'
    Forward Magnitude : 9.774287e+00
         Fd Magnitude : 9.774150e+00 (fd:forward)
    Absolute Error (Jfor  - Jfd) : 1.388623e-04 *

    Relative Error (Jfor  - Jfd) : 1.420710e-05 *

    Raw Forward Derivative (Jfor)
[[ 9.61001056  1.78448534]]

    Raw FD Derivative (Jfd)
[[ 9.60987922  1.78444024]]

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Use the driver scaled values during the check:

from openmdao.api import Problem, NonlinearBlockGS
from openmdao.test_suite.components.sellar import SellarDerivatives

prob = Problem()
prob.model = SellarDerivatives()
prob.model.nonlinear_solver = NonlinearBlockGS()

prob.model.add_design_var('x', lower=-100, upper=100, ref=100.0, ref0=-100.0)
prob.model.add_design_var('z', lower=-100, upper=100)
prob.model.add_objective('obj')
prob.model.add_constraint('con1', upper=0.0, ref=3.0)
prob.model.add_constraint('con2', upper=0.0, ref=20.0)

prob.setup()

# We don't call run_driver() here because we don't
# actually want the optimizer to run
prob.run_model()
NL: NLBGS Converged in 7 iterations
# check derivatives of all driver vars using the declared scaling
prob.check_totals(driver_scaling=True)
NL: NLBGS Converged in 3 iterations
NL: NLBGS Converged in 4 iterations
NL: NLBGS Converged in 3 iterations
-------------------------------------
Group: SellarDerivatives 'Full Model'
-------------------------------------
  Full Model: 'con_cmp1.con1' wrt 'px.x'
    Forward Magnitude : 6.537430e+01
         Fd Magnitude : 6.536618e+01 (fd:forward)
    Absolute Error (Jfor  - Jfd) : 8.116863e-03 *

    Relative Error (Jfor  - Jfd) : 1.241753e-04 *

    Raw Forward Derivative (Jfor)
[[-65.37429835]]

    Raw FD Derivative (Jfd)
[[-65.36618148]]

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  Full Model: 'con_cmp1.con1' wrt 'pz.z'
    Forward Magnitude : 3.213996e+00
         Fd Magnitude : 3.213951e+00 (fd:forward)
    Absolute Error (Jfor  - Jfd) : 4.628986e-05 *

    Relative Error (Jfor  - Jfd) : 1.440279e-05 *

    Raw Forward Derivative (Jfor)
[[-3.20334062 -0.26149719]]

    Raw FD Derivative (Jfd)
[[-3.20329684 -0.26148216]]

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  Full Model: 'con_cmp2.con2' wrt 'px.x'
    Forward Magnitude : 9.692762e-01
         Fd Magnitude : 9.691559e-01 (fd:forward)
    Absolute Error (Jfor  - Jfd) : 1.203438e-04 *

    Relative Error (Jfor  - Jfd) : 1.241738e-04 *

    Raw Forward Derivative (Jfor)
[[ 0.96927624]]

    Raw FD Derivative (Jfd)
[[ 0.9691559]]

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  Full Model: 'con_cmp2.con2' wrt 'pz.z'
    Forward Magnitude : 1.113909e-01
         Fd Magnitude : 1.113902e-01 (fd:forward)
    Absolute Error (Jfor  - Jfd) : 6.905434e-07

    Relative Error (Jfor  - Jfd) : 6.199319e-06 *

    Raw Forward Derivative (Jfor)
[[ 0.09749454  0.0538771 ]]

    Raw FD Derivative (Jfd)
[[ 0.09749388  0.05387688]]

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  Full Model: 'obj_cmp.obj' wrt 'px.x'
    Forward Magnitude : 5.961228e+02
         Fd Magnitude : 5.960986e+02 (fd:forward)
    Absolute Error (Jfor  - Jfd) : 2.414959e-02 *

    Relative Error (Jfor  - Jfd) : 4.051274e-05 *

    Raw Forward Derivative (Jfor)
[[ 596.1227827]]

    Raw FD Derivative (Jfd)
[[ 596.09863311]]

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  Full Model: 'obj_cmp.obj' wrt 'pz.z'
    Forward Magnitude : 9.774287e+00
         Fd Magnitude : 9.774150e+00 (fd:forward)
    Absolute Error (Jfor  - Jfd) : 1.388623e-04 *

    Relative Error (Jfor  - Jfd) : 1.420710e-05 *

    Raw Forward Derivative (Jfor)
[[ 9.61001056  1.78448534]]

    Raw FD Derivative (Jfd)
[[ 9.60987922  1.78444024]]

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Display the results in a compact format:

from openmdao.api import Problem, NonlinearBlockGS
from openmdao.test_suite.components.sellar import SellarDerivatives

prob = Problem()
prob.model = SellarDerivatives()
prob.model.nonlinear_solver = NonlinearBlockGS()

prob.model.add_design_var('x', lower=-100, upper=100)
prob.model.add_design_var('z', lower=-100, upper=100)
prob.model.add_objective('obj')
prob.model.add_constraint('con1', upper=0.0)
prob.model.add_constraint('con2', upper=0.0)

prob.setup()

# We don't call run_driver() here because we don't
# actually want the optimizer to run
prob.run_model()
NL: NLBGS Converged in 7 iterations
# check derivatives of all obj+constraints w.r.t all design variables
prob.check_totals(compact_print=True)
NL: NLBGS Converged in 3 iterations
NL: NLBGS Converged in 4 iterations
NL: NLBGS Converged in 3 iterations
'con_cmp1.con1'                wrt 'px.x'                         | 9.8061e-01 | 9.8049e-01 | 1.2175e-04 | 1.2418e-04
'con_cmp1.con1'                wrt 'pz.z'                         | 9.6420e+00 | 9.6419e+00 | 1.3887e-04 | 1.4403e-05
'con_cmp2.con2'                wrt 'px.x'                         | 9.6928e-02 | 9.6916e-02 | 1.2034e-05 | 1.2417e-04
'con_cmp2.con2'                wrt 'pz.z'                         | 2.2278e+00 | 2.2278e+00 | 1.3811e-05 | 6.1993e-06
'obj_cmp.obj'                  wrt 'px.x'                         | 2.9806e+00 | 2.9805e+00 | 1.2075e-04 | 4.0513e-05
'obj_cmp.obj'                  wrt 'pz.z'                         | 9.7743e+00 | 9.7741e+00 | 1.3886e-04 | 1.4207e-05
-------------------------------------
Group: SellarDerivatives 'Full Model'
-------------------------------------
'<output>'                     wrt '<variable>'                   | calc mag.  | check mag. | a(cal-chk) | r(cal-chk)
---------------------------------------------------------------------------------------------------------------------

Use complex step instead of finite difference for a more accurate check. We also change to a larger step size to trigger the nonlinear Gauss-Seidel solver to try to converge after the step.

from openmdao.api import Problem, NonlinearBlockGS
from openmdao.test_suite.components.sellar import SellarDerivatives

prob = Problem()
prob.model = SellarDerivatives()
prob.model.nonlinear_solver = NonlinearBlockGS()

prob.model.add_design_var('x', lower=-100, upper=100)
prob.model.add_design_var('z', lower=-100, upper=100)
prob.model.add_objective('obj')
prob.model.add_constraint('con1', upper=0.0)
prob.model.add_constraint('con2', upper=0.0)

prob.setup(force_alloc_complex=True)

# We don't call run_driver() here because we don't
# actually want the optimizer to run
prob.run_model()
NL: NLBGS Converged in 7 iterations
# check derivatives with complex step and a larger step size.
prob.check_totals(method='cs', step=1.0e-1)
NL: NLBGS Failed to Converge in 10 iterations
NL: NLBGS Failed to Converge in 10 iterations
NL: NLBGS Failed to Converge in 10 iterations
-------------------------------------
Group: SellarDerivatives 'Full Model'
-------------------------------------
  Full Model: 'con_cmp1.con1' wrt 'px.x'
    Forward Magnitude : 9.806145e-01
         Fd Magnitude : 9.806145e-01 (fd:forward)
    Absolute Error (Jfor  - Jfd) : 3.422115e-08

    Relative Error (Jfor  - Jfd) : 3.489766e-08

    Raw Forward Derivative (Jfor)
[[-0.98061448]]

    Raw FD Derivative (Jfd)
[[-0.98061451]]

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  Full Model: 'con_cmp1.con1' wrt 'pz.z'
    Forward Magnitude : 9.641989e+00
         Fd Magnitude : 9.641985e+00 (fd:forward)
    Absolute Error (Jfor  - Jfd) : 3.486804e-06 *

    Relative Error (Jfor  - Jfd) : 3.616272e-07

    Raw Forward Derivative (Jfor)
[[-9.61002186 -0.78449158]]

    Raw FD Derivative (Jfd)
[[-9.61001837 -0.7844916 ]]

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  Full Model: 'con_cmp2.con2' wrt 'px.x'
    Forward Magnitude : 9.692762e-02
         Fd Magnitude : 9.692745e-02 (fd:forward)
    Absolute Error (Jfor  - Jfd) : 1.711057e-07

    Relative Error (Jfor  - Jfd) : 1.765297e-06 *

    Raw Forward Derivative (Jfor)
[[ 0.09692762]]

    Raw FD Derivative (Jfd)
[[ 0.09692745]]

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  Full Model: 'con_cmp2.con2' wrt 'pz.z'
    Forward Magnitude : 2.227817e+00
         Fd Magnitude : 2.227833e+00 (fd:forward)
    Absolute Error (Jfor  - Jfd) : 1.743402e-05 *

    Relative Error (Jfor  - Jfd) : 7.825551e-06 *

    Raw Forward Derivative (Jfor)
[[ 1.94989072  1.0775421 ]]

    Raw FD Derivative (Jfd)
[[ 1.94990815  1.07754201]]

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  Full Model: 'obj_cmp.obj' wrt 'px.x'
    Forward Magnitude : 2.980614e+00
         Fd Magnitude : 2.980614e+00 (fd:forward)
    Absolute Error (Jfor  - Jfd) : 3.423605e-08

    Relative Error (Jfor  - Jfd) : 1.148624e-08

    Raw Forward Derivative (Jfor)
[[ 2.98061391]]

    Raw FD Derivative (Jfd)
[[ 2.98061395]]

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  Full Model: 'obj_cmp.obj' wrt 'pz.z'
    Forward Magnitude : 9.774287e+00
         Fd Magnitude : 9.774284e+00 (fd:forward)
    Absolute Error (Jfor  - Jfd) : 3.416581e-06 *

    Relative Error (Jfor  - Jfd) : 3.495480e-07

    Raw Forward Derivative (Jfor)
[[ 9.61001056  1.78448534]]

    Raw FD Derivative (Jfd)
[[ 9.61000714  1.78448537]]

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Turn off standard output and just view the derivatives in the return:

from openmdao.api import Problem, NonlinearBlockGS
from openmdao.test_suite.components.sellar import SellarDerivatives

prob = Problem()
prob.model = SellarDerivatives()
prob.model.nonlinear_solver = NonlinearBlockGS()

prob.model.add_design_var('x', lower=-100, upper=100)
prob.model.add_design_var('z', lower=-100, upper=100)
prob.model.add_objective('obj')
prob.model.add_constraint('con1', upper=0.0)
prob.model.add_constraint('con2', upper=0.0)

prob.setup()

# We don't call run_driver() here because we don't
# actually want the optimizer to run
prob.run_model()
NL: NLBGS Converged in 7 iterations
# check derivatives of all obj+constraints w.r.t all design variables
totals = prob.check_totals(out_stream=None)
NL: NLBGS Converged in 3 iterations
NL: NLBGS Converged in 4 iterations
NL: NLBGS Converged in 3 iterations
print(totals)
{('obj_cmp.obj', 'px.x'): {'J_fwd': array([[ 2.98061391]]), 'J_fd': array([[ 2.98049317]]), 'abs error': ErrorTuple(forward=0.00012074793172045517, reverse=None, forward_reverse=None), 'magnitude': MagnitudeTuple(forward=2.9806139134843366, reverse=None, fd=2.9804931655526161), 'rel error': ErrorTuple(forward=4.0512735649258626e-05, reverse=nan, forward_reverse=nan)}, ('obj_cmp.obj', 'pz.z'): {'J_fwd': array([[ 9.61001056,  1.78448534]]), 'J_fd': array([[ 9.60987922,  1.78444024]]), 'abs error': ErrorTuple(forward=0.00013886228655094472, reverse=None, forward_reverse=None), 'magnitude': MagnitudeTuple(forward=9.7742872281589364, reverse=None, fd=9.7741498668707187), 'rel error': ErrorTuple(forward=1.4207096109874026e-05, reverse=nan, forward_reverse=nan)}, ('con_cmp1.con1', 'px.x'): {'J_fwd': array([[-0.98061448]]), 'J_fd': array([[-0.98049272]]), 'abs error': ErrorTuple(forward=0.00012175295197092151, reverse=None, forward_reverse=None), 'magnitude': MagnitudeTuple(forward=0.98061447519504152, reverse=None, fd=0.9804927222430706), 'rel error': ErrorTuple(forward=0.000124175273522059, reverse=nan, forward_reverse=nan)}, ('con_cmp1.con1', 'pz.z'): {'J_fwd': array([[-9.61002186, -0.78449158]]), 'J_fd': array([[-9.60989052, -0.78444647]]), 'abs error': ErrorTuple(forward=0.00013886958946260141, reverse=None, forward_reverse=None), 'magnitude': MagnitudeTuple(forward=9.6419887538645792, reverse=None, fd=9.6418541793901085), 'rel error': ErrorTuple(forward=1.4402788807928803e-05, reverse=nan, forward_reverse=nan)}, ('con_cmp2.con2', 'px.x'): {'J_fwd': array([[ 0.09692762]]), 'J_fd': array([[ 0.09691559]]), 'abs error': ErrorTuple(forward=1.2034379274938889e-05, reverse=None, forward_reverse=None), 'magnitude': MagnitudeTuple(forward=0.09692762402477989, reverse=None, fd=0.096915589645504951), 'rel error': ErrorTuple(forward=0.00012417382300368697, reverse=nan, forward_reverse=nan)}, ('con_cmp2.con2', 'pz.z'): {'J_fwd': array([[ 1.94989072,  1.0775421 ]]), 'J_fd': array([[ 1.94987764,  1.07753764]]), 'abs error': ErrorTuple(forward=1.3810868113633928e-05, reverse=None, forward_reverse=None), 'magnitude': MagnitudeTuple(forward=2.2278174920224654, reverse=None, fd=2.22780389487618), 'rel error': ErrorTuple(forward=6.1993194936942718e-06, reverse=nan, forward_reverse=nan)}}