singular covariance matrices give traceback error for dream/amoeba/newton

[pkienzle]

Error message:

Error(s) Occurred:

Traceback (most recent call last):

File "/home/tec/sci/sasview-3.1.2/src/sas/perspectives/fitting/fit_thread.py", line 82, in compute

result = map(map_apply, inputs)

File "/home/tec/sci/sasview-3.1.2/src/sas/perspectives/fitting/fit_thread.py", line 14, in map_apply

return apply(arguments[0], arguments[1:])

File "/home/tec/sci/sasview-3.1.2/src/sas/perspectives/fitting/fit_thread.py", line 11, in map_getattr

return getattr(classInstance, classFunc)(*args)

File "/home/tec/sci/sasview-3.1.2/src/sas/fit/BumpsFitting.py", line 274, in fit

result = run_bumps(problem, handler, curr_thread)

File "/home/tec/sci/sasview-3.1.2/src/sas/fit/BumpsFitting.py", line 360, in run_bumps

'stderr': fitdriver.stderr() if success else None,

File "/usr/local/lib/python2.7/dist-packages/bumps/fitters.py", line 884, in stderr

H = lsqerror.hessian(self.problem, self.result[0])

File "/usr/local/lib/python2.7/dist-packages/bumps/lsqerror.py", line 79, in hessian

H = nd.Hessian(problem.nllf)(p)

File "/usr/local/lib/python2.7/dist-packages/bumps/numdifftools.py", line 1159, in call

return self.hessian(x00)

File "/usr/local/lib/python2.7/dist-packages/bumps/numdifftools.py", line 1170, in hessian

hess = self.hessdiag(x0)

File "/usr/local/lib/python2.7/dist-packages/bumps/numdifftools.py", line 1084, in hessdiag

return self._partial_der(x00)

File "/usr/local/lib/python2.7/dist-packages/bumps/numdifftools.py", line 720, in _partial_der

PD[ind] = self._derivative(fun, x0[ind], stepNom[ind])

File "/usr/local/lib/python2.7/dist-packages/bumps/numdifftools.py", line 424, in _derivative

der_romb, errors, h2 = self._romb_extrap(der_init, h1)

File "/usr/local/lib/python2.7/dist-packages/bumps/numdifftools.py", line 686, in _romb_extrap

errest = np.ones(der_init.shape) * hout

ValueError: operands could not be broadcast together with shapes (24,) (0,)

Steps: 266 of 1000 chisq: 43.6 ETA: 2s
M1.num_pearls: 1.29 | M1.radius: 15.3 | M1.scale: 0.00847

M1.sld_pearl: 1.5e-06 |

[pkienzle]

This error comes about when the covariance matrix is singular. This can happen for several reasons:

(1) correlated parameters. Running DREAM, I see that sld, scale and num_pearls are all correlated in this model.

(2) integer parameters. num_pearls rounds to the nearest integer in the code, which means the partial derivative wrt num_pearls will be zero.

(3) fit bounds. Running DREAM I see that radius is bounded by the top end of the range. When taking the partial derivative at the boundary, the parameter value is clipped to the boundary value, so the partial derivative wrt radius is zero. Freeing radius and letting it rise to its natural value of 100 removes the singularity.

The upstream author of the numerical derivatives package we are using has fixed the problem with singular matrices in the latest release, but this is not yet made its way into sasview. It will still give meaningless parameter uncertainties (probably inf or nan) but the error message will be gone.

[pkienzle]

failing fit

[butler]

As agreed on fortnightly call of June 21, am moving all minor priority tickets to next release (+1) given the resources and time available to get this release out.

[butler]

This may now be fixed as er P. K's comment? Needs testing. Also could provide a error trapping routine that warns user that the fit failed to converge properly and not to believe any of the results. Instead try again with different starting parameters and/or change constraints