Opened 3 years ago

Last modified 20 months ago

#495 new defect

singular covariance matrices give traceback error for dream/amoeba/newton

Reported by: pkienzle Owned by:
Priority: minor Milestone: SasView 4.3.0
Component: SasView Keywords:
Cc: Work Package: SasView Bug Fixing

Description

Error message:

Error(s) Occurred:

Traceback (most recent call last):

File "/home/tec/sci/sasview-3.1.2/src/sas/perspectives/fitting/fit_thread.py", line 82, in compute

result = map(map_apply, inputs)

File "/home/tec/sci/sasview-3.1.2/src/sas/perspectives/fitting/fit_thread.py", line 14, in map_apply

return apply(arguments[0], arguments[1:])

File "/home/tec/sci/sasview-3.1.2/src/sas/perspectives/fitting/fit_thread.py", line 11, in map_getattr

return getattr(classInstance, classFunc)(*args)

File "/home/tec/sci/sasview-3.1.2/src/sas/fit/BumpsFitting.py", line 274, in fit

result = run_bumps(problem, handler, curr_thread)

File "/home/tec/sci/sasview-3.1.2/src/sas/fit/BumpsFitting.py", line 360, in run_bumps

'stderr': fitdriver.stderr() if success else None,

File "/usr/local/lib/python2.7/dist-packages/bumps/fitters.py", line 884, in stderr

H = lsqerror.hessian(self.problem, self.result[0])

File "/usr/local/lib/python2.7/dist-packages/bumps/lsqerror.py", line 79, in hessian

H = nd.Hessian(problem.nllf)(p)

File "/usr/local/lib/python2.7/dist-packages/bumps/numdifftools.py", line 1159, in call

return self.hessian(x00)

File "/usr/local/lib/python2.7/dist-packages/bumps/numdifftools.py", line 1170, in hessian

hess = self.hessdiag(x0)

File "/usr/local/lib/python2.7/dist-packages/bumps/numdifftools.py", line 1084, in hessdiag

return self._partial_der(x00)

File "/usr/local/lib/python2.7/dist-packages/bumps/numdifftools.py", line 720, in _partial_der

PD[ind] = self._derivative(fun, x0[ind], stepNom[ind])

File "/usr/local/lib/python2.7/dist-packages/bumps/numdifftools.py", line 424, in _derivative

der_romb, errors, h2 = self._romb_extrap(der_init, h1)

File "/usr/local/lib/python2.7/dist-packages/bumps/numdifftools.py", line 686, in _romb_extrap

errest = np.ones(der_init.shape) * hout

ValueError: operands could not be broadcast together with shapes (24,) (0,)


Steps: 266 of 1000 chisq: 43.6 ETA: 2s
M1.num_pearls: 1.29 | M1.radius: 15.3 | M1.scale: 0.00847

M1.sld_pearl: 1.5e-06 |

Attachments (1)

FitPage1.fitv (37.6 KB) - added by pkienzle 3 years ago.
failing fit

Download all attachments as: .zip

Change History (4)

comment:1 Changed 3 years ago by pkienzle

  • Priority changed from major to minor

This error comes about when the covariance matrix is singular. This can happen for several reasons:

(1) correlated parameters. Running DREAM, I see that sld, scale and num_pearls are all correlated in this model.

(2) integer parameters. num_pearls rounds to the nearest integer in the code, which means the partial derivative wrt num_pearls will be zero.

(3) fit bounds. Running DREAM I see that radius is bounded by the top end of the range. When taking the partial derivative at the boundary, the parameter value is clipped to the boundary value, so the partial derivative wrt radius is zero. Freeing radius and letting it rise to its natural value of 100 removes the singularity.

The upstream author of the numerical derivatives package we are using has fixed the problem with singular matrices in the latest release, but this is not yet made its way into sasview. It will still give meaningless parameter uncertainties (probably inf or nan) but the error message will be gone.

Changed 3 years ago by pkienzle

failing fit

comment:2 Changed 3 years ago by butler

  • Milestone changed from SasView 4.0.0 to SasView Next Release +1

As agreed on fortnightly call of June 21, am moving all minor priority tickets to next release (+1) given the resources and time available to get this release out.

comment:3 Changed 20 months ago by butler

  • Milestone changed from SasView Next Release +1 to SasView 4.3.0

This may now be fixed as er P. K's comment? Needs testing. Also could provide a error trapping routine that warns user that the fit failed to converge properly and not to believe any of the results. Instead try again with different starting parameters and/or change constraints

Note: See TracTickets for help on using tickets.