opencl memory errors during fits

[pkienzle]

The following OpenCL memory error was reported from SasView. This may be because sasview_model.py (calculate_Iq) is allocating a new q vector each time the model is evaluated rather than reusing the same q vector across all fits. Restructuring the calculator so that the q values are held for the life of the model may fix the error.

2017-10-23 17:39:12 : Traceback (most recent call last):
 File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 82, in compute
 File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 14, in map_apply
 File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 11, in map_getattr
 File "sas/sascalc/fit/BumpsFitting.pyc", line 289, in fit
 File "sas/sascalc/fit/BumpsFitting.pyc", line 202, in theory
 File "sas/sascalc/fit/BumpsFitting.pyc", line 192, in _recalculate
 File "sas/sascalc/fit/AbstractFitEngine.pyc", line 242, in residuals
 File "/Users/lionel/.sasview/plugin_models/Cyl_CS_sphere.py", line 272, in evalDistribution
 File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/sasview_model.py", line 582, in evalDistribution
 File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/sasview_model.py", line 617, in calculate_Iq
 File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/sasview_model.py", line 638, in _calculate_Iq
 File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/kernelcl.py", line 567, in __call__
 File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/pyopencl/__init__.py", line 1088, in kernel_call
 File "<generated function enqueue_knl_cylinder_Iq>", line 169, in enqueue_knl_cylinder_Iq
LogicError: when processing argument #4 (1-based): clsetkernelarg failed: INVALID_MEM_OBJECT

=== Steps: 161 of 159  chisq: 70.4  ETA: -86s
  M2.p2_radius: 71.5       | M2.p2_sld_shell: -0.108     | M2.p2_thickness: 75.8      
   M1.p1_scale: 0.000499   | M1.p1_sld_solvent: 1.38       |    M1.p2_radius: 136       
M1.p2_radius.width: 0.851      |     M1.p2_scale: 0.0181     |  M1.p2_sld_core: 2.08      
M1.p2_thickness: 14.1       |    M4.p2_radius: 64.8       | M4.p2_sld_shell: -0.0451   
M4.p2_thickness: 79.6       |    M3.p2_radius: 69         | M3.p2_sld_shell: -0.199    
M3.p2_thickness: 83.7       | M3.p2_thickness.width: 0.794      |  
2017-10-23 17:42:59 : Traceback (most recent call last):
 File "sas/sascalc/data_util/calcthread.pyc", line 268, in _run
LogicError: when processing argument #4 (1-based): clsetkernelarg failed: INVALID_MEM_OBJECT

	2017-10-23 17:43:49 : ('   0.357', '   0.697')
	2017-10-23 17:43:50 : ('   0.357', '   0.697')
	2017-10-23 17:43:51 : ('   0.505', '      15')
	2017-10-23 17:43:51 : ('   0.505', '      15')
	2017-10-23 17:43:55 : ('  0.0847', '   0.268')
	2017-10-23 17:43:55 : ('  0.0847', '   0.268')
	2017-10-23 17:45:05 : Computation  completed!
	2017-10-23 17:45:07 : Computation  completed!
	2017-10-23 17:45:20 : Computation  completed!
	2017-10-23 17:45:21 : Computation  completed!
	2017-10-23 17:45:22 : Computation  completed!
	2017-10-23 17:45:25 : Computation  completed!
	2017-10-23 17:45:26 : Computation  completed!
	2017-10-23 17:45:27 : Computation  completed!
	2017-10-23 17:45:29 : Computation  completed!
	2017-10-23 17:45:40 : Computation  completed!
	2017-10-23 17:45:45 : Computation  completed!
	2017-10-23 17:46:15 : Computation  completed!
	2017-10-23 17:48:42 : Fitting is in progress...
	2017-10-23 17:48:42 : Fitting is in progress...
	2017-10-23 17:48:42 : Traceback (most recent call last):
 File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 82, in compute
 File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 14, in map_apply
 File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 11, in map_getattr
 File "sas/sascalc/fit/BumpsFitting.pyc", line 275, in fit
 File "sas/sascalc/fit/BumpsFitting.pyc", line 375, in run_bumps
AttributeError: 'FitDriver' object has no attribute 'fitter'

None 

[pkienzle]

The resulting error occurs also when a model is displayed on the fit page then you test a new OpenCL device:

LogicError: when processing argument #4 (1-based): clsetkernelarg failed: INVALID_MEM_OBJECT

In this case, I believe it is because the program was compiled in a different OpenCL context than the one used to allocate the q vectors. We need to keep all references to the buffers and kernels in the context object and retrieve them from when running. That way when the context is reset we can force a new set of buffers to be allocated and recompile the kernels.

Note: may want to store the compiled model as a class variable in the SasviewModel? wrapper so that it doesn't get recompiled each time you select a model on the fit page. At the top of SasviewModel._calculate_Iq use self.__class__._model = core.build_model(self._model_info). The model info attribute is already stored at the class level in _generate_model_attributes, but the build model step was being done lazily.

[pkienzle]

The work-around is to tell the user to reselect the model.

[wojciech]

I also experience this when switching between models and categories (cylinder and ellipsoids). I've been stress testing the GPU Options interface for 5.0.

[pkienzle]

Replying to wojciech:

I also experience this when switching between models and categories (cylinder and ellipsoids). I've been stress testing the GPU Options interface for 5.0.

Are you working from sasmodels/master or a tagged revision?

There was some additional cleanup that didn't get merged until the beta_approx branch was merged, though it's been too long for me to remember the details.