Loading a saved project is really really slow

[ajjackson]

Loading a saved project can take minutes if there are even a few datasets. This is observed on recent macs (core i7) running OS X 10.7

[butlerpd]

As of May 25 (presumably version 2.1.1) Andrew says:

File with 8 datasets and associated models in separate fit panels takes 40s to load on my 1.8 Ghz i7 mac. That seems a little long to me

So indeed still too slow but probably not worse than same on PC? should test that. This is probably because we chose to save just enough to recreate the state not the actual state? Thus we would need to recompute all the fits? also may be due to length of time to put up a plot from matplotlib?

If so not sure if this can be done for 2.2 — may have to be a 3.0 item with a refactoring unless we get some more help.

[butlerpd]

AS expected we do not have the manpower to refactor the saving function for this so moving to 3.3 milestone. Will revisit status after release 2.2 is done

[butler]

Actually we don't have a version 3.3 currently open so moving to wishlist to be evaluated later.

[pkienzle]

Putting a trace statement in sasmodels calculator, it appears that the model is recalculated and replotted many times in the process of loading. A single cylinder model with one data set gets replotted six times in the process of loading the project or analysis.

[ajj]

See attached file which contains 3 data sets and 3 fit panels.

Time to load in 4.0 release: 24 seconds on Macbook Pro 13" retina - 3.1 GhZ Core i7 dual core.

(For the record, at least this is faster than 3.1.2: 1 minute 42 seconds on the same machine and once loaded models were not correctly calculated and UI was broken on polydispersity. )

[krzywon]

I think I have a solution for this, but wanted to run it by everyone before I move forward on it. For saving 2D data projects (#827), I put all of the data into a single <Idata> element with the text of each child element (Qx, Qy, I, etc) as a comma delimited string of data points. This resulted in significantly faster load times, taking minutes to load in the 'traditional' format with each data point a separate <Idata> element, to a matter of seconds in this format. I could fairly quickly and easily do this for 1D data as well, while maintaining forward and backward compatibility with all other save states as well as CanSAS XML data. With the target date of 4.1 pushed back, it might even be ready by that time.

Example of what I would like to do:

<SASdata>
	<Idata>
		<Q unit="1/A">0.003797, 0.004197, ...</Q>
		<I unit="1/cm">4006.05896074, 1594.12921432, ...</I>
		<Idev unit="1/cm">160.350516888, 144.193913581, ...</Idev>
		<Qdev unit="1/A">0.00109, 0.001093, ...</Qdev>
	</Idata>
</SASdata>

Instead of:

<SASdata>
	<Idata>
		<Q unit="1/A">0.003797</Q>
		<I unit="1/cm">4006.05896074</I>
		<Idev unit="1/cm">160.350516888</Idev>
		<Qdev unit="1/A">0.00109</Qdev>
	</Idata>
	<Idata>
		<Q unit="1/A">0.004197</Q>
		<I unit="1/cm">1594.12921432</I>
		<Idev unit="1/cm">144.193913581</Idev>
		<Qdev unit="1/A">0.001093</Qdev>
	</Idata>

	...

</SASdata>

[pkienzle]

See ticket #869 for a possible solution.

[krzywon]

The computation cleanup suggested in #869 was in PR109. Loading a project is much faster thanks to it.