Changeset 7bd6860a in sasview for src/sas/sascalc/dataloader

Timestamp:

Aug 18, 2016 10:19:47 AM (8 years ago)

Author:

lewis

Branches:

master, ESS_GUI, ESS_GUI_Docs, ESS_GUI_batch_fitting, ESS_GUI_bumps_abstraction, ESS_GUI_iss1116, ESS_GUI_iss879, ESS_GUI_iss959, ESS_GUI_opencl, ESS_GUI_ordering, ESS_GUI_sync_sascalc, costrafo411, magnetic_scatt, release-4.1.1, release-4.1.2, release-4.2.2, ticket-1009, ticket-1094-headless, ticket-1242-2d-resolution, ticket-1243, ticket-1249, ticket885, unittest-saveload

Children:

e9e777c

Parents:

afbd172

Message:

Improve efficiency of CanSAS 2.0 reader

When it came to reading the Q and I data values, the reader was looping
through every single data point in the hdf5 file and appending it to an
array. This caused problems when loading very large files, rendering
SasView? unresponsive. Now, the reader saves the entire array from the
file, instead of looping through each point.

File:

• src/sas/sascalc/dataloader/readers/cansas_reader_HDF5.py (modified) (3 diffs)

src/sas/sascalc/dataloader/readers/cansas_reader_HDF5.py

@@ -62 +62 @@
         :return: List of Data1D/2D objects and/or a list of errors.
         """
-
         ## Reinitialize the class when loading a new data file to reset all class variables
         self.reset_class_variables()
@@ -136 +135 @@
                 ## If this is a dataset, store the data appropriately
                 data_set = data[key][:]
+                unit = self._get_unit(value)
+
+                ## I and Q Data
+                if key == u'I':
+                    if type(self.current_dataset) is plottable_2D:
+                        self.current_dataset.data = data_set.flatten()
+                        self.current_dataset.zaxis("Intensity", unit)
+                    else:
+                        self.current_dataset.y = data_set.flatten()
+                        self.current_dataset.yaxis("Intensity", unit)
+                    continue
+                elif key == u'Idev':
+                    if type(self.current_dataset) is plottable_2D:
+                        self.current_dataset.err_data = data_set.flatten()
+                    else:
+                        self.current_dataset.dy = data_set.flatten()
+                    continue
+                elif key == u'Q':
+                    self.current_dataset.xaxis("Q", unit)
+                    if type(self.current_dataset) is plottable_2D:
+                        self.current_dataset.q = data_set.flatten()
+                    else:
+                        self.current_dataset.x = data_set.flatten()
+                    continue
+                elif key == u'Qy':
+                    self.current_dataset.yaxis("Q_y", unit)
+                    self.current_dataset.qy_data = data_set.flatten()
+                    continue
+                elif key == u'Qydev':
+                    self.current_dataset.dqy_data = data_set.flatten()
+                    continue
+                elif key == u'Qx':
+                    self.current_dataset.xaxis("Q_x", unit)
+                    self.current_dataset.qx_data = data_set.flatten()
+                    continue
+                elif key == u'Qxdev':
+                    self.current_dataset.dqx_data = data_set.flatten()
+                    continue
+                elif key == u'Mask':
+                    self.current_dataset.mask = data_set.flatten()
+                    continue
 
                 for data_point in data_set:
                     ## Top Level Meta Data
-                    unit = self._get_unit(value)
                     if key == u'definition':
                         self.current_datainfo.meta_data['reader'] = data_point
@@ -148 +187 @@
                     elif key == u'SASnote':
                         self.current_datainfo.notes.append(data_point)
-
-                    ## I and Q Data
-                    elif key == u'I':
-                        if type(self.current_dataset) is plottable_2D:
-                            self.current_dataset.data = np.append(self.current_dataset.data, data_point)
-                            self.current_dataset.zaxis("Intensity", unit)
-                        else:
-                            self.current_dataset.y = np.append(self.current_dataset.y, data_point)
-                            self.current_dataset.yaxis("Intensity", unit)
-                    elif key == u'Idev':
-                        if type(self.current_dataset) is plottable_2D:
-                            self.current_dataset.err_data = np.append(self.current_dataset.err_data, data_point)
-                        else:
-                            self.current_dataset.dy = np.append(self.current_dataset.dy, data_point)
-                    elif key == u'Q':
-                        self.current_dataset.xaxis("Q", unit)
-                        if type(self.current_dataset) is plottable_2D:
-                            self.current_dataset.q = np.append(self.current_dataset.q, data_point)
-                        else:
-                            self.current_dataset.x = np.append(self.current_dataset.x, data_point)
-                    elif key == u'Qy':
-                        self.current_dataset.yaxis("Q_y", unit)
-                        self.current_dataset.qy_data = np.append(self.current_dataset.qy_data, data_point)
-                    elif key == u'Qydev':
-                        self.current_dataset.dqy_data = np.append(self.current_dataset.dqy_data, data_point)
-                    elif key == u'Qx':
-                        self.current_dataset.xaxis("Q_x", unit)
-                        self.current_dataset.qx_data = np.append(self.current_dataset.qx_data, data_point)
-                    elif key == u'Qxdev':
-                        self.current_dataset.dqx_data = np.append(self.current_dataset.dqx_data, data_point)
-                    elif key == u'Mask':
-                        self.current_dataset.mask = np.append(self.current_dataset.mask, data_point)
 
                     ## Sample Information