id summary reporter owner description type status priority milestone component resolution keywords cc workpackage 196 Be able to read semi-colon delimited files ajj "As per email discussion: On 18 nov 2013, at 10:25, stephen.king@stfc.ac.uk wrote: If it’s any consolation, Mantid won’t read Greg’s CSV files either; I just tried. It’s almost certainly a more complicated version of the ‘semicolon’ problem Andrew refers to (I’ve been there with Users on that too!). It’s all to do with the international settings in Windows. (For the benefit of those in the US, the continental Europeans – particularly the Germans - tend to use semicolons in their CSV files rather than commas). In Greg’s case he’s got thousands separators for number formatting ‘turned on’, which means when the program that wrote those files came to write them it needed a way to distinguish between the comma field separator and the comma thousands separator, so it encased the fields with comma thousands separators in speech quotes. Kind of understandable, but a right pain nonetheless. But I don’t see this as a ‘failure’ or ‘inadequacy’ of SasView. The User can easily rectify the situation themselves. Have to say that in 20-odd years this is the first time I’ve seen a User producing datafiles like Greg’s… As an aside, will the SasView ASCII reader read a CSV file that is semi-colon delimted??? That we SHOULD cater for. Steve From: Andrew Jackson [mailto:andrew.jackson@esss.se] Sent: 06 November 2013 13:16 To: Paul Butler Cc: sasview-developers@lists.sourceforge.net Subject: Re: [Sasview-developers] csv files - reader? Well, CSV is not really a strict format - I've been dealing recently with the fact that excel will sometimes export ""csv"" with the fields separated by semi-colons rather than commas. Unfortunately, the real answer is that ""Excel can read it"" should really be our basis. What it means, however, is that if a file cannot be auto loaded, we would have to have a dialog that allowed some choice about which lines to skip, what separator to use, how to handle number etc, much in the way excel does. In the immediate case, the user will have to go back to the data source and re-export with sensible output. Andrew On 6 nov 2013, at 14:01, Paul Butler wrote: Merci!! yes I should have remembered that csv files would be parsed by the generic ASCII reader. This can only be the most basic of ASCII parsers since to be general the reader has to apply some logic to figure out where the data is. As I recall it looks for the first instance with more than 5 or more rows containing X columns of numbers only AND where X is greater than 2 AND where X is exactly the same for all rows read. It then ignores everything before and after that block as being header and footer information of unknown formats. This would also explain why sasview did not read the first oscillation in the data where 2 points dropped below 1000 (and therefore did not have quotes). In principle I agree with Miguel that adding a special "".cvs"" reader seems like unneeded complexity (at a time when we are trying to simplify things:-). Also adding a check for quotes in the current reader would be adding complexity (specially since there are several permutations one might now have to consider). Which brings me to Andrew's point. I don't know csv rules, but from what I can find quotes are not used for strings or numbers unless it includes a comma that is not a delimeter. It is not clear to me whether a quote around a number with a comma is part of the strict definition of csv or not. However I note that excel does read it correctly. That said, it seems from what I'm hearing that we think that it is not necessary to support that variant? Certainly if the stric csv definition does not support this or if it is effectively not used this way, ignoring it is the right answer? Cheers, Paul On Wed, Nov 6, 2013 at 3:57 AM, gonzalez wrote: Salut Paul, you are completely right. All the lines having quotation marks are skipped. The csv files are read by the ASCII reader (ascii_reader.py), which basically reads a line, split its components and starts trying to convert to floating numbers the first two tokens, assumed to correspond to Q and Intensity, respectively. When it fails to do this, the line is simply omitted and this is what happens with all the lines having quotation marks. I guess that trying to treat this particular case could cause more problems than benefits, as we may end up with a lot of different possibilities in the data file that will be difficult to handle, so I would suggest that the easiest way forward is to edit or export the files avoiding the quotation marks and any separator in the numbers other than the decimal point. Cheers, Miguel. On 06/11/2013 04:07, Paul Butler wrote: Has anybody got experience with csv files? as I mentioned in my response to Greg, the problem he is having is sasview does not recognize quotation mark enclosed numbers. Does anybody know how we read csv files? are we using a csv reading package or are we trying to parse the file ourselves? Jeff - you have been working on the reader .. but focusing on cansas I know. while I cannot see the data ""as read"" with the datainfo (still broken) using the plot x,y printout in the info bar it would seem that the intensity is a bit lower than in the file (at least on the data sets sent) -- anybody have any thoughts on that? ------------------------------------------------------------------------------ November Webinars for C, C++, Fortran Developers Accelerate application performance with scalable programming models. Explore techniques for threading, error checking, porting, and tuning. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk _______________________________________________ Sasview-developers mailing list Sasview-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sasview-developers -- Dr. Miguel A. Gonzalez Institut Laue Langevin (ILL) 6, rue Jules Horowitz, BP 156 38042 Grenoble Cedex 9, FRANCE Tel: +33 (0)4.76.20.71.66 Fax: +33 (0)4.76.48.39.06 E-mail: gonzalez@ill.fr ------------------------------------------------------------------------------ November Webinars for C, C++, Fortran Developers Accelerate application performance with scalable programming models. Explore techniques for threading, error checking, porting, and tuning. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk _______________________________________________ Sasview-developers mailing list Sasview-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sasview-developers ------------------------------------------------------------------------------ November Webinars for C, C++, Fortran Developers Accelerate application performance with scalable programming models. Explore techniques for threading, error checking, porting, and tuning. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk_______________________________________________ Sasview-developers mailing list Sasview-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sasview-developers ____________________________________________________________________________ Andrew Jackson Instrument Scientist - Small Angle Scattering Adjunct Lecturer - Physical Chemistry European Spallation Source ESS AB Lund University P.O Box 176, SE-221 00 Lund, Sweden P.O. Box 124, SE-221 00 Lund, Sweden Street address: Tunavägen 24 (Medicon Village) Street address: Getingevägen 60 Phone: +46 46 888 3015 Mobile: +46 72 179 2015 E-mail: andrew.jackson@esss.se www.esss.se -- Scanned by iCritical. ____________________________________________________________________________ Andrew Jackson Instrument Scientist - Small Angle Scattering Adjunct Lecturer - Physical Chemistry European Spallation Source ESS AB Lund University P.O Box 176, SE-221 00 Lund, Sweden P.O. Box 124, SE-221 00 Lund, Sweden Street address: Tunavägen 24 (Medicon Village) Street address: Getingevägen 60 Phone: +46 46 888 3015 Mobile: +46 72 179 2015 E-mail: andrew.jackson@esss.se www.esss.se " defect closed minor SasView 3.0.0 SasView fixed