utest_sasview.py giving different results than run_one.py

[krzywon]

There seems to be an issue with unit testing. utest_sasview.py is giving 100% success on all unit tests, despite tests failing. Using run_one.py on the individual test files seems to be the only sure-fire way a developer can tell if the tests are successful. The build server caught the failing tests, despite showing 100% success in the log files.

Builds showing the issue:
​https://jenkins.esss.dk/sasview/job/SasView_Win7/718/
​https://jenkins.esss.dk/sasview/job/SasView_OSX10.10/598/

Ubuntu build showing success:
​https://jenkins.esss.dk/sasview/job/SasView_Ubuntu14.10/729/

Tests that failed:
​https://jenkins.esss.dk/sasview/job/SasView_Win7/718/testReport/

Excerpt from the console log showing successful utest_sasview.py results:
​https://jenkins.esss.dk/sasview/job/SasView_Win7/718/consoleFull

C:\jenkins\workspace\SasView_Win7\sasview\test>c:\python27\python utest_sasview.py
Result for utest_corfunc: SUCCESS
Result for utest_nxcansas_writer: SUCCESS
Result for utest_explorer: SUCCESS
Result for utest_invertor: SUCCESS
Result for utest_resolution_calculator: SUCCESS
Result for utest_sas_gen: SUCCESS
Result for utest_slit_length_calculator: SUCCESS
Result for utest_abs_reader: SUCCESS
Result for utest_ascii: SUCCESS
Result for utest_averaging: SUCCESS
Result for utest_cansas: SUCCESS
Result for utest_extension_registry: SUCCESS
Result for utest_generic_file_reader_class: SUCCESS
Result for utest_red2d_reader: SUCCESS
Result for utest_sesans: SUCCESS
Result for utest_manipulations: SUCCESS
Result for utest_data_handling: SUCCESS
Result for utest_use_cases: SUCCESS

----------------------------------------------
Results by test modules:
    PASSED: 18
    FAILED: 0    (0%)
Results by tests:
    Tests run:    161
    Tests failed: 0
    Test errors:  0
----------------------------------------------

[krzywon]

After further investigation, the unit tests that failed on the build all had Exceptions thrown by assertion test inputs so the tests were not actually run, thus no actual test failures. We will either have to manage uncaught exceptions through utest_sasview.py or require all unit tests have a try/except block that gives a unit test failure.

I am unsure how to do the first one without rewriting the entirety of utest_sasview and make use of the run_one code, but would be the cleaner way to do this. Using a try/except block on the unit tests would be as simple as the code below, but would require all future unit tests to have this same structure, so there is more potential for failure.

try
  // For expected exceptions, use self.assertRaises() here.
  <unit tests>
except Exception as inst:
  // For all unexpected exceptions.
  self.fail(inst.message)

[pkienzle]

The problem is with the parsing of the output:

FAILED (failures=1, errors=6)

The re is only testing for failures or errors alone, not both.

[krzywon]

The fix suggested above was applied to pull request 99 and merged into master yesterday. This ticket can be closed.