bryson/pacman
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

meson: handle XFAIL tests outside of TAP

Browse source 
This change causes expected fail tests to actually fail by eliding the
'# TODO' from the test plan. In turn, we can now properly use
'should_fail' in the meson test() rule and see these expected fail
tests in the output:

Before:
  ...
  320/332 upgrade077.py                   OK 0.12679290771484375 s
  321/332 upgrade078.py                   OK 0.12620115280151367 s
  322/332 upgrade080.py                   OK 0.1252129077911377 s
  ...

  Ok:                 332
  Expected Fail:      0
  Fail:               0
  Unexpected Pass:    0
  Skipped:            0
  Timeout:            0

After:
  ...
  320/332 upgrade077.py                   OK 0.12679290771484375 s
  321/332 upgrade078.py                   EXPECTEDFAIL0.12620115280151367 s
  322/332 upgrade080.py                   OK 0.1252129077911377 s
  ...

  Ok:                 326
  Expected Fail:      6
  Fail:               0
  Unexpected Pass:    0
  Skipped:            0
  Timeout:            0

Signed-off-by: Allan McRae <allan@archlinux.org>
This commit is contained in:
Dave Reisner 2020-05-02 12:12:32 -04:00 • committed by Allan McRae
parent 0b81fb64d3
commit cebda6d0ee
2 changed files with 20 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

13
test/pacman/meson.build
View file

@ -330,6 +330,17 @@ pacman_tests = [
'tests/xfercommand001.py',
]
xfail_tests = {
'tests/deprange001.py': true,
# expect failure on 32 bit machines
'tests/query006.py': cc.sizeof('ssize_t') < 8,
'tests/replace110.py': true,
'tests/sync-update-package-removing-required-provides.py': true,
'tests/sync403.py': true,
'tests/sync406.py': true,
'tests/upgrade078.py': true,
}
foreach input : pacman_tests
test_name = input.split('/')[1]
args = [
@ -351,7 +362,9 @@ foreach input : pacman_tests
test_name,
PYTHON,
protocol : 'tap',
env : ['RUNNING_UNDER_MESON=1'],
args : args,
timeout : 120,
should_fail : xfail_tests.get(input, false),
depends : [pacman_bin])
endforeach

7
test/pacman/pmenv.py
View file

@ -70,5 +70,10 @@ class pmenv(object):
t.run(self.pacman)
tap.diag("==> Checking rules")
tap.todo = t.expectfailure
# When running under meson, we don't emit 'todo' in the plan and instead
# handle expected failures in the test() objects. This really should be
# fixed in meson:
# https://github.com/mesonbuild/meson/issues/2923#issuecomment-614647076
tap.todo = (t.expectfailure and
not 'RUNNING_UNDER_MESON' in os.environ)
tap.subtest(lambda: t.check(), t.description)