震惊！allure统计结果居然与pytest结果不一致

背景

基于 pytest 框架写了两组 suites，一个用于测试成功的 test_success.py , 一个用于测试失败的 test_fail.py

test_success.py

import pytest
import warnings


def test_case1():
    """test pass"""
    assert 1


def test_case2():
    """test warning"""
    warnings.warn(UserWarning("This is a warning msg"))


@pytest.mark.skip()
def test_case3():
    """test skip"""
    assert 0

test_fail.py

import pytest


def test_case1():
    """test fail"""
    assert 0


def test_case2():
    """こんにちは世界"""
    raise IOError(123)


@pytest.fixture()
def user():
    a = "yoyo"
    assert a == "yoyo123"  # fixture failed with error
    return a


def test_case3(user):
    """test error"""
    assert user == "yoyo"

结果

用命令 pytest.main([‘./testsuites’, ‘-s’, ‘-q’, ‘–alluredir’, ‘./report’]) 执行，我们可以看到 pytest 的控制台输出结果

FFE..s
...
...
...
  /Workspace/DEV/study/pywebreport/test/testsuites/test_success.py:16: UserWarning: This is a warning msg
    warnings.warn(UserWarning("This is a warning msg"))

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED testsuites/test_fail.py::test_case1 - assert 0
FAILED testsuites/test_fail.py::test_case2 - OSError: 123
ERROR testsuites/test_fail.py::test_case3 - AssertionError: assert 'yoyo' == ...
2 failed, 2 passed, 1 skipped, 1 warning, 1 error in 0.09s

Process finished with exit code 0

从结果可以看到 test_fail.py 中的 test_case2 标记为 F(Failed)， test_case3 标记为 E(Error)>

再来看看 allure 报告这边

从结果可以看到 test_fail.py 中的 test_case2 标记为 Broken，test_case3 标记为 Failed

总结

allure 的结果统计在 failed 上跟 pytest 是不一样的，pytest 里面的 error 在 allure 上是 failed，而 pytest failed 的一种情况，在 allure 上被视为 broken