mirror of
https://gitlab.kitware.com/cmake/cmake.git
synced 2026-09-25 04:09:36 +03:00
Taylor Braun-JonesandTyler Yankee
5cced02b9d
When ctest repeats tests with its --repeat option, it repeats each test on its own. A fixture's setup and cleanup tests therefore run all of their repetitions back to back, and the tests they bracket repeat inside a single setup/cleanup pair: setup -> setup -> test -> test -> cleanup -> cleanup Add a FIXTURE_REPEAT_MODE test property to select how a fixture behaves when its tests are repeated: * AROUND_ALL_REPEATS: the fixture runs once, around all repetitions of the tests requiring it. * AROUND_EACH_REPEAT: the fixture and the tests requiring it repeat together, so every repetition gets a fresh setup and its own cleanup. * EACH_TEST_SEPARATELY: every test repeats on its own, as before. The property describes the fixture rather than the test carrying it, so setting it on any one of a fixture's setup or cleanup tests is enough. In AROUND_EACH_REPEAT mode the tests of a fixture form a repeat group that ctest re-queues as a whole once every test in it has finished. The --repeat condition then applies to the group the way it applies to an individual test: until-fail repeats while the whole group passes, until-pass repeats while any of it does not, and after-timeout repeats while any of it times out. Fixtures that share a test repeat together, so a test requiring two of them still runs once per repetition. A group is recorded the way a repeating test is: only once it stops repeating, and with the results of its last repetition. A group that until-pass makes pass therefore reports a pass rather than the failure that made it repeat, a test that DEPENDS on one of the group's tests waits for the last repetition rather than the first, and `ctest -F` resumes an interrupted group by running it again from the beginning. Fixtures that repeat together have to agree on the mode: a test cannot repeat with one fixture but not with another it takes part in, and a fixture whose setup and cleanup tests disagree has no coherent behavior. Report an error and run nothing in those cases rather than pick an order in which a test repeats after a fixture it requires has been cleaned up. Add policy CMP0224 to select AROUND_EACH_REPEAT as the default for fixtures whose setup and cleanup tests choose no mode themselves. Record the mode the policy chose in the generated test file under its own _CMAKE_DEFAULT_FIXTURE_REPEAT_MODE keyword, so that ctest reads a mode rather than the policy settings behind it, and so that a mode requested on one of a fixture's tests wins over the default recorded for its siblings. Only NEW needs recording: with nothing recorded, ctest already uses the behavior of CMake 4.4 and below. Fixtures are common, and the choice of mode matters only to those who run ctest --repeat, so warn about the unset policy only when the CMAKE_POLICY_WARNING_CMP0224 variable asks for it. discover_tests() and gtest_discover_tests() create their tests while ctest runs or at build time, too late for the policy to reach them, so carry the setting in effect at their call sites through to the tests they create. Report the repetition a grouped test belongs to in the "(run N/M)" suffix of its "Start" line, as ctest already does for a test repeating on its own. Co-authored-by: Tyler Yankee <tyler.yankee@kitware.com> Fixes: #21438
56 lines
2.0 KiB
ReStructuredText
56 lines
2.0 KiB
ReStructuredText
FIXTURES_CLEANUP
|
|
----------------
|
|
|
|
.. versionadded:: 3.7
|
|
|
|
Specifies a list of fixtures for which the test is to be treated as a cleanup
|
|
test. These fixture names are distinct from test case names and are not
|
|
required to have any similarity to the names of tests associated with them.
|
|
|
|
Fixture cleanup tests are ordinary tests with all of the usual test
|
|
functionality. Setting the ``FIXTURES_CLEANUP`` property for a test has two
|
|
primary effects:
|
|
|
|
- CTest will ensure the test executes after all other tests which list any of
|
|
the fixtures in its :prop_test:`FIXTURES_REQUIRED` property.
|
|
|
|
- If CTest is asked to run only a subset of tests (e.g. using regular
|
|
expressions or the ``--rerun-failed`` option) and the cleanup test is not in
|
|
the set of tests to run, it will automatically be added if any tests in the
|
|
set require any fixture listed in ``FIXTURES_CLEANUP``.
|
|
|
|
A cleanup test can have multiple fixtures listed in its ``FIXTURES_CLEANUP``
|
|
property. It will execute once for all of them, not once for each fixture. A
|
|
fixture can also have more than one cleanup test defined. If there are multiple
|
|
cleanup tests for a fixture, projects can control their order with the usual
|
|
:prop_test:`DEPENDS` test property if necessary.
|
|
|
|
A cleanup test is allowed to require other fixtures, but not any fixture listed
|
|
in its ``FIXTURES_CLEANUP`` property. For example:
|
|
|
|
.. code-block:: cmake
|
|
|
|
# Ok: Dependent fixture is different to cleanup
|
|
set_tests_properties(cleanupFoo PROPERTIES
|
|
FIXTURES_CLEANUP Foo
|
|
FIXTURES_REQUIRED Bar
|
|
)
|
|
|
|
# Error: cannot require same fixture as cleanup
|
|
set_tests_properties(cleanupFoo PROPERTIES
|
|
FIXTURES_CLEANUP Foo
|
|
FIXTURES_REQUIRED Foo
|
|
)
|
|
|
|
Cleanup tests will execute even if setup or regular tests for that fixture fail
|
|
or are skipped.
|
|
|
|
.. versionchanged:: 4.5
|
|
|
|
The :prop_test:`FIXTURE_REPEAT_MODE` property selects how a fixture behaves
|
|
when :manual:`ctest(1)` repeats tests with :ctest-option:`--repeat`. See
|
|
also :policy:`CMP0224`.
|
|
|
|
See :prop_test:`FIXTURES_REQUIRED` for a more complete discussion of how to use
|
|
test fixtures.
|