mirrors/cmake
2
0
Fork 0
mirror of https://gitlab.kitware.com/cmake/cmake.git synced 2026-09-25 04:09:36 +03:00
Code Activity
Files
master
cmake/Help/prop_test/FIXTURES_REQUIRED.rst
T
Taylor Braun-JonesandTyler Yankee 5cced02b9d CTest: Repeat test fixtures with the tests that require them 
When ctest repeats tests with its --repeat option, it repeats each test
on its own.  A fixture's setup and cleanup tests therefore run all of
their repetitions back to back, and the tests they bracket repeat inside
a single setup/cleanup pair:

  setup -> setup -> test -> test -> cleanup -> cleanup

Add a FIXTURE_REPEAT_MODE test property to select how a fixture behaves
when its tests are repeated:

* AROUND_ALL_REPEATS: the fixture runs once, around all repetitions of
  the tests requiring it.
* AROUND_EACH_REPEAT: the fixture and the tests requiring it repeat
  together, so every repetition gets a fresh setup and its own cleanup.
* EACH_TEST_SEPARATELY: every test repeats on its own, as before.

The property describes the fixture rather than the test carrying it, so
setting it on any one of a fixture's setup or cleanup tests is enough.

In AROUND_EACH_REPEAT mode the tests of a fixture form a repeat group that
ctest re-queues as a whole once every test in it has finished.  The --repeat
condition then applies to the group the way it applies to an individual
test: until-fail repeats while the whole group passes, until-pass repeats
while any of it does not, and after-timeout repeats while any of it times
out.  Fixtures that share a test repeat together, so a test requiring two of
them still runs once per repetition.

A group is recorded the way a repeating test is: only once it stops
repeating, and with the results of its last repetition.  A group that
until-pass makes pass therefore reports a pass rather than the failure that
made it repeat, a test that DEPENDS on one of the group's tests waits for
the last repetition rather than the first, and `ctest -F` resumes an
interrupted group by running it again from the beginning.

Fixtures that repeat together have to agree on the mode: a test cannot
repeat with one fixture but not with another it takes part in, and a fixture
whose setup and cleanup tests disagree has no coherent behavior.  Report an
error and run nothing in those cases rather than pick an order in which a
test repeats after a fixture it requires has been cleaned up.

Add policy CMP0224 to select AROUND_EACH_REPEAT as the default for fixtures
whose setup and cleanup tests choose no mode themselves.  Record the mode
the policy chose in the generated test file under its own
_CMAKE_DEFAULT_FIXTURE_REPEAT_MODE keyword, so that ctest reads a mode
rather than the policy settings behind it, and so that a mode requested on
one of a fixture's tests wins over the default recorded for its siblings.
Only NEW needs recording: with nothing recorded, ctest already uses the
behavior of CMake 4.4 and below.  Fixtures are common, and the choice of
mode matters only to those who run ctest --repeat, so warn about the unset
policy only when the CMAKE_POLICY_WARNING_CMP0224 variable asks for it.

discover_tests() and gtest_discover_tests() create their tests while ctest
runs or at build time, too late for the policy to reach them, so carry the
setting in effect at their call sites through to the tests they create.

Report the repetition a grouped test belongs to in the "(run N/M)" suffix
of its "Start" line, as ctest already does for a test repeating on its own.

Co-authored-by: Tyler Yankee <tyler.yankee@kitware.com>
Fixes: #21438
2026-09-18 08:29:28 -04:00

106 lines
5.1 KiB
ReStructuredText
Raw Permalink Blame History

FIXTURES_REQUIRED
-----------------
.. versionadded:: 3.7
Specifies a list of fixtures the test requires. Fixture names are case
sensitive and they are not required to have any similarity to test names.
Fixtures are a way to attach setup and cleanup tasks to a set of tests. If a
test requires a given fixture, then all tests marked as setup tasks for that
fixture will be executed first (once for the whole set of tests, not once per
test requiring the fixture). After all tests requiring a particular fixture
have completed, CTest will ensure all tests marked as cleanup tasks for that
fixture are then executed. Tests are marked as setup tasks with the
:prop_test:`FIXTURES_SETUP` property and as cleanup tasks with the
:prop_test:`FIXTURES_CLEANUP` property. If any of a fixture's setup tests fail,
all tests listing that fixture in their ``FIXTURES_REQUIRED`` property will not
be executed. The cleanup tests for the fixture will always be executed, even if
some setup tests fail.
When CTest is asked to execute only a subset of tests (e.g. by the use of
regular expressions or when run with the :option:`--rerun-failed <ctest --rerun-failed>`
command line option), it will automatically add any setup or cleanup tests for
fixtures required by any of the tests that are in the execution set. This
behavior can be overridden with the :option:`-FS <ctest -FS>`,
:option:`-FC <ctest -FC>` and :option:`-FA <ctest -FA>` command line options to
:manual:`ctest(1)` if desired.
.. versionchanged:: 4.5
The :prop_test:`FIXTURE_REPEAT_MODE` property selects how a fixture behaves
when :manual:`ctest(1)` repeats tests with :ctest-option:`--repeat`. See
also :policy:`CMP0224`.
Since setup and cleanup tasks are also tests, they can have an ordering
specified by the :prop_test:`DEPENDS` test property just like any other tests.
This can be exploited to implement setup or cleanup using multiple tests for a
single fixture to modularise setup or cleanup logic.
The concept of a fixture is different to that of a resource specified by
:prop_test:`RESOURCE_LOCK`, but they may be used together. A fixture defines a
set of tests which share setup and cleanup requirements, whereas a resource
lock has the effect of ensuring a particular set of tests do not run in
parallel. Some situations may need both, such as setting up a database,
serializing test access to that database and deleting the database again at the
end. For such cases, tests would populate both ``FIXTURES_REQUIRED`` and
:prop_test:`RESOURCE_LOCK` to combine the two behaviors. Names used for
:prop_test:`RESOURCE_LOCK` have no relationship with names of fixtures, so note
that a resource lock does not imply a fixture and vice versa.
Consider the following example which represents a database test scenario
similar to that mentioned above:
.. code-block:: cmake
add_test(NAME testsDone COMMAND emailResults)
add_test(NAME fooOnly COMMAND testFoo)
add_test(NAME dbOnly COMMAND testDb)
add_test(NAME dbWithFoo COMMAND testDbWithFoo)
add_test(NAME createDB COMMAND initDB)
add_test(NAME setupUsers COMMAND userCreation)
add_test(NAME cleanupDB COMMAND deleteDB)
add_test(NAME cleanupFoo COMMAND removeFoos)
set_tests_properties(setupUsers PROPERTIES DEPENDS createDB)
set_tests_properties(createDB PROPERTIES FIXTURES_SETUP DB)
set_tests_properties(setupUsers PROPERTIES FIXTURES_SETUP DB)
set_tests_properties(cleanupDB PROPERTIES FIXTURES_CLEANUP DB)
set_tests_properties(cleanupFoo PROPERTIES FIXTURES_CLEANUP Foo)
set_tests_properties(testsDone PROPERTIES FIXTURES_CLEANUP "DB;Foo")
set_tests_properties(fooOnly PROPERTIES FIXTURES_REQUIRED Foo)
set_tests_properties(dbOnly PROPERTIES FIXTURES_REQUIRED DB)
set_tests_properties(dbWithFoo PROPERTIES FIXTURES_REQUIRED "DB;Foo")
set_tests_properties(dbOnly dbWithFoo createDB setupUsers cleanupDB
PROPERTIES RESOURCE_LOCK DbAccess)
Key points from this example:
- Two fixtures are defined: ``DB`` and ``Foo``. Tests can require a single
fixture as ``fooOnly`` and ``dbOnly`` do, or they can depend on multiple
fixtures like ``dbWithFoo`` does.
- A ``DEPENDS`` relationship is set up to ensure ``setupUsers`` happens after
``createDB``, both of which are setup tests for the ``DB`` fixture and will
therefore be executed before the ``dbOnly`` and ``dbWithFoo`` tests
automatically.
- No explicit ``DEPENDS`` relationships were needed to make the setup tests run
before or the cleanup tests run after the regular tests.
- The ``Foo`` fixture has no setup tests defined, only a single cleanup test.
- ``testsDone`` is a cleanup test for both the ``DB`` and ``Foo`` fixtures.
Therefore, it will only execute once regular tests for both fixtures have
finished (i.e. after ``fooOnly``, ``dbOnly`` and ``dbWithFoo``). No
``DEPENDS`` relationship was specified for ``testsDone``, so it is free to
run before, after or concurrently with other cleanup tests for either
fixture.
- The setup and cleanup tests never list the fixtures they are for in their own
``FIXTURES_REQUIRED`` property, as that would result in a dependency on
themselves and be considered an error.
View Git Blame Copy Permalink