mirrors/cmake
2
0
Fork 0
mirror of https://gitlab.kitware.com/cmake/cmake.git synced 2026-09-25 04:09:36 +03:00
Code Activity
Files
master
cmake/Help/command/discover_tests.rst
T
Taylor Braun-JonesandTyler Yankee 5cced02b9d CTest: Repeat test fixtures with the tests that require them 
When ctest repeats tests with its --repeat option, it repeats each test
on its own.  A fixture's setup and cleanup tests therefore run all of
their repetitions back to back, and the tests they bracket repeat inside
a single setup/cleanup pair:

  setup -> setup -> test -> test -> cleanup -> cleanup

Add a FIXTURE_REPEAT_MODE test property to select how a fixture behaves
when its tests are repeated:

* AROUND_ALL_REPEATS: the fixture runs once, around all repetitions of
  the tests requiring it.
* AROUND_EACH_REPEAT: the fixture and the tests requiring it repeat
  together, so every repetition gets a fresh setup and its own cleanup.
* EACH_TEST_SEPARATELY: every test repeats on its own, as before.

The property describes the fixture rather than the test carrying it, so
setting it on any one of a fixture's setup or cleanup tests is enough.

In AROUND_EACH_REPEAT mode the tests of a fixture form a repeat group that
ctest re-queues as a whole once every test in it has finished.  The --repeat
condition then applies to the group the way it applies to an individual
test: until-fail repeats while the whole group passes, until-pass repeats
while any of it does not, and after-timeout repeats while any of it times
out.  Fixtures that share a test repeat together, so a test requiring two of
them still runs once per repetition.

A group is recorded the way a repeating test is: only once it stops
repeating, and with the results of its last repetition.  A group that
until-pass makes pass therefore reports a pass rather than the failure that
made it repeat, a test that DEPENDS on one of the group's tests waits for
the last repetition rather than the first, and `ctest -F` resumes an
interrupted group by running it again from the beginning.

Fixtures that repeat together have to agree on the mode: a test cannot
repeat with one fixture but not with another it takes part in, and a fixture
whose setup and cleanup tests disagree has no coherent behavior.  Report an
error and run nothing in those cases rather than pick an order in which a
test repeats after a fixture it requires has been cleaned up.

Add policy CMP0224 to select AROUND_EACH_REPEAT as the default for fixtures
whose setup and cleanup tests choose no mode themselves.  Record the mode
the policy chose in the generated test file under its own
_CMAKE_DEFAULT_FIXTURE_REPEAT_MODE keyword, so that ctest reads a mode
rather than the policy settings behind it, and so that a mode requested on
one of a fixture's tests wins over the default recorded for its siblings.
Only NEW needs recording: with nothing recorded, ctest already uses the
behavior of CMake 4.4 and below.  Fixtures are common, and the choice of
mode matters only to those who run ctest --repeat, so warn about the unset
policy only when the CMAKE_POLICY_WARNING_CMP0224 variable asks for it.

discover_tests() and gtest_discover_tests() create their tests while ctest
runs or at build time, too late for the policy to reach them, so carry the
setting in effect at their call sites through to the tests they create.

Report the repetition a grouped test belongs to in the "(run N/M)" suffix
of its "Start" line, as ctest already does for a test repeating on its own.

Co-authored-by: Tyler Yankee <tyler.yankee@kitware.com>
Fixes: #21438
2026-09-18 08:29:28 -04:00

125 lines
4.4 KiB
ReStructuredText
Raw Permalink Blame History

discover_tests
--------------
.. versionadded:: 4.4
Register tests with names and properties discovered at test time by
:manual:`ctest(1)`.
.. code-block:: cmake
discover_tests(COMMAND <command> [<arg>...] [COMMAND_EXPAND_LISTS]
[CONFIGURATIONS <config>...]
DISCOVERY_ARGS <arg>...
DISCOVERY_MATCH <regex>
[DISCOVERY_PROPERTIES <key> <value> [<key> <value>]...]
TEST_NAME <replacement>
TEST_ARGS <replacement>...
[TEST_PROPERTIES <key> <replacement> [<key> <replacement>]...]
)
This command configures test discovery rather than defining a single test at
configure time. During test execution, :manual:`ctest(1)` runs the specified
discovery command, parses its output, and registers one or more tests based on
the provided regular expression and replacement strings.
``discover_tests`` options are:
``COMMAND``
Specify the command-line used for test discovery.
The command is executed by :manual:`ctest(1)` at test time (not by CMake at
configure time). With ``DISCOVERY_ARGS`` appended, it must print the list of
available tests in a format matched by ``DISCOVERY_MATCH``.
If ``<command>`` specifies an executable target created by
:command:`add_executable`:
* It will automatically be replaced by the location of the executable
created at build time.
* The target's :prop_tgt:`CROSSCOMPILING_EMULATOR`, if set, will be
used to run the command on the host::
<emulator> <command>
The emulator is used only when
:variable:`cross-compiling <CMAKE_CROSSCOMPILING>`.
* The target's :prop_tgt:`TEST_LAUNCHER`, if set, will be used to launch the
command::
<launcher> <command>
If the :prop_tgt:`CROSSCOMPILING_EMULATOR` is also set, both are used::
<launcher> <emulator> <command>
The command may be specified using
:manual:`generator expressions <cmake-generator-expressions(7)>`.
``COMMAND_EXPAND_LISTS``
Lists in ``COMMAND`` arguments will be expanded, including those created with
:manual:`generator expressions <cmake-generator-expressions(7)>`.
``CONFIGURATIONS``
Restrict the test discovery only to the named configurations.
``DISCOVERY_ARGS``
Additional arguments passed to ``COMMAND`` when performing discovery.
``DISCOVERY_MATCH``
Regular expression used to parse each line produced by the discovery command.
Capturing groups may be referenced by ``TEST_NAME``, ``TEST_ARGS``, and
values in ``TEST_PROPERTIES`` using ``\1``, ``\2``, etc.
``DISCOVERY_PROPERTIES``
Specify properties for the discovery run itself.
``TEST_NAME``
Replacement string used to generate the test name for each discovered test.
It may reference capture groups from ``DISCOVERY_MATCH``.
``TEST_ARGS``
Replacement strings used to generate the arguments passed to the discovered
test. Each argument may reference capture groups from ``DISCOVERY_MATCH``.
``TEST_PROPERTIES``
Specify test properties to set on each discovered test. Values are
replacement strings and may reference capture groups from
``DISCOVERY_MATCH``.
.. versionchanged:: 4.5
If the properties make the discovered tests part of a test fixture, and
they do not set :prop_test:`FIXTURE_REPEAT_MODE`, policy :policy:`CMP0224`
determines the behavior when the :ctest-option:`--repeat` option is used.
CTest executes the discovery step to obtain the list of tests and then runs
each discovered test using the command-line produced by ``COMMAND`` together
with ``TEST_ARGS``. The pass/fail behavior of each discovered test follows
the usual CTest rules (exit code ``0`` indicates success unless inverted by
the :prop_test:`WILL_FAIL` property). Output written to stdout or stderr is
captured by :manual:`ctest(1)` and only affects the pass/fail status via the
:prop_test:`PASS_REGULAR_EXPRESSION`, :prop_test:`FAIL_REGULAR_EXPRESSION`,
or :prop_test:`SKIP_REGULAR_EXPRESSION` test properties.
Example usage:
.. code-block:: cmake
discover_tests(COMMAND testDriver --exe $<TARGET_FILE:myexe>
DISCOVERY_ARGS --list-tests
DISCOVERY_MATCH "^([^,]+),([^,]+),([^,]+),(.*)$"
TEST_NAME "${PROJECT_NAME}.\\1.\\2"
TEST_ARGS --run-test "\\1.\\2"
TEST_PROPERTIES
PROCESSORS "\\3"
LABELS "\\4"
)
This example configures discovery by running ``testDriver --list-tests``.
For each line of output that matches ``DISCOVERY_MATCH``, a test name is
generated using ``TEST_NAME``, the per-test command-line is generated using
``TEST_ARGS``, and test properties are populated from the remaining capture
groups.
View Git Blame Copy Permalink