When ctest repeats tests with its --repeat option, it repeats each test
on its own. A fixture's setup and cleanup tests therefore run all of
their repetitions back to back, and the tests they bracket repeat inside
a single setup/cleanup pair:
setup -> setup -> test -> test -> cleanup -> cleanup
Add a FIXTURE_REPEAT_MODE test property to select how a fixture behaves
when its tests are repeated:
* AROUND_ALL_REPEATS: the fixture runs once, around all repetitions of
the tests requiring it.
* AROUND_EACH_REPEAT: the fixture and the tests requiring it repeat
together, so every repetition gets a fresh setup and its own cleanup.
* EACH_TEST_SEPARATELY: every test repeats on its own, as before.
The property describes the fixture rather than the test carrying it, so
setting it on any one of a fixture's setup or cleanup tests is enough.
In AROUND_EACH_REPEAT mode the tests of a fixture form a repeat group that
ctest re-queues as a whole once every test in it has finished. The --repeat
condition then applies to the group the way it applies to an individual
test: until-fail repeats while the whole group passes, until-pass repeats
while any of it does not, and after-timeout repeats while any of it times
out. Fixtures that share a test repeat together, so a test requiring two of
them still runs once per repetition.
A group is recorded the way a repeating test is: only once it stops
repeating, and with the results of its last repetition. A group that
until-pass makes pass therefore reports a pass rather than the failure that
made it repeat, a test that DEPENDS on one of the group's tests waits for
the last repetition rather than the first, and `ctest -F` resumes an
interrupted group by running it again from the beginning.
Fixtures that repeat together have to agree on the mode: a test cannot
repeat with one fixture but not with another it takes part in, and a fixture
whose setup and cleanup tests disagree has no coherent behavior. Report an
error and run nothing in those cases rather than pick an order in which a
test repeats after a fixture it requires has been cleaned up.
Add policy CMP0224 to select AROUND_EACH_REPEAT as the default for fixtures
whose setup and cleanup tests choose no mode themselves. Record the mode
the policy chose in the generated test file under its own
_CMAKE_DEFAULT_FIXTURE_REPEAT_MODE keyword, so that ctest reads a mode
rather than the policy settings behind it, and so that a mode requested on
one of a fixture's tests wins over the default recorded for its siblings.
Only NEW needs recording: with nothing recorded, ctest already uses the
behavior of CMake 4.4 and below. Fixtures are common, and the choice of
mode matters only to those who run ctest --repeat, so warn about the unset
policy only when the CMAKE_POLICY_WARNING_CMP0224 variable asks for it.
discover_tests() and gtest_discover_tests() create their tests while ctest
runs or at build time, too late for the policy to reach them, so carry the
setting in effect at their call sites through to the tests they create.
Report the repetition a grouped test belongs to in the "(run N/M)" suffix
of its "Start" line, as ctest already does for a test repeating on its own.
Co-authored-by: Tyler Yankee <tyler.yankee@kitware.com>
Fixes: #21438
This makes the output more user friendly when using CTest's repeat
options (`until-fail`, `until-pass` `after-timeout`).
Closes: #28039
Co-authored-by: Tyler Yankee <tyler.yankee@kitware.com>
As the number of areas in CMake for which we've provided JSON schemas
has expanded, testing infrastructure has mostly been copy-pasted across
the relevant test suites. Unify it to a single reusable validator
script and CMake helper.
Commit 433721b7a7 (ctest: Pass CTEST_SITE/CTEST_BUILD_NAME to configure
step, 2026-06-15) updated ctest_configure() to pass `-DSITE=...` and
`-DBUILDNAME=...` to the configure command whenever CTEST_SITE or
CTEST_BUILD_NAME are set. Projects that don't include the CTest module never
reference these cache variables, causing CMake to report an unused-cli warning.
Use `-C <initial-cache>` instead of `-D` to seed SITE/BUILDNAME. Cache entries
set using `-C` are not tracked by CMake's unused-cli diagnostic, so projects
using the CTest module still get DartConfiguration.tcl populated correctly,
while projects that never reference SITE/BUILDNAME no longer see the warning.
Fixes: #27953
Add an --out-of-date option to ctest which runs only those tests with
recorded build dependencies whose timestamps are newer than the timestamp
at which the test last finished.
Fixes: #27614
When performing the configure step in dashboard client mode, propagate
`CTEST_SITE` and `CTEST_BUILD_NAME` as the `SITE` and `BUILDNAME`
respective cache variables.
This causes the values recorded in `DartConfiguration.tcl` to match those
specified by the user, allowing subsequent ctest calls to reuse them
automatically.
Prior to this commit, ctest would sometimes use the current working directory
as its binary directory rather than honoring `binaryDir` from a configure
preset. This would occur when using ctest in dashboard client mode to perform
an initial configuration with a preset.
`ctest --source-dir=<dir> -T Configure -D CTEST_PRESET=<preset>`
An explicit `--build-dir` still takes precedence over the preset's `binaryDir`.
Add the --source-dir argument to ctest to allow it to perform an initial
configuration step for an empty binary directory when operating in
dashboard client mode.
Add support for `ctest [options] -- <args>` to forward extra arguments
to every executed test's command line. Arguments after `--` are appended
after each test's own arguments in the order given.
This enables passing options to test frameworks like Google Test or
doctest without needing per-test ARGS properties:
ctest -R my_test -- --gtest_filter=SomeTest.Case
ctest --verbose -- --success
An error is raised if `--` is combined with `--build-and-test`, `-S`,
or `-D`/`-T`/`-M` modes, which have their own argument forwarding.
A bare `--` with no following arguments is silently ignored.
Fixes: #20470
Do not run the custom pre-/post- test tasks when instructed to only show
the tests that would be executed. Otherwise it's impossible to reliably
parse the output of this as is will be contaminated by all output
produced by the pre-/post-scripts!
Fixes: #23254
9dc3edbba8 ctest: Restore default test timeout for command-line `-T Test` step
8745e6308e ctest: Restore default of no time limit for command-line `-T Test` step
Acked-by: Kitware Robot <kwrobot@kitware.com>
Merge-request: !11094
In commit 774fcbe49c (CTest: Base command line mode on top of scripting
commands, 2024-10-05, v4.0.0-rc1~653^2) the implementation of this step
was converted to use `ctest_test` internally. That command has its own
default test timeout of 600s if the ctest "TimeOut" configuration value
is not set. If "TimeOut" is not set then default it to our previous
command-line mode default test timeout.
Fixes: #27139
Update commit 712cb8cb14 (ctest: Restore support for -T Test without
dashboard configuration, 2025-03-06, v4.0.0-rc4~11^2) to avoid
conflating coverage of CMake itself with the coverage mode of ctest
being tested.
When `--schedule-random` is used in automated CI jobs, failures may
occur due to test order. We now log the seed. Provide a way for
developers to re-run the same order by specifying the seed.
Fixes: #26760
Co-authored-by: Brad King <brad.king@kitware.com>
Since commit 774fcbe49c (CTest: Base command line mode on top of
scripting commands, 2024-10-05, v4.0.0-rc1~653^2) we drive dashboard
client steps through `cmCTestHandlerCommand::ExecuteHandlerCommand`
instead of calling `ProcessHandler` directly. This requires the
`BuildDirectory` to be known to establish a work directory.
Fixes: #26743
Co-authored-by: Brad King <brad.king@kitware.com>
Make sure that all CMake variables that are translated into
CTest options in `cmCTest*Command` implementations are translated
from CTest options into CMake variables before the functions are
called. This back-and-forth translation should be temporary.
It is a necessary prerequisite for refactoring `cmCTest*Handler`
implementations to operate on CMake variables directly rather
than CTest options.
In a complex test environment, there may be a desire to add a test
property label to indicate context. Here is one such example:
set_property(TEST mygoofytest PROPERTY LABELS might-fail-on-config-change)
Upon looking at the failed test summary, one can quickly decide whether
the failure might be expected or not.
198 - mygoofytest (Failed) might-fail-on-config-change
Taylor Braun-JonesandTyler Yankee
Brad King
Nerixyz
Maxime HaselbauerandTyler Yankee
Arha Gatram
Zack Galbreath
Martin Duffy
scivision
Mickaël Germain
Sebastian Sura
Sertonix
AJIOB
Craig Scott
Daniel GoldbergandBrad King
Daniel PfeiferandBrad King
Wim Colgate
castro-ma