When ctest repeats tests with its --repeat option, it repeats each test
on its own. A fixture's setup and cleanup tests therefore run all of
their repetitions back to back, and the tests they bracket repeat inside
a single setup/cleanup pair:
setup -> setup -> test -> test -> cleanup -> cleanup
Add a FIXTURE_REPEAT_MODE test property to select how a fixture behaves
when its tests are repeated:
* AROUND_ALL_REPEATS: the fixture runs once, around all repetitions of
the tests requiring it.
* AROUND_EACH_REPEAT: the fixture and the tests requiring it repeat
together, so every repetition gets a fresh setup and its own cleanup.
* EACH_TEST_SEPARATELY: every test repeats on its own, as before.
The property describes the fixture rather than the test carrying it, so
setting it on any one of a fixture's setup or cleanup tests is enough.
In AROUND_EACH_REPEAT mode the tests of a fixture form a repeat group that
ctest re-queues as a whole once every test in it has finished. The --repeat
condition then applies to the group the way it applies to an individual
test: until-fail repeats while the whole group passes, until-pass repeats
while any of it does not, and after-timeout repeats while any of it times
out. Fixtures that share a test repeat together, so a test requiring two of
them still runs once per repetition.
A group is recorded the way a repeating test is: only once it stops
repeating, and with the results of its last repetition. A group that
until-pass makes pass therefore reports a pass rather than the failure that
made it repeat, a test that DEPENDS on one of the group's tests waits for
the last repetition rather than the first, and `ctest -F` resumes an
interrupted group by running it again from the beginning.
Fixtures that repeat together have to agree on the mode: a test cannot
repeat with one fixture but not with another it takes part in, and a fixture
whose setup and cleanup tests disagree has no coherent behavior. Report an
error and run nothing in those cases rather than pick an order in which a
test repeats after a fixture it requires has been cleaned up.
Add policy CMP0224 to select AROUND_EACH_REPEAT as the default for fixtures
whose setup and cleanup tests choose no mode themselves. Record the mode
the policy chose in the generated test file under its own
_CMAKE_DEFAULT_FIXTURE_REPEAT_MODE keyword, so that ctest reads a mode
rather than the policy settings behind it, and so that a mode requested on
one of a fixture's tests wins over the default recorded for its siblings.
Only NEW needs recording: with nothing recorded, ctest already uses the
behavior of CMake 4.4 and below. Fixtures are common, and the choice of
mode matters only to those who run ctest --repeat, so warn about the unset
policy only when the CMAKE_POLICY_WARNING_CMP0224 variable asks for it.
discover_tests() and gtest_discover_tests() create their tests while ctest
runs or at build time, too late for the policy to reach them, so carry the
setting in effect at their call sites through to the tests they create.
Report the repetition a grouped test belongs to in the "(run N/M)" suffix
of its "Start" line, as ctest already does for a test repeating on its own.
Co-authored-by: Tyler Yankee <tyler.yankee@kitware.com>
Fixes: #21438
CheckResume() reads the checkpoint `ctest -F` resumes from and removes each
test it names. RemoveTest() erased the iterator std::find returned without
checking it, so a checkpoint naming a test the current run does not have
pending -- one written before the test list changed, say -- erased the end
iterator of a std::list, which is undefined behavior. Skip such an entry
instead.
cmCTestRunTest counted the repetitions of a test with two members whose
meaning overlapped: how many runs were left, and how many there were in
total. The "(run N/M)" suffix then had to recover N by subtracting one
from the other, and the rule deciding which run prints the "completed/total"
prefix read as a pair of conditions on the remaining count.
Count up instead. Keep the number of the run being made and the number of
runs the test may be given, and let the rule say which run the prefix
belongs to.
Prepare for a future commit which will reuse this functionality. Add a
structure to hold the resolved fields so that we can more easily
distinguish between errors during preset resolution and cases when
presets simply aren't used.
This makes the output more user friendly when using CTest's repeat
options (`until-fail`, `until-pass` `after-timeout`).
Closes: #28039
Co-authored-by: Tyler Yankee <tyler.yankee@kitware.com>
The ctest(1) command line allows repeating -L/-LE to form an AND
filter over test labels, but the ctest_test() and ctest_memcheck()
scripting commands accepted only a single INCLUDE_LABEL/EXCLUDE_LABEL
value, so a dashboard script could not express that filter.
The label-matching engine already stores the include/exclude
expressions as vectors and applies them with AND semantics; only the
command binding was single-value. Change both keywords to multi-value
(NonEmpty<vector<string>>) and hand the collected list straight to the
engine. ctest_memcheck() gains the same behavior for free, since it
inherits the option set.
Fixes : #27497
Commit 433721b7a7 (ctest: Pass CTEST_SITE/CTEST_BUILD_NAME to configure
step, 2026-06-15) updated ctest_configure() to pass `-DSITE=...` and
`-DBUILDNAME=...` to the configure command whenever CTEST_SITE or
CTEST_BUILD_NAME are set. Projects that don't include the CTest module never
reference these cache variables, causing CMake to report an unused-cli warning.
Use `-C <initial-cache>` instead of `-D` to seed SITE/BUILDNAME. Cache entries
set using `-C` are not tracked by CMake's unused-cli diagnostic, so projects
using the CTest module still get DartConfiguration.tcl populated correctly,
while projects that never reference SITE/BUILDNAME no longer see the warning.
Fixes: #27953
Use a named semaphore polled from the libuv event loop to share GNU Make
job slots on Windows. Cover token ordering, bounded draining, and
teardown, and document the new support.
Fixes: #27968
Extend the interrupt-flush mechanism already used for cmakeBuild and
cmakeInstall to ctest, so that a user interrupt (e.g. Ctrl+C) still writes
the overall ctest instrumentation snippet, marked with interruptSignal,
before exiting.
On interrupt CTest stops scheduling further tests at every launch site,
skips the postCTest hook, forces a non-zero exit status, and preserves the
checkpoint file so that a later 'ctest -F' can resume the interrupted test
set.
Issue: #27859
cmCTestLaunch, which backs ctest --launch and ctest --instrument,
derived its exit code from the child's SpawnResult and ExitStatus but
ignored TermSignal. On POSIX a command killed by a signal reports
ExitStatus 0 and TermSignal N, so the launcher reported success for a
crashed or killed command, masking failed compile/link/custom rules on
the dashboard and in instrumentation snippets.
Classify the child with the ordered triple SpawnResult, TermSignal,
ExitStatus, mapping a signal-killed child to a non-zero exit code.
Native Windows is unaffected: abnormal termination already arrives as a
non-zero exit code there.
Fixes: #27921
Refactor CTest's logging to replace custom color code handling with the
infrastructure introduced in commit 329d755dbd (StdIo: Add a Terminal
abstraction to print color text, 2025-05-06).
Avoid ANSI escape sequences in files passed to `--output-log-file`.
Fixes: #27915
Add an --out-of-date option to ctest which runs only those tests with
recorded build dependencies whose timestamps are newer than the timestamp
at which the test last finished.
Fixes: #27614
When performing the configure step in dashboard client mode, propagate
`CTEST_SITE` and `CTEST_BUILD_NAME` as the `SITE` and `BUILDNAME`
respective cache variables.
This causes the values recorded in `DartConfiguration.tcl` to match those
specified by the user, allowing subsequent ctest calls to reuse them
automatically.
Add `stdout` and `stderr` reporting to instrumentation for `compile`, `link`,
`custom`, `install` and `test` snippets when the `captureOutput` option is
enabled.
Fixes: #26704
CMake writes ConfigureCommand and MakeCommand entries into
DartConfiguration.tcl when configuring a project that uses the CTest module.
These values prevented users from being able to use configure/build presets
in dashboard client mode from a pre-configured build directory.
This commit updates ctest_configure() and ctest_build() to ignore
CTEST_CONFIGURE_COMMAND / CTEST_BUILD_COMMAND when a preset is specified.
This allows the requested preset to be honored.
Introduce the capability to use the Clang Source code coverage capability
to generate coverage information for a submission to CDash.
This handler will:
* Recursively glob the binary directory for coverage files (`*.profdata`)
* Use the `llvm-profdata` tool to merge all found files into a single
`default.profdata` file
* Recursively glob the binary directory for .o files
* Use the `llvm-cov export` capability to parse the coverage information
found for each object file
Since the coverage information contains branch coverage, add new attributes
to the CoverageLog Line object to capture the number of branches that are
covered and uncovered
See documentation page here
https://clang.llvm.org/docs/SourceBasedCodeCoverage.html
Brad King
AJIOB
Taylor Braun-JonesandTyler Yankee
Martin Duffy
Nerixyz
Maxime HaselbauerandTyler Yankee
Arha Gatram
Daniel Pfeifer
Daksh Mamodiya
Zack Galbreath
Joseph Snyder