When ctest repeats tests with its --repeat option, it repeats each test
on its own. A fixture's setup and cleanup tests therefore run all of
their repetitions back to back, and the tests they bracket repeat inside
a single setup/cleanup pair:
setup -> setup -> test -> test -> cleanup -> cleanup
Add a FIXTURE_REPEAT_MODE test property to select how a fixture behaves
when its tests are repeated:
* AROUND_ALL_REPEATS: the fixture runs once, around all repetitions of
the tests requiring it.
* AROUND_EACH_REPEAT: the fixture and the tests requiring it repeat
together, so every repetition gets a fresh setup and its own cleanup.
* EACH_TEST_SEPARATELY: every test repeats on its own, as before.
The property describes the fixture rather than the test carrying it, so
setting it on any one of a fixture's setup or cleanup tests is enough.
In AROUND_EACH_REPEAT mode the tests of a fixture form a repeat group that
ctest re-queues as a whole once every test in it has finished. The --repeat
condition then applies to the group the way it applies to an individual
test: until-fail repeats while the whole group passes, until-pass repeats
while any of it does not, and after-timeout repeats while any of it times
out. Fixtures that share a test repeat together, so a test requiring two of
them still runs once per repetition.
A group is recorded the way a repeating test is: only once it stops
repeating, and with the results of its last repetition. A group that
until-pass makes pass therefore reports a pass rather than the failure that
made it repeat, a test that DEPENDS on one of the group's tests waits for
the last repetition rather than the first, and `ctest -F` resumes an
interrupted group by running it again from the beginning.
Fixtures that repeat together have to agree on the mode: a test cannot
repeat with one fixture but not with another it takes part in, and a fixture
whose setup and cleanup tests disagree has no coherent behavior. Report an
error and run nothing in those cases rather than pick an order in which a
test repeats after a fixture it requires has been cleaned up.
Add policy CMP0224 to select AROUND_EACH_REPEAT as the default for fixtures
whose setup and cleanup tests choose no mode themselves. Record the mode
the policy chose in the generated test file under its own
_CMAKE_DEFAULT_FIXTURE_REPEAT_MODE keyword, so that ctest reads a mode
rather than the policy settings behind it, and so that a mode requested on
one of a fixture's tests wins over the default recorded for its siblings.
Only NEW needs recording: with nothing recorded, ctest already uses the
behavior of CMake 4.4 and below. Fixtures are common, and the choice of
mode matters only to those who run ctest --repeat, so warn about the unset
policy only when the CMAKE_POLICY_WARNING_CMP0224 variable asks for it.
discover_tests() and gtest_discover_tests() create their tests while ctest
runs or at build time, too late for the policy to reach them, so carry the
setting in effect at their call sites through to the tests they create.
Report the repetition a grouped test belongs to in the "(run N/M)" suffix
of its "Start" line, as ctest already does for a test repeating on its own.
Co-authored-by: Tyler Yankee <tyler.yankee@kitware.com>
Fixes: #21438
Add an --out-of-date option to ctest which runs only those tests with
recorded build dependencies whose timestamps are newer than the timestamp
at which the test last finished.
Fixes: #27614
Add a `CTEST_TEST_COVERAGE_TOOL` variable to tell CTest to integrate
with `llvm-cov` source-based code coverage by setting the
`LLVM_PROFILE_FILE` environment variable for each test.
Issue: #26932
Introduce a new internal testProperty object called CTestDirectory.
This directory is originally set to the directory were the test was created,
but will not be overwritten when the WORKING_DIRECTORY property is
updated on the test.
Add support for `ctest [options] -- <args>` to forward extra arguments
to every executed test's command line. Arguments after `--` are appended
after each test's own arguments in the order given.
This enables passing options to test frameworks like Google Test or
doctest without needing per-test ARGS properties:
ctest -R my_test -- --gtest_filter=SomeTest.Case
ctest --verbose -- --success
An error is raised if `--` is combined with `--build-and-test`, `-S`,
or `-D`/`-T`/`-M` modes, which have their own argument forwarding.
A bare `--` with no following arguments is silently ignored.
Fixes: #20470
In commit fd55647d16 (ctest: Record `StartTestTime` entry for each test
in Test.xml, 2026-02-12) we added the field with milliseconds to match
instrumentation. However, all other times reported by CTest are in
seconds, including the global `StartTestTime` field. Preserve
sub-second resolution using a floating-point representation.
Issue: #26859
`include-what-you-use` diagnostics, in practice, are specific to
the environment's compiler and standard library. Update includes
to satisfy IWYU for our CI job under Debian 13. Some patterns:
* Types named in virtual `override` signatures no longer require
includes since the overridden signature already names them.
* A function argument's type needs to be included even if its constructor
is called only by implicit conversion. For example, constructing a
`std::function` from a lambda now requires `<functional>`.
* Some prior mysterious `<type_traits>` inclusions are no longer required.
When `--schedule-random` is used in automated CI jobs, failures may
occur due to test order. We now log the seed. Provide a way for
developers to re-run the same order by specifying the seed.
Fixes: #26760
Co-authored-by: Brad King <brad.king@kitware.com>
Run the `clang-format.bash` script to update all our C and C++ code to a
new style defined by `.clang-format`, now with "east const" enforcement.
Use `clang-format` version 18.
* If you reached this commit for a line in `git blame`, re-run the blame
operation starting at the parent of this commit to see older history
for the content.
* See the parent commit for instructions to rebase a change across this
style transition commit.
Issue: #26123
Make sure that all CMake variables that are translated into
CTest options in `cmCTest*Command` implementations are translated
from CTest options into CMake variables before the functions are
called. This back-and-forth translation should be temporary.
It is a necessary prerequisite for refactoring `cmCTest*Handler`
implementations to operate on CMake variables directly rather
than CTest options.
If the options added by
* commit 022f20f663 (ctest: add command line option to run the tests listed
in a given file, 2023-11-29, v3.29.0-rc1~66^2~2)
* commit dbacc1d5a8 (ctest: add command line option to exclude tests listed
in a given file, 2023-11-30, v3.29.0-rc1~66^2~1)
* commit 701029726f (ctest_test: add options INCLUDE_FROM_FILE and
EXCLUDE_FROM_FILE, 2023-12-03, v3.29.0-rc1~66^2)
are given a missing file, fail instead of ignoring it.
Fixes: #25740
Add `--tests-from-file <filename>` to run only the tests listed in the
given file. The test names must match exactly, no regexps or something.
The listed tests can still be filtered with a regexp using -R.
Issue: #25455
`include-what-you-use` diagnostics, in practice, are specific to
the environment's compiler and standard library. Update includes
to satisfy IWYU for our CI job under Debian 12.
CTest normally terminates test processes on timeout using `SIGKILL`.
Offer tests a chance to exit gracefully, on platforms supporting POSIX
signals, by setting `TIMEOUT_SIGNAL_{NAME,GRACE_PERIOD}` properties.
Fixes: #17288
Remove a comment that has not been relevant since commit 177edc5ed1
(Fixed ctest -N segfault issue. Further refactored ctest. Enabled
failover for ctest, 2009-08-27, v2.8.0~250).
Add `--test-output-truncation` to `ctest`. This option can be used to
customize which part of the test output is being truncated. Currently
supported values are `tail`, `middle` and `head`.
Also add equivalent `CTEST_CUSTOM_TEST_OUTPUT_TRUNCATION` variable.
Fixes: #23206
This property allows projects to modify environment variables at test
time rather than trying to guess what the state should be based on what
is present at configure time. Of particular interest is the ability to
use a `PATH` present at test time while adding entries known to be
necessary for the test itself.
There are multiple operations provided to modify variables, including:
- setting and unsetting
- appending and prepending as:
- strings
- path lists
- CMake lists
Additionally, a `reset` action is provided to cancel any prior
modifications to that particular variable in the case of incremental
additions to the test property.
Teach CTest to parse output for <CTestMeasurement> in addition to
<DartMeasurement> for measurements defined at runtime.
Use a new class (cmCTestTestMeasurementXMLParser) derived from cmXMLParser
to parse the data and attributes these XML elements. This is an improvement
over our previous approach of using a series of regular expressions.
As part of this commit we also rename some member variables and methods
to make their purpose more clear.
DartStuff -> AllTestMeasurementsRegex
DartStuff1 -> SingleTestMeasurementRegex
DartString -> TestMeasurementsOutput
GenerateDartOutput() -> GenerateCTestXML()
GenerateRegressionImages() -> RecordCustomTestMeasurements()
cmCTestRunTest::DartProcessing() -> ParseOutputForMeasurements()
Allow tests to specify files to upload at runtime. Previously this was
only possible to specify at configure time with the ATTACHED_FILES
test properties.
This commit also fixes a bug in how our test data tarballs were generated
by CTest. Previously, if you tried to attach a file outside of the binary
directory, CTest would generate a tar file with a relative path, and tar
would not allow you to extract it. We resolve this problem by creating
tar files with a flat directory structure instead.
Fixes: #22284
Taylor Braun-JonesandTyler Yankee
Arha Gatram
Martin Duffy
Zack Galbreath
Joseph Snyder
Mickaël Germain
Brad King
Daniel Pfeifer
Daniel GoldbergandBrad King
Kitware Robot
Daniel Sim
Alex Neundorf
Kyle Edwards
Frank Winklmeier
Rose
Marc Chevrier
Ben Boeckel