When ctest repeats tests with its --repeat option, it repeats each test
on its own. A fixture's setup and cleanup tests therefore run all of
their repetitions back to back, and the tests they bracket repeat inside
a single setup/cleanup pair:
setup -> setup -> test -> test -> cleanup -> cleanup
Add a FIXTURE_REPEAT_MODE test property to select how a fixture behaves
when its tests are repeated:
* AROUND_ALL_REPEATS: the fixture runs once, around all repetitions of
the tests requiring it.
* AROUND_EACH_REPEAT: the fixture and the tests requiring it repeat
together, so every repetition gets a fresh setup and its own cleanup.
* EACH_TEST_SEPARATELY: every test repeats on its own, as before.
The property describes the fixture rather than the test carrying it, so
setting it on any one of a fixture's setup or cleanup tests is enough.
In AROUND_EACH_REPEAT mode the tests of a fixture form a repeat group that
ctest re-queues as a whole once every test in it has finished. The --repeat
condition then applies to the group the way it applies to an individual
test: until-fail repeats while the whole group passes, until-pass repeats
while any of it does not, and after-timeout repeats while any of it times
out. Fixtures that share a test repeat together, so a test requiring two of
them still runs once per repetition.
A group is recorded the way a repeating test is: only once it stops
repeating, and with the results of its last repetition. A group that
until-pass makes pass therefore reports a pass rather than the failure that
made it repeat, a test that DEPENDS on one of the group's tests waits for
the last repetition rather than the first, and `ctest -F` resumes an
interrupted group by running it again from the beginning.
Fixtures that repeat together have to agree on the mode: a test cannot
repeat with one fixture but not with another it takes part in, and a fixture
whose setup and cleanup tests disagree has no coherent behavior. Report an
error and run nothing in those cases rather than pick an order in which a
test repeats after a fixture it requires has been cleaned up.
Add policy CMP0224 to select AROUND_EACH_REPEAT as the default for fixtures
whose setup and cleanup tests choose no mode themselves. Record the mode
the policy chose in the generated test file under its own
_CMAKE_DEFAULT_FIXTURE_REPEAT_MODE keyword, so that ctest reads a mode
rather than the policy settings behind it, and so that a mode requested on
one of a fixture's tests wins over the default recorded for its siblings.
Only NEW needs recording: with nothing recorded, ctest already uses the
behavior of CMake 4.4 and below. Fixtures are common, and the choice of
mode matters only to those who run ctest --repeat, so warn about the unset
policy only when the CMAKE_POLICY_WARNING_CMP0224 variable asks for it.
discover_tests() and gtest_discover_tests() create their tests while ctest
runs or at build time, too late for the policy to reach them, so carry the
setting in effect at their call sites through to the tests they create.
Report the repetition a grouped test belongs to in the "(run N/M)" suffix
of its "Start" line, as ctest already does for a test repeating on its own.
Co-authored-by: Tyler Yankee <tyler.yankee@kitware.com>
Fixes: #21438
Move some target related enumeration types from `cmStateTypes.h` to a
new header, as these really aren't "state" types. Also, make
`TargetType` a strongly-typed enumeration.
Add an --out-of-date option to ctest which runs only those tests with
recorded build dependencies whose timestamps are newer than the timestamp
at which the test last finished.
Fixes: #27614
Extend the test_prep/<test> and test_prep/all build targets, generated
from add_test(BUILD_DEPENDS) when CMAKE_TEST_BUILD_DEPENDS is enabled, to
the Makefile generators. Previously only the Ninja and FASTBuild
generators provided them.
Resolve each test's dependencies in a shared layer so all command-line
generators consume one result. cmTestGenerator now filters target
dependencies to build-system targets and records, for each file
dependency, the single target that produces it as a primary
custom-command output (cmGlobalGenerator::FindOutputOwningTarget). The
Makefile generators emit phony rules in CMakeFiles/Makefile2 depending on
<target>.dir/all, with top-level forwarding rules. Sharing the
resolution also stops Ninja and FASTBuild from emitting a dead test_prep
edge for a non-buildable (e.g. INTERFACE) BUILD_DEPENDS target.
On the Makefile generators a file dependency that no single target
produces, and a test whose name contains ':', cannot be expressed as a
recursive-make rule and are reported with a warning and skipped.
Fixes: #27879
Add `test_prep/<test_name>` convenience targets to the ninja generator
to build dependencies of tests, and `test_prep/all` to build the dependencies
of all tests.
Build dependencies of a test include:
- Executables directly invoked by test `COMMAND`.
- Targets mentioned in generator expressionf os test `COMMAND`.
- Anything explicitly listed with the new `BUILD_DEPENDS` argument of `add_test`.
Issue: #27613
Add and use a couple helper functions for issuing policy warnings. This
improves consistency and allows some simplification of many call sites.
(One or two instances in particular are greatly simplified.)
Add new diagnostic categories CMD_POLICY and CMD_EXPERIMENTAL, both as
children of CMD_AUTHOR. Use the former when issuing policy warnings. Use
the latter when issuing experimental feature warnings.
Find places that are currently relying on diagnostic-specific message
types to issue diagnostics and replace these with calls to the new
diagnostic methods.
Run the `clang-format.bash` script to update all our C and C++ code to a
new style defined by `.clang-format`, now with "east const" enforcement.
Use `clang-format` version 18.
* If you reached this commit for a line in `git blame`, re-run the blame
operation starting at the parent of this commit to see older history
for the content.
* See the parent commit for instructions to rebase a change across this
style transition commit.
Issue: #26123
This will now preserve empty values in the TEST_LAUNCHER and
CROSSCOMPILING_EMULATOR target properties for tests added by:
- The add_test() command.
- The ExternalData_Add_Test() command from the ExternalData module.
- The gtest_add_tests() or gtest_discover_tests() commands from the
GoogleTest module.
For the gtest_add_tests() and gtest_discover_tests() commands,
empty elements in the values passed after the EXTRA_ARGS keyword
are also now preserved.
Policy CMP0178 is added to provide backward compatibility with the
old behavior where empty values were silently discarded from the
above cases.
Fixes: #26337
Evaluate generator expressions in these properties, as they apply to
`add_test`, `add_custom_command`, and `add_custom_target`.
The `CMAKE_CROSSCOMPILING_EMULATOR` variable's `try_run` behavior occurs
at configure time and so cannot support generator expressions.
This was regressed by refactoring in commit e08ba229ee (CMake code rely
on cmList class for CMake lists management (part. 1), 2023-04-14,
v3.27.0-rc1~174^2). Fix it and add a test case.
Fixes: #24986
Restore the change from commit f84af8e270 (add_test: Allow special
characters in test name, 2020-05-16, v3.18.0-rc1~142^2) that had to be
reverted by commit f84af8e270 (add_test: Allow special characters in
test name, 2020-05-16, v3.18.0-rc1~142^2) for compatibility.
Add policy CMP0110 to make the change in a compatible way.
Also, support even more characters than before by generating the
test scripts using bracket arguments around the test names.
Fixes: #19391
Signed-off-by: Deniz Bahadir <dbahadir@benocs.com>
Revert commit f84af8e270 (add_test: Allow special characters in test
name, 2020-05-16, v3.18.0-rc1~142^2). Unfortunately the fix breaks
projects that were working around the limitation with manual escaping.
The fix can be re-introduced with a policy in a future version.
Also add a 3.18.1 release note explaining the change.
Fixes: #21017, #20965
Issue: #19391
The `std::endl` manipulator, except inserting `\n` character, also
performs `os.flush()`, which may lead to undesired effects (like
disk I/O in the middle of forming data strings). For the
`std::stringstream` it also has no meaning.
* replace multiple `operator<<` calls on a string literal w/
the only call and the only (bigger) string literal;
* replace one character string literal used in `operator<<`
w/ a char literal.
An old workaround for `std::allocator_traits<>::value_type` lints from
IWYU on `std::vector<>` usage breaks IWYU's handling of `<memory>`.
Convert the workaround to use the same approach we already use for a
workaround of `std::__decay_and_strip<>::::__type` lints. Then update
the `<memory>` inclusions to follow the now-correct IWYU lints.
Add a `COMMAND_EXPAND_LISTS` option to the `add_test` command to cause
`;`-separated lists produced by generator expressions to be expanded
into multiple arguments. The `add_custom_command` command already
has such an option.
Fixes: #17284
Suppress some cases in `Source/cmGeneratorExpressionNode.cxx` and
`Source/cmUVHandlePtr.h` where a few older compilers require a
user-defined default constructor (with `{}`).
If CROSSCOMPILING_EMULATOR was set to an empty string, and a test
was generated with the executable as the command, CMake would segfault
upon trying to generate the test file. Fix this.
Fixes: #18819
Run the `clang-format.bash` script to update all our C and C++ code to a
new style defined by `.clang-format`. Use `clang-format` version 6.0.
* If you reached this commit for a line in `git blame`, re-run the blame
operation starting at the parent of this commit to see older history
for the content.
* See the parent commit for instructions to rebase a change across this
style transition commit.
Taylor Braun-JonesandTyler Yankee
Martin Duffy
Matthew Woehlke
Daksh Mamodiya
Brad King
Daniel Pfeifer
Kitware Robot
Craig Scott
Ralf Habacker
Marc Chevrier
Kyle Edwards
Oleksandr Koval
Deniz Bahadir
Vitaly Stakhovsky
Alex Turbov
Sebastian Holtermann
Sergey Bobrenok
Regina Pfeifer
Artur Ryt
Justin Goshi