As the number of areas in CMake for which we've provided JSON schemas
has expanded, testing infrastructure has mostly been copy-pasted across
the relevant test suites. Unify it to a single reusable validator
script and CMake helper.
Commit 433721b7a7 (ctest: Pass CTEST_SITE/CTEST_BUILD_NAME to configure
step, 2026-06-15) updated ctest_configure() to pass `-DSITE=...` and
`-DBUILDNAME=...` to the configure command whenever CTEST_SITE or
CTEST_BUILD_NAME are set. Projects that don't include the CTest module never
reference these cache variables, causing CMake to report an unused-cli warning.
Use `-C <initial-cache>` instead of `-D` to seed SITE/BUILDNAME. Cache entries
set using `-C` are not tracked by CMake's unused-cli diagnostic, so projects
using the CTest module still get DartConfiguration.tcl populated correctly,
while projects that never reference SITE/BUILDNAME no longer see the warning.
Fixes: #27953
Add an --out-of-date option to ctest which runs only those tests with
recorded build dependencies whose timestamps are newer than the timestamp
at which the test last finished.
Fixes: #27614
When performing the configure step in dashboard client mode, propagate
`CTEST_SITE` and `CTEST_BUILD_NAME` as the `SITE` and `BUILDNAME`
respective cache variables.
This causes the values recorded in `DartConfiguration.tcl` to match those
specified by the user, allowing subsequent ctest calls to reuse them
automatically.
Prior to this commit, ctest would sometimes use the current working directory
as its binary directory rather than honoring `binaryDir` from a configure
preset. This would occur when using ctest in dashboard client mode to perform
an initial configuration with a preset.
`ctest --source-dir=<dir> -T Configure -D CTEST_PRESET=<preset>`
An explicit `--build-dir` still takes precedence over the preset's `binaryDir`.
Add the --source-dir argument to ctest to allow it to perform an initial
configuration step for an empty binary directory when operating in
dashboard client mode.
Add support for `ctest [options] -- <args>` to forward extra arguments
to every executed test's command line. Arguments after `--` are appended
after each test's own arguments in the order given.
This enables passing options to test frameworks like Google Test or
doctest without needing per-test ARGS properties:
ctest -R my_test -- --gtest_filter=SomeTest.Case
ctest --verbose -- --success
An error is raised if `--` is combined with `--build-and-test`, `-S`,
or `-D`/`-T`/`-M` modes, which have their own argument forwarding.
A bare `--` with no following arguments is silently ignored.
Fixes: #20470
Do not run the custom pre-/post- test tasks when instructed to only show
the tests that would be executed. Otherwise it's impossible to reliably
parse the output of this as is will be contaminated by all output
produced by the pre-/post-scripts!
Fixes: #23254
9dc3edbba8 ctest: Restore default test timeout for command-line `-T Test` step
8745e6308e ctest: Restore default of no time limit for command-line `-T Test` step
Acked-by: Kitware Robot <kwrobot@kitware.com>
Merge-request: !11094
In commit 774fcbe49c (CTest: Base command line mode on top of scripting
commands, 2024-10-05, v4.0.0-rc1~653^2) the implementation of this step
was converted to use `ctest_test` internally. That command has its own
default test timeout of 600s if the ctest "TimeOut" configuration value
is not set. If "TimeOut" is not set then default it to our previous
command-line mode default test timeout.
Fixes: #27139
Update commit 712cb8cb14 (ctest: Restore support for -T Test without
dashboard configuration, 2025-03-06, v4.0.0-rc4~11^2) to avoid
conflating coverage of CMake itself with the coverage mode of ctest
being tested.
When `--schedule-random` is used in automated CI jobs, failures may
occur due to test order. We now log the seed. Provide a way for
developers to re-run the same order by specifying the seed.
Fixes: #26760
Co-authored-by: Brad King <brad.king@kitware.com>
Since commit 774fcbe49c (CTest: Base command line mode on top of
scripting commands, 2024-10-05, v4.0.0-rc1~653^2) we drive dashboard
client steps through `cmCTestHandlerCommand::ExecuteHandlerCommand`
instead of calling `ProcessHandler` directly. This requires the
`BuildDirectory` to be known to establish a work directory.
Fixes: #26743
Co-authored-by: Brad King <brad.king@kitware.com>
Make sure that all CMake variables that are translated into
CTest options in `cmCTest*Command` implementations are translated
from CTest options into CMake variables before the functions are
called. This back-and-forth translation should be temporary.
It is a necessary prerequisite for refactoring `cmCTest*Handler`
implementations to operate on CMake variables directly rather
than CTest options.
In a complex test environment, there may be a desire to add a test
property label to indicate context. Here is one such example:
set_property(TEST mygoofytest PROPERTY LABELS might-fail-on-config-change)
Upon looking at the failed test summary, one can quickly decide whether
the failure might be expected or not.
198 - mygoofytest (Failed) might-fail-on-config-change
Arha Gatram
Tyler Yankee
Brad King
Zack Galbreath
Martin Duffy
scivision
Mickaël Germain
Sebastian Sura
Sertonix
AJIOB
Craig Scott
Daniel GoldbergandBrad King
Daniel PfeiferandBrad King
Wim Colgate
castro-ma
Jake D'Esposito