mirrors/cmake
2
0
Fork 0
mirror of https://gitlab.kitware.com/cmake/cmake.git synced 2026-09-25 04:09:36 +03:00
Code Activity
Files
master
cmake/Tests/RunCMake/GoogleTest/RunCMakeTest.cmake
T
Taylor Braun-JonesandTyler Yankee 5cced02b9d CTest: Repeat test fixtures with the tests that require them 
When ctest repeats tests with its --repeat option, it repeats each test
on its own.  A fixture's setup and cleanup tests therefore run all of
their repetitions back to back, and the tests they bracket repeat inside
a single setup/cleanup pair:

  setup -> setup -> test -> test -> cleanup -> cleanup

Add a FIXTURE_REPEAT_MODE test property to select how a fixture behaves
when its tests are repeated:

* AROUND_ALL_REPEATS: the fixture runs once, around all repetitions of
  the tests requiring it.
* AROUND_EACH_REPEAT: the fixture and the tests requiring it repeat
  together, so every repetition gets a fresh setup and its own cleanup.
* EACH_TEST_SEPARATELY: every test repeats on its own, as before.

The property describes the fixture rather than the test carrying it, so
setting it on any one of a fixture's setup or cleanup tests is enough.

In AROUND_EACH_REPEAT mode the tests of a fixture form a repeat group that
ctest re-queues as a whole once every test in it has finished.  The --repeat
condition then applies to the group the way it applies to an individual
test: until-fail repeats while the whole group passes, until-pass repeats
while any of it does not, and after-timeout repeats while any of it times
out.  Fixtures that share a test repeat together, so a test requiring two of
them still runs once per repetition.

A group is recorded the way a repeating test is: only once it stops
repeating, and with the results of its last repetition.  A group that
until-pass makes pass therefore reports a pass rather than the failure that
made it repeat, a test that DEPENDS on one of the group's tests waits for
the last repetition rather than the first, and `ctest -F` resumes an
interrupted group by running it again from the beginning.

Fixtures that repeat together have to agree on the mode: a test cannot
repeat with one fixture but not with another it takes part in, and a fixture
whose setup and cleanup tests disagree has no coherent behavior.  Report an
error and run nothing in those cases rather than pick an order in which a
test repeats after a fixture it requires has been cleaned up.

Add policy CMP0224 to select AROUND_EACH_REPEAT as the default for fixtures
whose setup and cleanup tests choose no mode themselves.  Record the mode
the policy chose in the generated test file under its own
_CMAKE_DEFAULT_FIXTURE_REPEAT_MODE keyword, so that ctest reads a mode
rather than the policy settings behind it, and so that a mode requested on
one of a fixture's tests wins over the default recorded for its siblings.
Only NEW needs recording: with nothing recorded, ctest already uses the
behavior of CMake 4.4 and below.  Fixtures are common, and the choice of
mode matters only to those who run ctest --repeat, so warn about the unset
policy only when the CMAKE_POLICY_WARNING_CMP0224 variable asks for it.

discover_tests() and gtest_discover_tests() create their tests while ctest
runs or at build time, too late for the policy to reach them, so carry the
setting in effect at their call sites through to the tests they create.

Report the repetition a grouped test belongs to in the "(run N/M)" suffix
of its "Start" line, as ctest already does for a test repeating on its own.

Co-authored-by: Tyler Yankee <tyler.yankee@kitware.com>
Fixes: #21438
2026-09-18 08:29:28 -04:00

549 lines
15 KiB
CMake
Raw Permalink Blame History

include(RunCMake)
# Isolate our ctest runs from external environment.
unset(ENV{CTEST_PARALLEL_LEVEL})
unset(ENV{CTEST_OUTPUT_ON_FAILURE})
if(RunCMake_GENERATOR STREQUAL "Borland Makefiles" OR
RunCMake_GENERATOR STREQUAL "Watcom WMake")
set(fs_delay 3)
else()
set(fs_delay 1.125)
endif()
run_cmake(PropertiesOdd)
# Policy CMP0224 has to reach the tests that discovery creates.
run_cmake(FixtureRepeatMode-NEW)
run_cmake(FixtureRepeatMode-OLD)
function(run_GoogleTest DISCOVERY_MODE)
# Use a single build tree for a few tests without cleaning.
set(RunCMake_TEST_BINARY_DIR ${RunCMake_BINARY_DIR}/GoogleTest-build)
if(NOT RunCMake_GENERATOR_IS_MULTI_CONFIG)
set(RunCMake_TEST_OPTIONS -DCMAKE_BUILD_TYPE=Debug)
endif()
run_cmake_with_options(GoogleTest -DCMAKE_GTEST_DISCOVER_TESTS_DISCOVERY_MODE=${DISCOVERY_MODE})
set(RunCMake_TEST_NO_CLEAN 1)
run_cmake_command(GoogleTest-build
${CMAKE_COMMAND}
--build .
--config Debug
--target fake_gtest
)
run_cmake_command(GoogleTest-property-timeout-exe
${CMAKE_COMMAND}
--build .
--config Debug
--target property_timeout_test
)
run_cmake_command(GoogleTest-test1
${CMAKE_CTEST_COMMAND}
-C Debug
-L TEST1
--no-label-summary
)
run_cmake_command(GoogleTest-test2
${CMAKE_CTEST_COMMAND}
-C Debug
-L TEST2
--no-label-summary
)
run_cmake_command(GoogleTest-test3
${CMAKE_CTEST_COMMAND}
-C Debug
-L TEST3
--no-label-summary
)
run_cmake_command(GoogleTest-test4
${CMAKE_CTEST_COMMAND}
-C Debug
-L TEST4
--no-label-summary
)
run_cmake_command(GoogleTest-test5
${CMAKE_CTEST_COMMAND}
-C Debug
-L TEST5
--no-label-summary
)
run_cmake_command(GoogleTest-test6
${CMAKE_CTEST_COMMAND}
-C Debug
-L TEST6
--no-label-summary
)
run_cmake_command(GoogleTest-test7
${CMAKE_CTEST_COMMAND}
-C Debug
-L TEST7
--no-label-summary
)
run_cmake_command(GoogleTest-test8
${CMAKE_CTEST_COMMAND}
-C Debug
-L TEST8
--no-label-summary
)
run_cmake_command(GoogleTest-test9
${CMAKE_CTEST_COMMAND}
-C Debug
-L TEST9
--no-label-summary
)
file(WRITE "${RunCMake_TEST_BINARY_DIR}/test9-names.txt"
"TEST:newline_name/suite.case/trailing_param!9\n"
)
run_cmake_command(GoogleTest-test9-from-file
${CMAKE_CTEST_COMMAND}
-C Debug
--tests-from-file "${RunCMake_TEST_BINARY_DIR}/test9-names.txt"
--no-label-summary
)
run_cmake_command(GoogleTest-test-missing
${CMAKE_CTEST_COMMAND}
-C Debug
-R no_tests_defined
--no-label-summary
)
run_cmake_command(GoogleTest-property-timeout1
${CMAKE_CTEST_COMMAND}
-C Debug
-R property_timeout\\.case_no_discovery
--no-label-summary
)
run_cmake_command(GoogleTest-property-timeout2
${CMAKE_CTEST_COMMAND}
-C Debug
-R property_timeout\\.case_with_discovery
--no-label-summary
)
run_cmake_command(GoogleTest-build
${CMAKE_COMMAND}
--build .
--config Debug
--target skip_test
)
run_cmake_command(GoogleTest-skip-test
${CMAKE_CTEST_COMMAND}
-C Debug
-R skip_test
--no-label-summary
)
endfunction()
function(run_Launcher_CMP0178 DISCOVERY_MODE cmp0178)
if(CMAKE_C_COMPILER_ID STREQUAL "MSVC" AND CMAKE_C_COMPILER_VERSION VERSION_LESS "14.0")
return()
endif()
if(CMAKE_VS_PLATFORM_NAME STREQUAL "ARM64" AND CMAKE_C_COMPILER_ID STREQUAL "MSVC" AND CMAKE_C_COMPILER_VERSION VERSION_LESS "19.36")
return()
endif()
# Use a single build tree for a few tests without cleaning.
set(RunCMake_TEST_BINARY_DIR ${RunCMake_BINARY_DIR}/Launcher-CMP0178-${cmp0178}-build)
if(NOT RunCMake_GENERATOR_IS_MULTI_CONFIG)
set(RunCMake_TEST_OPTIONS -DCMAKE_BUILD_TYPE=Debug)
endif()
run_cmake_with_options(Launcher-CMP0178-${cmp0178}
-DCMAKE_GTEST_DISCOVER_TESTS_DISCOVERY_MODE=${DISCOVERY_MODE}
)
set(RunCMake_TEST_NO_CLEAN 1)
# do not issue any warnings on stderr that would cause the build to fail
set(RunCMake_TEST_OUTPUT_MERGE 1)
run_cmake_command(Launcher-CMP0178-${cmp0178}-build
${CMAKE_COMMAND}
--build .
--config Debug
)
unset(RunCMake_TEST_OUTPUT_MERGE)
run_cmake_command(Launcher-CMP0178-${cmp0178}-test
${CMAKE_CTEST_COMMAND}
-C Debug
-V
--no-label-summary
)
endfunction()
function(run_GoogleTestXML DISCOVERY_MODE)
# Use a single build tree for a few tests without cleaning.
set(RunCMake_TEST_BINARY_DIR ${RunCMake_BINARY_DIR}/GoogleTestXML-build)
if(NOT RunCMake_GENERATOR_IS_MULTI_CONFIG)
set(RunCMake_TEST_OPTIONS -DCMAKE_BUILD_TYPE=Debug)
endif()
run_cmake_with_options(GoogleTestXML -DCMAKE_GTEST_DISCOVER_TESTS_DISCOVERY_MODE=${DISCOVERY_MODE})
set(RunCMake_TEST_NO_CLEAN 1)
run_cmake_command(GoogleTestXML-discovery
${CMAKE_COMMAND}
--build .
--config Debug
--target xml_output
)
run_cmake_command(GoogleTestXML-result
${CMAKE_CTEST_COMMAND}
-C Debug
-R GoogleTestXML
--no-label-summary
)
run_cmake_command(GoogleTestXML-special-result
${CMAKE_CTEST_COMMAND}
-C Debug
-R GoogleTestXMLSpecial
--no-label-summary
)
endfunction()
function(run_GoogleTest_discovery_timeout DISCOVERY_MODE)
# Use a single build tree for a few tests without cleaning.
set(RunCMake_TEST_BINARY_DIR ${RunCMake_BINARY_DIR}/GoogleTest-discovery-timeout)
if(NOT RunCMake_GENERATOR_IS_MULTI_CONFIG)
set(RunCMake_TEST_OPTIONS -DCMAKE_BUILD_TYPE=Debug)
endif()
run_cmake_with_options(GoogleTestDiscoveryTimeout -DDISCOVERY_MODE=${DISCOVERY_MODE})
set(RunCMake_TEST_NO_CLEAN 1)
set(RunCMake_TEST_OUTPUT_MERGE 1)
run_cmake_command(GoogleTest-discovery-${DISCOVERY_MODE}-timeout-build
${CMAKE_COMMAND}
--build .
--config Debug
--target discovery_timeout_test
)
set(RunCMake_TEST_OUTPUT_MERGE 0)
run_cmake_command(GoogleTest-discovery-${DISCOVERY_MODE}-timeout-test
${CMAKE_CTEST_COMMAND}
-C Debug
-R discovery_timeout_test
--no-label-summary
)
endfunction()
function(run_GoogleTest_discovery_arg_change DISCOVERY_MODE)
# Use a single build tree for a few tests without cleaning.
set(RunCMake_TEST_BINARY_DIR ${RunCMake_BINARY_DIR}/GoogleTest-discovery-arg-change)
run_cmake_with_options(GoogleTestDiscoveryArgChange
-DCMAKE_GTEST_DISCOVER_TESTS_DISCOVERY_MODE=${DISCOVERY_MODE}
-DTEST_FILTER=basic
)
set(RunCMake_TEST_NO_CLEAN 1)
run_cmake_command(GoogleTest-discovery-arg-change-build
${CMAKE_COMMAND}
--build .
--config Release
--target fake_gtest
)
run_cmake_command(GoogleTest-discovery-arg-change-basic
${CMAKE_CTEST_COMMAND}
-C Release
-N
)
execute_process(COMMAND ${CMAKE_COMMAND} -E sleep ${fs_delay}) # handle 1s resolution
run_cmake_with_options(GoogleTestDiscoveryArgChange
-DCMAKE_GTEST_DISCOVER_TESTS_DISCOVERY_MODE=${DISCOVERY_MODE}
-DTEST_FILTER=typed
)
run_cmake_command(GoogleTest-discovery-arg-change-build
${CMAKE_COMMAND}
--build .
--config Release
--target fake_gtest
)
run_cmake_command(GoogleTest-discovery-arg-change-typed
${CMAKE_CTEST_COMMAND}
-C Release
-N
)
endfunction()
function(run_GoogleTest_discovery_multi_config)
# Use a single build tree for a few tests without cleaning.
set(RunCMake_TEST_BINARY_DIR ${RunCMake_BINARY_DIR}/GoogleTest-discovery-multi-config)
run_cmake(GoogleTestDiscoveryMultiConfig)
set(RunCMake_TEST_NO_CLEAN 1)
run_cmake_command(GoogleTest-build-release
${CMAKE_COMMAND}
--build .
--config Release
--target configuration_gtest
)
run_cmake_command(GoogleTest-build-debug
${CMAKE_COMMAND}
--build .
--config Debug
--target configuration_gtest
)
run_cmake_command(GoogleTest-configuration-release
${CMAKE_CTEST_COMMAND}
-C Release
-L CONFIG
-N
)
run_cmake_command(GoogleTest-configuration-debug
${CMAKE_CTEST_COMMAND}
-C Debug
-L CONFIG
-N
)
endfunction()
function(run_GoogleTest_discovery_test_list DISCOVERY_MODE)
# Use a single build tree for a few tests without cleaning.
set(RunCMake_TEST_BINARY_DIR ${RunCMake_BINARY_DIR}/GoogleTest-discovery-test-list-build)
if(NOT RunCMake_GENERATOR_IS_MULTI_CONFIG)
set(RunCMake_TEST_OPTIONS -DCMAKE_BUILD_TYPE=Debug)
endif()
run_cmake_with_options(GoogleTestDiscoveryTestList -DCMAKE_GTEST_DISCOVER_TESTS_DISCOVERY_MODE=${DISCOVERY_MODE})
set(RunCMake_TEST_NO_CLEAN 1)
run_cmake_command(GoogleTest-discovery-test-list-build
${CMAKE_COMMAND}
--build .
--config Debug
--target test_list_test
)
run_cmake_command(GoogleTest-discovery-test-list-test
${CMAKE_CTEST_COMMAND}
-C Debug
--no-label-summary
)
endfunction()
function(run_GoogleTest_discovery_flush_script DISCOVERY_MODE)
# Use a single build tree for a few tests without cleaning.
set(RunCMake_TEST_BINARY_DIR ${RunCMake_BINARY_DIR}/GoogleTest-discovery-flush-script-build)
if(NOT RunCMake_GENERATOR_IS_MULTI_CONFIG)
set(RunCMake_TEST_OPTIONS -DCMAKE_BUILD_TYPE=Debug)
endif()
run_cmake_with_options(GoogleTestDiscoveryFlushScript -DCMAKE_GTEST_DISCOVER_TESTS_DISCOVERY_MODE=${DISCOVERY_MODE})
set(RunCMake_TEST_NO_CLEAN 1)
run_cmake_command(GoogleTest-discovery-flush-script-build
${CMAKE_COMMAND}
--build .
--config Debug
--target flush_script_test
)
run_cmake_command(GoogleTest-discovery-flush-script-test
${CMAKE_CTEST_COMMAND}
-C Debug
--no-label-summary
)
endfunction()
function(run_GoogleTest_discovery_post_build_race)
# Use a single build tree for this test without cleaning.
set(RunCMake_TEST_BINARY_DIR ${RunCMake_BINARY_DIR}/GoogleTest-discovery-post-build-race)
if(NOT RunCMake_GENERATOR_IS_MULTI_CONFIG)
set(RunCMake_TEST_OPTIONS -DCMAKE_BUILD_TYPE=Debug)
endif()
run_cmake(GoogleTestDiscoveryPostBuildRace)
set(RunCMake_TEST_NO_CLEAN 1)
run_cmake_command(GoogleTest-discovery-post-build-race-build
${CMAKE_COMMAND}
--build .
--config Debug
-j 16
--target check_discovery_json_files
)
endfunction()
function(run_GoogleTest_discovery_test_list_scoped DISCOVERY_MODE)
# Use a single build tree for a few tests without cleaning.
set(RunCMake_TEST_BINARY_DIR ${RunCMake_BINARY_DIR}/GoogleTest-discovery-test-list-scoped-build)
if(NOT RunCMake_GENERATOR_IS_MULTI_CONFIG)
set(RunCMake_TEST_OPTIONS -DCMAKE_BUILD_TYPE=Debug)
endif()
run_cmake_with_options(GoogleTestDiscoveryTestListScoped -DCMAKE_GTEST_DISCOVER_TESTS_DISCOVERY_MODE=${DISCOVERY_MODE})
set(RunCMake_TEST_NO_CLEAN 1)
run_cmake_command(GoogleTest-discovery-test-list-scoped-build
${CMAKE_COMMAND}
--build .
--config Debug
--target test_list_scoped_test
)
run_cmake_command(GoogleTest-discovery-test-list-scoped-test
${CMAKE_CTEST_COMMAND}
-C Debug
--no-label-summary
)
endfunction()
function(run_GoogleTest_discovery_test_list_extra_args DISCOVERY_MODE)
# Use a single build tree for a few tests without cleaning.
set(RunCMake_TEST_BINARY_DIR ${RunCMake_BINARY_DIR}/GoogleTest-discovery-test-list-extra-args-build)
if(NOT RunCMake_GENERATOR_IS_MULTI_CONFIG)
set(RunCMake_TEST_OPTIONS -DCMAKE_BUILD_TYPE=Debug)
endif()
run_cmake_with_options(GoogleTestDiscoveryTestListExtraArgs -DCMAKE_GTEST_DISCOVER_TESTS_DISCOVERY_MODE=${DISCOVERY_MODE})
set(RunCMake_TEST_NO_CLEAN 1)
run_cmake_command(GoogleTest-discovery-test-list-extra-args-build
${CMAKE_COMMAND}
--build .
--config Debug
--target test_list_extra_args
)
run_cmake_command(GoogleTest-discovery-test-list-extra-args-test
${CMAKE_CTEST_COMMAND}
-C Debug
--no-label-summary
)
endfunction()
function(run_GoogleTest_LegacyParser)
# Use a single build tree for a few tests without cleaning.
set(RunCMake_TEST_BINARY_DIR ${RunCMake_BINARY_DIR}/GoogleTestLegacyParser-build)
if(NOT RunCMake_GENERATOR_IS_MULTI_CONFIG)
set(RunCMake_TEST_OPTIONS -DCMAKE_BUILD_TYPE=Debug)
endif()
set(ENV{NO_GTEST_JSON_OUTPUT} 1)
run_cmake(GoogleTestLegacyParser)
set(RunCMake_TEST_NO_CLEAN 1)
run_cmake_command(GoogleTestLegacyParser-build
${CMAKE_COMMAND}
--build .
--config Debug
--target fake_gtest
)
set(RunCMake-stdout-file GoogleTest-test1-stdout.txt)
run_cmake_command(GoogleTestLegacyParser-test
${CMAKE_CTEST_COMMAND}
-C Debug
--no-label-summary
)
unset(ENV{NO_GTEST_JSON_OUTPUT})
endfunction()
function(run_GoogleTest_DEF_SOURCE_LINE)
# Use a single build tree for a few tests without cleaning.
set(RunCMake_TEST_BINARY_DIR ${RunCMake_BINARY_DIR}/GoogleTest-DEF_SOURCE_LINE-build)
if(NOT RunCMake_GENERATOR_IS_MULTI_CONFIG)
set(RunCMake_TEST_OPTIONS -DCMAKE_BUILD_TYPE=Debug)
endif()
run_cmake(GoogleTestDefSourceLine)
set(RunCMake_TEST_NO_CLEAN 1)
run_cmake_command(GoogleTest-DEF_SOURCE_LINE-build
${CMAKE_COMMAND}
--build .
--config Debug
--target fake_gtest
)
run_cmake_command(GoogleTest-DEF_SOURCE_LINE
${CMAKE_CTEST_COMMAND}
-C Debug
-R "^basic\\.case_"
--show-only=json-v1
)
endfunction()
function(run_GoogleTest_discovery_duplicate)
# Use a single build tree for a few tests without cleaning.
set(RunCMake_TEST_BINARY_DIR ${RunCMake_BINARY_DIR}/GoogleTest-discovery-duplicate-build)
if(NOT RunCMake_GENERATOR_IS_MULTI_CONFIG)
set(RunCMake_TEST_OPTIONS -DCMAKE_BUILD_TYPE=Debug)
endif()
run_cmake(GoogleTestDiscoveryDuplicate)
set(RunCMake_TEST_NO_CLEAN 1)
set(RunCMake_TEST_OUTPUT_MERGE 1)
run_cmake_command(GoogleTest-discovery-duplicate-build
${CMAKE_COMMAND}
--build .
--config Debug
--target fake_gtest
)
unset(RunCMake_TEST_OUTPUT_MERGE)
endfunction()
foreach(DISCOVERY_MODE POST_BUILD PRE_TEST)
message(STATUS "Testing ${DISCOVERY_MODE} discovery mode via CMAKE_GTEST_DISCOVER_TESTS_DISCOVERY_MODE global override...")
run_GoogleTest(${DISCOVERY_MODE})
run_GoogleTestXML(${DISCOVERY_MODE})
run_Launcher_CMP0178(${DISCOVERY_MODE} NEW)
run_Launcher_CMP0178(${DISCOVERY_MODE} OLD)
run_Launcher_CMP0178(${DISCOVERY_MODE} WARN)
message(STATUS "Testing ${DISCOVERY_MODE} discovery mode via DISCOVERY_MODE option...")
run_GoogleTest_discovery_timeout(${DISCOVERY_MODE})
run_GoogleTest_discovery_arg_change(${DISCOVERY_MODE})
run_GoogleTest_discovery_test_list(${DISCOVERY_MODE})
run_GoogleTest_discovery_test_list_scoped(${DISCOVERY_MODE})
run_GoogleTest_discovery_test_list_extra_args(${DISCOVERY_MODE})
run_GoogleTest_discovery_flush_script(${DISCOVERY_MODE})
endforeach()
if(RunCMake_GENERATOR_IS_MULTI_CONFIG)
message(STATUS "Testing PRE_TEST discovery multi configuration...")
run_GoogleTest_discovery_multi_config()
endif()
block(SCOPE_FOR VARIABLES)
# Use a single build tree for a few tests without cleaning.
set(RunCMake_TEST_BINARY_DIR ${RunCMake_BINARY_DIR}/WorkDirWithSpaces-build)
if(NOT RunCMake_GENERATOR_IS_MULTI_CONFIG)
set(RunCMake_TEST_OPTIONS -DCMAKE_BUILD_TYPE=Debug)
endif()
run_cmake(WorkDirWithSpaces)
set(RunCMake_TEST_NO_CLEAN 1)
run_cmake_command(WorkDirWithSpaces-build
${CMAKE_COMMAND}
--build .
--config Debug
--target test_workdir
)
run_cmake_command(WorkDirWithSpaces-test
${CMAKE_CTEST_COMMAND}
-C Debug
--no-label-summary
--output-on-failure
)
endblock()
run_GoogleTest_LegacyParser()
run_GoogleTest_DEF_SOURCE_LINE()
# This test is meaningful only for generators which support parallel builds.
if (NOT RunCMake_GENERATOR MATCHES "(Borland|NMake|Watcom)")
run_GoogleTest_discovery_post_build_race()
endif()
run_GoogleTest_discovery_duplicate()
View Git Blame Copy Permalink