diff --git a/Help/command/discover_tests.rst b/Help/command/discover_tests.rst index 6ef6b66ff2..2b46a80d4c 100644 --- a/Help/command/discover_tests.rst +++ b/Help/command/discover_tests.rst @@ -89,6 +89,11 @@ the provided regular expression and replacement strings. replacement strings and may reference capture groups from ``DISCOVERY_MATCH``. + .. versionchanged:: 4.5 + If the properties make the discovered tests part of a test fixture, and + they do not set :prop_test:`FIXTURE_REPEAT_MODE`, policy :policy:`CMP0224` + determines the behavior when the :ctest-option:`--repeat` option is used. + CTest executes the discovery step to obtain the list of tests and then runs each discovered test using the command-line produced by ``COMMAND`` together with ``TEST_ARGS``. The pass/fail behavior of each discovered test follows diff --git a/Help/manual/cmake-policies.7.rst b/Help/manual/cmake-policies.7.rst index d311e53a75..9be50d662e 100644 --- a/Help/manual/cmake-policies.7.rst +++ b/Help/manual/cmake-policies.7.rst @@ -100,6 +100,7 @@ Policies Introduced by CMake 4.5 .. toctree:: :maxdepth: 1 + CMP0224: Test fixtures use AROUND_EACH_REPEAT mode by default with ctest --repeat. CMP0223: An empty path is not a prefix of any path. CMP0222: The if() command supports path prefix tests using PATH_IS_PREFIX operator. CMP0221: cmake_host_system_information() DISTRIB_* queries read the host os-release. diff --git a/Help/manual/cmake-properties.7.rst b/Help/manual/cmake-properties.7.rst index 97d0073b0e..f06bfa52ca 100644 --- a/Help/manual/cmake-properties.7.rst +++ b/Help/manual/cmake-properties.7.rst @@ -579,6 +579,7 @@ Properties on Tests /prop_test/ENVIRONMENT /prop_test/ENVIRONMENT_MODIFICATION /prop_test/FAIL_REGULAR_EXPRESSION + /prop_test/FIXTURE_REPEAT_MODE /prop_test/FIXTURES_CLEANUP /prop_test/FIXTURES_REQUIRED /prop_test/FIXTURES_SETUP diff --git a/Help/policy/CMP0224.rst b/Help/policy/CMP0224.rst new file mode 100644 index 0000000000..0b44bea6de --- /dev/null +++ b/Help/policy/CMP0224.rst @@ -0,0 +1,46 @@ +CMP0224 +------- + +.. versionadded:: 4.5 + +Test fixtures use ``AROUND_EACH_REPEAT`` mode by default with +:option:`ctest --repeat`. + +When :manual:`ctest(1)` repeats tests with its :ctest-option:`--repeat` +option, CMake 4.4 and below repeated fixture setup and cleanup tests along +with the tests that require them. Each test, including the fixture tests, +ran all of its repetitions before the next test started:: + + setup (xN) -> test (xN) -> cleanup (xN) + +CMake 4.5 and above prefer to repeat the fixture as a whole, so that each +repetition of a test is surrounded by its own fixture setup and cleanup:: + + (setup -> test -> cleanup) xN + +This policy provides compatibility with projects that rely on the old +behavior. It determines the default behavior when the +:prop_test:`FIXTURE_REPEAT_MODE` test property is not set. The ``OLD`` +behavior for this policy is ``EACH_TEST_SEPARATELY``. The ``NEW`` behavior +for this policy is ``AROUND_EACH_REPEAT``. + +The property describes a fixture rather than one test, so setting it on any +one of a fixture's setup or cleanup tests is enough to make this policy +irrelevant to that fixture. + +Tests are not always created during the configure step by :command:`add_test`. +The :command:`discover_tests` and :command:`gtest_discover_tests` commands +defer test discovery to the build or test step, but they apply the policy +setting as recorded when they were called. If a project adds tests through +its own script listed in :prop_dir:`TEST_INCLUDE_FILES`, it should set +:prop_test:`FIXTURE_REPEAT_MODE` on its fixture tests explicitly. + +.. |INTRODUCED_IN_CMAKE_VERSION| replace:: 4.5 +.. |WARNS_OR_DOES_NOT_WARN| replace:: does *not* warn by default +.. include:: include/STANDARD_ADVICE.rst + +See documentation of the +:variable:`CMAKE_POLICY_WARNING_CMP0224 >` +variable to control the warning. + +.. include:: include/DEPRECATED.rst diff --git a/Help/prop_test/FIXTURES_CLEANUP.rst b/Help/prop_test/FIXTURES_CLEANUP.rst index aa043da467..80ec421be4 100644 --- a/Help/prop_test/FIXTURES_CLEANUP.rst +++ b/Help/prop_test/FIXTURES_CLEANUP.rst @@ -20,10 +20,10 @@ primary effects: set require any fixture listed in ``FIXTURES_CLEANUP``. A cleanup test can have multiple fixtures listed in its ``FIXTURES_CLEANUP`` -property. It will execute only once for the whole CTest run, not once for each -fixture. A fixture can also have more than one cleanup test defined. If there -are multiple cleanup tests for a fixture, projects can control their order with -the usual :prop_test:`DEPENDS` test property if necessary. +property. It will execute once for all of them, not once for each fixture. A +fixture can also have more than one cleanup test defined. If there are multiple +cleanup tests for a fixture, projects can control their order with the usual +:prop_test:`DEPENDS` test property if necessary. A cleanup test is allowed to require other fixtures, but not any fixture listed in its ``FIXTURES_CLEANUP`` property. For example: @@ -45,5 +45,11 @@ in its ``FIXTURES_CLEANUP`` property. For example: Cleanup tests will execute even if setup or regular tests for that fixture fail or are skipped. +.. versionchanged:: 4.5 + + The :prop_test:`FIXTURE_REPEAT_MODE` property selects how a fixture behaves + when :manual:`ctest(1)` repeats tests with :ctest-option:`--repeat`. See + also :policy:`CMP0224`. + See :prop_test:`FIXTURES_REQUIRED` for a more complete discussion of how to use test fixtures. diff --git a/Help/prop_test/FIXTURES_REQUIRED.rst b/Help/prop_test/FIXTURES_REQUIRED.rst index 7a056012b7..e89f911d61 100644 --- a/Help/prop_test/FIXTURES_REQUIRED.rst +++ b/Help/prop_test/FIXTURES_REQUIRED.rst @@ -26,6 +26,12 @@ behavior can be overridden with the :option:`-FS `, :option:`-FC ` and :option:`-FA ` command line options to :manual:`ctest(1)` if desired. +.. versionchanged:: 4.5 + + The :prop_test:`FIXTURE_REPEAT_MODE` property selects how a fixture behaves + when :manual:`ctest(1)` repeats tests with :ctest-option:`--repeat`. See + also :policy:`CMP0224`. + Since setup and cleanup tasks are also tests, they can have an ordering specified by the :prop_test:`DEPENDS` test property just like any other tests. This can be exploited to implement setup or cleanup using multiple tests for a diff --git a/Help/prop_test/FIXTURES_SETUP.rst b/Help/prop_test/FIXTURES_SETUP.rst index 04a09d86cc..20e4715c33 100644 --- a/Help/prop_test/FIXTURES_SETUP.rst +++ b/Help/prop_test/FIXTURES_SETUP.rst @@ -20,10 +20,10 @@ primary effects: set require any fixture listed in ``FIXTURES_SETUP``. A setup test can have multiple fixtures listed in its ``FIXTURES_SETUP`` -property. It will execute only once for the whole CTest run, not once for each -fixture. A fixture can also have more than one setup test defined. If there are -multiple setup tests for a fixture, projects can control their order with the -usual :prop_test:`DEPENDS` test property if necessary. +property. It will execute once for all of them, not once for each fixture. A +fixture can also have more than one setup test defined. If there are multiple +setup tests for a fixture, projects can control their order with the usual +:prop_test:`DEPENDS` test property if necessary. A setup test is allowed to require other fixtures, but not any fixture listed in its ``FIXTURES_SETUP`` property. For example: @@ -46,5 +46,11 @@ If any of a fixture's setup tests fail, none of the tests listing that fixture in its :prop_test:`FIXTURES_REQUIRED` property will be run. Cleanup tests will, however, still be executed. +.. versionchanged:: 4.5 + + The :prop_test:`FIXTURE_REPEAT_MODE` property selects how a fixture behaves + when :manual:`ctest(1)` repeats tests with :ctest-option:`--repeat`. See + also :policy:`CMP0224`. + See :prop_test:`FIXTURES_REQUIRED` for a more complete discussion of how to use test fixtures. diff --git a/Help/prop_test/FIXTURE_REPEAT_MODE.rst b/Help/prop_test/FIXTURE_REPEAT_MODE.rst new file mode 100644 index 0000000000..20a0b6e0c0 --- /dev/null +++ b/Help/prop_test/FIXTURE_REPEAT_MODE.rst @@ -0,0 +1,87 @@ +FIXTURE_REPEAT_MODE +------------------- + +.. versionadded:: 4.5 + +Specifies how a test fixture behaves when :manual:`ctest(1)` repeats tests +with its :ctest-option:`--repeat` option. + +The property describes the fixture rather than the test carrying it, so +setting it on any one of the fixture's setup or cleanup tests is enough. It +has no effect on a test that has neither a :prop_test:`FIXTURES_SETUP` nor a +:prop_test:`FIXTURES_CLEANUP` property, although :manual:`ctest(1)` still +rejects a value it does not recognize. + +The value must be one of the following: + +``AROUND_ALL_REPEATS`` + The fixture runs once, around all repetitions of the tests requiring it:: + + setup -> test -> test -> test -> cleanup + +``AROUND_EACH_REPEAT`` + The fixture repeats with the tests requiring it, so that every repetition + of a test gets a fresh setup and its own cleanup:: + + setup -> test -> cleanup -> setup -> test -> cleanup -> ... + +``EACH_TEST_SEPARATELY`` + No test is treated specially: every test, including the fixture's own setup + and cleanup tests, runs all of its repetitions before the next test starts:: + + setup -> setup -> setup -> test -> test -> test -> cleanup -> ... + + This is the behavior of CMake 4.4 and below. + +If the property is not set on any of a fixture's setup or cleanup tests, the +behavior is determined by policy :policy:`CMP0224`. + +Fixtures that repeat together must agree on the mode. All of a fixture's own +setup and cleanup tests must request the same mode, and so must any two +fixtures that share a test, since one test cannot repeat with one fixture and +not with another it takes part in. :manual:`ctest(1)` reports an error and +runs nothing if they disagree. + +In ``AROUND_EACH_REPEAT`` mode the fixture's tests repeat as a unit, so +:ctest-option:`--repeat` applies its condition to the unit as a whole. +``until-fail`` repeats while every test in the unit passes, ``until-pass`` +repeats while any of them does not pass, and ``after-timeout`` repeats while +any of them times out. A test that requires two fixtures in +``AROUND_EACH_REPEAT`` mode makes those fixtures repeat together. + +Only the last repetition of the unit is reported, as for a test repeating on +its own, and a test that lists one of the unit's tests in its +:prop_test:`DEPENDS` property runs after that last repetition. If +:manual:`ctest(1)` is interrupted part-way through the repetitions, +:ctest-option:`-F` resumes by running the whole unit again. + +Example +^^^^^^^ + +.. code-block:: cmake + + add_test(NAME start_server COMMAND start_server) + set_tests_properties(start_server PROPERTIES + FIXTURES_SETUP Server + FIXTURE_REPEAT_MODE AROUND_EACH_REPEAT) + + add_test(NAME query_server COMMAND query_server) + set_tests_properties(query_server PROPERTIES FIXTURES_REQUIRED Server) + + add_test(NAME stop_server COMMAND stop_server) + set_tests_properties(stop_server PROPERTIES FIXTURES_CLEANUP Server) + +With ``ctest --repeat until-fail:3``, this runs a fresh server for each +attempt at ``query_server``:: + + start_server -> query_server -> stop_server + start_server -> query_server -> stop_server + start_server -> query_server -> stop_server + +See Also +^^^^^^^^ + +* :prop_test:`FIXTURES_SETUP` +* :prop_test:`FIXTURES_CLEANUP` +* :prop_test:`FIXTURES_REQUIRED` +* :policy:`CMP0224` diff --git a/Help/release/dev/ctest-fixture-repeat.rst b/Help/release/dev/ctest-fixture-repeat.rst new file mode 100644 index 0000000000..8c19996589 --- /dev/null +++ b/Help/release/dev/ctest-fixture-repeat.rst @@ -0,0 +1,6 @@ +ctest-fixture-repeat +-------------------- + +* The :prop_test:`FIXTURE_REPEAT_MODE` test property was added to select how + a test fixture behaves when :manual:`ctest(1)` repeats tests with its + :ctest-option:`--repeat` option. See policy :policy:`CMP0224`. diff --git a/Help/variable/CMAKE_POLICY_WARNING_CMPNNNN.rst b/Help/variable/CMAKE_POLICY_WARNING_CMPNNNN.rst index f0b1ed6077..f6872ceedf 100644 --- a/Help/variable/CMAKE_POLICY_WARNING_CMPNNNN.rst +++ b/Help/variable/CMAKE_POLICY_WARNING_CMPNNNN.rst @@ -44,6 +44,8 @@ only for the policies that do not warn by default: policy :policy:`CMP0206`. * ``CMAKE_POLICY_WARNING_CMP0220`` controls the warning for policy :policy:`CMP0220`. +* ``CMAKE_POLICY_WARNING_CMP0224`` controls the warning for + policy :policy:`CMP0224`. This variable should not be set by a project in CMake code. Project developers running CMake may set this variable in their cache to diff --git a/Modules/GoogleTest.cmake b/Modules/GoogleTest.cmake index 9f836f06cf..8b144892f3 100644 --- a/Modules/GoogleTest.cmake +++ b/Modules/GoogleTest.cmake @@ -256,6 +256,11 @@ same as the Google Test name (i.e. ``suite.testcase``); see also Specifies additional properties to be set on all tests discovered by this invocation of ``gtest_discover_tests()``. + .. versionchanged:: 4.5 + If the properties make the discovered tests part of a test fixture, and + they do not set :prop_test:`FIXTURE_REPEAT_MODE`, policy :policy:`CMP0224` + determines the behavior when the :ctest-option:`--repeat` option is used. + ``TEST_LIST var`` Make the list of tests available in the variable ``var``, rather than the default ``_TESTS``. This can be useful when the same test @@ -595,6 +600,17 @@ function(gtest_discover_tests target) endif() endif() + # For policy CMP0224, only NEW needs carrying through. With nothing + # recorded, ctest already uses the EACH_TEST_SEPARATELY behavior of + # CMake 4.4 and below. + cmake_policy(GET CMP0224 _CMP0224 + PARENT_SCOPE # undocumented, do not use outside of CMake + ) + if(_CMP0224 STREQUAL "NEW") + list(APPEND arg_PROPERTIES + _CMAKE_DEFAULT_FIXTURE_REPEAT_MODE AROUND_EACH_REPEAT) + endif() + get_property(test_launcher TARGET ${target} PROPERTY TEST_LAUNCHER diff --git a/Source/CTest/cmCTestMultiProcessHandler.cxx b/Source/CTest/cmCTestMultiProcessHandler.cxx index ea51ccfea3..268b0070be 100644 --- a/Source/CTest/cmCTestMultiProcessHandler.cxx +++ b/Source/CTest/cmCTestMultiProcessHandler.cxx @@ -9,11 +9,15 @@ #include // IWYU pragma: keep #include #include +#include #include #include #include +#include +#include #include #include +#include #include #include #include @@ -21,6 +25,7 @@ #include #include #include +#include #include #include @@ -42,6 +47,7 @@ #include "cmStringAlgorithms.h" #include "cmSystemTools.h" #include "cmUVJobServerClient.h" +#include "cmUnreachable.h" #include "cmWorkingDirectory.h" namespace { @@ -95,6 +101,36 @@ cm::optional splitCostLine(cm::string_view line) return CostEntry{ line.substr(0, pos1), prev, cost }; } +std::string FixtureRepeatModeString(cmCTestTestHandler::FixtureRepeatMode mode) +{ + switch (mode) { + case cmCTestTestHandler::FixtureRepeatMode::AroundAllRepeats: + return "AROUND_ALL_REPEATS"; + case cmCTestTestHandler::FixtureRepeatMode::AroundEachRepeat: + return "AROUND_EACH_REPEAT"; + case cmCTestTestHandler::FixtureRepeatMode::EachTestSeparately: + return "EACH_TEST_SEPARATELY"; + } + CM_UNREACHABLE; +} + +// A fixture and everything ctest needs to decide how it repeats. +struct Fixture +{ + // The fixture's own setup and cleanup tests. + std::set SetupAndCleanupTests; + // Those tests and the tests requiring the fixture, which together are + // everything that repeats with it in AROUND_EACH_REPEAT mode. + std::set AllTests; + // The modes the setup and cleanup tests request. Each is mapped to one + // test that requests it, so that a report of conflicting modes can name + // the tests responsible. + std::map RequestedModes; + // The mode CMake recorded as the CMP0224 default, taken from the first + // setup or cleanup test that carries one. + cm::optional DefaultMode; +}; + } namespace cmsys { @@ -144,7 +180,8 @@ bool cmCTestMultiProcessHandler::SetTests(TestMap tests, this->HasCycles = !this->CheckCycles(); this->HasInvalidGeneratedResourceSpec = !this->CheckGeneratedResourceSpec(); - if (this->HasCycles || this->HasInvalidGeneratedResourceSpec) { + if (this->HasCycles || this->HasInvalidGeneratedResourceSpec || + !this->ComputeFixtureRepetition()) { return false; } this->CreateTestCostList(); @@ -152,6 +189,158 @@ bool cmCTestMultiProcessHandler::SetTests(TestMap tests, return true; } +bool cmCTestMultiProcessHandler::ComputeFixtureRepetition() +{ + using Mode = cmCTestTestHandler::FixtureRepeatMode; + + if (this->RepeatMode == cmCTest::Repeat::Never || this->RepeatCount <= 1) { + return true; + } + + // Collect the tests taking part in each fixture and the modes its own + // setup and cleanup tests ask for. + std::map fixtures; + for (auto const& p : this->Properties) { + int const test = p.first; + auto const& props = *p.second; + for (std::string const& name : props.FixturesRequired) { + fixtures[name].AllTests.insert(test); + } + for (std::set const* own : + { &props.FixturesSetup, &props.FixturesCleanup }) { + for (std::string const& name : *own) { + Fixture& fixture = fixtures[name]; + fixture.SetupAndCleanupTests.insert(test); + fixture.AllTests.insert(test); + if (props.RequestedFixtureRepeatMode) { + fixture.RequestedModes.emplace(*props.RequestedFixtureRepeatMode, + test); + } else if (!fixture.DefaultMode) { + fixture.DefaultMode = props.DefaultFixtureRepeatMode; + } + } + } + } + + // Resolve each fixture's mode. A mode requested by one of its own tests + // applies to the whole fixture, so its own tests must agree. + std::map fixtureMode; + for (auto const& fi : fixtures) { + Fixture const& fixture = fi.second; + if (fixture.RequestedModes.size() > 1) { + std::string e = + cmStrCat("Error: the setup and cleanup tests of fixture \"", fi.first, + "\" request conflicting FIXTURE_REPEAT_MODE values:\n"); + for (auto const& r : fixture.RequestedModes) { + e += cmStrCat(" \"", this->GetName(r.second), "\" requests ", + FixtureRepeatModeString(r.first), '\n'); + } + e += "All setup and cleanup tests of a fixture must request the same " + "mode.\n"; + cmCTestLog(this->CTest, ERROR_MESSAGE, e); + return false; + } + fixtureMode[fi.first] = !fixture.RequestedModes.empty() + ? fixture.RequestedModes.begin()->first + : fixture.DefaultMode.value_or(Mode::EachTestSeparately); + } + + // Fixtures that share a test repeat together, so they must agree as well. + // Otherwise a test could repeat with one fixture after another fixture it + // takes part in has been cleaned up. + for (auto const& p : this->Properties) { + auto const& props = *p.second; + std::map modes; + for (std::set const* used : + { &props.FixturesSetup, &props.FixturesCleanup, + &props.FixturesRequired }) { + for (std::string const& name : *used) { + modes.emplace(fixtureMode[name], name); + } + } + if (modes.size() > 1) { + std::string e = cmStrCat("Error: test \"", this->GetName(p.first), + "\" takes part in fixtures with conflicting " + "FIXTURE_REPEAT_MODE values:\n"); + for (auto const& m : modes) { + e += cmStrCat(" fixture \"", m.second, "\" uses ", + FixtureRepeatModeString(m.first), '\n'); + } + e += "Fixtures that share a test must use the same mode.\n"; + cmCTestLog(this->CTest, ERROR_MESSAGE, e); + return false; + } + } + + // Record what the resolved modes ask of each fixture's tests. + for (auto const& fi : fixtures) { + switch (fixtureMode[fi.first]) { + case Mode::AroundAllRepeats: + this->TestsRunOnce.insert(fi.second.SetupAndCleanupTests.begin(), + fi.second.SetupAndCleanupTests.end()); + break; + case Mode::AroundEachRepeat: + this->AddRepeatGroup(fi.second.AllTests); + break; + case Mode::EachTestSeparately: + break; + } + } + + // Record what each group needs to repeat itself. + for (auto& group : this->RepeatGroups) { + group.RepetitionsLeft = this->RepeatCount - 1; + group.Unfinished = group.Tests.size(); + for (auto& t : group.Tests) { + // Keep only dependencies within the group. Dependencies on other + // tests are satisfied once, by the group's first repetition. + for (int depend : this->PendingTests[t.first].Depends) { + if (group.Tests.count(depend) != 0) { + t.second.insert(depend); + } + } + } + } + + return true; +} + +void cmCTestMultiProcessHandler::AddRepeatGroup(std::set const& tests) +{ + // Reuse the group of any test that is already in one, so that fixtures + // sharing a test end up repeating together. + int groupNumber = -1; + for (int test : tests) { + auto const gi = this->RepeatGroupOfTest.find(test); + if (gi != this->RepeatGroupOfTest.end()) { + groupNumber = gi->second; + break; + } + } + if (groupNumber < 0) { + groupNumber = static_cast(this->RepeatGroups.size()); + this->RepeatGroups.emplace_back(); + } + RepeatGroup& group = this->RepeatGroups[groupNumber]; + + for (int test : tests) { + auto const gi = this->RepeatGroupOfTest.find(test); + if (gi == this->RepeatGroupOfTest.end()) { + this->RepeatGroupOfTest[test] = groupNumber; + group.Tests[test]; + } else if (gi->second != groupNumber) { + // Move the tests of the other group over. It is left behind empty in + // RepeatGroups, where no test refers to it any more. + RepeatGroup& other = this->RepeatGroups[gi->second]; + for (auto const& t : other.Tests) { + this->RepeatGroupOfTest[t.first] = groupNumber; + } + group.Tests.insert(other.Tests.begin(), other.Tests.end()); + other.Tests.clear(); + } + } +} + // Set the max number of tests that can be run at the same time. void cmCTestMultiProcessHandler::SetParallelLevel(cm::optional level) { @@ -273,8 +462,15 @@ void cmCTestMultiProcessHandler::StartTestProcess(int test) auto testRun = cm::make_unique(*this, test); if (this->RepeatMode != cmCTest::Repeat::Never) { - testRun->SetRepeatMode(this->RepeatMode); - testRun->SetRunNumber(1, this->RepeatCount); + auto const gi = this->RepeatGroupOfTest.find(test); + if (gi != this->RepeatGroupOfTest.end()) { + // The group repeats the test, so report the repetition it is running. + int const left = this->RepeatGroups[gi->second].RepetitionsLeft; + testRun->SetRunNumber(this->RepeatCount - left, this->RepeatCount); + } else if (this->TestsRunOnce.count(test) == 0) { + testRun->SetRepeatMode(this->RepeatMode); + testRun->SetRunNumber(1, this->RepeatCount); + } } if (this->UseResourceSpec) { testRun->SetUseAllocatedResources(true); @@ -829,19 +1025,36 @@ void cmCTestMultiProcessHandler::FinishTestProcess( this->Failed->push_back(properties->Name); } - for (auto& t : this->PendingTests) { - t.second.Depends.erase(test); + // Let the tests waiting on this one start. A test that its fixture + // repeats releases only the other tests of its group; the tests outside + // it have to wait for the group's last repetition, so FinishRepeatGroup + // releases those. + auto const group = this->RepeatGroupOfTest.find(test); + bool const repeatsWithFixture = group != this->RepeatGroupOfTest.end(); + for (auto& pending : this->PendingTests) { + if (repeatsWithFixture) { + auto const pendingGroup = this->RepeatGroupOfTest.find(pending.first); + if (pendingGroup == this->RepeatGroupOfTest.end() || + pendingGroup->second != group->second) { + continue; + } + } + pending.second.Depends.erase(test); } // A test killed by the interrupt (e.g. Ctrl+C) never truly finished, so do // not record it in the checkpoint; otherwise `ctest -F` would skip it when - // resuming this interrupted run. - if (cmInstrumentationInterrupt::PendingInterruptSignal() == 0) { + // resuming this interrupted run. A test that its fixture repeats is + // recorded by FinishRepeatGroup instead, once its group is done. + if (cmInstrumentationInterrupt::PendingInterruptSignal() == 0 && + !repeatsWithFixture) { this->WriteCheckpoint(test); } this->DeallocateResources(test); this->UnlockResources(test); + this->FinishRepeatGroupTest(test, testResult.TestStatus); + runner.reset(); if (this->JobServerClient) { @@ -850,6 +1063,90 @@ void cmCTestMultiProcessHandler::FinishTestProcess( this->StartNextTestsOnIdle(); } +void cmCTestMultiProcessHandler::FinishRepeatGroupTest(int test, + int testStatus) +{ + auto const gi = this->RepeatGroupOfTest.find(test); + if (gi == this->RepeatGroupOfTest.end()) { + return; + } + RepeatGroup& group = this->RepeatGroups[gi->second]; + + group.AllCompleted &= testStatus == cmCTestTestHandler::COMPLETED; + group.AnyTimedOut |= testStatus == cmCTestTestHandler::TIMEOUT; + if (--group.Unfinished > 0) { + return; + } + + // The group finished a repetition. Repeat it under the same conditions + // that make an individual test repeat. See cmCTestRunTest::NeedsToRepeat. + bool const conditionMet = + (this->RepeatMode == cmCTest::Repeat::UntilFail && group.AllCompleted) || + (this->RepeatMode == cmCTest::Repeat::UntilPass && !group.AllCompleted) || + (this->RepeatMode == cmCTest::Repeat::AfterTimeout && group.AnyTimedOut); + bool const runEnding = this->StopTimePassed || + (this->CheckStopOnFailure() && !this->Failed->empty()) || + cmInstrumentationInterrupt::PendingInterruptSignal() != 0; + if (group.RepetitionsLeft == 0 || !conditionMet || runEnding) { + this->FinishRepeatGroup(group); + return; + } + + group.RepetitionsLeft--; + group.Unfinished = group.Tests.size(); + group.AllCompleted = true; + group.AnyTimedOut = false; + this->RequeueRepeatGroup(group); +} + +void cmCTestMultiProcessHandler::FinishRepeatGroup(RepeatGroup const& group) +{ + // The group has run its last repetition, so record its tests the way a + // test repeating on its own is recorded once it stops repeating: in the + // checkpoint, unless the interrupt cut the group short, and as satisfying + // the tests outside the group that were waiting on them. + bool const interrupted = + cmInstrumentationInterrupt::PendingInterruptSignal() != 0; + for (auto const& t : group.Tests) { + if (!interrupted) { + this->WriteCheckpoint(t.first); + } + for (auto& pending : this->PendingTests) { + pending.second.Depends.erase(t.first); + } + } +} + +void cmCTestMultiProcessHandler::RequeueRepeatGroup(RepeatGroup const& group) +{ + // Only the last repetition of the group counts, as for a test repeating on + // its own, so drop the results of the repetition just finished. Leaving a + // failure behind would also keep the tests requiring the fixture from + // running again, since their dependency on it has to have succeeded. + for (auto const& t : group.Tests) { + std::string const& name = this->Properties[t.first]->Name; + cm::erase(*this->Passed, name); + cm::erase(*this->Failed, name); + } + cm::erase_if(*this->TestResults, + [&group](cmCTestTestHandler::cmCTestTestResult const& result) { + return group.Tests.count(result.TestCount) != 0; + }); + this->Completed -= group.Tests.size(); + + for (auto const& t : group.Tests) { + TestInfo info; + info.Depends = t.second; + this->PendingTests[t.first] = std::move(info); + this->OrderedTests.push_back(t.first); + } + cmCTestOptionalLog(this->CTest, HANDLER_VERBOSE_OUTPUT, + "Re-queued " << group.Tests.size() + << " tests to repeat their fixture" + << std::endl, + this->Quiet); +} + void cmCTestMultiProcessHandler::UpdateCostData() { std::string fname = this->CTest->GetCostDataFile(); @@ -1256,6 +1553,13 @@ static Json::Value DumpCTestProperties( raw ? rawProperties["FIXTURES_SETUP"] : DumpToJsonArray(testProperties.FixturesSetup))); } + if (testProperties.RequestedFixtureRepeatMode) { + properties.append( + DumpCTestProperty("FIXTURE_REPEAT_MODE", + raw ? rawProperties["FIXTURE_REPEAT_MODE"] + : Json::Value(FixtureRepeatModeString( + *testProperties.RequestedFixtureRepeatMode)))); + } if (!testProperties.GeneratedResourceSpecFile.empty()) { properties.append( DumpCTestProperty("GENERATED_RESOURCE_SPEC_FILE", diff --git a/Source/CTest/cmCTestMultiProcessHandler.h b/Source/CTest/cmCTestMultiProcessHandler.h index b85284b6ab..5799de1395 100644 --- a/Source/CTest/cmCTestMultiProcessHandler.h +++ b/Source/CTest/cmCTestMultiProcessHandler.h @@ -177,6 +177,46 @@ private: cm::optional ResourceSpecSetupTest; bool HasInvalidGeneratedResourceSpec = false; + // Tests that ctest --repeat repeats as a unit because they take part in a + // fixture using AROUND_EACH_REPEAT mode. + struct RepeatGroup + { + // The group's tests, each mapped to its dependencies within the group. + // Dependencies on tests outside the group are satisfied by the first + // repetition and are not restored for the later ones. + std::map Tests; + // Repetitions still to run after the current one. + int RepetitionsLeft = 0; + // Tests that have not finished the current repetition. + std::size_t Unfinished = 0; + // Whether every test finished the current repetition successfully. + bool AllCompleted = true; + // Whether any test timed out during the current repetition. + bool AnyTimedOut = false; + }; + std::vector RepeatGroups; + // Index into RepeatGroups of each test that belongs to a group. + std::map RepeatGroupOfTest; + // Fixture setup and cleanup tests that run only once, because their + // fixture brackets all repetitions of the tests requiring it. + std::set TestsRunOnce; + + // Work out how ctest --repeat repeats each test of a fixture, filling in + // the members above. Returns false if fixtures that repeat together + // disagree on the mode. + bool ComputeFixtureRepetition(); + // Put the given tests in one repeat group, merging any group they already + // belong to. + void AddRepeatGroup(std::set const& tests); + // Record the result of a test belonging to a repeat group, and re-queue + // the group if it just finished a repetition and another one is due. + void FinishRepeatGroupTest(int test, int testStatus); + // Record the group's tests once it has run its last repetition. + void FinishRepeatGroup(RepeatGroup const& group); + // Run the group's tests again, dropping the results of the repetition it + // just finished. + void RequeueRepeatGroup(RepeatGroup const& group); + // Tests pending selection to start. They may have dependencies. TestMap PendingTests; // List of pending test indexes, ordered by cost. diff --git a/Source/CTest/cmCTestRunTest.cxx b/Source/CTest/cmCTestRunTest.cxx index c1399fe0d5..eb948405dc 100644 --- a/Source/CTest/cmCTestRunTest.cxx +++ b/Source/CTest/cmCTestRunTest.cxx @@ -391,6 +391,7 @@ cmCTestRunTest::EndTestResult cmCTestRunTest::EndTest(size_t completed, } cmCTestRunTest::EndTestResult testResult; testResult.Passed = passed || skipped; + testResult.TestStatus = this->TestResult.Status; if (res == cmProcess::State::Expired && this->TestProcess->GetTimeoutReason() == cmProcess::TimeoutReason::StopTime) { @@ -1001,7 +1002,7 @@ void cmCTestRunTest::WriteLogOutputTop(size_t completed, size_t total) // the test, and blanks on its other runs. Which run that is has to be // decided before the run finishes: with until-fail it is the last run, and // with until-pass and after-timeout the repetitions may end early, so it - // is the first. + // is the first. A test that its fixture repeats records every run. bool countThisRun = true; switch (this->RepeatMode) { case cmCTest::Repeat::Never: diff --git a/Source/CTest/cmCTestRunTest.h b/Source/CTest/cmCTestRunTest.h index 3d2cd6527a..42ad7087e2 100644 --- a/Source/CTest/cmCTestRunTest.h +++ b/Source/CTest/cmCTestRunTest.h @@ -31,8 +31,8 @@ public: explicit cmCTestRunTest(cmCTestMultiProcessHandler& multiHandler, int index); // Report this run as run `number` of `count` in the test's "(run N/M)" - // suffix. A test that repeats on its own starts at run 1 and counts up - // itself. + // suffix. A test repeating on its own starts at run 1 and counts up + // itself; a test repeated by its fixture is told which run it is. void SetRunNumber(int number, int count) { this->RunNumber = number; @@ -76,6 +76,7 @@ public: { bool Passed = false; bool StopTimePassed = false; + int TestStatus = cmCTestTestHandler::NOT_RUN; }; // launch the test process, return whether it started correctly @@ -160,6 +161,8 @@ private: std::vector>> AllocatedResources; + // Never unless this test repeats itself. A test repeated by its fixture + // leaves this Never; the fixture decides when to repeat. cmCTest::Repeat RepeatMode = cmCTest::Repeat::Never; int RunNumber = 1; // which run of the test this is int RunCount = 1; // how many runs it may be given diff --git a/Source/CTest/cmCTestTestHandler.cxx b/Source/CTest/cmCTestTestHandler.cxx index 9e7e7a5cf3..ce2c26355b 100644 --- a/Source/CTest/cmCTestTestHandler.cxx +++ b/Source/CTest/cmCTestTestHandler.cxx @@ -2351,6 +2351,27 @@ std::string cmCTestTestHandler::cmCTestTestProperties::GetStampFile() return cmCryptoHash(cmCryptoHash::AlgoMD5).HashString(this->Name) + ".stamp"; } +namespace { +cm::optional ParseFixtureRepeatMode( + cmCTestTestHandler::cmCTestTestProperties& rt, std::string const& val) +{ + using Mode = cmCTestTestHandler::FixtureRepeatMode; + if (val == "AROUND_ALL_REPEATS"_s) { + return Mode::AroundAllRepeats; + } + if (val == "AROUND_EACH_REPEAT"_s) { + return Mode::AroundEachRepeat; + } + if (val == "EACH_TEST_SEPARATELY"_s) { + return Mode::EachTestSeparately; + } + rt.AppendError(cmStrCat("FIXTURE_REPEAT_MODE \"", val, + "\" not recognized. Must be AROUND_ALL_REPEATS, " + "AROUND_EACH_REPEAT, or EACH_TEST_SEPARATELY.")); + return cm::nullopt; +} +} + bool cmCTestTestHandler::SetTestsProperties( std::vector const& args) { @@ -2433,6 +2454,11 @@ bool cmCTestTestHandler::SetTestsProperties( cmList lval{ val }; rt.FixturesRequired.insert(lval.begin(), lval.end()); + } else if (key == "FIXTURE_REPEAT_MODE"_s) { + rt.RawProperties[key] = val; + rt.RequestedFixtureRepeatMode = ParseFixtureRepeatMode(rt, val); + } else if (key == "_CMAKE_DEFAULT_FIXTURE_REPEAT_MODE"_s) { + rt.DefaultFixtureRepeatMode = ParseFixtureRepeatMode(rt, val); } else if (key == "TIMEOUT"_s) { rt.RawProperties[key] = val; rt.Timeout = cmDuration(atof(val.c_str())); diff --git a/Source/CTest/cmCTestTestHandler.h b/Source/CTest/cmCTestTestHandler.h index b7c3a1f3b2..0c5b8654fe 100644 --- a/Source/CTest/cmCTestTestHandler.h +++ b/Source/CTest/cmCTestTestHandler.h @@ -143,6 +143,15 @@ public: std::string Name; }; + // How ctest --repeat treats a fixture. See the FIXTURE_REPEAT_MODE test + // property and policy CMP0224. + enum class FixtureRepeatMode + { + AroundAllRepeats, // the fixture runs once, around all repetitions + AroundEachRepeat, // the fixture repeats with the tests requiring it + EachTestSeparately // every test repeats on its own + }; + struct cmCTestTestProperties { void AppendError(cm::string_view err); @@ -221,6 +230,10 @@ public: std::set FixturesRequired; std::string FixturesRequiredRaw; std::set RequireSuccessDepends; + // For the fixtures this test sets up or cleans up. A mode requested by + // any of a fixture's own tests wins over the default recorded for them. + cm::optional RequestedFixtureRepeatMode; + cm::optional DefaultFixtureRepeatMode; std::vector> ResourceGroups; std::string ResourceGroupsRaw; std::string GeneratedResourceSpecFile; diff --git a/Source/cmDiscoverTestsCommand.cxx b/Source/cmDiscoverTestsCommand.cxx index dc2e75dcdc..93dbf63995 100644 --- a/Source/cmDiscoverTestsCommand.cxx +++ b/Source/cmDiscoverTestsCommand.cxx @@ -8,6 +8,7 @@ #include #include +#include #include #include "cmArgumentParser.h" @@ -17,6 +18,7 @@ #include "cmListFileCache.h" #include "cmLocalGenerator.h" #include "cmMakefile.h" +#include "cmPolicies.h" #include "cmScriptGenerator.h" #include "cmTestDiscovery.h" #include "cmTestGenerator.h" @@ -89,6 +91,17 @@ private: cmListFileBacktrace Backtrace; }; +bool SetsFixtureRepeatMode(std::vector const& properties) +{ + for (std::size_t i = 0; i < properties.size(); i += 2) { + if (properties[i] == "FIXTURE_REPEAT_MODE"_s || + properties[i] == "_CMAKE_DEFAULT_FIXTURE_REPEAT_MODE"_s) { + return true; + } + } + return false; +} + } // namespace bool cmDiscoverTestsCommand(std::vector const& args, @@ -121,6 +134,18 @@ bool cmDiscoverTestsCommand(std::vector const& args, } cmMakefile& mf = status.GetMakefile(); + + // The discovered tests are created while ctest runs, too late for the + // policy to reach them. Only NEW needs carrying through. With nothing + // recorded, ctest already uses the EACH_TEST_SEPARATELY behavior of + // CMake 4.4 and below. + if (mf.GetPolicyStatus(cmPolicies::CMP0224) == cmPolicies::NEW && + !SetsFixtureRepeatMode(arguments.TestProperties)) { + arguments.TestProperties.emplace_back( + "_CMAKE_DEFAULT_FIXTURE_REPEAT_MODE"); + arguments.TestProperties.emplace_back("AROUND_EACH_REPEAT"); + } + mf.AddTestGenerator(cm::make_unique(std::move(arguments), mf.GetBacktrace())); return true; diff --git a/Source/cmGlobalGenerator.cxx b/Source/cmGlobalGenerator.cxx index e83d10c093..4bebfdc7d9 100644 --- a/Source/cmGlobalGenerator.cxx +++ b/Source/cmGlobalGenerator.cxx @@ -1566,6 +1566,11 @@ void cmGlobalGenerator::AddCMP0068WarnTarget(std::string const& target) this->CMP0068WarnTargets.insert(target); } +void cmGlobalGenerator::AddCMP0224WarnTest(std::string const& test) +{ + this->CMP0224WarnTests.insert(test); +} + bool cmGlobalGenerator::ShouldWarnCMP0210(std::string const& lang) { return this->WarnedCMP0210Languages.insert(lang).second; @@ -1686,6 +1691,9 @@ bool cmGlobalGenerator::Compute() // clear targets to issue warning CMP0068 for this->CMP0068WarnTargets.clear(); + // clear tests to issue warning CMP0224 for + this->CMP0224WarnTests.clear(); + // Check whether this generator is allowed to run. if (!this->CheckALLOW_DUPLICATE_CUSTOM_TARGETS()) { return false; @@ -1966,6 +1974,22 @@ void cmGlobalGenerator::Generate() this->GetCMakeInstance()->IssueDiagnostic(cmDiagnostics::CMD_POLICY, w.str()); } + + if (!this->CMP0224WarnTests.empty()) { + std::ostringstream w; + /* clang-format off */ + w << + cmPolicies::GetPolicyWarning(cmPolicies::CMP0224) << "\n" + "For compatibility with older versions of CMake, the fixtures of the " + "following tests will use EACH_TEST_SEPARATELY mode:\n" + ; + /* clang-format on */ + for (std::string const& t : this->CMP0224WarnTests) { + w << ' ' << t << '\n'; + } + this->GetCMakeInstance()->IssueDiagnostic(cmDiagnostics::CMD_POLICY, + w.str()); + } } #if !defined(CMAKE_BOOTSTRAP) diff --git a/Source/cmGlobalGenerator.h b/Source/cmGlobalGenerator.h index 3184a28e60..2af685cb63 100644 --- a/Source/cmGlobalGenerator.h +++ b/Source/cmGlobalGenerator.h @@ -696,6 +696,7 @@ public: cmExportBuildFileGenerator* GetExportedTargetsFile( std::string const& filename) const; void AddCMP0068WarnTarget(std::string const& target); + void AddCMP0224WarnTest(std::string const& test); virtual bool SupportsShortObjectNames() const; bool UseShortObjectNames( @@ -1035,6 +1036,9 @@ private: // track targets to issue CMP0068 warning for. std::set CMP0068WarnTargets; + // track tests to issue CMP0224 warning for. + std::set CMP0224WarnTests; + std::unordered_set WarnedCMP0210Languages; std::unordered_set WarnedExperimental; diff --git a/Source/cmPolicies.h b/Source/cmPolicies.h index 80eebc3d59..97979d9d68 100644 --- a/Source/cmPolicies.h +++ b/Source/cmPolicies.h @@ -672,7 +672,11 @@ class cmMakefile; "PATH_IS_PREFIX operator.", \ 4, 5, 0, WARN) \ SELECT(POLICY, CMP0223, "An empty path is not a prefix of any path.", 4, 5, \ - 0, WARN) + 0, WARN) \ + SELECT(POLICY, CMP0224, \ + "Test fixtures use AROUND_EACH_REPEAT mode by default with " \ + "ctest --repeat.", \ + 4, 5, 0, WARN) #define CM_SELECT_ID(F, A1, A2, A3, A4, A5, A6) F(A1) #define CM_FOR_EACH_POLICY_ID(POLICY) \ diff --git a/Source/cmTest.cxx b/Source/cmTest.cxx index 75a81b840a..274c6d9c11 100644 --- a/Source/cmTest.cxx +++ b/Source/cmTest.cxx @@ -13,6 +13,7 @@ cmTest::cmTest(cmMakefile* mf) : Backtrace(mf->GetBacktrace()) , PolicyStatusCMP0158(mf->GetPolicyStatus(cmPolicies::CMP0158)) , PolicyStatusCMP0178(mf->GetPolicyStatus(cmPolicies::CMP0178)) + , PolicyStatusCMP0224(mf->GetPolicyStatus(cmPolicies::CMP0224)) { this->Makefile = mf; this->OldStyle = true; diff --git a/Source/cmTest.h b/Source/cmTest.h index ee7d935441..22dc100120 100644 --- a/Source/cmTest.h +++ b/Source/cmTest.h @@ -83,6 +83,12 @@ public: this->PolicyStatusCMP0178 = p; } + /** Get the CMP0224 policy setting */ + cmPolicies::PolicyStatus GetCMP0224() const + { + return this->PolicyStatusCMP0224; + } + /** Set/Get whether lists in command lines should be expanded. */ bool GetCommandExpandLists() const; void SetCommandExpandLists(bool b); @@ -100,4 +106,5 @@ private: cmListFileBacktrace Backtrace; cmPolicies::PolicyStatus PolicyStatusCMP0158; cmPolicies::PolicyStatus PolicyStatusCMP0178; + cmPolicies::PolicyStatus PolicyStatusCMP0224; }; diff --git a/Source/cmTestGenerator.cxx b/Source/cmTestGenerator.cxx index 2666e87d81..b0b56d4ea5 100644 --- a/Source/cmTestGenerator.cxx +++ b/Source/cmTestGenerator.cxx @@ -314,6 +314,7 @@ void cmTestGenerator::GenerateScriptForConfig(std::ostream& os, os << " _CMAKE_TEST_BUILD_DEPENDS " << cmScriptGenerator::Quote(depList.to_string()); } + this->GenerateDefaultFixtureRepeatMode(os); os << ' '; this->GenerateBacktrace(os, this->Test->GetBacktrace()); os << ")\n"; @@ -367,11 +368,46 @@ void cmTestGenerator::GenerateOldStyle(std::ostream& fout, Indent indent) for (auto const& i : this->Test->GetProperties().GetList()) { fout << " " << i.first << " " << cmScriptGenerator::Quote(i.second); } + this->GenerateDefaultFixtureRepeatMode(fout); fout << ' '; this->GenerateBacktrace(fout, this->Test->GetBacktrace()); fout << ")\n"; } +void cmTestGenerator::GenerateDefaultFixtureRepeatMode(std::ostream& os) +{ + // Nothing to choose for a test that is not part of a fixture, or that + // names a mode itself. + if (this->Test->GetProperty("FIXTURE_REPEAT_MODE") || + (!this->Test->GetProperty("FIXTURES_SETUP") && + !this->Test->GetProperty("FIXTURES_CLEANUP"))) { + return; + } + + // Write the mode the policy chose into the test file, so that ctest reads + // a mode rather than the policy settings behind it. Only NEW needs + // writing: with nothing written, ctest already uses the + // EACH_TEST_SEPARATELY behavior of CMake 4.4 and below. + switch (this->Test->GetCMP0224()) { + case cmPolicies::WARN: + // Warn only on request. Fixtures are common, and the choice of mode + // matters only to those who run ctest --repeat. Collect the tests + // rather than warning about each: a project that sets its fixtures up + // in an add_test() wrapper would fill the console. + if (this->Test->GetMakefile()->PolicyOptionalWarningEnabled( + "CMAKE_POLICY_WARNING_CMP0224")) { + this->LG->GetGlobalGenerator()->AddCMP0224WarnTest( + this->Test->GetName()); + } + CM_FALLTHROUGH; + case cmPolicies::OLD: + break; + case cmPolicies::NEW: + os << " _CMAKE_DEFAULT_FIXTURE_REPEAT_MODE AROUND_EACH_REPEAT"; + break; + } +} + void cmTestGenerator::GenerateBacktrace(std::ostream& os, cmListFileBacktrace bt) { diff --git a/Source/cmTestGenerator.h b/Source/cmTestGenerator.h index 2982f2bc57..d3357a3862 100644 --- a/Source/cmTestGenerator.h +++ b/Source/cmTestGenerator.h @@ -67,6 +67,7 @@ private: protected: void GenerateBacktrace(std::ostream& os, cmListFileBacktrace bt); + void GenerateDefaultFixtureRepeatMode(std::ostream& os); void GenerateCommand(std::ostream& os, std::vector const& command, std::string const& config, bool expand, diff --git a/Tests/RunCMake/CTestCommandLine/RunCMakeTest.cmake b/Tests/RunCMake/CTestCommandLine/RunCMakeTest.cmake index 376d75308d..d4f10ce0f6 100644 --- a/Tests/RunCMake/CTestCommandLine/RunCMakeTest.cmake +++ b/Tests/RunCMake/CTestCommandLine/RunCMakeTest.cmake @@ -104,6 +104,84 @@ endfunction() run_repeat_until_fail_tests(--repeat-until-fail 3) run_repeat_until_fail_tests(--repeat until-fail:3) +function(run_repeat_fixture_test case) + # Each case runs one fixture around one test with `--repeat until-fail:3`. + # The FIXTURE_REPEAT_MODE property and policy CMP0224 decide whether the + # fixture repeats with the test, around it, or on its own. + set(RunCMake_TEST_BINARY_DIR ${RunCMake_BINARY_DIR}/repeat-fixture-${case}-build) + run_cmake_with_options(repeat-fixture-${case}-cmake ${ARGN}) + set(RunCMake_TEST_NO_CLEAN 1) + run_cmake_command(repeat-fixture-${case}-ctest + ${CMAKE_CTEST_COMMAND} -C Debug --repeat until-fail:3 + ) +endfunction() +# CMP0224 selects the default mode for fixture tests with no explicit mode. +# Its warning is off by default, so the WARN case asks for it. +run_repeat_fixture_test(default) # not set, and silent +run_repeat_fixture_test(warn -DCMAKE_POLICY_WARNING_CMP0224=ON) +run_repeat_fixture_test(old) # OLD: EACH_TEST_SEPARATELY +run_repeat_fixture_test(new) # NEW: AROUND_EACH_REPEAT +# An explicit FIXTURE_REPEAT_MODE property applies whatever the policy says. +run_repeat_fixture_test(around-all) +run_repeat_fixture_test(around-each) +run_repeat_fixture_test(separately) +# The property describes the fixture, so one of its tests may carry it alone. +run_repeat_fixture_test(setup-only) +run_repeat_fixture_test(cleanup-only) +# Fixtures that share a test repeat as one unit. +run_repeat_fixture_test(merge) +# A test depending on one inside the group waits for the last repetition. +run_repeat_fixture_test(depends) + +block() + # A repeat group records its tests in the checkpoint once, when it stops + # repeating, so that `ctest -F` re-runs an interrupted group from its first + # repetition. A checkpoint naming a test twice, or naming one this run + # does not have, must not upset the resume either. + set(RunCMake_TEST_BINARY_DIR + ${RunCMake_BINARY_DIR}/repeat-fixture-resume-build) + run_cmake(repeat-fixture-resume-cmake) + set(RunCMake_TEST_NO_CLEAN 1) + file(WRITE "${RunCMake_TEST_BINARY_DIR}/Testing/Temporary/CTestCheckpoint.txt" + "1\n2\n3\n1\n99\n") + run_cmake_command(repeat-fixture-resume-ctest + ${CMAKE_CTEST_COMMAND} -C Debug -F --repeat until-fail:3 + ) +endblock() + +function(run_repeat_fixture_until_pass_test) + set(RunCMake_TEST_BINARY_DIR + ${RunCMake_BINARY_DIR}/repeat-fixture-until-pass-build) + run_cmake(repeat-fixture-until-pass-cmake) + set(RunCMake_TEST_NO_CLEAN 1) + run_cmake_command(repeat-fixture-until-pass-ctest + ${CMAKE_CTEST_COMMAND} -C Debug --repeat until-pass:3 + ) +endfunction() +run_repeat_fixture_until_pass_test() + +function(run_repeat_fixture_conflict_test case) + # Fixtures that repeat together must agree on the mode; ctest refuses to + # run when they do not. + set(RunCMake_TEST_BINARY_DIR + ${RunCMake_BINARY_DIR}/repeat-fixture-conflict-${case}-build) + run_cmake(repeat-fixture-conflict-${case}-cmake) + set(RunCMake_TEST_NO_CLEAN 1) + run_cmake_command(repeat-fixture-conflict-${case}-ctest + ${CMAKE_CTEST_COMMAND} -C Debug --repeat until-fail:3 + ) +endfunction() +run_repeat_fixture_conflict_test(fixture) # one fixture, disagreeing tests +run_repeat_fixture_conflict_test(shared) # two fixtures sharing a test + +function(run_repeat_fixture_bad_mode_test) + set(RunCMake_TEST_BINARY_DIR ${RunCMake_BINARY_DIR}/repeat-fixture-bad-mode-build) + run_cmake(repeat-fixture-bad-mode-cmake) + set(RunCMake_TEST_NO_CLEAN 1) + run_cmake_command(repeat-fixture-bad-mode-ctest ${CMAKE_CTEST_COMMAND} -C Debug) +endfunction() +run_repeat_fixture_bad_mode_test() + block() set(RunCMake_TEST_BINARY_DIR ${RunCMake_BINARY_DIR}/rerun) set(RunCMake_TEST_NO_CLEAN 1) diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-around-all-cmake.cmake b/Tests/RunCMake/CTestCommandLine/repeat-fixture-around-all-cmake.cmake new file mode 100644 index 0000000000..c165fdb376 --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-around-all-cmake.cmake @@ -0,0 +1,2 @@ +set(FIXTURE_REPEAT_MODE AROUND_ALL_REPEATS) +include(${CMAKE_CURRENT_LIST_DIR}/repeat-fixture-common.cmake) diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-around-all-ctest-stdout.txt b/Tests/RunCMake/CTestCommandLine/repeat-fixture-around-all-ctest-stdout.txt new file mode 100644 index 0000000000..18c72a153e --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-around-all-ctest-stdout.txt @@ -0,0 +1,15 @@ +^Test project .*/Tests/RunCMake/CTestCommandLine/repeat-fixture-around-all-build + Start 1: fixture_setup +1/3 Test #1: fixture_setup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 1/3\) + Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 2/3\) + Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 3/3\) +2/3 Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: fixture_cleanup +3/3 Test #3: fixture_cleanup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec ++ +100% tests passed out of 3 ++ +Total Test time \(real\) = +[0-9\.]+ sec$ diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-around-each-cmake.cmake b/Tests/RunCMake/CTestCommandLine/repeat-fixture-around-each-cmake.cmake new file mode 100644 index 0000000000..42224a83ea --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-around-each-cmake.cmake @@ -0,0 +1,2 @@ +set(FIXTURE_REPEAT_MODE AROUND_EACH_REPEAT) +include(${CMAKE_CURRENT_LIST_DIR}/repeat-fixture-common.cmake) diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-around-each-ctest-stdout.txt b/Tests/RunCMake/CTestCommandLine/repeat-fixture-around-each-ctest-stdout.txt new file mode 100644 index 0000000000..73c9e2fed7 --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-around-each-ctest-stdout.txt @@ -0,0 +1,23 @@ +^Test project .*/Tests/RunCMake/CTestCommandLine/repeat-fixture-around-each-build + Start 1: fixture_setup \(run 1/3\) +1/3 Test #1: fixture_setup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 1/3\) +2/3 Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: fixture_cleanup \(run 1/3\) +3/3 Test #3: fixture_cleanup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 1: fixture_setup \(run 2/3\) +1/3 Test #1: fixture_setup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 2/3\) +2/3 Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: fixture_cleanup \(run 2/3\) +3/3 Test #3: fixture_cleanup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 1: fixture_setup \(run 3/3\) +1/3 Test #1: fixture_setup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 3/3\) +2/3 Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: fixture_cleanup \(run 3/3\) +3/3 Test #3: fixture_cleanup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec ++ +100% tests passed out of 3 ++ +Total Test time \(real\) = +[0-9\.]+ sec$ diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-bad-mode-cmake.cmake b/Tests/RunCMake/CTestCommandLine/repeat-fixture-bad-mode-cmake.cmake new file mode 100644 index 0000000000..144fbe8f78 --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-bad-mode-cmake.cmake @@ -0,0 +1,2 @@ +set(FIXTURE_REPEAT_MODE BOGUS) +include(${CMAKE_CURRENT_LIST_DIR}/repeat-fixture-common.cmake) diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-bad-mode-ctest-result.txt b/Tests/RunCMake/CTestCommandLine/repeat-fixture-bad-mode-ctest-result.txt new file mode 100644 index 0000000000..45a4fb75db --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-bad-mode-ctest-result.txt @@ -0,0 +1 @@ +8 diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-bad-mode-ctest-stderr.txt b/Tests/RunCMake/CTestCommandLine/repeat-fixture-bad-mode-ctest-stderr.txt new file mode 100644 index 0000000000..ba4235defb --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-bad-mode-ctest-stderr.txt @@ -0,0 +1 @@ +Errors while running CTest diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-bad-mode-ctest-stdout.txt b/Tests/RunCMake/CTestCommandLine/repeat-fixture-bad-mode-ctest-stdout.txt new file mode 100644 index 0000000000..22fb851d64 --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-bad-mode-ctest-stdout.txt @@ -0,0 +1,19 @@ +^Test project .*/Tests/RunCMake/CTestCommandLine/repeat-fixture-bad-mode-build + Start 1: fixture_setup +FIXTURE_REPEAT_MODE "BOGUS" not recognized\. Must be AROUND_ALL_REPEATS, AROUND_EACH_REPEAT, or EACH_TEST_SEPARATELY\. +1/3 Test #1: fixture_setup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\*\*\*Not Run +[0-9\.]+ sec + Start 2: test_with_fixture +Failed test dependencies: fixture_setup +2/3 Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\*\*\*Not Run +[0-9\.]+ sec + Start 3: fixture_cleanup +FIXTURE_REPEAT_MODE "BOGUS" not recognized\. Must be AROUND_ALL_REPEATS, AROUND_EACH_REPEAT, or EACH_TEST_SEPARATELY\. +3/3 Test #3: fixture_cleanup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\*\*\*Not Run +[0-9\.]+ sec ++ +0% tests passed, 3 tests failed out of 3 ++ +Total Test time \(real\) = +[0-9\.]+ sec ++ +The following tests FAILED: +[ ]+1 - fixture_setup \(Not Run\) +[ ]+2 - test_with_fixture \(Not Run\) +[ ]+3 - fixture_cleanup \(Not Run\)$ diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-cleanup-only-cmake.cmake b/Tests/RunCMake/CTestCommandLine/repeat-fixture-cleanup-only-cmake.cmake new file mode 100644 index 0000000000..b191f3559f --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-cleanup-only-cmake.cmake @@ -0,0 +1,5 @@ +# The mode set on the cleanup test governs the whole fixture, so the setup +# test must not repeat on its own. +set(FIXTURE_REPEAT_MODE AROUND_ALL_REPEATS) +set(FIXTURE_REPEAT_TESTS fixture_cleanup) +include(${CMAKE_CURRENT_LIST_DIR}/repeat-fixture-common.cmake) diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-cleanup-only-ctest-stdout.txt b/Tests/RunCMake/CTestCommandLine/repeat-fixture-cleanup-only-ctest-stdout.txt new file mode 100644 index 0000000000..5f83662e9e --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-cleanup-only-ctest-stdout.txt @@ -0,0 +1,15 @@ +^Test project .*/Tests/RunCMake/CTestCommandLine/repeat-fixture-cleanup-only-build + Start 1: fixture_setup +1/3 Test #1: fixture_setup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 1/3\) + Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 2/3\) + Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 3/3\) +2/3 Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: fixture_cleanup +3/3 Test #3: fixture_cleanup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec ++ +100% tests passed out of 3 ++ +Total Test time \(real\) = +[0-9\.]+ sec$ diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-common.cmake b/Tests/RunCMake/CTestCommandLine/repeat-fixture-common.cmake new file mode 100644 index 0000000000..1965ac0e48 --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-common.cmake @@ -0,0 +1,20 @@ +enable_testing() + +add_test(NAME fixture_setup COMMAND ${CMAKE_COMMAND} -E true) +set_tests_properties(fixture_setup PROPERTIES FIXTURES_SETUP MyFixture) + +add_test(NAME test_with_fixture COMMAND ${CMAKE_COMMAND} -E true) +set_tests_properties(test_with_fixture PROPERTIES FIXTURES_REQUIRED MyFixture) + +add_test(NAME fixture_cleanup COMMAND ${CMAKE_COMMAND} -E true) +set_tests_properties(fixture_cleanup PROPERTIES FIXTURES_CLEANUP MyFixture) + +# The mode applies to the whole fixture, so cases that check that setting it +# on one of its tests is enough name only that test in FIXTURE_REPEAT_TESTS. +if(FIXTURE_REPEAT_MODE) + if(NOT FIXTURE_REPEAT_TESTS) + set(FIXTURE_REPEAT_TESTS fixture_setup fixture_cleanup) + endif() + set_tests_properties(${FIXTURE_REPEAT_TESTS} PROPERTIES + FIXTURE_REPEAT_MODE ${FIXTURE_REPEAT_MODE}) +endif() diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-conflict-fixture-cmake.cmake b/Tests/RunCMake/CTestCommandLine/repeat-fixture-conflict-fixture-cmake.cmake new file mode 100644 index 0000000000..616624e53d --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-conflict-fixture-cmake.cmake @@ -0,0 +1,10 @@ +# The setup and cleanup tests of one fixture disagree on the mode. +enable_testing() +add_test(NAME fixture_setup COMMAND ${CMAKE_COMMAND} -E true) +add_test(NAME fixture_cleanup COMMAND ${CMAKE_COMMAND} -E true) +add_test(NAME test_with_fixture COMMAND ${CMAKE_COMMAND} -E true) +set_tests_properties(fixture_setup PROPERTIES + FIXTURES_SETUP MyFixture FIXTURE_REPEAT_MODE AROUND_EACH_REPEAT) +set_tests_properties(fixture_cleanup PROPERTIES + FIXTURES_CLEANUP MyFixture FIXTURE_REPEAT_MODE AROUND_ALL_REPEATS) +set_tests_properties(test_with_fixture PROPERTIES FIXTURES_REQUIRED MyFixture) diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-conflict-fixture-ctest-result.txt b/Tests/RunCMake/CTestCommandLine/repeat-fixture-conflict-fixture-ctest-result.txt new file mode 100644 index 0000000000..45a4fb75db --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-conflict-fixture-ctest-result.txt @@ -0,0 +1 @@ +8 diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-conflict-fixture-ctest-stderr.txt b/Tests/RunCMake/CTestCommandLine/repeat-fixture-conflict-fixture-ctest-stderr.txt new file mode 100644 index 0000000000..b18b0e151f --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-conflict-fixture-ctest-stderr.txt @@ -0,0 +1,7 @@ +Error: the setup and cleanup tests of fixture "MyFixture" request conflicting FIXTURE_REPEAT_MODE values: + "fixture_cleanup" requests AROUND_ALL_REPEATS + "fixture_setup" requests AROUND_EACH_REPEAT +All setup and cleanup tests of a fixture must request the same mode\. +Errors while running CTest +Output from these tests are in: .*/Testing/Temporary/LastTest\.log +Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely\. diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-conflict-fixture-ctest-stdout.txt b/Tests/RunCMake/CTestCommandLine/repeat-fixture-conflict-fixture-ctest-stdout.txt new file mode 100644 index 0000000000..1d54424178 --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-conflict-fixture-ctest-stdout.txt @@ -0,0 +1 @@ +^Test project .*/Tests/RunCMake/CTestCommandLine/repeat-fixture-conflict-fixture-build$ diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-conflict-shared-cmake.cmake b/Tests/RunCMake/CTestCommandLine/repeat-fixture-conflict-shared-cmake.cmake new file mode 100644 index 0000000000..521b411de1 --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-conflict-shared-cmake.cmake @@ -0,0 +1,16 @@ +# Two fixtures disagree on the mode, and one test takes part in both. +enable_testing() +add_test(NAME setupA COMMAND ${CMAKE_COMMAND} -E true) +add_test(NAME cleanupA COMMAND ${CMAKE_COMMAND} -E true) +add_test(NAME setupB COMMAND ${CMAKE_COMMAND} -E true) +add_test(NAME cleanupB COMMAND ${CMAKE_COMMAND} -E true) +add_test(NAME needs_both COMMAND ${CMAKE_COMMAND} -E true) +set_tests_properties(setupA PROPERTIES FIXTURES_SETUP A + FIXTURE_REPEAT_MODE AROUND_EACH_REPEAT) +set_tests_properties(cleanupA PROPERTIES FIXTURES_CLEANUP A + FIXTURE_REPEAT_MODE AROUND_EACH_REPEAT) +set_tests_properties(setupB PROPERTIES FIXTURES_SETUP B + FIXTURE_REPEAT_MODE AROUND_ALL_REPEATS) +set_tests_properties(cleanupB PROPERTIES FIXTURES_CLEANUP B + FIXTURE_REPEAT_MODE AROUND_ALL_REPEATS) +set_tests_properties(needs_both PROPERTIES FIXTURES_REQUIRED "A;B") diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-conflict-shared-ctest-result.txt b/Tests/RunCMake/CTestCommandLine/repeat-fixture-conflict-shared-ctest-result.txt new file mode 100644 index 0000000000..45a4fb75db --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-conflict-shared-ctest-result.txt @@ -0,0 +1 @@ +8 diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-conflict-shared-ctest-stderr.txt b/Tests/RunCMake/CTestCommandLine/repeat-fixture-conflict-shared-ctest-stderr.txt new file mode 100644 index 0000000000..8b67d6665e --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-conflict-shared-ctest-stderr.txt @@ -0,0 +1,7 @@ +Error: test "needs_both" takes part in fixtures with conflicting FIXTURE_REPEAT_MODE values: + fixture "B" uses AROUND_ALL_REPEATS + fixture "A" uses AROUND_EACH_REPEAT +Fixtures that share a test must use the same mode\. +Errors while running CTest +Output from these tests are in: .*/Testing/Temporary/LastTest\.log +Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely\. diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-conflict-shared-ctest-stdout.txt b/Tests/RunCMake/CTestCommandLine/repeat-fixture-conflict-shared-ctest-stdout.txt new file mode 100644 index 0000000000..ba4b038763 --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-conflict-shared-ctest-stdout.txt @@ -0,0 +1 @@ +^Test project .*/Tests/RunCMake/CTestCommandLine/repeat-fixture-conflict-shared-build$ diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-default-cmake.cmake b/Tests/RunCMake/CTestCommandLine/repeat-fixture-default-cmake.cmake new file mode 100644 index 0000000000..e12c991925 --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-default-cmake.cmake @@ -0,0 +1 @@ +include(${CMAKE_CURRENT_LIST_DIR}/repeat-fixture-common.cmake) diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-default-ctest-stdout.txt b/Tests/RunCMake/CTestCommandLine/repeat-fixture-default-ctest-stdout.txt new file mode 100644 index 0000000000..2b0ad229b6 --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-default-ctest-stdout.txt @@ -0,0 +1,23 @@ +^Test project .*/Tests/RunCMake/CTestCommandLine/repeat-fixture-default-build + Start 1: fixture_setup \(run 1/3\) + Test #1: fixture_setup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 1: fixture_setup \(run 2/3\) + Test #1: fixture_setup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 1: fixture_setup \(run 3/3\) +1/3 Test #1: fixture_setup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 1/3\) + Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 2/3\) + Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 3/3\) +2/3 Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: fixture_cleanup \(run 1/3\) + Test #3: fixture_cleanup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: fixture_cleanup \(run 2/3\) + Test #3: fixture_cleanup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: fixture_cleanup \(run 3/3\) +3/3 Test #3: fixture_cleanup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec ++ +100% tests passed out of 3 ++ +Total Test time \(real\) = +[0-9\.]+ sec$ diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-depends-cmake.cmake b/Tests/RunCMake/CTestCommandLine/repeat-fixture-depends-cmake.cmake new file mode 100644 index 0000000000..9553ae55fc --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-depends-cmake.cmake @@ -0,0 +1,7 @@ +# A test outside a repeat group that DEPENDS on a test inside it has to wait +# for the group's last repetition, not its first. +cmake_policy(SET CMP0224 NEW) +include(${CMAKE_CURRENT_LIST_DIR}/repeat-fixture-common.cmake) + +add_test(NAME after_fixture COMMAND ${CMAKE_COMMAND} -E true) +set_tests_properties(after_fixture PROPERTIES DEPENDS test_with_fixture) diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-depends-ctest-stdout.txt b/Tests/RunCMake/CTestCommandLine/repeat-fixture-depends-ctest-stdout.txt new file mode 100644 index 0000000000..d76910f749 --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-depends-ctest-stdout.txt @@ -0,0 +1,29 @@ +^Test project .*/Tests/RunCMake/CTestCommandLine/repeat-fixture-depends-build + Start 1: fixture_setup \(run 1/3\) +1/4 Test #1: fixture_setup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 1/3\) +2/4 Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: fixture_cleanup \(run 1/3\) +3/4 Test #3: fixture_cleanup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 1: fixture_setup \(run 2/3\) +1/4 Test #1: fixture_setup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 2/3\) +2/4 Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: fixture_cleanup \(run 2/3\) +3/4 Test #3: fixture_cleanup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 1: fixture_setup \(run 3/3\) +1/4 Test #1: fixture_setup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 3/3\) +2/4 Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: fixture_cleanup \(run 3/3\) +3/4 Test #3: fixture_cleanup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 4: after_fixture \(run 1/3\) + Test #4: after_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 4: after_fixture \(run 2/3\) + Test #4: after_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 4: after_fixture \(run 3/3\) +4/4 Test #4: after_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec ++ +100% tests passed out of 4 ++ +Total Test time \(real\) = +[0-9\.]+ sec$ diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-merge-cmake.cmake b/Tests/RunCMake/CTestCommandLine/repeat-fixture-merge-cmake.cmake new file mode 100644 index 0000000000..96c919fb17 --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-merge-cmake.cmake @@ -0,0 +1,18 @@ +# Fixture C is required by a test of fixture A and by a test of fixture B, so +# all three repeat as one unit. The DEPENDS chain fixes the order in which +# the unit runs its tests. +enable_testing() +cmake_policy(SET CMP0224 NEW) + +foreach(f A B C) + add_test(NAME setup${f} COMMAND ${CMAKE_COMMAND} -E true) + set_tests_properties(setup${f} PROPERTIES FIXTURES_SETUP ${f}) +endforeach() +set_tests_properties(setupB PROPERTIES DEPENDS setupA) +set_tests_properties(setupC PROPERTIES DEPENDS setupB) + +add_test(NAME testA COMMAND ${CMAKE_COMMAND} -E true) +set_tests_properties(testA PROPERTIES FIXTURES_REQUIRED "A;C") + +add_test(NAME testB COMMAND ${CMAKE_COMMAND} -E true) +set_tests_properties(testB PROPERTIES FIXTURES_REQUIRED "B;C" DEPENDS testA) diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-merge-ctest-stdout.txt b/Tests/RunCMake/CTestCommandLine/repeat-fixture-merge-ctest-stdout.txt new file mode 100644 index 0000000000..438de53524 --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-merge-ctest-stdout.txt @@ -0,0 +1,35 @@ +^Test project .*/Tests/RunCMake/CTestCommandLine/repeat-fixture-merge-build + Start 1: setupA \(run 1/3\) +1/5 Test #1: setupA \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: setupB \(run 1/3\) +2/5 Test #2: setupB \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: setupC \(run 1/3\) +3/5 Test #3: setupC \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 4: testA \(run 1/3\) +4/5 Test #4: testA \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 5: testB \(run 1/3\) +5/5 Test #5: testB \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 1: setupA \(run 2/3\) +1/5 Test #1: setupA \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: setupB \(run 2/3\) +2/5 Test #2: setupB \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: setupC \(run 2/3\) +3/5 Test #3: setupC \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 4: testA \(run 2/3\) +4/5 Test #4: testA \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 5: testB \(run 2/3\) +5/5 Test #5: testB \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 1: setupA \(run 3/3\) +1/5 Test #1: setupA \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: setupB \(run 3/3\) +2/5 Test #2: setupB \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: setupC \(run 3/3\) +3/5 Test #3: setupC \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 4: testA \(run 3/3\) +4/5 Test #4: testA \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 5: testB \(run 3/3\) +5/5 Test #5: testB \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec ++ +100% tests passed out of 5 ++ +Total Test time \(real\) = +[0-9\.]+ sec$ diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-new-cmake.cmake b/Tests/RunCMake/CTestCommandLine/repeat-fixture-new-cmake.cmake new file mode 100644 index 0000000000..9a19f67ec8 --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-new-cmake.cmake @@ -0,0 +1,2 @@ +cmake_policy(SET CMP0224 NEW) +include(${CMAKE_CURRENT_LIST_DIR}/repeat-fixture-common.cmake) diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-new-ctest-stdout.txt b/Tests/RunCMake/CTestCommandLine/repeat-fixture-new-ctest-stdout.txt new file mode 100644 index 0000000000..35c85d7c6d --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-new-ctest-stdout.txt @@ -0,0 +1,23 @@ +^Test project .*/Tests/RunCMake/CTestCommandLine/repeat-fixture-new-build + Start 1: fixture_setup \(run 1/3\) +1/3 Test #1: fixture_setup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 1/3\) +2/3 Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: fixture_cleanup \(run 1/3\) +3/3 Test #3: fixture_cleanup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 1: fixture_setup \(run 2/3\) +1/3 Test #1: fixture_setup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 2/3\) +2/3 Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: fixture_cleanup \(run 2/3\) +3/3 Test #3: fixture_cleanup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 1: fixture_setup \(run 3/3\) +1/3 Test #1: fixture_setup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 3/3\) +2/3 Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: fixture_cleanup \(run 3/3\) +3/3 Test #3: fixture_cleanup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec ++ +100% tests passed out of 3 ++ +Total Test time \(real\) = +[0-9\.]+ sec$ diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-old-cmake.cmake b/Tests/RunCMake/CTestCommandLine/repeat-fixture-old-cmake.cmake new file mode 100644 index 0000000000..c629859558 --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-old-cmake.cmake @@ -0,0 +1,2 @@ +cmake_policy(SET CMP0224 OLD) +include(${CMAKE_CURRENT_LIST_DIR}/repeat-fixture-common.cmake) diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-old-ctest-stdout.txt b/Tests/RunCMake/CTestCommandLine/repeat-fixture-old-ctest-stdout.txt new file mode 100644 index 0000000000..3be868b1bb --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-old-ctest-stdout.txt @@ -0,0 +1,23 @@ +^Test project .*/Tests/RunCMake/CTestCommandLine/repeat-fixture-old-build + Start 1: fixture_setup \(run 1/3\) + Test #1: fixture_setup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 1: fixture_setup \(run 2/3\) + Test #1: fixture_setup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 1: fixture_setup \(run 3/3\) +1/3 Test #1: fixture_setup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 1/3\) + Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 2/3\) + Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 3/3\) +2/3 Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: fixture_cleanup \(run 1/3\) + Test #3: fixture_cleanup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: fixture_cleanup \(run 2/3\) + Test #3: fixture_cleanup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: fixture_cleanup \(run 3/3\) +3/3 Test #3: fixture_cleanup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec ++ +100% tests passed out of 3 ++ +Total Test time \(real\) = +[0-9\.]+ sec$ diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-resume-cmake.cmake b/Tests/RunCMake/CTestCommandLine/repeat-fixture-resume-cmake.cmake new file mode 100644 index 0000000000..5179ad245e --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-resume-cmake.cmake @@ -0,0 +1,5 @@ +# `ctest -F` resumes from the checkpoint that records the tests already run. +cmake_policy(SET CMP0224 NEW) +include(${CMAKE_CURRENT_LIST_DIR}/repeat-fixture-common.cmake) + +add_test(NAME not_yet_run COMMAND ${CMAKE_COMMAND} -E true) diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-resume-ctest-stdout.txt b/Tests/RunCMake/CTestCommandLine/repeat-fixture-resume-ctest-stdout.txt new file mode 100644 index 0000000000..b6f8fb5913 --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-resume-ctest-stdout.txt @@ -0,0 +1,11 @@ +^Test project .*/Tests/RunCMake/CTestCommandLine/repeat-fixture-resume-build + Start 4: not_yet_run \(run 1/3\) + Test #4: not_yet_run \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 4: not_yet_run \(run 2/3\) + Test #4: not_yet_run \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 4: not_yet_run \(run 3/3\) +4/4 Test #4: not_yet_run \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec ++ +100% tests passed out of 1 ++ +Total Test time \(real\) = +[0-9\.]+ sec$ diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-separately-cmake.cmake b/Tests/RunCMake/CTestCommandLine/repeat-fixture-separately-cmake.cmake new file mode 100644 index 0000000000..5b2b0eb2f2 --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-separately-cmake.cmake @@ -0,0 +1,2 @@ +set(FIXTURE_REPEAT_MODE EACH_TEST_SEPARATELY) +include(${CMAKE_CURRENT_LIST_DIR}/repeat-fixture-common.cmake) diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-separately-ctest-stdout.txt b/Tests/RunCMake/CTestCommandLine/repeat-fixture-separately-ctest-stdout.txt new file mode 100644 index 0000000000..24007695ee --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-separately-ctest-stdout.txt @@ -0,0 +1,23 @@ +^Test project .*/Tests/RunCMake/CTestCommandLine/repeat-fixture-separately-build + Start 1: fixture_setup \(run 1/3\) + Test #1: fixture_setup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 1: fixture_setup \(run 2/3\) + Test #1: fixture_setup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 1: fixture_setup \(run 3/3\) +1/3 Test #1: fixture_setup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 1/3\) + Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 2/3\) + Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 3/3\) +2/3 Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: fixture_cleanup \(run 1/3\) + Test #3: fixture_cleanup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: fixture_cleanup \(run 2/3\) + Test #3: fixture_cleanup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: fixture_cleanup \(run 3/3\) +3/3 Test #3: fixture_cleanup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec ++ +100% tests passed out of 3 ++ +Total Test time \(real\) = +[0-9\.]+ sec$ diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-setup-only-cmake.cmake b/Tests/RunCMake/CTestCommandLine/repeat-fixture-setup-only-cmake.cmake new file mode 100644 index 0000000000..cfb450af63 --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-setup-only-cmake.cmake @@ -0,0 +1,6 @@ +# The mode set on the setup test governs the whole fixture, so the default +# CMP0224 records for the cleanup test must not override or conflict with it. +cmake_policy(SET CMP0224 NEW) +set(FIXTURE_REPEAT_MODE AROUND_ALL_REPEATS) +set(FIXTURE_REPEAT_TESTS fixture_setup) +include(${CMAKE_CURRENT_LIST_DIR}/repeat-fixture-common.cmake) diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-setup-only-ctest-stdout.txt b/Tests/RunCMake/CTestCommandLine/repeat-fixture-setup-only-ctest-stdout.txt new file mode 100644 index 0000000000..85b93a3e72 --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-setup-only-ctest-stdout.txt @@ -0,0 +1,15 @@ +^Test project .*/Tests/RunCMake/CTestCommandLine/repeat-fixture-setup-only-build + Start 1: fixture_setup +1/3 Test #1: fixture_setup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 1/3\) + Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 2/3\) + Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 3/3\) +2/3 Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: fixture_cleanup +3/3 Test #3: fixture_cleanup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec ++ +100% tests passed out of 3 ++ +Total Test time \(real\) = +[0-9\.]+ sec$ diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-until-pass-cmake.cmake b/Tests/RunCMake/CTestCommandLine/repeat-fixture-until-pass-cmake.cmake new file mode 100644 index 0000000000..fcf33ab438 --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-until-pass-cmake.cmake @@ -0,0 +1,18 @@ +# The fixture's setup test fails on its first run. With `--repeat until-pass` +# the whole unit runs again, and only the repetition that passed is reported. +cmake_policy(SET CMP0224 NEW) +enable_testing() + +set(TEST_OUTPUT_FILE "${CMAKE_CURRENT_BINARY_DIR}/test_output.txt") +file(WRITE "${TEST_OUTPUT_FILE}" "0") + +add_test(NAME fixture_setup COMMAND ${CMAKE_COMMAND} + "-DTEST_OUTPUT_FILE=${TEST_OUTPUT_FILE}" + -P "${CMAKE_CURRENT_SOURCE_DIR}/test1-pass.cmake") +set_tests_properties(fixture_setup PROPERTIES FIXTURES_SETUP MyFixture) + +add_test(NAME test_with_fixture COMMAND ${CMAKE_COMMAND} -E true) +set_tests_properties(test_with_fixture PROPERTIES FIXTURES_REQUIRED MyFixture) + +add_test(NAME fixture_cleanup COMMAND ${CMAKE_COMMAND} -E true) +set_tests_properties(fixture_cleanup PROPERTIES FIXTURES_CLEANUP MyFixture) diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-until-pass-ctest-stdout.txt b/Tests/RunCMake/CTestCommandLine/repeat-fixture-until-pass-ctest-stdout.txt new file mode 100644 index 0000000000..b32c36fb46 --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-until-pass-ctest-stdout.txt @@ -0,0 +1,18 @@ +^Test project .*/Tests/RunCMake/CTestCommandLine/repeat-fixture-until-pass-build + Start 1: fixture_setup \(run 1/3\) +1/3 Test #1: fixture_setup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\*\*\*Failed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 1/3\) +Failed test dependencies: fixture_setup +2/3 Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\*\*\*Not Run +[0-9\.]+ sec + Start 3: fixture_cleanup \(run 1/3\) +3/3 Test #3: fixture_cleanup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 1: fixture_setup \(run 2/3\) +1/3 Test #1: fixture_setup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 2/3\) +2/3 Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: fixture_cleanup \(run 2/3\) +3/3 Test #3: fixture_cleanup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec ++ +100% tests passed out of 3 ++ +Total Test time \(real\) = +[0-9\.]+ sec$ diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-warn-cmake-stderr.txt b/Tests/RunCMake/CTestCommandLine/repeat-fixture-warn-cmake-stderr.txt new file mode 100644 index 0000000000..b90e7ef026 --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-warn-cmake-stderr.txt @@ -0,0 +1,14 @@ +CMake Warning \(policy\): + Policy CMP0224 is not set: Test fixtures use AROUND_EACH_REPEAT mode by + default with ctest --repeat\. Run "cmake --help-policy CMP0224" for policy + details\. Use the cmake_policy command to set the policy and suppress this + warning\. + + For compatibility with older versions of CMake, the fixtures of the + following tests will use EACH_TEST_SEPARATELY mode: + + fixture_cleanup + fixture_setup + +This warning is for project developers\. Use -Wno-author or -Wno-policy to +suppress it\. diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-warn-cmake.cmake b/Tests/RunCMake/CTestCommandLine/repeat-fixture-warn-cmake.cmake new file mode 100644 index 0000000000..e12c991925 --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-warn-cmake.cmake @@ -0,0 +1 @@ +include(${CMAKE_CURRENT_LIST_DIR}/repeat-fixture-common.cmake) diff --git a/Tests/RunCMake/CTestCommandLine/repeat-fixture-warn-ctest-stdout.txt b/Tests/RunCMake/CTestCommandLine/repeat-fixture-warn-ctest-stdout.txt new file mode 100644 index 0000000000..4b0d962876 --- /dev/null +++ b/Tests/RunCMake/CTestCommandLine/repeat-fixture-warn-ctest-stdout.txt @@ -0,0 +1,23 @@ +^Test project .*/Tests/RunCMake/CTestCommandLine/repeat-fixture-warn-build + Start 1: fixture_setup \(run 1/3\) + Test #1: fixture_setup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 1: fixture_setup \(run 2/3\) + Test #1: fixture_setup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 1: fixture_setup \(run 3/3\) +1/3 Test #1: fixture_setup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 1/3\) + Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 2/3\) + Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: test_with_fixture \(run 3/3\) +2/3 Test #2: test_with_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: fixture_cleanup \(run 1/3\) + Test #3: fixture_cleanup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: fixture_cleanup \(run 2/3\) + Test #3: fixture_cleanup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 3: fixture_cleanup \(run 3/3\) +3/3 Test #3: fixture_cleanup \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec ++ +100% tests passed out of 3 ++ +Total Test time \(real\) = +[0-9\.]+ sec$ diff --git a/Tests/RunCMake/GoogleTest/FixtureRepeatMode-NEW-check.cmake b/Tests/RunCMake/GoogleTest/FixtureRepeatMode-NEW-check.cmake new file mode 100644 index 0000000000..a155b83781 --- /dev/null +++ b/Tests/RunCMake/GoogleTest/FixtureRepeatMode-NEW-check.cmake @@ -0,0 +1,2 @@ +set(expect_mode "AROUND_EACH_REPEAT") +include(${CMAKE_CURRENT_LIST_DIR}/FixtureRepeatMode-verify.cmake) diff --git a/Tests/RunCMake/GoogleTest/FixtureRepeatMode-NEW.cmake b/Tests/RunCMake/GoogleTest/FixtureRepeatMode-NEW.cmake new file mode 100644 index 0000000000..63d7d395c1 --- /dev/null +++ b/Tests/RunCMake/GoogleTest/FixtureRepeatMode-NEW.cmake @@ -0,0 +1,2 @@ +cmake_policy(SET CMP0224 NEW) +include(${CMAKE_CURRENT_LIST_DIR}/FixtureRepeatMode-common.cmake) diff --git a/Tests/RunCMake/GoogleTest/FixtureRepeatMode-OLD-check.cmake b/Tests/RunCMake/GoogleTest/FixtureRepeatMode-OLD-check.cmake new file mode 100644 index 0000000000..1c5d45e91b --- /dev/null +++ b/Tests/RunCMake/GoogleTest/FixtureRepeatMode-OLD-check.cmake @@ -0,0 +1,2 @@ +set(expect_mode "") +include(${CMAKE_CURRENT_LIST_DIR}/FixtureRepeatMode-verify.cmake) diff --git a/Tests/RunCMake/GoogleTest/FixtureRepeatMode-OLD.cmake b/Tests/RunCMake/GoogleTest/FixtureRepeatMode-OLD.cmake new file mode 100644 index 0000000000..72f3589a7a --- /dev/null +++ b/Tests/RunCMake/GoogleTest/FixtureRepeatMode-OLD.cmake @@ -0,0 +1,2 @@ +cmake_policy(SET CMP0224 OLD) +include(${CMAKE_CURRENT_LIST_DIR}/FixtureRepeatMode-common.cmake) diff --git a/Tests/RunCMake/GoogleTest/FixtureRepeatMode-common.cmake b/Tests/RunCMake/GoogleTest/FixtureRepeatMode-common.cmake new file mode 100644 index 0000000000..4ab34b0df3 --- /dev/null +++ b/Tests/RunCMake/GoogleTest/FixtureRepeatMode-common.cmake @@ -0,0 +1,9 @@ +# The discovered tests are created by a script that ctest runs, so check the +# generated file that carries their properties. +add_executable(example IMPORTED) +set_property(TARGET example PROPERTY IMPORTED_LOCATION + ${CMAKE_CURRENT_BINARY_DIR}/example) +include(GoogleTest) +gtest_discover_tests(example DISCOVERY_MODE PRE_TEST + PROPERTIES FIXTURES_SETUP ExampleFixture +) diff --git a/Tests/RunCMake/GoogleTest/FixtureRepeatMode-verify.cmake b/Tests/RunCMake/GoogleTest/FixtureRepeatMode-verify.cmake new file mode 100644 index 0000000000..0275ca146a --- /dev/null +++ b/Tests/RunCMake/GoogleTest/FixtureRepeatMode-verify.cmake @@ -0,0 +1,23 @@ +# Verify whether the generated discovery scripts carry the mode chosen by +# policy CMP0224 through to the tests they create. ${expect_mode} is the +# mode expected, or empty if nothing should be recorded. A multi-config +# generator writes one script per configuration. +set(property "_CMAKE_DEFAULT_FIXTURE_REPEAT_MODE") +file(GLOB discovery_scripts + "${RunCMake_TEST_BINARY_DIR}/example_*_discovery.cmake") +if(NOT discovery_scripts) + set(RunCMake_TEST_FAILED "No test discovery script was generated.") + return() +endif() +foreach(discovery_script IN LISTS discovery_scripts) + file(READ "${discovery_script}" content) + if(expect_mode STREQUAL "") + if(content MATCHES "${property}") + set(RunCMake_TEST_FAILED + "${property} should not appear in ${discovery_script}:\n${content}") + endif() + elseif(NOT content MATCHES "${property}[^A-Za-z_]+${expect_mode}") + set(RunCMake_TEST_FAILED + "${property} ${expect_mode} missing from ${discovery_script}:\n${content}") + endif() +endforeach() diff --git a/Tests/RunCMake/GoogleTest/RunCMakeTest.cmake b/Tests/RunCMake/GoogleTest/RunCMakeTest.cmake index 8233628485..daa08e034b 100644 --- a/Tests/RunCMake/GoogleTest/RunCMakeTest.cmake +++ b/Tests/RunCMake/GoogleTest/RunCMakeTest.cmake @@ -13,6 +13,10 @@ endif() run_cmake(PropertiesOdd) +# Policy CMP0224 has to reach the tests that discovery creates. +run_cmake(FixtureRepeatMode-NEW) +run_cmake(FixtureRepeatMode-OLD) + function(run_GoogleTest DISCOVERY_MODE) # Use a single build tree for a few tests without cleaning. set(RunCMake_TEST_BINARY_DIR ${RunCMake_BINARY_DIR}/GoogleTest-build) diff --git a/Tests/RunCMake/discover_tests/RunCMakeTest.cmake b/Tests/RunCMake/discover_tests/RunCMakeTest.cmake index 4042d0acc1..c392895659 100644 --- a/Tests/RunCMake/discover_tests/RunCMakeTest.cmake +++ b/Tests/RunCMake/discover_tests/RunCMakeTest.cmake @@ -28,6 +28,17 @@ block() run_cmake(bad-test-properties-cmake) endblock() +block() + # Policy CMP0224 has to reach the tests that discovery creates. + set(RunCMake_TEST_BINARY_DIR "${RunCMake_BINARY_DIR}/fixture-repeat-mode-build") + run_cmake(fixture-repeat-mode) + set(RunCMake_TEST_NO_CLEAN 1) + run_cmake_command(fixture-repeat-mode-build + ${CMAKE_COMMAND} --build . --config Debug) + run_cmake_command(fixture-repeat-mode-test + ${CMAKE_CTEST_COMMAND} -C Debug --repeat until-fail:2) +endblock() + function(run_case CASE) set(RunCMake_TEST_BINARY_DIR ${RunCMake_BINARY_DIR}/${CASE}-build) run_cmake(${CASE}) diff --git a/Tests/RunCMake/discover_tests/fixture-repeat-mode-test-stdout.txt b/Tests/RunCMake/discover_tests/fixture-repeat-mode-test-stdout.txt new file mode 100644 index 0000000000..742ea06b5b --- /dev/null +++ b/Tests/RunCMake/discover_tests/fixture-repeat-mode-test-stdout.txt @@ -0,0 +1,13 @@ +^Test project .*/Tests/RunCMake/discover_tests/fixture-repeat-mode-build + Start 1: setup_case_one \(run 1/2\) +1/2 Test #1: setup_case_one \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: needs_fixture \(run 1/2\) +2/2 Test #2: needs_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 1: setup_case_one \(run 2/2\) +1/2 Test #1: setup_case_one \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec + Start 2: needs_fixture \(run 2/2\) +2/2 Test #2: needs_fixture \.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\. Passed +[0-9\.]+ sec ++ +100% tests passed out of 2 ++ +Total Test time \(real\) = +[0-9\.]+ sec$ diff --git a/Tests/RunCMake/discover_tests/fixture-repeat-mode.cmake b/Tests/RunCMake/discover_tests/fixture-repeat-mode.cmake new file mode 100644 index 0000000000..5d899b64a6 --- /dev/null +++ b/Tests/RunCMake/discover_tests/fixture-repeat-mode.cmake @@ -0,0 +1,20 @@ +# Tests created by discovery cannot be reached by policy CMP0224 when it is +# read, so discover_tests() records the mode the policy chose for them. +cmake_policy(SET CMP0224 NEW) +enable_language(C) +enable_testing() + +add_executable(fake_discovery fake_discovery.c) + +discover_tests(COMMAND fake_discovery + DISCOVERY_ARGS --list_tests + DISCOVERY_MATCH "^(case_one),LBL1$" + TEST_NAME "setup_\\1" + TEST_ARGS "\\1" + TEST_PROPERTIES + FIXTURES_SETUP DiscoveredFixture +) + +add_test(NAME needs_fixture COMMAND ${CMAKE_COMMAND} -E true) +set_tests_properties(needs_fixture PROPERTIES + FIXTURES_REQUIRED DiscoveredFixture)