From a8161b7dc67a71ba12eaba2e9ddc0b6e8b156392 Mon Sep 17 00:00:00 2001 From: Mark Kittisopikul Date: Thu, 30 Jul 2026 15:32:26 -0400 Subject: [PATCH] Stabilize nvhpc CI: force Open MPI off UCX to avoid flaky UCP worker failures (#6554) GitHub-hosted runners have no real InfiniBand, and the Open MPI bundled with NVIDIA HPC-X intermittently has UCX select a non-functional verbs interface (e.g. ud_verbs/mana_0), causing: ucp_worker.c:1487 UCX ERROR uct_iface_open(ud_verbs/mana_0:1) failed pml_ucx.c:319 Error: Failed to create UCP worker MPI_INIT has failed because at least one MPI process is unreachable which fails h5_api_test_parallel_* tests in the "Run Tests" step. This only affects nvhpc.yml (HPC-X's bundled Open MPI ships UCX); main-par.yml uses Ubuntu's openmpi-bin package, which doesn't use UCX and is unaffected. Co-authored-by: Claude Sonnet 5 --- .github/workflows/nvhpc.yml | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/.github/workflows/nvhpc.yml b/.github/workflows/nvhpc.yml index 4690033733b..8b210c02739 100644 --- a/.github/workflows/nvhpc.yml +++ b/.github/workflows/nvhpc.yml @@ -23,6 +23,16 @@ jobs: CUDAVERDOT: "13.2" NVERDOT: "26.5" NVERDASH: "26-5" + # CI stabilization: GitHub-hosted runners have no real InfiniBand, and the + # Open MPI bundled with NVIDIA HPC-X sometimes has UCX select a + # non-functional verbs interface (e.g. ud_verbs/mana_0), causing + # intermittent "Failed to create UCP worker" / MPI_Init unreachable (-12) + # failures. Force ob1 + tcp/shared-memory transports and skip UCX. + OMPI_MCA_pml: ob1 + OMPI_MCA_btl: self,tcp,sm + OMPI_MCA_mtl: ^ucx + UCX_TLS: tcp,self,sm + OMPI_MCA_btl_tcp_if_exclude: lo,docker0 steps: - name: Get Sources