* Fix release-progress badges to read native issue-level Priority field
The Project Priority board field was consolidated into a Priority field
managed at the issue level (GitHub's org-wide custom issue fields), which
the GraphQL API mirrors into projects as ProjectV2ItemIssueFieldValue
instead of ProjectV2ItemFieldSingleSelectValue. The badge script's query
had no fragment for that type, so it silently found zero priority values
and would raise ProjectFieldMissingError on every run.
* Show TBD on the Next Release badge when no milestone due date is set
Previously the badge silently omitted any due-date annotation when the
milestone had none, which read as if the badge just hadn't picked one up.
* Update release-schedule image and plantuml source from add-next-release-badge branch
* Don't flag CODEOWNERS auto-assignment as a manual reviewer add on a PR's first pass
GitHub's own CODEOWNERS engine fires an identical review_requested webhook
(sender type User, not Bot) when it auto-assigns owners at PR-open time. If
one of those survives the cancel-in-progress race as the run that actually
executes, it was mistaken for a human deliberately requesting that reviewer,
permanently flagging them "manually added" and requiring separate approval —
e.g. a catch-all "*" owner also named on a touched area's CODEOWNERS line got
flagged on every PR touching that area, even though no human ever picked them.
Gate the manual-add detection and the sticky per-area assignment on
!isFirstCoordinationPass (no checklist comment posted yet), since that's
exactly the window where GitHub's own auto-assignment is indistinguishable
from a real human pick.
Added 3 new functions to support this: H5TSset_internal_threads(), H5Pset_io_threads(), and H5Pget_io_threads().
This feature internally parallelizes read operations on chunked datasets. H5TSset_internal_threads() is used to enable the feature globally, while H5Pset_io_threads() can be used to disable the feature on a per-operation basis. These functions are only available when the library is configured with HDF5_ENABLE_CONCURRENCY=ON. When performing an internally threaded read, the library will concurrently read from disk, unfilter, and scatter to memory all chunks in a read operation on a chunked dataset. Currently each of these sub-operations is serialized (protected by a mutex), so there is not yet likely to be any performance improvement.
Checkout deliberately uses persist-credentials: false so the workflow
token never lands in git config, but the add-and-commit step never got
a token of its own to push with -- it silently relied on credentials
checkout no longer leaves behind. Every push to any branch failed with
"could not read Username for 'https://github.com'" once clang-format
found something to fix, regardless of the repo's Actions write-permission
setting.
* Close datatype IDs derived from the memory type in the JNI translate helpers
The object-tree read/write helpers in h5util.c derive a base datatype from
the memory type with H5Tget_super() for the variable-length, array and
complex classes, but never closed it. Because an hid_t is not reclaimed when
the native method returns, every read or write of such data leaked at least
one datatype ID for the lifetime of the process, and nested types leaked
one per level.
This PR updates the helpers to close the derived type in their done: blocks,
which covers both the success and the error paths, and to reset the id in the
compound loops to avoid the potential for double closes.
It also has the helpers release the class references that the per-element
helpers look up on entry. These are local references, so they were reclaimed
when the enclosing native method returned, but a compound read calls the helper
once per member per element and held one set per call until then. Releasing
them at the single exit bounds the count of references to the recursion depth.
* Add CHANGELOG entry for the JNI datatype ID leak fix
* Restrict the derived datatype close guards to strictly positive IDs
hid_t 0 is not a valid datatype ID, so H5Tclose(0) would fail.
* Revert "Restrict the derived datatype close guards to strictly positive IDs"
An hid_t of 0 not being a valid ID is a property of the current H5I
encoding rather than a documented guarantee, so the JNI helpers should
not depend on it.
The hdf5_vol_rest_fetchcontent job fails at "Install HSDS dependencies"
because HSDS pins numpy==2.4.6, which publishes no distributions for
Python 3.10:
ERROR: Could not find a version that satisfies the requirement
numpy==2.4.6 (from versions: ... 2.2.5, 2.2.6)
numpy 2.4.6 declares requires_python >= 3.11, and HSDS's pyproject.toml
now says requires-python = ">=3.11". Move the matrix to Python 3.12,
which is what HSDS uses for its own unix-socket test job - the same
configuration this job exercises.
Also drop the "Fix requests version" step that force-downgraded to
requests==2.31.0. HSDS now pins requests-unixsocket==0.4.1, which fixed
the requests 2.32 breakage that step worked around, so the downgrade only
fights HSDS's own pins. Upstream HSDS has already commented the same step
out of its socket-test job.
Fixes#6657
Claude-Session: https://claude.ai/code/session_011qUw9C11arB1WftiT56EA4
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* review-checklist: don't lose a reviewer who only left comments
GitHub un-requests a reviewer the instant they submit any review,
including a comment-only one from batching several inline comments into
a single submission — not just Approve/Request-changes. Unlike a stale
approval, this never produces a DISMISSED transition, so the checklist
had no way to tell "abandoned the area" apart from "still reviewing,
just hasn't finished yet."
This silently dropped the reviewer's mention from the checklist display
the moment they left comments, and risked the next push's additive-fill
picker handing their area to a completely different load-balanced
reviewer (observed on PR #6645: jhendersonHDF's batched review comments
repeatedly vanished him from src/test's rows mid-review).
Track a sticky area assignee as still engaged whenever they have no
APPROVED/CHANGES_REQUESTED/DISMISSED review on record, regardless of
whether GitHub currently lists them as requested, and use that in the
read-only display path, resolveAreaPicks, and the additive-fill picker.
* draft-pr-policy: a checked keepalive checkbox is real activity too
lastRealActivityAt() deliberately ignores bot comments so a metadata-only
bump can't dodge the staleness check forever. But checking the "Still
working on this" checkbox is an edit to the bot's own keepalive comment,
not a new comment of the human's own — its author stays
github-actions[bot], so the edit was invisible to the activity check too.
Confirming via the checkbox removed the label and posted "Thanks for
confirming" without ever moving the underlying 60-day clock, so the very
next scheduled run saw the same stale last-activity timestamp and
immediately re-flagged it — contradicting the checkbox's own promise
that checking it "resets this".
Observed on #6326: once its true last activity fell behind the 60-day
window, and only the checkbox (never a new commit or comment) was used
to confirm it, the label thrashed on and off on a roughly 1-2 day loop.
Now a checked keepalive checkbox counts via the comment's updated_at
(when it was toggled), same as any other real activity signal.
The issue_comment trigger let every PR comment boot a full runner just
to make one API call checking draft status, even though the only
comments that matter are ones on PRs already labeled draft-stale. That
label snapshot is already present in the webhook payload, so gate the
job on it directly and drop the now-redundant draft-status API step.
Adds HDF5Examples/C/H5PAR/ph5_comm_split.c, showing parallel HDF5 use on a
communicator produced by MPI_Comm_split rather than on MPI_COMM_WORLD.
The example splits MPI_COMM_WORLD by rank parity. The even ranks form a
writer group that collectively creates a dataset with one row per writer
rank, then reopens the file read-only on the same sub-communicator and
verifies what it wrote. The odd ranks stand in for a group with other
responsibilities and make no HDF5 calls at all.
The comments cover the two rules this pattern depends on: MPI_Comm_split is
collective over the parent communicator, so every rank must call it even if
it never opens the file; and once a file is opened with a FAPL carrying a
sub-communicator, every collective HDF5 call on that file is collective over
that sub-communicator alone.
The example runs with any number of MPI ranks.
h5close_f reset its count of the objects created by h5open_f with
CALL h5fget_obj_count_f(INT(H5F_OBJ_ALL_F,HID_T), H5F_OBJ_ALL_F, &
H5OPEN_NUM_OBJ, error)
passing the H5F module variable H5OPEN_NUM_OBJ as the actual argument for
the INTENT(OUT) obj_count dummy, while h5fget_obj_count_f also reads
H5OPEN_NUM_OBJ by use association. F2018 15.5.2.13 prohibits referencing a
variable through use association once it has been redefined through a dummy
argument in the same call, so the result depended on how the compiler
implemented argument association. Where the actual argument was passed by
reference the subtraction collapsed to 0 - 0 and produced the intended zero;
where the compiler used copy-in/copy-out it evaluated 0 - H5OPEN_NUM_OBJ and
left the count negative.
A negative count then defeats the guard at the top of h5open_f, which returns
early when H5OPEN_NUM_OBJ is non-zero. h5open_f reported success without
calling h5init_types_c, leaving H5T_NATIVE_INTEGER and the other predefined
types holding identifiers that h5close_f had released.
h5close_f now assigns the count directly, which is what the comment there has
always described. h5fget_obj_count_f computes into a local variable so that no
caller can reintroduce the aliasing; note that this change alone would make
the old h5close_f call site produce the negative count on every compiler
rather than only on some, so the two belong together. H5OPEN_NUM_OBJ is also
given an initial value, since h5open_f tests it before anything assigns to it.
h5fget_obj_count_f subtracted every object created by h5open_f from a count of
a single object type, so with the interface open a query such as
CALL h5fget_obj_count_f(INT(H5F_OBJ_ALL_F,HID_T), H5F_OBJ_FILE_F, n, error)
returned a negative n and hdferr of 0. h5open_f now records what it leaves open
per object type, and a count is adjusted by the recorded value for the types
being counted, so the adjustment does not depend on which types the
initialization creates. The check for a negative count runs both on the value
returned by H5Fget_obj_count and after the adjustment.
h5fget_obj_ids_f applied no such adjustment, so it returned the identifiers
h5open_f opened alongside the application's own and disagreed with
h5fget_obj_count_f about the same query: with only a file and a group open,
H5F_OBJ_ALL_F counted 2 objects but listed 62. The C API reports 2 and 2. An
application walking the list found datatypes it never opened, and closing them
breaks the Fortran interface. h5fget_obj_ids_f now excludes those identifiers,
requesting enough from H5Fget_obj_ids that max_objs of the application's own
can still be returned when the two are interleaved.
The h5open/h5close test verified its object counts by calling
h5fget_obj_count_f after h5close_f, when h5open_f is the only call the Fortran
interface permits. Those checks move to after the interface is reopened, where
they additionally confirm that the predefined types are valid again, that the
preceding h5close_f released the previous h5open_f's types, and that
h5fget_obj_ids_f agrees with h5fget_obj_count_f.
The Fortran tests also aborted unrecoverable failures with STOP, which exits
with a success status whether the stop code is absent or is a string, so a run
that died part way through reported no failure to CTest. They now exit through
h5_exit_f(1). The STOPs that end a run normally, in fflush1 and in the async
test's skip path for a build without MPI_THREAD_MULTIPLE, are unchanged.
Fixes#6642Fixes#6648
Reported and diagnosed by Dom Heinzeller.
* Reject chunked datasets with mismatched chunk/dspace rank
H5D__chunk_construct() validates that the chunk layout dimensionality
matches the dataspace rank, but that runs only at dataset creation time.
When an existing dataset is opened, H5D__chunk_init() didn't repeat the
check, so a file whose stored chunk rank disagreed with its dataspace rank
was accepted. During chunk I/O the memory-selection rank (from the
dataspace) and the file-selection rank (chunk ndims - 1) then differ, which
produces a zero stride that causes a divide-by-zero in
H5S__hyper_iter_get_seq_list().
H5D__chunk_init() now performs the same dimensionality check on open (the
stored chunk rank includes the extra element-size dimension, so it must be
exactly one greater than the dataspace rank) and rejects a mismatch with an
error.
Added test_chunk_dims_mismatch() as a regression test in test/dsets.c
Fixes#6491
* Fix typo
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Clarify element-vs-byte wording
* Validate chunk/dataspace rank at layout decode time
Move the stored-chunk-rank vs dataspace-rank consistency check out of
H5D__chunk_init() and into H5O__layout_decode(), so a malformed chunked
layout is rejected as the message is decoded (mirroring the fill/datatype
size check in the fill message decode).
* Update release_docs/CHANGELOG.md
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
* Update CHANGELOG
* Pin format version bounds in bad chunk layout generator
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
The Fortran example set sel_type = H5S_BLOCK_F on the branch where a rank
calls H5Sselect_none (no data to contribute) and H5S_ALL_F on the branch
where a rank sets a real hyperslab, which is backwards. The equivalent C
example (ph5_filtered_writes_no_sel.c) does the opposite:
H5Dwrite(..., no_selection ? H5S_ALL : H5S_BLOCK, ...);
i.e. H5S_ALL when there is no selection (mirrors the file dataspace's own
empty selection) and H5S_BLOCK when a hyperslab was set (memory buffer
matches the block shape of the file selection).
With the inversion, H5Dwrite_f feeds a bogus selection into the collective
filtered I/O path, corrupting the element/byte-length accounting in
H5D__mpio_collective_filtered_chunk_update, which segfaults inside
H5D_select_io_mem -> H5VM_memcpyvv on 12 MPI ranks. Reproduces on current
develop; the bug has been present since the example was added in #3916.
When pushing errors to an error stack that is already full, the
library skipped the push operation but didn't inform calling code
about what happened. This resulted in calling code leaking memory,
leaving reference counts on IDs incremented and causing an infinite
loop while closing the library.
* Build the examples as C++11 to match the HDF5 C++ library
The standalone examples build forces CMAKE_CXX_STANDARD 98, but
H5public.h includes <cinttypes>, which requires C++11. Any C++
translation unit that includes hdf5.h is therefore affected, not just
users of the C++ API, and the HDF5 C++ library itself is built as C++11
(config/flags/HDFCompilerCXXFlags.cmake). The C++ examples do not
compile as a result, against static or shared HDF5 alike.
Only the standalone build is affected, which is why this is not visible
in ordinary use. The C++98 setting lives in BASIC_SETTINGS, and
HDF5Examples/CMakeLists.txt skips that whole block when
EXAMPLES_EXTERNALLY_CONFIGURED is set -- which HDF5 does for its own
in-tree example build (config/cmake/HDF5ExampleCache.cmake). Built in
tree, the examples inherit HDF5's C++11 and compile normally, and that
is the path the CI workflows exercise. The standalone path, where the
C++98 setting does apply, is driven by the release scripts rather than
by the workflows, and has the C++ examples off by default.
* Select the examples' HL, Fortran and C++ libraries on the right variable
When the examples are built standalone against an installed HDF5, the
HL, Fortran and C++ branches choose between the shared and static
libraries using BUILD_SHARED_LIBS, while the C branch just above them
uses H5EXAMPLE_USE_SHARED_LIBS.
H5EXAMPLE_USE_SHARED_LIBS is what decides whether the "shared" or the
"static" component is requested from find_package, so only the matching
HDF5_<linkage>_<lang>_FOUND variables are ever set. BUILD_SHARED_LIBS
cannot select a linkage on its own; it can only agree or fail to match.
Of its four combinations with H5EXAMPLE_USE_SHARED_LIBS, three produce
no observable difference. In the fourth, H5EXAMPLE_USE_SHARED_LIBS=ON
with BUILD_SHARED_LIBS unset, the shared branch is not taken and the
static branch cannot be, so the HL, Fortran and C++ examples are
disabled with "libs not found" even though the libraries are installed
and were found.
Use H5EXAMPLE_USE_SHARED_LIBS, which is the declared option and is
already what the C branch uses.
A build driven through config/examples/CTestScript.cmake does not reach
the broken combination, because it configures with
HDF5Examples/config/cmake/cacheinit.cmake, which forces
BUILD_SHARED_LIBS=ON. A direct cmake invocation without that cache file
does. In either case the HL, Fortran and C++ examples are off by
default, so this is only visible once they are enabled.
BUILD_SHARED_LIBS remains documented as a user option in
config/examples/HDF5_Examples_options.cmake but no longer influences
library selection; that comment should be revisited separately.
These are made no-ops by the cmake_minimum_required(VERSION 3.26)
* CMP0074 by CMake 3.12
* CMP0077 by CMake 3.13
* CMP0083 by CMake 3.14
* CMP0127 by CMake 3.22
* CMP0135 by CMake 3.24
When called on a FAPL without ROS3 VFD block caching parameters
set on it, the H5Pget_fapl_ros3_block_caching() function previously
returned the VFD's default page buffer size value instead of its
default I/O block cache size value.
* Abort on infinite loop even when error output is disabled
* Remove abort() on infinite close loop
* Update CHANGELOG.md
* Reference the fixed issue in the CHANGELOG entry
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
The base type lookups in the object-tree helpers tested the
returned hid_t for truth rather than for a negative value. A failed
lookup returns H5I_INVALID_HID (-1), resolving true when checked, so any
failure wouldn't be caught until later.
When using the '-s' (h5ls) or '-r' (h5dump) option to display 1-byte integer datasets and attributes as ASCII characters, a closing double-quote character for data values was dropped in some cases.
Every CODEOWNERS "avalanche" (multiple owners of one area simultaneously
requested — on PR creation, on ready_for_review, or when a push first
touches an already-covered area) was resolved by re-running the
load-balancer from scratch, with no memory of who was already the
settled reviewer for that area. Since open-PR review load drifts over
time, this made the pick non-deterministic across repeated events and
caused three related symptoms:
- marking a draft ready for review could swap out reviewers who were
already assigned (manually or from an earlier pass) for different
people
- a routine follow-up push could re-avalanche an area and bump the
already-engaged reviewer for whoever currently has a lighter queue
- a manually re-requested reviewer survived the triggering event (the
existing forced-pick handling already covers that) but got silently
removed again on a later event, since two now-requested owners look
identical to an unpruned avalanche without that memory
Add a persisted per-area "sticky assignment" record (ASSIGNED_PREFIX,
alongside the existing exclusion and manually-added markers) and a
resolveAreaPicks() helper that prefers a valid sticky pick, then a lone
already-requested owner, before ever falling back to a fresh
load-balanced pick. Manual review_requested actions now write to this
record too, so a deliberate reviewer choice holds up across future
runs, not just the run it was made on.
Co-authored-by: H. Joe Lee <hyoklee@hdfgroup.org>
HDF5 discovers Java twice, through mechanisms with opposite search
precedence, and never reconciles them:
find_package(Java) in CMakeLists.txt sets Java_JAVA_EXECUTABLE,
Java_JAVAC_EXECUTABLE and Java_VERSION_STRING. FindJava searches
JAVA_HOME via HINTS, which outrank $PATH.
project(HDF5_JAVA C Java) in java/CMakeLists.txt sets
CMAKE_Java_COMPILER/RUNTIME/ARCHIVE. CMakeDetermineJavaCompiler searches
$ENV{JAVA_HOME}/bin only as a PATHS entry, the lowest priority slot,
below $PATH, and ignores the CMake variable JAVA_HOME entirely.
When JAVA_HOME and PATH refer to different JDKs the two diverge, so the
FFM/JNI version gate in java/CMakeLists.txt is evaluated against one JDK
while the code is compiled and tested with another. On macOS both resolve
to the /usr/bin stubs, which re-select a JDK from JAVA_HOME on every
invocation, so javac and java can differ inside a single build and every
Java test fails to load its class files with UnsupportedClassVersionError.
Propagate the probed toolchain before the Java language is enabled, filling
each variable independently. The guard in CMakeDetermineJavaCompiler.cmake
is if(NOT CMAKE_Java_COMPILER) and it closes only after the runtime and
archive lookups, so a caller passing just
-DJAVA_HOME=$JAVA_HOME -DCMAKE_Java_COMPILER=$JAVA_HOME/bin/javac
suppresses the other two lookups entirely and ends up with an empty
CMAKE_Java_RUNTIME and an empty TEST_JAVA in the generated CTest files.
Filling them one at a time keeps an explicit override working while still
completing the toolchain.
Verified two ways. Configuring with PATH -> JDK 11 and JAVA_HOME -> JDK 20
previously reported Java 20 and selected the JNI implementation while
CMAKE_Java_COMPILER, CMAKE_Java_RUNTIME and the generated TEST_JAVA all
pointed at JDK 11; they now all resolve to JDK 20. On macOS 26 arm64 with
JAVA_HOME -> JDK 25 the same build went from 95 failing Java tests to none.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Document the Metadata Cache Image block and header message
Adds docs/doxygen/dox/H5.format.4.0.dox coverage for:
- Level 1J: the Metadata Cache Image Block (subsec_fmt4_infra_mdci)
- IV.A.3.y: the Metadata Cache Image header message (0x0018)
- TOC/cross-reference entries and the superblock-extension list entry for both
Also clarifies the 'Size of Header Message #n Data' field description
for v1 and v2 object header messages to note it includes flags/reserved
(v1) or flags/creation-order (v2) fields, not just type and length.
* Address review feedback on MDCI message docs
- Clarify that the Metadata Cache Image message and its image block
are only removed/reclaimed on the next file open when the file is
opened for read/write access; read-only opens leave them unchanged.
- Fix a malformed table row (missing closing </td>) in the Status
field of the IV.A.3.y message table.
Address all zizmor 1.25.2 findings in the Maven/Java-themed workflows
with no behavioral change:
- template-injection: move attacker-controllable ${{ }} expressions
(matrix.*, runner.workspace, github.actor, github.base_ref,
steps.*.outputs.*, needs.*.outputs.*) out of run: script bodies into
step-level env: blocks referenced as shell/pwsh variables.
- artipacked: add persist-credentials: false to all actions/checkout
steps (none of these workflows push to git).
- excessive-permissions: add/tighten explicit permissions. Reduce
overly broad workflow-level packages:/pull-requests: grants to
contents: read, granting minimal packages: read/write only to the
specific jobs that need it.
- secrets-inherit (release.yml): remove secrets: inherit on the
test-maven-packages.yml call, which declares no workflow_call secrets.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
* Harden workflow checkout credential hygiene + permissions
Address zizmor static-analysis findings:
- artipacked: add `persist-credentials: false` to actions/checkout
steps in codeql.yml (2), test-binary-installation.yml (2),
clang-format-check.yml, freebsd.yml, h5py.yml, openbsd.yml,
signed-plugins.yml, and update-progress.yml. None of these jobs
push to git, so dropping the persisted checkout credential is safe.
- excessive-permissions: restrict call-workflows.yml top-level
permissions to `contents: read`, and grant `packages: write` /
`pull-requests: write` only on the two jobs that call the
maven-staging reusable workflow (which declares those needs).
No behavioral change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Removed extra blank lines.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
* Harden scheduled/downstream build workflows against zizmor findings
Fix all zizmor static-analysis findings in the scheduled/downstream build
workflows without changing behavior:
- template-injection: move ${{ }} expressions out of run: script bodies
into step-level env: blocks referenced as shell variables
- artipacked: add persist-credentials: false to actions/checkout steps
(none of these jobs push to git)
- excessive-permissions: add top-level 'permissions: contents: read' to
macos-26-matrix.yml (other files already restrict permissions)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Address Copilot review comments on PR #6544
- Drop unnecessary command substitution in daily-schedule.yml's
FILE_NAME_BASE assignment
- Name the two env-block steps in daily-build.yml that previously ran
unnamed (easier to scan in logs)
- Quote $GITHUB_OUTPUT in the getinputs step
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Remove extra blank lines.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
Address GitHub Actions static-analysis (zizmor/CodeQL) findings in the
S3 publishing workflows:
- publish-release.yml: hoist every inputs/secrets/vars value used inside
run blocks into job-level env vars and reference plain shell variables
($USE_TAG, $FILE_NAME, $TARGET_DIR, $S3_BUCKET, $TARGET_PATH, $DRY_RUN),
eliminating 41 template-injection findings. Add persist-credentials:
false to the checkout (artipacked).
- publish-branch.yml: add persist-credentials: false to the checkout
(artipacked); its S3 sync already used env vars.
No behavioral change: the same values are used, only via the shell
environment instead of direct ${{ }} expansion into the script body.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
On SunOS the C and C++ flag files appended the single string
"-erroff=%none -DBSD_COMP" to HDF5_CMAKE_C/CXX_FLAGS. Because that is one
list element, target_compile_options() hands it to the compiler as one
argument, and -erroff is an Oracle Developer Studio option to begin with.
gcc silently ignores the result (it parses the leading -e as the linker
entry-symbol option), but clang errors out:
clang: error: unknown argument: '-erroff=%none -DBSD_COMP'
which breaks the build on the first object file. As a side effect,
-DBSD_COMP was never actually reaching the compiler as a define.
Emit -erroff=%none only for SunPro, and append -DBSD_COMP as its own list
element so it is passed as a separate argument on all Solaris compilers.
Fixes#6569
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
GitHub-hosted runners have no real InfiniBand, and the Open MPI bundled
with NVIDIA HPC-X intermittently has UCX select a non-functional verbs
interface (e.g. ud_verbs/mana_0), causing:
ucp_worker.c:1487 UCX ERROR uct_iface_open(ud_verbs/mana_0:1) failed
pml_ucx.c:319 Error: Failed to create UCP worker
MPI_INIT has failed because at least one MPI process is unreachable
which fails h5_api_test_parallel_* tests in the "Run Tests" step. This
only affects nvhpc.yml (HPC-X's bundled Open MPI ships UCX); main-par.yml
uses Ubuntu's openmpi-bin package, which doesn't use UCX and is unaffected.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Add Fortran ABI compatibility check to abi-report workflow
Mirrors the existing C/HL/C++ abi-dumper + abi-compliance-checker
steps for libhdf5_fortran.so, plus a supplementary nm-based exported
symbol diff since gfortran's mangled names don't encode argument
lists and DWARF-based diffing has not been validated against
Fortran-specific constructs (array descriptors, derived types).
* Auto-detect ABI reference release instead of hardcoding it
file_ref was pinned to '2.0.0' in both daily-build.yml and release.yml
and had gone stale across two releases (2.1.0, 2.1.1) without being
bumped, silently comparing against a two-versions-old baseline. Have
abi-report.yml query the GitHub API for the latest published
HDFGroup/hdf5 release when file_ref isn't explicitly given, so callers
no longer need to remember to update a pinned tag after every release.
Verified the resolution command returns 2.1.1 against the live API.
Also documents the expected false-positive pattern in the new Fortran
ABI check: abi-dumper misreads gfortran's DWARF encoding of
assumed-shape array descriptors as fixed array bounds, producing bulk
Low-severity noise on the KIND/RANK-generated H5_gen.F90 procedures
that isn't a real interface change.
* Various improvement in documentation and a decoding function
- Improves documentation on the type size when creating/accessing a compound datatype with no predefined struct (GH issue #5371)
- Provides better description of the min_meta_perc and min_raw_perc arguments in the H5Pset_page_buffer_size() (GH issue #5711)
- Adds error checkings to an internal decoding function
* Corrected the checks, the base address equals the end-of-file is valid.
* Skip for multi-file and split drivers when validating addresses against stored_eof
* Committing clang-format changes
* Remove incorrect name
* Fix typos
* Modified description of type_size arguments
* Used a more robust condition when checking EOF
* Corrected incorrect conflict resolving
* Update src/H5Fsuper_cache.c
Co-authored-by: Neil Fortner <fortnern@gmail.com>
* Update src/H5Fsuper_cache.c
Co-authored-by: Neil Fortner <fortnern@gmail.com>
* Committing clang-format changes
* Modified per feedbacks.
* Fix comparison per feedbacks
* Omit unused parameter name in a catch block
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Neil Fortner <fortnern@gmail.com>
* fix: pre-size filter output buffer to avoid realloc fragmentation on Windows
When reading a filtered (compressed) chunk, filters like deflate initialize
their output buffer using *buf_size (the pipeline's buffer capacity hint).
With *buf_size set to chunk_disk_size (the compressed size), deflate grows
its output buffer via repeated realloc() doubling. On Windows this fragments
the heap and causes read times to increase steadily across iterations.
Set buf_alloc to MAX(chunk_disk_size, chunk_size) immediately before calling
H5Z_pipeline so filters pre-allocate their output at the full uncompressed
size and avoid realloc. The inbuf is still allocated at chunk_disk_size —
only the hint passed to the pipeline is enlarged — so there is no
double-large-allocation overhead and peak memory stays at
chunk_disk_size + chunk_size rather than 2 * chunk_size.
For incompressible chunks where chunk_disk_size > chunk_size the hint
remains at chunk_disk_size, satisfying H5Z_pipeline's post-filter size
validation.
Fixes#4481 and #4513.
* fix: return nbytes instead of *buf_size in nbit and scaleoffset no-op paths
Per the HDF5 filter API, nbytes is the count of valid data bytes in the
buffer, while *buf_size is the allocated buffer capacity. These two values
were historically always equal on the read path, so either worked in
practice. The pre-sizing change (which sets buf_alloc = chunk_size before
the filter pipeline) makes *buf_size > nbytes for compressed reads,
exposing the latent bug.
In H5Znbit.c: the no-compress pass-through (cd_values[1] == 1) was
returning *buf_size, causing the next filter (e.g. fletcher32) to treat
the full pre-sized buffer as valid data and compute a checksum over
uninitialised bytes.
In H5Zscaleoffset.c: the no-process path had the same pattern. No current
test exercises this path through a multi-filter pipeline where the size
mismatch would be observable, but the fix is correct by the same API
reasoning.
* experiment: deflate decompress reuses inbuf via H5resize_memory
Instead of allocating a fresh output buffer (H5MM_malloc), copy the
compressed input to a small temp buffer and resize *buf in-place via
H5resize_memory. On Windows, HeapReAlloc can often extend an existing
heap block without moving it, avoiding the cost of finding and committing
a fresh large allocation for every chunk read.
This is combined with the pre-sizing hint in H5Dchunk.c that sets
buf_alloc = chunk_size before the pipeline call, so the resize goes
directly to chunk_size in one step with no realloc loop.
* fix: pre-resize chunk buf in H5Dchunk.c so all filters see correct capacity
Replace the hint-only buf_alloc enlargement with an actual
H5D__chunk_mem_realloc() before calling H5Z_pipeline. Every filter now
sees *buf_size equal to the real buffer capacity, not just an advisory
hint. This fixes a bounds-check regression in H5Zscaleoffset where the
read path uses *buf_size as the end-of-buffer sentinel in
H5_IS_BUFFER_OVERFLOW and as the input-size argument to
H5Z__scaleoffset_decompress; an inflated hint caused false-negative
overflow checks and potential over-reads when scaleoffset is the on-disk
filter.
H5Zdeflate is simplified accordingly: the up-front H5resize_memory(*buf,
nalloc) is removed since the buffer is already at nalloc on entry. The
inbuf copy is retained (still needed to read compressed input while
writing uncompressed output into the same buffer), as is the
realloc-doubling loop for the uncommon case where output exceeds
chunk_size.
Also add test/chunk_deflate_perf.c, a standalone benchmark that times
per-chunk deflate reads over multiple passes to detect the steady read-time
increase caused by heap fragmentation on Windows (issues #4481 / #4513).
* Committing clang-format changes
* test: add file path argument to chunk_deflate_perf benchmark
Add optional 4th argument to specify the output HDF5 file path
(default: chunk_deflate_perf.h5 in CWD). Allows running develop
and fix builds against separate files so pass 1 is cold-cache for
both and the two runs don't share page-cache state.
* test: remove chunk_deflate_perf benchmark
Not suitable for the test suite; intended for manual Windows validation
only. Keep locally if needed.
* fix: pre-resize chunk buf in H5Dchunk.c so all filters see correct capacity
Allocate the chunk read buffer at MAX(chunk_disk_size, chunk_size) from
the start rather than allocating at chunk_disk_size and immediately
reallocating. The read only fills chunk_disk_size bytes regardless of
buffer size, so there is no cost to the larger initial allocation and the
separate realloc step is eliminated.
Every filter now receives *buf_size equal to the actual buffer capacity
with no additional allocation needed. On Windows a single HeapAlloc at
the correct size avoids the repeated realloc-doubling in the deflate
filter that fragments the heap and causes read times to increase over
successive iterations (issues #4481 / #4513). Also fixes the
scaleoffset bounds-check regression where *buf_size was used as an
end-of-buffer sentinel.
* revert: restore H5Dchunk.c and H5Zdeflate.c to pre-experiment state
Reverts the deflate in-place decompression experiment and the H5Dchunk.c
pre-sizing changes back to the state at 9ab5e191d7, keeping the nbit
and scaleoffset no-op path fixes.
* fix: allocate chunk buf at MAX(disk_size, chunk_size) before filter pipeline
The previous approach allocated the buffer at chunk_disk_size and then
bumped buf_alloc to chunk_size as a hint to the pipeline, causing *buf_size
to misrepresent the actual allocation. Any filter that writes up to
*buf_size bytes into *buf would overflow.
Allocate at MAX(chunk_disk_size, chunk_size) upfront so the buffer and
the hint given to filters are always consistent. For incompressible chunks
where chunk_disk_size >= chunk_size the allocation is unchanged.
* Use HGOTO_DONE(nbytes) for the scaleoffset no-op passthrough
Matches the macro convention used for the other early-return in this
function per bmribler's review comment on PR #6389.
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Scot Breitenfeld
Neil Fortner
Nayyar
Matt L
H. Joe LeeandClaude Opus 5
tbeu
dependabot[bot]
vchoi-hdfgroup
jhendersonHDF
scivision
bmribler
