Added 3 new functions to support this: H5TSset_internal_threads(), H5Pset_io_threads(), and H5Pget_io_threads().
This feature internally parallelizes read operations on chunked datasets. H5TSset_internal_threads() is used to enable the feature globally, while H5Pset_io_threads() can be used to disable the feature on a per-operation basis. These functions are only available when the library is configured with HDF5_ENABLE_CONCURRENCY=ON. When performing an internally threaded read, the library will concurrently read from disk, unfilter, and scatter to memory all chunks in a read operation on a chunked dataset. Currently each of these sub-operations is serialized (protected by a mutex), so there is not yet likely to be any performance improvement.
* Reject chunked datasets with mismatched chunk/dspace rank
H5D__chunk_construct() validates that the chunk layout dimensionality
matches the dataspace rank, but that runs only at dataset creation time.
When an existing dataset is opened, H5D__chunk_init() didn't repeat the
check, so a file whose stored chunk rank disagreed with its dataspace rank
was accepted. During chunk I/O the memory-selection rank (from the
dataspace) and the file-selection rank (chunk ndims - 1) then differ, which
produces a zero stride that causes a divide-by-zero in
H5S__hyper_iter_get_seq_list().
H5D__chunk_init() now performs the same dimensionality check on open (the
stored chunk rank includes the extra element-size dimension, so it must be
exactly one greater than the dataspace rank) and rejects a mismatch with an
error.
Added test_chunk_dims_mismatch() as a regression test in test/dsets.c
Fixes#6491
* Fix typo
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Clarify element-vs-byte wording
* Validate chunk/dataspace rank at layout decode time
Move the stored-chunk-rank vs dataspace-rank consistency check out of
H5D__chunk_init() and into H5O__layout_decode(), so a malformed chunked
layout is rejected as the message is decoded (mirroring the fill/datatype
size check in the fill message decode).
* Update release_docs/CHANGELOG.md
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
* Update CHANGELOG
* Pin format version bounds in bad chunk layout generator
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
When pushing errors to an error stack that is already full, the
library skipped the push operation but didn't inform calling code
about what happened. This resulted in calling code leaking memory,
leaving reference counts on IDs incremented and causing an infinite
loop while closing the library.
When called on a FAPL without ROS3 VFD block caching parameters
set on it, the H5Pget_fapl_ros3_block_caching() function previously
returned the VFD's default page buffer size value instead of its
default I/O block cache size value.
* Fix NULL pointer access when H5A_operator2_t is NULL
Passing NULL for the callback function pointer to H5Aiterate2 and
H5Aiterate_by_name was not detected, leading to a subsequent access
of an uninitialized pointer. Add a check for this "no operator
specified" case in both functions so they fail gracefully instead.
Fixes GHSA-r7g4-hv2f-5c66 - CVE-2025-9274
* Fix format
* Fix Java test to handle NULL callback to H5Aiterate2
* Avoid converting fill value from memory to file type when reading vlen
data from a nonexistent chunk.
* Committing clang-format changes
* Add Changelog note
* Spelling
* Update changelog note
* Improve test description
* Hopefully make formatter happy
* Hyphenate read-only
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* bound SOHM list decode to list_max messages
* Add tsohm test for out-of-range SOHM list message count
Create a file with a shared-message list index, corrupt the on-disk
message count so it exceeds list_max (repairing the table checksum),
and confirm reopening rejects the file instead of overrunning the
list image buffer and message array.
---------
Co-authored-by: H. Joe Lee <hyoklee@hdfgroup.org>
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
* ROS3 VFD block cache feature
Implements a minimal I/O block cache for files without paged allocation
enabled to better optimize I/O and reduce requests to S3
I/O is cached in fixed-size blocks (16MiB blocks by default) and are kept
in a simple LRU cache that evicts the oldest used block when a new block
needs to be cached
Reading and caching of the initial bytes of a file has been delayed
from file open to the first read for a file instead
API functions have been added for setting/getting the I/O block caching
parameters to be used
* Address comments from review
* Rename 'page' -> 'block'
* Change default block cache size and separate from page buffer logic
* Update CHANGELOG
The ROS3 VFD appended the raw object key to the HTTP request path.
Because the signing configuration disables use_double_uri_encode (the
correct setting for S3), the SigV4 signer uses the request path
verbatim, so keys containing characters that AWS requires to be
percent-encoded -- such as '=' in Hive-style "key=value" partition
prefixes, '+', or spaces -- produced signatures that disagree with
S3's server-side recomputation. S3 rejects such requests with
SignatureDoesNotMatch, surfaced as a bodyless HTTP 403 that is
indistinguishable from a permissions error on a HEAD request, even
though other S3 clients (AWS CLI, boto3, s3fs) could read the same
objects.
feat: add optional digital signature verification for HDF5 filter plugins
Introduce an opt-in plugin signing and verification system that allows
HDF5 deployments to require cryptographically signed filter plugins before
loading them. Disabled by default (HDF5_REQUIRE_SIGNED_PLUGINS=OFF).
New tool: h5sign
- Signs plugin shared libraries by appending an RSA signature and a
14-byte footer (algo_id | sig_len | 8-byte magic | format_ver) to the
binary without modifying the original content.
- Supports SHA-512 (default), SHA-256, SHA-384, and their PSS variants
(-a/--algorithm flag).
- Detects already-signed plugins; --force strips the old signature and
re-signs.
- Security hardened: keeps the file descriptor open through hashing and
appending (no TOCTOU window), enforces a 2048-bit minimum RSA key size,
rolls back partial writes on failure, and rejects paths that are not
regular files.
Verification (H5PLsig.c)
- At plugin load time, reads the footer, validates the magic and format
version, then checks the RSA signature against all public keys found in
the KeyStore directory.
- File is hashed once; per-key verification operates on the pre-computed
digest (no redundant I/O for multi-key keystores).
- Plugins whose signature hash appears in revoked_signatures.txt are
rejected regardless of key validity.
- Runtime debug output via HDF5_DEBUG=pl.
KeyStore management
- Trusted public keys are PEM files in a directory specified by
HDF5_PLUGIN_KEYSTORE_DIR (build time) or HDF5_PLUGIN_KEYSTORE (env var).
- HDF5_LOCK_PLUGIN_KEYSTORE cmake option disables the env-var override for
security-hardened deployments.
Test infrastructure
- h5signverifytest: positive, negative, tamper, re-sign, and revocation
test cases.
- CTest fixture-based dependency graph (FIXTURES_SETUP/FIXTURES_REQUIRED)
replaces fragile DEPENDS chains so tests remain correct under -R filtering.
- Dedicated signed-plugins.yml CI workflow; full test suite scoped to
H5SIGN and H5PLUGIN-signature tests to avoid unrelated flaky failures.
- Cross-platform: Linux, macOS, and Windows (MSVC-compatible, BIO-based
OpenSSL I/O, HDsleep/HDsetenv portability wrappers).
Documentation: docs/PLUGIN_SIGNATURE_README.md covers usage, footer
format, revocation file format, FAQ, and troubleshooting.
Co-authored-by: Glenn Song <gsong@hdfgroup.org>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Verify uncompressed chunks are the right size after being uncompressed (reverse filtered)
Verify that the buffer returned from the filter callback is large enough to hold the returned data size
Fix bug in deflate filter that caused it to report the wrong buffer size
Fix bug in chunk copy code that could cause a background buffer overflow
Fix bug in chunk copy code that could cause a double free if the filter realloced the data buffer
Other general cleanup
When a global API version is set (e.g., H5_USE_16_API), functions
introduced after that version now default to their earliest version
(version 1) instead of the latest. This prevents breakage when an
application uses an older API setting but calls functions that were
later versioned.
Fix issue where chunked datasets could get setup with an incorrect
chunking index type in parallel HDF5
Fix issue where metadata cache images with an undefined address
and size of 0 couldn't be properly decoded
Fix issue where a flag in H5Cimage.c wasn't getting set correctly
for release builds of the library, leading to incorrect error
checking when reconstructing metadata cache entries
Intel's icx compiler defines _WIN32 (with underscore) but not WIN32,
causing unistd.h and other POSIX headers to be incorrectly included
on Windows builds.
Closes can_apply and set_local filter callbacks skipped for H5T_VARIABLE dtypes #5942. A more detailed explanation of the rationale for this PR is in the linked issue.
Mandatory filters can be now applied to variable-length data. This notably includes h5py object strings and NpyStrings.
Optional filters applied to variable-length data no longer skip can_apply and set_local, only to jump directly to filter. This fixes hdf5_blosc and HDF5_Blosc2, which rely on set_local. Note that all filters which have neither can_apply nor set_local have always worked fine with vlen data, as long as they were tagged as optional. Notable examples are deflate, lzf, bzip2 (from hdf5plugin), and lz4 (also from hdf5plugin).
Lossy filters, such as scale-offset, are and remain untested and most likely broken, as I expect them to treat the vlen metadata as numbers and corrupt it. The right place to ensure proper behaviour is their can_apply function, which however is broken (filters: can_apply does nothing for optional filters #6161).
Refactor the cross-compilation infrastructure to rely on the standard
CMAKE_CROSSCOMPILING_EMULATOR variable instead of the custom
CROSSCOMPILING_PATH environment hack.
Key changes:
- Removed `ENVIRONMENT "${CROSSCOMPILING_PATH}"` from all test definitions.
- Updated custom commands (e.g., H5match_types, H5_gen) to invoke the
emulator explicitly when cross-compiling.
- Updated ConfigureChecks.cmake and HDF5UseFortran.cmake to enable
`try_run` checks when an emulator is defined.
- Added config/toolchain/mingw-w64-x86-64-wine.sh wrapper to automatically
locate MinGW runtime libraries (like libgfortran) for Wine execution.
- Cleaned up toolchain files to remove hardcoded emulator paths, deferring
configuration to the build environment.
Fixes security issue by treating non-NULL buffer with size 0 as length-only query in get_name API functions.
Behavior:
Modify get_name API functions to treat (buffer != NULL, size == 0) as length-only queries, preventing undefined behavior.
Fix applied to H5Aget_name, H5Aget_name_by_idx, H5Fget_name, H5Gget_objname_by_idx, H5Iget_name, H5Lget_name_by_idx, H5Rget_file_name, H5Rget_obj_name, H5Rget_attr_name, and 8 other functions.
Tests:
Update test/links.c, test/tattr.c, test/tfile.c, test/titerate.c, and test/trefer.c to verify new behavior with non-null buffer and size 0.
Documentation:
Update comments in H5A.c, H5F.c, H5Gdeprec.c, H5I.c, H5L.c, H5R.c, and H5Rdeprec.c to reflect new behavior.t]@users.noreply.github.com>
* Enable data sieving for chunks that can't be cached
Fixed an issue that prevented use of a data sieve buffer for I/O on dataset
chunks when those chunks couldn't be cached by the library. This issue
could result in worst-case behavior of I/O on a single data element at a
time when chunks are non-contiguous with respect to memory layout.
Added a test to attempt to catch performance regressions in I/O on dataset
chunks that are non-contiguous with respect to memory layout
Updated the External File List logic to set the data sieve buffer size to
the smaller of the dataset size and the size set in the FAPL, similar to
the logic elsewhere in the library
Adds predefined datatypes for FP6 data in E2M3 and E3M2 formats
Does not add support for any native FP6 types; datatype conversions are performed in software
Fix zero-element read failure in H5D__virtual_read() and H5D__virtual_write() in H5Dvirtual.c by skipping H5S_SELECT_BOUNDS if nelmts is 0.
Add test_vds_empty_slice() in rtree.c to verify zero-element read behavior on virtual datasets with r-trees.
Various related changes, including refactoring part of dataset creation, and reworking how layout versions are calculated.
Needs more testing of filters that create very large chunks, but that will need code to turn it off in cases where we can't allocate >4GiB buffers. I tested this manually by increasing the expansion ration in the expand2 test in dsets, and it passed everything up until it tried to expand the datasets with H5Dset_extent() and my laptop ran out of disk space.
We should add code and testing to handle the case where the "size of size" is set less than 8 bytes. This is not a new issue, since it can be set to 2.
I will file issues for these, but I don't think they are necessary for the release.
* Add instruction to verify library version compliance with sematic
versioning with link to semantic versioning wiki page.
Update H5.c and version tests for move of major and minor versions to
1st and 2nd version numbers.
* WILL_FAIL for the tcheck_version doesn't need changing for release
branches - removed that instruction from RELEASE_PROCESS.
Change release version instructions to use x.y.z.1 for pre-release
instead of x.y.z-1 as for develop snapshots.
Adds predefined datatypes for FP8 data in E4M3 and E5M2 formats
Does not add support for any native FP8 types; datatype conversions are performed in software
In order to reduce hash collisions and take advantage of modern memory capacity, the default hash table size for the chunk cache has been increased from 521 to 8191. This means the hash table will consume approximately 64 KiB per open dataset. This value can be changed with H5Pset_cache() or H5Pset_chunk_cache(). This value was chosen because it is a prime number close to 8K.
* Change new chunk indexing methods to always encode chunk size as a 64
bit (size of lengths) integer, when using the 2.0 file format.
* Add CHANGELOG.md note
* Spelling
* Fix errors in parallel build
* Committing clang-format changes
* More parallel fixes.
* Committing clang-format changes
* Another parallel fix
* Fix parallel for real this time I hope
* Update function descriptions in dsets.c
* Fix spelling
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Change default file format to 1.8 across various tests and examples, updating file creation and access logic accordingly.
Behavior:
Default file format version changed to 1.8 in H5Pfapl.c.
Updated file creation and access to use 1.8 format in h5ex_g_compact.c and test_file_image.c.
Set earliest file format in multiple test files including cache_tagging.c, dtypes.c, and links.c.
Tests:
Modified expected output in tools/test/misc/expected/*.ls files to reflect new file format locations.
Adjusted test logic in test_file_image.c and cache_tagging.c to accommodate format changes.
Misc:
Added comments and TODOs for future format testing in test_file_image.c.
Minor variable renaming for clarity in test_file_image.c.
Implement the special filename handling that makes the rtree test compatible with the family VFD (and any other VFD that changes how filenames are treated).
Make rtree.c test VFD-compatible by implementing dynamic filename handling and updating function signatures for file access property lists.
Behavior:
Modify HDF5UseFortran.cmake to handle cross-compilation by providing default KINDs and sizes when CMAKE_CROSSCOMPILING is true.
Default values for INTEGER and REAL KINDs and their sizes are set when cross-compiling.
Error messages are adjusted to reflect cross-compilation scenarios.
Logic:
Conditional checks added for CMAKE_CROSSCOMPILING to determine if default values should be used.
Default values include INTEGER KINDs {1,2,4,8,16} and REAL KINDs {4,8,10,16}.
Default sizes for native kinds are set to 4 for INTEGER and REAL, and 8 for DOUBLE PRECISION when cross-compiling.
Misc:
Adjustments to verbose messages to indicate when defaults are used during cross-compilation.
Optimize VDS operations using R-tree spatial index, adding new API functions and tests for improved performance.
Behavior:
Introduces R-tree spatial index for optimizing VDS operations in H5Dvirtual.c.
Adds H5Pset_dset_use_spatial_tree() and H5Pget_dset_use_spatial_tree() to control R-tree usage.
Default behavior uses R-tree for VDS with more than 1000 mappings.
Implementation:
Adds H5RT.c, H5RTprivate.h, and H5RTpkg.h for R-tree implementation.
Updates H5Pdapl.c and H5Pdcpl.c to include R-tree properties.
Modifies H5Dvirtual.c to integrate R-tree in VDS I/O operations.
Testing:
Adds rtree.c for testing R-tree creation, search, and copy operations.
Tests R-tree integration with VDS in test/dsets.c.
Verifies R-tree behavior with different dataset access property list settings.
Implement an r-tree data structure in a new module. It has three exposed methods: H5RT_create(), H5RT_search(), and H5RT_free().
The STR algorithm used during creation is based on the one described here.
Updates CMakeLists.txt to include H5RT.c and related headers.
Adds rtree.c test file to validate R-tree creation and search functionalities.
Adds predefined datatypes for little- and big-endian bfloat16 data
Does not add support for any native bfloat16 types; datatype conversions are performed in software
Also adds missing float16 predefined types to fortran
During testing, specific VFDs create files with names that
are different than the "base" filename passed to H5Fcreate().
The unicode tests are setup to expect specific filenames, so
these tests are skipped when the test utility function
h5_fixname() would modify the "base" filename.
Treat filenames as UTF-8 initially and fall back to ANSI functions
if conversion to UTF-16 fails
Adds new HDF5_PREFER_WINDOWS_CODE_PAGE environment variable to
prefer interpreting filenames according to the active Windows code
page rather than assuming UTF-8 encoding
Neil Fortner
Nayyar
jhendersonHDF
bmribler
Aleksandar Jelenak
phfix
Guido Imperiale
Martin Diehl
Julien Schueller
efferre79
Neil Fortner
Allen Byrne