* ROS3 VFD block cache feature
Implements a minimal I/O block cache for files without paged allocation
enabled to better optimize I/O and reduce requests to S3
I/O is cached in fixed-size blocks (16MiB blocks by default) and are kept
in a simple LRU cache that evicts the oldest used block when a new block
needs to be cached
Reading and caching of the initial bytes of a file has been delayed
from file open to the first read for a file instead
API functions have been added for setting/getting the I/O block caching
parameters to be used
* Address comments from review
* Rename 'page' -> 'block'
* Change default block cache size and separate from page buffer logic
* Update CHANGELOG
For objects with multiple hard links, use hash table to map between
object tokens and pathnames during traversal to avoid linear scan over
all previous objects for each hard link seen
Use separate hash table for h5trav "table" interface to map between
object tokens and an index into the table of visited objects. This
facilitates quick lookups of objects when adding hard link name aliases
for h5repack processing
* Update Visual Studio versions in workflows
* Revert change to Windows ARM workflows
* Pin Intel OneAPI workflows to windows-2025
* Update more instances of issues
Fix issue where chunked datasets could get setup with an incorrect
chunking index type in parallel HDF5
Fix issue where metadata cache images with an undefined address
and size of 0 couldn't be properly decoded
Fix issue where a flag in H5Cimage.c wasn't getting set correctly
for release builds of the library, leading to incorrect error
checking when reconstructing metadata cache entries
During file creation, the direct I/O VFD only checked data alignment
requirements for writes and assumed they were the same for reads.
Checks are now performed separately for writes and reads. In order
to avoid EINVAL errors, the VFD is also now slightly more conservative
about when it tries to avoid performing extra work when data alignment
isn't required.
* Re-write zlib/szip CMake logic for clarity
* Update external libaec, zlib-ng and zlib builds to not use patching process
* Add FindZLIBNG module to locate zlib-ng on system
* Rework HDF5 filter plugins support
Ensure that the Subfiling VFD's stub file gets truncated when subfiles are
being truncated to avoid a potentially unreadable file when the EOF doesn't
match the stored EOF value
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
Removes the force-setting of ZLIB_USE_EXTERNAL and SZIP_USE_EXTERNAL to
ON when HDF5_ALLOW_EXTERNAL_SUPPORT is GIT or TGZ so that zlib and
szip can be independently built from the system or externally as desired
* Enable data sieving for chunks that can't be cached
Fixed an issue that prevented use of a data sieve buffer for I/O on dataset
chunks when those chunks couldn't be cached by the library. This issue
could result in worst-case behavior of I/O on a single data element at a
time when chunks are non-contiguous with respect to memory layout.
Added a test to attempt to catch performance regressions in I/O on dataset
chunks that are non-contiguous with respect to memory layout
Updated the External File List logic to set the data sieve buffer size to
the smaller of the dataset size and the size set in the FAPL, similar to
the logic elsewhere in the library
Adds predefined datatypes for FP6 data in E2M3 and E3M2 formats
Does not add support for any native FP6 types; datatype conversions are performed in software
Adds predefined datatypes for FP8 data in E4M3 and E5M2 formats
Does not add support for any native FP8 types; datatype conversions are performed in software
When discarding a metadata cache entry after flushing it, errors
during the discard process could cause the library to skip calling
the 'free_icr' callback for the entry. This could result in resource
leaks and the inability of the cache to be fully flushed and closed
due to issues such as pinned entries remaining in the cache. This
has been fixed by noting errors during the discard process, but
attempting to fully free a cache entry before signalling that an
error has occurred.
Fixes CVE-2025-7068
Adds predefined datatypes for little- and big-endian bfloat16 data
Does not add support for any native bfloat16 types; datatype conversions are performed in software
Also adds missing float16 predefined types to fortran
An HDF5 file could be specifically constructed such that an object
header contained a corrupted continuation message which pointed
back to itself. This eventually resulted in an internal buffer being
allocated with too small of a size, leading to a heap buffer overflow
when encoding an object header message into it. This has been fixed
by checking the expected number of deserialized object header chunks
against the actual value as chunks are being deserialized.
Fixes CVE-2025-6816, CVE-2025-6856, CVE-2025-2923
When linking a file free space section into a free space manager's
internal data structures, the library previously wouldn't unlink
the free space section when it failed to update the free space
manager's internal data structures. This could eventually result
in a use-after-free issue due to the stale reference kept around.
During testing, specific VFDs create files with names that
are different than the "base" filename passed to H5Fcreate().
The unicode tests are setup to expect specific filenames, so
these tests are skipped when the test utility function
h5_fixname() would modify the "base" filename.
Treat filenames as UTF-8 initially and fall back to ANSI functions
if conversion to UTF-16 fails
Adds new HDF5_PREFER_WINDOWS_CODE_PAGE environment variable to
prefer interpreting filenames according to the active Windows code
page rather than assuming UTF-8 encoding
Implemented missing --help/-h, -c, -echo and -shlib/-noshlib options
Added new -nohl option to avoid building and linking against
high-level HDF5 libraries if desired
Implemented adding of HDF5 installation library directory to rpath of
resulting executable/library and added new -norpath option to avoid
this behavior if desired
Added parsing of HDF5_USE_SHLIB environment variable to determine
whether to link against shared or static HDF5 libraries
Added parsing of HDF5_PKG_CONFIG_ARGS environment variable to separate
pkg-config-specific options from compiler-specific options and prevent
conflicts
Adds workflow to build ROS3 VFD and optionally build aws-c-s3 library
from source or use package managers
Adds testing of ROS3 VFD with s3proxy and docker
Adds new H5Pset_fapl_ros3_endpoint()/H5Pget_fapl_ros3_endpoint() API
functions to set/get an alternative endpoint URL to use when opening
files with the ROS3 VFD
Cleans up warnings in tools and tests related to ROS3 VFD structure size
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
Co-authored-by: Allen Byrne <50328838+byrnHDF@users.noreply.github.com>
* Re-write ROS3 VFD S3 backend to use AWS' C S3 library
Replaces the ROS3 VFD's libcurl-based S3 backend with a new
backend which uses the aws-c-s3 library
Adds support for sourcing credentials from standard AWS
configuration files, environment variables, etc. with a
fallback to anonymous credentials when credentials can't be
found
Adds support for sourcing AWS region from standard AWS
configuration files and environment variables
Adds support for new environment variables HDF5_ROS3_VFD_DEBUG,
HDF5_ROS3_VFD_LOG_LEVEL and HDF5_ROS3_VFD_LOG_FILE for debugging
purposes
Replaces function to load testing credentials in the ros3.c and
s3comms.c tests with new function that delegates to the aws-c-s3
library for better compatibility with standard AWS configuration
files and environment variables
Adds support for s3:// object URIs
Adds HTTP User-Agent header in form of
'libhdf5/<hdf5_version> (vfd:ros3) libaws-c-s3'
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
Co-authored-by: Allen Byrne <50328838+byrnHDF@users.noreply.github.com>
Fix an incorrect range check that would prevent reads
from being served from the ROS3 VFD's internal cache
when the offset + length of the read extends exactly
to the end of the cached range of bytes.
Build test generator programs against shared libraries, if available,
by default. Otherwise, build them against static libraries. If
HDF5_BUILD_STATIC_TOOLS is set to ON, the tools test generator
programs will be built against static libraries even if shared
libraries are available.
Moves call to TestAlarmOn() in testframe framework back into
the PerformTests() function due to the timing of when the
framework processes command-line arguments
Adds new -testexpress command-line option to the testframe testing
framework to allow setting or overriding of the TestExpress level
at runtime
Adds macros for the different currently defined TestExpress levels
Makes testing parameters specified with AddTest non-const again when
passed to test functions. This allows test functions to update their
parameters during execution and is particularly useful for tests
which use those parameters to accumulate statistics, error counts,
etc. without resorting to global variables.
* Implement support for complex number datatypes
Adds the new datatype class H5T_COMPLEX
Adds the new API function H5Tcomplex_create which creates a complex
number datatype from an ID of a base floating-point datatype
Adds the new feature check macros H5_HAVE_COMPLEX_NUMBERS and
H5_HAVE_C99_COMPLEX_NUMBERS
Adds the new datatype size macros H5_SIZEOF_FLOAT_COMPLEX,
H5_SIZEOF_DOUBLE_COMPLEX and H5_SIZEOF_LONG_DOUBLE_COMPLEX
Adds the new datatype ID macros H5T_NATIVE_FLOAT_COMPLEX,
H5T_NATIVE_DOUBLE_COMPLEX, H5T_NATIVE_LDOUBLE_COMPLEX,
H5T_CPLX_IEEE_F16LE, H5T_CPLX_IEEE_F16BE,
H5T_CPLX_IEEE_F32LE, H5T_CPLX_IEEE_F32BE,
H5T_CPLX_IEEE_F64LE and H5T_CPLX_IEEE_F64BE
Adds hard and soft datatype conversion paths between complex number
datatypes and all the integer and floating-point datatypes, as well as
between other complex number datatypes
Adds a special conversion path between complex number datatypes and
array or compound datatypes where the in-memory layout of data is the
same between the datatypes and data can be converted directly
Adds support for complex number datatypes to the h5dump, h5ls and
h5diff/ph5diff tools. Allows h5dump '-m' option to change floating-point
printing format for float complex and double complex datatypes, as well
as long double complex if it has the same size as double complex
Adds minimal support to the h5watch and h5import tools
Adds support for the predefined complex number datatypes and
H5Tcomplex_create function to the Java wrappers. Also adds initial,
untested support to the JNI for future use with HDFView
Adds support for just the H5T_COMPLEX datatype class to the Fortran
wrappers
Adds support for the predefined complex number datatypes and
H5Tcomplex_create function to the high level library H5LT interface
for use with the H5LTtext_to_dtype and H5LTdtype_to_text functions
Changes some usages of "complex" in the library since it conflicts with
the "complex" keyword from the complex.h header. Also changes various
usages of the word "complex" throughout the library to distinguish
compound datatypes from complex datatypes.
jhendersonHDF
