* Enable data sieving for chunks that can't be cached
Fixed an issue that prevented use of a data sieve buffer for I/O on dataset
chunks when those chunks couldn't be cached by the library. This issue
could result in worst-case behavior of I/O on a single data element at a
time when chunks are non-contiguous with respect to memory layout.
Added a test to attempt to catch performance regressions in I/O on dataset
chunks that are non-contiguous with respect to memory layout
Updated the External File List logic to set the data sieve buffer size to
the smaller of the dataset size and the size set in the FAPL, similar to
the logic elsewhere in the library
Adds predefined datatypes for FP6 data in E2M3 and E3M2 formats
Does not add support for any native FP6 types; datatype conversions are performed in software
Adds predefined datatypes for FP8 data in E4M3 and E5M2 formats
Does not add support for any native FP8 types; datatype conversions are performed in software
When discarding a metadata cache entry after flushing it, errors
during the discard process could cause the library to skip calling
the 'free_icr' callback for the entry. This could result in resource
leaks and the inability of the cache to be fully flushed and closed
due to issues such as pinned entries remaining in the cache. This
has been fixed by noting errors during the discard process, but
attempting to fully free a cache entry before signalling that an
error has occurred.
Fixes CVE-2025-7068
Adds predefined datatypes for little- and big-endian bfloat16 data
Does not add support for any native bfloat16 types; datatype conversions are performed in software
Also adds missing float16 predefined types to fortran
An HDF5 file could be specifically constructed such that an object
header contained a corrupted continuation message which pointed
back to itself. This eventually resulted in an internal buffer being
allocated with too small of a size, leading to a heap buffer overflow
when encoding an object header message into it. This has been fixed
by checking the expected number of deserialized object header chunks
against the actual value as chunks are being deserialized.
Fixes CVE-2025-6816, CVE-2025-6856, CVE-2025-2923
When linking a file free space section into a free space manager's
internal data structures, the library previously wouldn't unlink
the free space section when it failed to update the free space
manager's internal data structures. This could eventually result
in a use-after-free issue due to the stale reference kept around.
During testing, specific VFDs create files with names that
are different than the "base" filename passed to H5Fcreate().
The unicode tests are setup to expect specific filenames, so
these tests are skipped when the test utility function
h5_fixname() would modify the "base" filename.
Treat filenames as UTF-8 initially and fall back to ANSI functions
if conversion to UTF-16 fails
Adds new HDF5_PREFER_WINDOWS_CODE_PAGE environment variable to
prefer interpreting filenames according to the active Windows code
page rather than assuming UTF-8 encoding
Implemented missing --help/-h, -c, -echo and -shlib/-noshlib options
Added new -nohl option to avoid building and linking against
high-level HDF5 libraries if desired
Implemented adding of HDF5 installation library directory to rpath of
resulting executable/library and added new -norpath option to avoid
this behavior if desired
Added parsing of HDF5_USE_SHLIB environment variable to determine
whether to link against shared or static HDF5 libraries
Added parsing of HDF5_PKG_CONFIG_ARGS environment variable to separate
pkg-config-specific options from compiler-specific options and prevent
conflicts
Adds workflow to build ROS3 VFD and optionally build aws-c-s3 library
from source or use package managers
Adds testing of ROS3 VFD with s3proxy and docker
Adds new H5Pset_fapl_ros3_endpoint()/H5Pget_fapl_ros3_endpoint() API
functions to set/get an alternative endpoint URL to use when opening
files with the ROS3 VFD
Cleans up warnings in tools and tests related to ROS3 VFD structure size
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
Co-authored-by: Allen Byrne <50328838+byrnHDF@users.noreply.github.com>
* Re-write ROS3 VFD S3 backend to use AWS' C S3 library
Replaces the ROS3 VFD's libcurl-based S3 backend with a new
backend which uses the aws-c-s3 library
Adds support for sourcing credentials from standard AWS
configuration files, environment variables, etc. with a
fallback to anonymous credentials when credentials can't be
found
Adds support for sourcing AWS region from standard AWS
configuration files and environment variables
Adds support for new environment variables HDF5_ROS3_VFD_DEBUG,
HDF5_ROS3_VFD_LOG_LEVEL and HDF5_ROS3_VFD_LOG_FILE for debugging
purposes
Replaces function to load testing credentials in the ros3.c and
s3comms.c tests with new function that delegates to the aws-c-s3
library for better compatibility with standard AWS configuration
files and environment variables
Adds support for s3:// object URIs
Adds HTTP User-Agent header in form of
'libhdf5/<hdf5_version> (vfd:ros3) libaws-c-s3'
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
Co-authored-by: Allen Byrne <50328838+byrnHDF@users.noreply.github.com>
Fix an incorrect range check that would prevent reads
from being served from the ROS3 VFD's internal cache
when the offset + length of the read extends exactly
to the end of the cached range of bytes.
Build test generator programs against shared libraries, if available,
by default. Otherwise, build them against static libraries. If
HDF5_BUILD_STATIC_TOOLS is set to ON, the tools test generator
programs will be built against static libraries even if shared
libraries are available.
Moves call to TestAlarmOn() in testframe framework back into
the PerformTests() function due to the timing of when the
framework processes command-line arguments
Adds new -testexpress command-line option to the testframe testing
framework to allow setting or overriding of the TestExpress level
at runtime
Adds macros for the different currently defined TestExpress levels
Makes testing parameters specified with AddTest non-const again when
passed to test functions. This allows test functions to update their
parameters during execution and is particularly useful for tests
which use those parameters to accumulate statistics, error counts,
etc. without resorting to global variables.
* Implement support for complex number datatypes
Adds the new datatype class H5T_COMPLEX
Adds the new API function H5Tcomplex_create which creates a complex
number datatype from an ID of a base floating-point datatype
Adds the new feature check macros H5_HAVE_COMPLEX_NUMBERS and
H5_HAVE_C99_COMPLEX_NUMBERS
Adds the new datatype size macros H5_SIZEOF_FLOAT_COMPLEX,
H5_SIZEOF_DOUBLE_COMPLEX and H5_SIZEOF_LONG_DOUBLE_COMPLEX
Adds the new datatype ID macros H5T_NATIVE_FLOAT_COMPLEX,
H5T_NATIVE_DOUBLE_COMPLEX, H5T_NATIVE_LDOUBLE_COMPLEX,
H5T_CPLX_IEEE_F16LE, H5T_CPLX_IEEE_F16BE,
H5T_CPLX_IEEE_F32LE, H5T_CPLX_IEEE_F32BE,
H5T_CPLX_IEEE_F64LE and H5T_CPLX_IEEE_F64BE
Adds hard and soft datatype conversion paths between complex number
datatypes and all the integer and floating-point datatypes, as well as
between other complex number datatypes
Adds a special conversion path between complex number datatypes and
array or compound datatypes where the in-memory layout of data is the
same between the datatypes and data can be converted directly
Adds support for complex number datatypes to the h5dump, h5ls and
h5diff/ph5diff tools. Allows h5dump '-m' option to change floating-point
printing format for float complex and double complex datatypes, as well
as long double complex if it has the same size as double complex
Adds minimal support to the h5watch and h5import tools
Adds support for the predefined complex number datatypes and
H5Tcomplex_create function to the Java wrappers. Also adds initial,
untested support to the JNI for future use with HDFView
Adds support for just the H5T_COMPLEX datatype class to the Fortran
wrappers
Adds support for the predefined complex number datatypes and
H5Tcomplex_create function to the high level library H5LT interface
for use with the H5LTtext_to_dtype and H5LTdtype_to_text functions
Changes some usages of "complex" in the library since it conflicts with
the "complex" keyword from the complex.h header. Also changes various
usages of the word "complex" throughout the library to distinguish
compound datatypes from complex datatypes.
Added setup and cleanup callback parameters to TestInit() to perform
setup and cleanup tasks once for whole test program
Removed TestCleanup() function since its functionality is covered by
PerformTests()
Added check of the HDF5_NOCLEANUP environment variable in
GetTestCleanup()
Added new testframe.h header to document testing framework functions and
split them away from h5test.h and from test programs that don't
integrate with the testframe.c testing framework
Added new test setup callback to testframe.c testing framework
Added parameters to AddTest() to specify size of test parameters so they
can be copied for later use
Enabled HDF5 error stacks in testframe.c framework by default and added
some error stack suppressions to some testhdf5 tests
Added new maxthreads option to testframe.c framework to allow specifying
the maximum number of threads a multi-threaded test can use
Moved TestExpress functionality out of testframe.c and into more general
h5test.c for wider use by tests through getter and setter
Updated some tests to not mix and match functionality between h5test.c/h
and testframe.c/h
Moved some functionality from testphdf5.h into testpar.h for parallel
tests that aren't part of testphdf5
Added new parallel test library that contains common shared
functionality for parallel tests (similar to h5test library)
Fixes a bug where the vector I/O sizes weren't being extended when
one of the entries in the array is 0. This caused an over-read of
the I/O sizes buffer and on some machines would cause a memory
allocation failure due to the calculated I/O vector size being too
large.
* Fixes and cleanup for ph5diff
Fixes concurrency issues in ph5diff that can cause interleaved
output
Fixes an issue where output can sometimes be dropped if it ended
up in ph5diff's output overflow file
Fixes an issue where MPI_Init is called after HDF5 has been
initialized, preventing the library from setting up an MPI
attribute to perform cleanup on MPI_Finalize
Fixes an issue in config/cmake/runTest.cmake where the CMake
logic would try to access an invalid list index if the number
of lines in a test's output and reference files don't match
* Add release note
Run configure checks with and without CFLAGS/CMAKE_C_FLAGS since some
compilers work in one case while not working in the other case
Sync CMake configure checks with Autotools
Fixes an issue where invalid datatype IDs are passed to application conversion
functions in the case where the top-level conversion function is a library-internal
function that operates on a container-like datatype, but one or more of the
base datatype members are converted with an application conversion function.
Converts an assertion in H5G_loc_find into a normal error
check that checks for empty link names
Initializes H5O_link_t structure early in H5G__ent_to_link
to avoid trying to free potentially uninitialized memory
Checks for an empty link name after H5MM_strndup in
H5G__ent_to_link
Fixes GitHub #4307
Add configure option to enable or disable extension features in general
Add configure option to enable or disable _Float16 support
Add new config options to various settings files
H5PB_read previously did not account for the fact that the size of the
read it's performing could overflow the page buffer pointer, depending
on the calculated offset for the read. This has been fixed by adjusting
the size of the read if it's determined that it would overflow the page.
When the H5_addr_overlap macro was updated to use H5_RANGE_OVERLAP,
it failed to take into account that H5_RANGE_OVERLAP expects the
range to be inclusive. This lead to an assertion failure in
H5MM_memcpy due to a memcpy operation on overlapping memory.
This has been fixed by subtracting 1 from the calculated high
bound values passed to H5_RANGE_OVERLAP
jhendersonHDF
