Fix issue where chunked datasets could get setup with an incorrect
chunking index type in parallel HDF5
Fix issue where metadata cache images with an undefined address
and size of 0 couldn't be properly decoded
Fix issue where a flag in H5Cimage.c wasn't getting set correctly
for release builds of the library, leading to incorrect error
checking when reconstructing metadata cache entries
Intel's icx compiler defines _WIN32 (with underscore) but not WIN32,
causing unistd.h and other POSIX headers to be incorrectly included
on Windows builds.
Closes can_apply and set_local filter callbacks skipped for H5T_VARIABLE dtypes #5942. A more detailed explanation of the rationale for this PR is in the linked issue.
Mandatory filters can be now applied to variable-length data. This notably includes h5py object strings and NpyStrings.
Optional filters applied to variable-length data no longer skip can_apply and set_local, only to jump directly to filter. This fixes hdf5_blosc and HDF5_Blosc2, which rely on set_local. Note that all filters which have neither can_apply nor set_local have always worked fine with vlen data, as long as they were tagged as optional. Notable examples are deflate, lzf, bzip2 (from hdf5plugin), and lz4 (also from hdf5plugin).
Lossy filters, such as scale-offset, are and remain untested and most likely broken, as I expect them to treat the vlen metadata as numbers and corrupt it. The right place to ensure proper behaviour is their can_apply function, which however is broken (filters: can_apply does nothing for optional filters #6161).
Refactor the cross-compilation infrastructure to rely on the standard
CMAKE_CROSSCOMPILING_EMULATOR variable instead of the custom
CROSSCOMPILING_PATH environment hack.
Key changes:
- Removed `ENVIRONMENT "${CROSSCOMPILING_PATH}"` from all test definitions.
- Updated custom commands (e.g., H5match_types, H5_gen) to invoke the
emulator explicitly when cross-compiling.
- Updated ConfigureChecks.cmake and HDF5UseFortran.cmake to enable
`try_run` checks when an emulator is defined.
- Added config/toolchain/mingw-w64-x86-64-wine.sh wrapper to automatically
locate MinGW runtime libraries (like libgfortran) for Wine execution.
- Cleaned up toolchain files to remove hardcoded emulator paths, deferring
configuration to the build environment.
Fixes security issue by treating non-NULL buffer with size 0 as length-only query in get_name API functions.
Behavior:
Modify get_name API functions to treat (buffer != NULL, size == 0) as length-only queries, preventing undefined behavior.
Fix applied to H5Aget_name, H5Aget_name_by_idx, H5Fget_name, H5Gget_objname_by_idx, H5Iget_name, H5Lget_name_by_idx, H5Rget_file_name, H5Rget_obj_name, H5Rget_attr_name, and 8 other functions.
Tests:
Update test/links.c, test/tattr.c, test/tfile.c, test/titerate.c, and test/trefer.c to verify new behavior with non-null buffer and size 0.
Documentation:
Update comments in H5A.c, H5F.c, H5Gdeprec.c, H5I.c, H5L.c, H5R.c, and H5Rdeprec.c to reflect new behavior.t]@users.noreply.github.com>
* Enable data sieving for chunks that can't be cached
Fixed an issue that prevented use of a data sieve buffer for I/O on dataset
chunks when those chunks couldn't be cached by the library. This issue
could result in worst-case behavior of I/O on a single data element at a
time when chunks are non-contiguous with respect to memory layout.
Added a test to attempt to catch performance regressions in I/O on dataset
chunks that are non-contiguous with respect to memory layout
Updated the External File List logic to set the data sieve buffer size to
the smaller of the dataset size and the size set in the FAPL, similar to
the logic elsewhere in the library
Adds predefined datatypes for FP6 data in E2M3 and E3M2 formats
Does not add support for any native FP6 types; datatype conversions are performed in software
Fix zero-element read failure in H5D__virtual_read() and H5D__virtual_write() in H5Dvirtual.c by skipping H5S_SELECT_BOUNDS if nelmts is 0.
Add test_vds_empty_slice() in rtree.c to verify zero-element read behavior on virtual datasets with r-trees.
Various related changes, including refactoring part of dataset creation, and reworking how layout versions are calculated.
Needs more testing of filters that create very large chunks, but that will need code to turn it off in cases where we can't allocate >4GiB buffers. I tested this manually by increasing the expansion ration in the expand2 test in dsets, and it passed everything up until it tried to expand the datasets with H5Dset_extent() and my laptop ran out of disk space.
We should add code and testing to handle the case where the "size of size" is set less than 8 bytes. This is not a new issue, since it can be set to 2.
I will file issues for these, but I don't think they are necessary for the release.
* Add instruction to verify library version compliance with sematic
versioning with link to semantic versioning wiki page.
Update H5.c and version tests for move of major and minor versions to
1st and 2nd version numbers.
* WILL_FAIL for the tcheck_version doesn't need changing for release
branches - removed that instruction from RELEASE_PROCESS.
Change release version instructions to use x.y.z.1 for pre-release
instead of x.y.z-1 as for develop snapshots.
Adds predefined datatypes for FP8 data in E4M3 and E5M2 formats
Does not add support for any native FP8 types; datatype conversions are performed in software
In order to reduce hash collisions and take advantage of modern memory capacity, the default hash table size for the chunk cache has been increased from 521 to 8191. This means the hash table will consume approximately 64 KiB per open dataset. This value can be changed with H5Pset_cache() or H5Pset_chunk_cache(). This value was chosen because it is a prime number close to 8K.
* Change new chunk indexing methods to always encode chunk size as a 64
bit (size of lengths) integer, when using the 2.0 file format.
* Add CHANGELOG.md note
* Spelling
* Fix errors in parallel build
* Committing clang-format changes
* More parallel fixes.
* Committing clang-format changes
* Another parallel fix
* Fix parallel for real this time I hope
* Update function descriptions in dsets.c
* Fix spelling
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Change default file format to 1.8 across various tests and examples, updating file creation and access logic accordingly.
Behavior:
Default file format version changed to 1.8 in H5Pfapl.c.
Updated file creation and access to use 1.8 format in h5ex_g_compact.c and test_file_image.c.
Set earliest file format in multiple test files including cache_tagging.c, dtypes.c, and links.c.
Tests:
Modified expected output in tools/test/misc/expected/*.ls files to reflect new file format locations.
Adjusted test logic in test_file_image.c and cache_tagging.c to accommodate format changes.
Misc:
Added comments and TODOs for future format testing in test_file_image.c.
Minor variable renaming for clarity in test_file_image.c.
Implement the special filename handling that makes the rtree test compatible with the family VFD (and any other VFD that changes how filenames are treated).
Make rtree.c test VFD-compatible by implementing dynamic filename handling and updating function signatures for file access property lists.
Behavior:
Modify HDF5UseFortran.cmake to handle cross-compilation by providing default KINDs and sizes when CMAKE_CROSSCOMPILING is true.
Default values for INTEGER and REAL KINDs and their sizes are set when cross-compiling.
Error messages are adjusted to reflect cross-compilation scenarios.
Logic:
Conditional checks added for CMAKE_CROSSCOMPILING to determine if default values should be used.
Default values include INTEGER KINDs {1,2,4,8,16} and REAL KINDs {4,8,10,16}.
Default sizes for native kinds are set to 4 for INTEGER and REAL, and 8 for DOUBLE PRECISION when cross-compiling.
Misc:
Adjustments to verbose messages to indicate when defaults are used during cross-compilation.
Optimize VDS operations using R-tree spatial index, adding new API functions and tests for improved performance.
Behavior:
Introduces R-tree spatial index for optimizing VDS operations in H5Dvirtual.c.
Adds H5Pset_dset_use_spatial_tree() and H5Pget_dset_use_spatial_tree() to control R-tree usage.
Default behavior uses R-tree for VDS with more than 1000 mappings.
Implementation:
Adds H5RT.c, H5RTprivate.h, and H5RTpkg.h for R-tree implementation.
Updates H5Pdapl.c and H5Pdcpl.c to include R-tree properties.
Modifies H5Dvirtual.c to integrate R-tree in VDS I/O operations.
Testing:
Adds rtree.c for testing R-tree creation, search, and copy operations.
Tests R-tree integration with VDS in test/dsets.c.
Verifies R-tree behavior with different dataset access property list settings.
Implement an r-tree data structure in a new module. It has three exposed methods: H5RT_create(), H5RT_search(), and H5RT_free().
The STR algorithm used during creation is based on the one described here.
Updates CMakeLists.txt to include H5RT.c and related headers.
Adds rtree.c test file to validate R-tree creation and search functionalities.
Adds predefined datatypes for little- and big-endian bfloat16 data
Does not add support for any native bfloat16 types; datatype conversions are performed in software
Also adds missing float16 predefined types to fortran
During testing, specific VFDs create files with names that
are different than the "base" filename passed to H5Fcreate().
The unicode tests are setup to expect specific filenames, so
these tests are skipped when the test utility function
h5_fixname() would modify the "base" filename.
Treat filenames as UTF-8 initially and fall back to ANSI functions
if conversion to UTF-16 fails
Adds new HDF5_PREFER_WINDOWS_CODE_PAGE environment variable to
prefer interpreting filenames according to the active Windows code
page rather than assuming UTF-8 encoding
Move the portion of test_reference_external() in charge of test file creation to its own dedicated function. This will eventually be helpful if/when we change the test file creation here to used a shared routine.
* H5FDset_fapl_ros3() sets page buffer size to 64MB by default. Added
H5F_PAGE_BUFFER_SIZE_DEFAULT macro indicating the page buffer size
setting is the default, and associated machinery and tests.
Add hash tables for detecting repeated source file and dataset names.
Share these repeated strings between mappings in memory. Add new
encoding format for VDS for shared names. Add tests and documentation
for these changes.
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Matthew Larson <matthewjlar@gmail.com>
Adds workflow to build ROS3 VFD and optionally build aws-c-s3 library
from source or use package managers
Adds testing of ROS3 VFD with s3proxy and docker
Adds new H5Pset_fapl_ros3_endpoint()/H5Pget_fapl_ros3_endpoint() API
functions to set/get an alternative endpoint URL to use when opening
files with the ROS3 VFD
Cleans up warnings in tools and tests related to ROS3 VFD structure size
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
Co-authored-by: Allen Byrne <50328838+byrnHDF@users.noreply.github.com>
* Re-write ROS3 VFD S3 backend to use AWS' C S3 library
Replaces the ROS3 VFD's libcurl-based S3 backend with a new
backend which uses the aws-c-s3 library
Adds support for sourcing credentials from standard AWS
configuration files, environment variables, etc. with a
fallback to anonymous credentials when credentials can't be
found
Adds support for sourcing AWS region from standard AWS
configuration files and environment variables
Adds support for new environment variables HDF5_ROS3_VFD_DEBUG,
HDF5_ROS3_VFD_LOG_LEVEL and HDF5_ROS3_VFD_LOG_FILE for debugging
purposes
Replaces function to load testing credentials in the ros3.c and
s3comms.c tests with new function that delegates to the aws-c-s3
library for better compatibility with standard AWS configuration
files and environment variables
Adds support for s3:// object URIs
Adds HTTP User-Agent header in form of
'libhdf5/<hdf5_version> (vfd:ros3) libaws-c-s3'
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
Co-authored-by: Allen Byrne <50328838+byrnHDF@users.noreply.github.com>
* Refactor warning macros
Replaces the scheme in H5warnings.h with a more compiler-agnostic
one that relocates ifdef complexity to the warnings header file
and uses better naming.
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* Enable bitwise comparison of file image in file_image.c for all file
format versions. Add RELEASE.txt note for file image fix with latest
format (#1915).
* Fix error message in file image test.
jhendersonHDF
phfix
Scot Breitenfeld
Guido Imperiale
Martin Diehl
Julien Schueller
bmribler
efferre79
Matt L
H. Joe Lee
Neil Fortner
Larry Knox
Allen Byrne
Dana Robinsonandgithub-actions