RFC-HDFG-2026-001 v5 reverses the earlier "no on-disk format change"
decision: the verbatim key=value string passed to H5Pappend_filter is
now retained on the pipeline entry and written to the file, so it can be
recovered losslessly without loading the filter plugin.
H5O_PLINE_VERSION_3 appends [config_length:2][config] after each filter's
name. H5O_pline_set_version selects v3 only when a filter carries a
stored string and the file's high bound admits it; otherwise it stays at
v2 and the string is omitted, keeping string-free files byte-identical.
The new bound H5F_LIBVER_V300 (now H5F_LIBVER_LATEST) gates emission and
is mirrored in every *_ver_bounds[] table, H5trace, h5test, and the
Fortran/Java constants.
The string is threaded through the property machinery: H5Pappend_filter
retains it, H5Z_append/decode initialize it, H5O__pline copy/size/reset
and the H5Pencode/H5Pdecode pipeline enc/dec carry it, H5Z_modify
(H5Pmodify_filter) clears it, and H5Z_delete frees it.
H5Pget_filter_params_by_idx now returns the stored string first, falling
back to get_config then a cd_values listing; h5dump -p shows it for every
filter. H5Z_get_config_func_t's doc no longer frames cd_values packing
as normative (it is filter-private).
Tests (test/tfilter2.c) cover the verbatim round-trip recovered without
the plugin, the silent v2 downgrade when the libver bound is too low,
H5Pmodify_filter clearing, and survival through H5Pencode/H5Pdecode and
H5Ocopy. Updated the h5repack plugin_test_cfg reference to the verbatim
PARAMS_STRING.
write_blob/read_blob/close_blob and the corresponding H5Z_PARAMS_BLOB
enum value and has_blob_callbacks info field were speculative
reservations for a not-yet-designed feature. Pre-reserving generic
void* slots doesn't avoid the ABI/version bump the feature will need
once its callback signatures are actually decided, so there's no
benefit to carrying them now. Drop them from the public struct, the
internal filter table entry, all built-in filter registrations, the
Java bindings (JNI and Panama/FFM), and the test plugin/test suite.
Comments should describe current behavior, not cite external documents
by section number that will drift as the RFC is revised -- and should
stick to ASCII. Sweeps beyond this session's own edits since these
patterns were already present in this feature's earlier commits: em-dashes,
arrows, curly quotes, ellipses, and multiplication signs replaced with
ASCII equivalents; "RFC-HDFG-2026-001 SecN" citations replaced with
plain descriptions of the behavior itself.
The trailing "2" follows HDF5's convention for versioned/deprecated
symbols (H5Dopen2, H5Gcreate2, etc.), implying this is a successor to
H5Zget_filter_info registered in H5vers.txt -- it isn't. The two functions
are independent: H5Zget_filter_info returns only encode/decode config
flags, H5Zget_filter_class_info additionally exposes name, description,
and set_config/get_config presence. Renamed across C, Fortran, and both
Java binding variants (JNI + FFM) since it hadn't shipped yet.
* Fix H5DreadVL failing for pre-allocate cmpd-of-seq dsets
* Fix bad vlen of cmpd with null slot read
* Fix bad cmpd of cmpd read in java
`translate_rbuf`'s H5T_VLEN case had a similar bug where when `found_jList` was set to false due to an entyr in `ret_buf` being null, `ret_buf.add()` would be invoked on an array of objects without the list .add() method. This would occur whenever a read was invoked of a vlen sequence with a null (non-preallocated) entry. The pre-existing tests only tested the pre-allocated cases.
I removed the use of the `found_jList` flag, since it conflated the passing of an unallocated slot with `ret_buf` not being an array. Instead use `ret_buflen == 0` as the check to match the pattern in H5T_INTEGER and other branches.
The test for this fix is testH5Dread_vlen_of_compound_nullslot.
---
`translate_atomic_rebuf` had two issues related to handling of nested compounds. First, it discarded recursive returns, resulting in the construction of empty lists. Secondly, its member offset (`char_buf + i * typeSize + memb_offset`) was incorrect. In this case, `i` was the member index and `memberSize` was the entire cmpd size, so the offset would be erroneously large. It seems like this came from copying of the offset computation from `translate_rbuf`, which had to advance over entire elements of compound data. This error was duplicated on the write side in `translate_atomic_wbuf`'s H5T_COMPOUND case (h5util.c:4611).
I changed `translate_atomic_rbuf` to capture the resultant object, and dropped the `i * typeSize` term in both routines.
The new test verifying the fix works is `testH5Dread_vlen_of_nested_compound`.
* Add exception checks
* Update NULL checks in translate_wbuf
* Correct potentially bad array length check
* Clang format
* Fix readVL/writeVL crash on malformed buffer
* Committing clang-format changes
* Add bufSize checks to wbuf/rbuf translation
* Remove vlen pre-allocation support
* Harden JNI buffer interface
* Handle opaque types as byte[] and document JNI buffer data model
Opaque elements were grouped with H5T_INTEGER in the nested-type
translation path, which boxed them as Integer/Long and rejected
arbitrary-sized opaque blobs. Treat H5T_OPAQUE like H5T_REFERENCE
(a byte[] per element) in translate_atomic_rbuf, translate_atomic_wbuf,
and h5validate_atomic_wbuf so nested opaque round-trips correctly.
Also add "Buffer data model" header comments on translate_rbuf() and
translate_wbuf() and note the reference/opaque byte[] leaves in the
H5.java javadocv.
* Initialize typeSize to fix -Werror=maybe-uninitialized
typeSize was assigned only inside the vl_data_class branch but read in
a second, separate vl_data_class branch, which gcc -O2 flags as
maybe-uninitialized under -Werror. Initialize it to 0 at declaration in
H5Aread/H5Awrite/H5Dread/H5Dwrite, matching the existing vl_array_len
pattern.
* Port nested cmpd/vlen tests to java/test and sync reference
The legacy java/test tree's JUnit-TestH5D.txt reference listed the new
nested compound/vlen tests, but the corresponding @Test methods existed
only in java/src-jni/test/TestH5D.java. Port the 10 tests and the
writeCompoundOfVlenDataset helper into java/test/TestH5D.java, remove
debug prints, and
regenerate the reference to match the actual JUnit output.
* Support nested vlen/compound datatypes in Java FFM compat layer
The FFM compatibility layer (java/hdf) lacked the vlen/compound read and
write support that the JNI interface gained, so the nested cmpd/vlen tests
ported into java/test (TestH5D) failed and leaked an id.
VLDataConverter now has recursive encodeValue/decodeValue helpers that pack
and unpack any member class (integer, float, fixed/vl string, nested
compound, and VLEN) in the native HDF5 in-memory layout. These are wired
into convertCompoundDatatype, readCompoundDatatype and convertRawDataToArrayList,
and a type-aware convertToHVLAuto handles top-level VLEN-of-compound writes.
Compound reads now reclaim VL memory, and type/count mismatches raise
IllegalArgumentException instead of silently corrupting data.
H5DwriteVL rejects an undersized buffer up front and routes VLEN writes
through convertToHVLAuto. The JUnit-TestH5D reference regains its trailing
blank line to match the actual JUnit output.
* Committing clang-format changes
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
The FFM TestH5Z.java gained 8 new test methods covering the new filter
string-config API (H5Pappend_filter, H5Pget_filter_params_by_idx,
H5Zconfig_get_param_*), but the expected-output reference file was left
at 5 tests, causing the JUnit-TestH5Z CTest comparison to fail.
Three bugs in H5Pget_filter2:
1. cd_nelmts_segment was allocated as JAVA_INT (4 bytes) but size_t* needs
8 bytes on 64-bit — the C write overflowed into cd_values_segment.
2. cd_nelmts[0] was read back from cd_values_segment (wrong) instead of
cd_nelmts_segment.
3. cd_values and flags were never copied back from their native segments.
Fix: allocate cd_nelmts_segment as JAVA_LONG, seed it with the caller's
capacity on input, and copy all three output arrays back correctly.
H5Pappend_filter with CDVALUES is documented as identical to H5Pset_filter.
Routing through H5Pset_filter avoids constructing an H5Z_params_t struct
manually in FFM heap memory, which was silently producing cd_nelmts=0.
SymbolLookup.loaderLookup() only finds symbols loaded via
System.loadLibrary(); jextract loads libhdf5 via its own mechanism so
loaderLookup() never finds the RFC symbols at runtime.
Replace all SymbolLookup.loaderLookup()+MethodHandle patterns in the RFC
methods with direct hdf5_h.* calls, consistent with every other method in
H5.java:
H5Pappend_filter (both overloads)
H5Pget_filter_params_by_idx
H5Zget_filter_info2
H5Zconfig_has_key
H5Zconfig_get_param (long[], double[], boolean[], String[])
Two test failures from the h5dump nesting fix (2604f5dd6f5):
1. tools/test/h5repack/expected/deflate_limit.h5repack_layout.h5.ddl and
h5repack_layout.h5-plugin_test.ddl had PARAMS_STRING (and DESCRIPTION)
as flat siblings of the filter block rather than nested inside it —
the h5dump fixture files were updated in 2604f5dd6f5 but these two
h5repack expected files were missed.
2. java/src-jni/jni/h5zImp.c called CALL_CONSTRUCTOR with a 6-arg array
and signature "(IILjava/lang/String;Ljava/lang/String;ZZ)V" but
H5Z_class_info_t's constructor takes 7 args (adds has_blob_callbacks).
GetMethodID failed at runtime with the wrong arity. Add args[6] =
JNI_FALSE and update the descriptor to ZZZ)V.
Adds a human-readable key=value parameter string API for HDF5 filters,
alongside the existing integer cd_values arrays.
New C API:
- H5Pappend_filter(plist, filter_id, flags, params) — appends a filter
using either a key=value string or raw cd_values (H5Z_params_t)
- H5Pget_filter_params_by_idx(plist, idx, buf, buf_size, content_len) —
retrieves the parameter string for a filter by pipeline index
- H5Zconfig_get_int/double/bool/str — typed accessors for use inside
filter set_config callbacks
- H5Z_filter_id_by_name(name) — look up a filter id by registered name
- H5Zget_filter_info2(id, info) — extended filter info including v3 fields
New H5Z_class3_t fields: name, description, set_config, get_config,
and reserved blob-callback placeholders (write_blob/read_blob/close_blob).
H5Z_pipeline gains dxpl_id, scaled[], and ndims arguments threaded
through from all call sites so v3 filter callbacks have full context.
All six built-in filters (deflate, shuffle, fletcher32, nbit, szip,
scaleoffset) implement set_config/get_config callbacks.
TOML subset parser: tomlc17 (MIT) vendored in src/tomlc17/ and compiled
unconditionally into libhdf5. Hex-float literals are transparently
rewritten to decimal before parsing. tomlc17 symbols are hidden via
-fvisibility=hidden to prevent namespace collisions.
On-disk format: no new pipeline version. Parameter strings are converted
to cd_values by set_config at H5Pappend_filter time and stored using the
existing v2 pipeline message. On read, get_config reconstructs the string.
Full backward read compatibility is preserved.
Fortran, C++, and Java bindings added. Tests in test/tfilter2.c
(~2300 lines) and testpar/t_filters_parallel.c (par-01–par-04).
h5dump displays filter parameter strings; h5repack accepts TOML-form
UD= filter specs.
Code-review fixes included: tomlc17 visibility, H5Pget_filter_params_by_idx
arg validation and true-length two-pass contract, flags re-validation after
set_config, H5Z_register3 runtime plugin validation, Java two-pass protocol
and h5libraryError() consistency, CHANGELOG corrections.
Fixes GitHub issue #6153
Build-tree exports can't diverge from install-tree exports — the export(EXPORT ...) reads directly from the install export sets. No manual list to keep in sync.
Removed 3 global variables (HDF5_STATIC_LIBRARIES_TO_EXPORT, HDF5_JAVA_LIBRARIES_TO_EXPORT, HDF5_UTILS_TO_EXPORT) and their ~21 set_global_variable calls across tool/utility files.
Fixed the static-only build bug in the PR where the base export set was guarded by BUILD_SHARED_LIBS, breaking tools export.
Removed redundant utils in export files — the PR was dumping tools into all three build-tree export files (java, static, shared). Now they correctly appear only in the base export.
Addresses the "multiple @ param documentation sections" warnings are caused by the interaction of three things:
1. MACRO_EXPANSION = YES with EXPAND_ONLY_PREDEF = NO in Doxyfile.in
2. Versioned API macros in H5version.h like:
\def documentation in APIVersions.dox like:
3. \def H5Tarray_create \api_vers_2{H5Tarray_create,H5Tarray_create1,H5Tarray_create2}
Because Doxygen expands all macros, it sees H5Tarray_create as an alias for H5Tarray_create1. It then associates documentation from both the \def block (for the macro) and the function's own doc block (with \param entries) with the same function — resulting in "multiple @ param documentation sections."
* Update Doxygen version from 1.16.0 to 1.16.1
Use Doxygen 1.16.1 for all platforms. This version includes the latest
bug fixes and improvements while maintaining compatibility with our
documentation format.
* Fix Windows Doxygen installation for version 1.16.1
Handle different Windows binary naming conventions across Doxygen versions:
- 1.16.1 uses: doxygen-VERSION.windows.x64.bin.zip
- 1.16.0 uses: doxygen-VERSION.x64.bin.zip
Try the newer naming convention first, then fall back to the older one
if the download fails. This ensures compatibility across versions.
* Use ssciwr/doxygen-install in setup-doxygen action
Replace direct installation of official Doxygen binaries with
ssciwr/doxygen-install action. This avoids Java documentation parsing
errors that occur with official binaries but not with distribution
builds.
The setup-doxygen action now serves as a thin wrapper around
ssciwr/doxygen-install@f13be16c83 (v1),
using commit hash for security and reproducibility.
* Remove redundant version specification from workflows
The setup-doxygen action already has a default version (1.16.1),
so explicit version specification in workflows is unnecessary.
This follows DRY principles and makes version updates simpler.
Update version to 2.1 and derive version information from H5public.h, removing h5vers script and updating CMake and Java configurations.
Versioning:
Update version to 2.1 in H5public.h.
Derive version strings in H5public.h using macros.
CMake:
Extract version from H5public.h in HDF5config.cmake and HDF5AsSubdirMacros.cmake.
Configure README.md and CHANGELOG.md using CMakeLists.txt.
Java:
Generate H5Version.java from H5public.h for version consistency.
Update H5.java to use H5Version for version constants.
Removals:
Delete bin/h5vers script, previously used for version management.
Adds predefined datatypes for FP6 data in E2M3 and E3M2 formats
Does not add support for any native FP6 types; datatype conversions are performed in software
FFM build requires Java 25, Jextract 25.
Generates FFM bindings during configure.
JNI is default when the requirements are not met or can be forced.
Presets added for maven and FFM - JNI is default selection.
Enhanced Maven options will work with either JNI or FFM
New Workflows for testing and maven uploads.
Extensive documentation changes for java.
Adds predefined datatypes for FP8 data in E4M3 and E5M2 formats
Does not add support for any native FP8 types; datatype conversions are performed in software
In order to reduce hash collisions and take advantage of modern memory capacity, the default hash table size for the chunk cache has been increased from 521 to 8191. This means the hash table will consume approximately 64 KiB per open dataset. This value can be changed with H5Pset_cache() or H5Pset_chunk_cache(). This value was chosen because it is a prime number close to 8K.
Change default file format to 1.8 across various tests and examples, updating file creation and access logic accordingly.
Behavior:
Default file format version changed to 1.8 in H5Pfapl.c.
Updated file creation and access to use 1.8 format in h5ex_g_compact.c and test_file_image.c.
Set earliest file format in multiple test files including cache_tagging.c, dtypes.c, and links.c.
Tests:
Modified expected output in tools/test/misc/expected/*.ls files to reflect new file format locations.
Adjusted test logic in test_file_image.c and cache_tagging.c to accommodate format changes.
Misc:
Added comments and TODOs for future format testing in test_file_image.c.
Minor variable renaming for clarity in test_file_image.c.
Behavior:
Modify HDF5UseFortran.cmake to handle cross-compilation by providing default KINDs and sizes when CMAKE_CROSSCOMPILING is true.
Default values for INTEGER and REAL KINDs and their sizes are set when cross-compiling.
Error messages are adjusted to reflect cross-compilation scenarios.
Logic:
Conditional checks added for CMAKE_CROSSCOMPILING to determine if default values should be used.
Default values include INTEGER KINDs {1,2,4,8,16} and REAL KINDs {4,8,10,16}.
Default sizes for native kinds are set to 4 for INTEGER and REAL, and 8 for DOUBLE PRECISION when cross-compiling.
Misc:
Adjustments to verbose messages to indicate when defaults are used during cross-compilation.
Optimize VDS operations using R-tree spatial index, adding new API functions and tests for improved performance.
Behavior:
Introduces R-tree spatial index for optimizing VDS operations in H5Dvirtual.c.
Adds H5Pset_dset_use_spatial_tree() and H5Pget_dset_use_spatial_tree() to control R-tree usage.
Default behavior uses R-tree for VDS with more than 1000 mappings.
Implementation:
Adds H5RT.c, H5RTprivate.h, and H5RTpkg.h for R-tree implementation.
Updates H5Pdapl.c and H5Pdcpl.c to include R-tree properties.
Modifies H5Dvirtual.c to integrate R-tree in VDS I/O operations.
Testing:
Adds rtree.c for testing R-tree creation, search, and copy operations.
Tests R-tree integration with VDS in test/dsets.c.
Verifies R-tree behavior with different dataset access property list settings.
Adds predefined datatypes for little- and big-endian bfloat16 data
Does not add support for any native bfloat16 types; datatype conversions are performed in software
Also adds missing float16 predefined types to fortran
* Remove hbool_t from the public API
This was previously removed from the source code, but the API calls
retained hbool_t in the public API calls. This commit removes
hbool_t from the public APIs.
hbool_t is retained as a typedef for bool in H5public.h so existing
code doesn't break.
* Refactor warning macros
Replaces the scheme in H5warnings.h with a more compiler-agnostic
one that relocates ifdef complexity to the warnings header file
and uses better naming.
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Scot Breitenfeld
Matt Landgithub-actions
Orion Poplawski
jhendersonHDF
Allen Byrne
Larry Knox
Neil Fortner
Dana Robinson
Aleksandar Jelenak