Two CI failures, both from the preceding commits.
Missing prototypes (hdf5 dev cmake CI). H5Pmodify_filter_by_idx_str_c
and _raw_c were defined in fortran/src/H5Zf.c but never declared in
H5f90proto.h alongside the H5Pappend_filter_*_c helpers. That broke two
different jobs for one reason:
- "Special Workflows / gcc REL -Werror (build only)" builds with
-Werror=missing-prototypes and rejected both definitions outright.
- The Intel and windows-oneapi jobs failed at link with LNK2019 on
both symbols, because H5FC_DLL on the declaration is what carries
the dllexport attribute; with no declaration the symbols were never
exported from hdf5_fortran.dll.
Adding the two H5FC_DLL declarations fixes both. Verified locally by
compiling H5Zf.c with -Werror=missing-prototypes, which the ordinary
build does not enable.
Formatting (clang-format Commit Changes). This job was green on the
commit before this series and has failed on every commit since; the
fault was entirely mine. Ran clang-format 17 -- the version CI pins --
over every file this series touched, C and Java alike, since the check
covers .java here too. The whole tree now passes a local run using CI's
source path and exclusion list.
Full build clean; ctest 3458/3459, the one failure being the
pre-existing ph5_f90_filtered_writes_no_sel example bug fixed upstream
in HDFGroup/hdf5#6633.
Implements the two overloads specified in RFC-HDFG-2026-001
sec:modify-bindings -- one taking a String, one an int[] -- mirroring
the existing H5Pappend_filter overloads.
Both Java binding paths are covered, since the tree maintains two:
- JNI (java/src-jni): two private native methods dispatched from the
public overloads, plus the JNI implementations and declarations in
h5pDCPLImp.c/.h, following the H5Pappend_filter_str/_raw pattern
exactly.
- Panama/FFM (java/hdf): the H5Z_params_t struct is built directly in
an Arena and passed to the generated org.hdfgroup.javahdf5.hdf5_h
downcall, as the append string form already does. Unlike the append
case the cd_values form cannot delegate to H5Pmodify_filter, which
addresses an entry by filter ID and resolves to the first match
rather than by index, so it constructs the struct explicitly. The
generated binding is produced by jextract from the public headers,
so it picks up the new declaration automatically.
Adds testH5Pmodify_filter_by_idx_string and
testH5Pmodify_filter_by_idx_cdvalues to TestH5Z.java and updates the
JUnit golden file. Placement and the test count were derived from
@FixMethodOrder(MethodSorters.NAME_ASCENDING) rather than guessed, and
the file's surrounding format is unchanged.
Not compile-verified locally: this configuration has HDF5_BUILD_JAVA
OFF and the machine has a JRE but no JDK (no javac) and no jextract, so
nothing under java/ can be built here. The C library still builds clean
and the full C/Fortran suite is unchanged. What was checked statically:
the JNI symbol names against the mangling rule (underscores in the Java
method name become _1) and against the existing append pair; the
H5Z_params_t offsets (int type at 0, union at 8, cd_nelmts at 8,
cd_values at 16, 24 bytes total on 64-bit) and the enum values
(H5Z_PARAMS_CDVALUES 0, H5Z_PARAMS_STRING 1) against H5Zpublic.h; and
the H5Pget_filter call in the new test against the working call in
testH5Pappend_filter_cdvalues. Java CI should be treated as the real
gate.
JUnit4's default method sorter (MethodSorters.DEFAULT) orders test
methods by a hash of the method name, not declaration order -- so any
rename or addition shifts the whole run order unpredictably relative
to a hand-maintained golden file. Both java/test/TestH5Z.java (added
this session, 13 methods) and java/src-jni/test/TestH5Z.java (5
methods, unmodified but apparently never correctly regenerated after
the testH5Zget_filter_info2 -> testH5Zget_filter_class_info rename)
had golden JUnit-TestH5Z.txt files that didn't match the real
hash-sorted order, both confirmed by computing Java's documented
String.hashCode() algorithm directly and matching it byte-for-byte
against actual CI failure output.
Verified against upstream develop (an unrelated, currently-open
develop-based PR): this test passes cleanly there, confirming the
mismatch was never a general JUnit/reflection flakiness -- it was
always a genuinely wrong reference file.
Fixed by adding @FixMethodOrder(MethodSorters.NAME_ASCENDING) to both
classes (alphabetical by method name, a properly documented and stable
JUnit4 ordering mode) and regenerating both reference files to match
the resulting deterministic order. This also makes the test immune to
the same class of breakage going forward: alphabetical order is
computable by inspection, unlike the hash order.
The trailing "2" follows HDF5's convention for versioned/deprecated
symbols (H5Dopen2, H5Gcreate2, etc.), implying this is a successor to
H5Zget_filter_info registered in H5vers.txt -- it isn't. The two functions
are independent: H5Zget_filter_info returns only encode/decode config
flags, H5Zget_filter_class_info additionally exposes name, description,
and set_config/get_config presence. Renamed across C, Fortran, and both
Java binding variants (JNI + FFM) since it hadn't shipped yet.
* Fix H5DreadVL failing for pre-allocate cmpd-of-seq dsets
* Fix bad vlen of cmpd with null slot read
* Fix bad cmpd of cmpd read in java
`translate_rbuf`'s H5T_VLEN case had a similar bug where when `found_jList` was set to false due to an entyr in `ret_buf` being null, `ret_buf.add()` would be invoked on an array of objects without the list .add() method. This would occur whenever a read was invoked of a vlen sequence with a null (non-preallocated) entry. The pre-existing tests only tested the pre-allocated cases.
I removed the use of the `found_jList` flag, since it conflated the passing of an unallocated slot with `ret_buf` not being an array. Instead use `ret_buflen == 0` as the check to match the pattern in H5T_INTEGER and other branches.
The test for this fix is testH5Dread_vlen_of_compound_nullslot.
---
`translate_atomic_rebuf` had two issues related to handling of nested compounds. First, it discarded recursive returns, resulting in the construction of empty lists. Secondly, its member offset (`char_buf + i * typeSize + memb_offset`) was incorrect. In this case, `i` was the member index and `memberSize` was the entire cmpd size, so the offset would be erroneously large. It seems like this came from copying of the offset computation from `translate_rbuf`, which had to advance over entire elements of compound data. This error was duplicated on the write side in `translate_atomic_wbuf`'s H5T_COMPOUND case (h5util.c:4611).
I changed `translate_atomic_rbuf` to capture the resultant object, and dropped the `i * typeSize` term in both routines.
The new test verifying the fix works is `testH5Dread_vlen_of_nested_compound`.
* Add exception checks
* Update NULL checks in translate_wbuf
* Correct potentially bad array length check
* Clang format
* Fix readVL/writeVL crash on malformed buffer
* Committing clang-format changes
* Add bufSize checks to wbuf/rbuf translation
* Remove vlen pre-allocation support
* Harden JNI buffer interface
* Handle opaque types as byte[] and document JNI buffer data model
Opaque elements were grouped with H5T_INTEGER in the nested-type
translation path, which boxed them as Integer/Long and rejected
arbitrary-sized opaque blobs. Treat H5T_OPAQUE like H5T_REFERENCE
(a byte[] per element) in translate_atomic_rbuf, translate_atomic_wbuf,
and h5validate_atomic_wbuf so nested opaque round-trips correctly.
Also add "Buffer data model" header comments on translate_rbuf() and
translate_wbuf() and note the reference/opaque byte[] leaves in the
H5.java javadocv.
* Initialize typeSize to fix -Werror=maybe-uninitialized
typeSize was assigned only inside the vl_data_class branch but read in
a second, separate vl_data_class branch, which gcc -O2 flags as
maybe-uninitialized under -Werror. Initialize it to 0 at declaration in
H5Aread/H5Awrite/H5Dread/H5Dwrite, matching the existing vl_array_len
pattern.
* Port nested cmpd/vlen tests to java/test and sync reference
The legacy java/test tree's JUnit-TestH5D.txt reference listed the new
nested compound/vlen tests, but the corresponding @Test methods existed
only in java/src-jni/test/TestH5D.java. Port the 10 tests and the
writeCompoundOfVlenDataset helper into java/test/TestH5D.java, remove
debug prints, and
regenerate the reference to match the actual JUnit output.
* Support nested vlen/compound datatypes in Java FFM compat layer
The FFM compatibility layer (java/hdf) lacked the vlen/compound read and
write support that the JNI interface gained, so the nested cmpd/vlen tests
ported into java/test (TestH5D) failed and leaked an id.
VLDataConverter now has recursive encodeValue/decodeValue helpers that pack
and unpack any member class (integer, float, fixed/vl string, nested
compound, and VLEN) in the native HDF5 in-memory layout. These are wired
into convertCompoundDatatype, readCompoundDatatype and convertRawDataToArrayList,
and a type-aware convertToHVLAuto handles top-level VLEN-of-compound writes.
Compound reads now reclaim VL memory, and type/count mismatches raise
IllegalArgumentException instead of silently corrupting data.
H5DwriteVL rejects an undersized buffer up front and routes VLEN writes
through convertToHVLAuto. The JUnit-TestH5D reference regains its trailing
blank line to match the actual JUnit output.
* Committing clang-format changes
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
The FFM TestH5Z.java gained 8 new test methods covering the new filter
string-config API (H5Pappend_filter, H5Pget_filter_params_by_idx,
H5Zconfig_get_param_*), but the expected-output reference file was left
at 5 tests, causing the JUnit-TestH5Z CTest comparison to fail.
Adds a human-readable key=value parameter string API for HDF5 filters,
alongside the existing integer cd_values arrays.
New C API:
- H5Pappend_filter(plist, filter_id, flags, params) — appends a filter
using either a key=value string or raw cd_values (H5Z_params_t)
- H5Pget_filter_params_by_idx(plist, idx, buf, buf_size, content_len) —
retrieves the parameter string for a filter by pipeline index
- H5Zconfig_get_int/double/bool/str — typed accessors for use inside
filter set_config callbacks
- H5Z_filter_id_by_name(name) — look up a filter id by registered name
- H5Zget_filter_info2(id, info) — extended filter info including v3 fields
New H5Z_class3_t fields: name, description, set_config, get_config,
and reserved blob-callback placeholders (write_blob/read_blob/close_blob).
H5Z_pipeline gains dxpl_id, scaled[], and ndims arguments threaded
through from all call sites so v3 filter callbacks have full context.
All six built-in filters (deflate, shuffle, fletcher32, nbit, szip,
scaleoffset) implement set_config/get_config callbacks.
TOML subset parser: tomlc17 (MIT) vendored in src/tomlc17/ and compiled
unconditionally into libhdf5. Hex-float literals are transparently
rewritten to decimal before parsing. tomlc17 symbols are hidden via
-fvisibility=hidden to prevent namespace collisions.
On-disk format: no new pipeline version. Parameter strings are converted
to cd_values by set_config at H5Pappend_filter time and stored using the
existing v2 pipeline message. On read, get_config reconstructs the string.
Full backward read compatibility is preserved.
Fortran, C++, and Java bindings added. Tests in test/tfilter2.c
(~2300 lines) and testpar/t_filters_parallel.c (par-01–par-04).
h5dump displays filter parameter strings; h5repack accepts TOML-form
UD= filter specs.
Code-review fixes included: tomlc17 visibility, H5Pget_filter_params_by_idx
arg validation and true-length two-pass contract, flags re-validation after
set_config, H5Z_register3 runtime plugin validation, Java two-pass protocol
and h5libraryError() consistency, CHANGELOG corrections.
Fixes GitHub issue #6153
FFM build requires Java 25, Jextract 25.
Generates FFM bindings during configure.
JNI is default when the requirements are not met or can be forced.
Presets added for maven and FFM - JNI is default selection.
Enhanced Maven options will work with either JNI or FFM
New Workflows for testing and maven uploads.
Extensive documentation changes for java.
In order to reduce hash collisions and take advantage of modern memory capacity, the default hash table size for the chunk cache has been increased from 521 to 8191. This means the hash table will consume approximately 64 KiB per open dataset. This value can be changed with H5Pset_cache() or H5Pset_chunk_cache(). This value was chosen because it is a prime number close to 8K.
Change default file format to 1.8 across various tests and examples, updating file creation and access logic accordingly.
Behavior:
Default file format version changed to 1.8 in H5Pfapl.c.
Updated file creation and access to use 1.8 format in h5ex_g_compact.c and test_file_image.c.
Set earliest file format in multiple test files including cache_tagging.c, dtypes.c, and links.c.
Tests:
Modified expected output in tools/test/misc/expected/*.ls files to reflect new file format locations.
Adjusted test logic in test_file_image.c and cache_tagging.c to accommodate format changes.
Misc:
Added comments and TODOs for future format testing in test_file_image.c.
Minor variable renaming for clarity in test_file_image.c.
Behavior:
Modify HDF5UseFortran.cmake to handle cross-compilation by providing default KINDs and sizes when CMAKE_CROSSCOMPILING is true.
Default values for INTEGER and REAL KINDs and their sizes are set when cross-compiling.
Error messages are adjusted to reflect cross-compilation scenarios.
Logic:
Conditional checks added for CMAKE_CROSSCOMPILING to determine if default values should be used.
Default values include INTEGER KINDs {1,2,4,8,16} and REAL KINDs {4,8,10,16}.
Default sizes for native kinds are set to 4 for INTEGER and REAL, and 8 for DOUBLE PRECISION when cross-compiling.
Misc:
Adjustments to verbose messages to indicate when defaults are used during cross-compilation.
* Implement support for complex number datatypes
Adds the new datatype class H5T_COMPLEX
Adds the new API function H5Tcomplex_create which creates a complex
number datatype from an ID of a base floating-point datatype
Adds the new feature check macros H5_HAVE_COMPLEX_NUMBERS and
H5_HAVE_C99_COMPLEX_NUMBERS
Adds the new datatype size macros H5_SIZEOF_FLOAT_COMPLEX,
H5_SIZEOF_DOUBLE_COMPLEX and H5_SIZEOF_LONG_DOUBLE_COMPLEX
Adds the new datatype ID macros H5T_NATIVE_FLOAT_COMPLEX,
H5T_NATIVE_DOUBLE_COMPLEX, H5T_NATIVE_LDOUBLE_COMPLEX,
H5T_CPLX_IEEE_F16LE, H5T_CPLX_IEEE_F16BE,
H5T_CPLX_IEEE_F32LE, H5T_CPLX_IEEE_F32BE,
H5T_CPLX_IEEE_F64LE and H5T_CPLX_IEEE_F64BE
Adds hard and soft datatype conversion paths between complex number
datatypes and all the integer and floating-point datatypes, as well as
between other complex number datatypes
Adds a special conversion path between complex number datatypes and
array or compound datatypes where the in-memory layout of data is the
same between the datatypes and data can be converted directly
Adds support for complex number datatypes to the h5dump, h5ls and
h5diff/ph5diff tools. Allows h5dump '-m' option to change floating-point
printing format for float complex and double complex datatypes, as well
as long double complex if it has the same size as double complex
Adds minimal support to the h5watch and h5import tools
Adds support for the predefined complex number datatypes and
H5Tcomplex_create function to the Java wrappers. Also adds initial,
untested support to the JNI for future use with HDFView
Adds support for just the H5T_COMPLEX datatype class to the Fortran
wrappers
Adds support for the predefined complex number datatypes and
H5Tcomplex_create function to the high level library H5LT interface
for use with the H5LTtext_to_dtype and H5LTdtype_to_text functions
Changes some usages of "complex" in the library since it conflicts with
the "complex" keyword from the complex.h header. Also changes various
usages of the word "complex" throughout the library to distinguish
compound datatypes from complex datatypes.
This is where most people will expect to find license information. The
COPYING_LBNL_HDF5 file has also been renamed to LICENSE_LBNL_HDF5.
The licenses are unchanged.
Cleanup and prepare for thread-safety changes.
Big ideas:
* Wrap H5VL_class_t with H5VL_connector_t, so use of the class can be refcounted within the H5VL package, instead of relying on storing an ID within the H5VL_t struct and incrementing & decrementing the ID's refcount.
* Register H5VL_connector_t* for VOL connector IDs, instead of the H5VL_class_t*
* Stop other packages from rummaging around inside H5VL_connector_t and H5VL_object_t data structures, so that the H5VL package can change implementation details without coupled changes throughout the library
Small things:
* Simplified the coding for creating links
* Moved some routines into more logical locations
* H5F_LIBVER_LATEST changes for move to 1.15
* Add new default api check
* Format fixes
* Fix default configure
* fix lib version tests
* Fix another version variable
* Add 1.14 doc link
* Add compound and refactor out atomic types
* Add Array String tests back
* Convert Attribute version of compound example
* Update transfer atom8ic read to return object
Scot Breitenfeld
Matt Landgithub-actions
Larry Knox
Allen Byrne
Neil Fortner
Aleksandar Jelenak
jhendersonHDF
Dana Robinson
Quincey Koziol