* Close datatype IDs derived from the memory type in the JNI translate helpers
The object-tree read/write helpers in h5util.c derive a base datatype from
the memory type with H5Tget_super() for the variable-length, array and
complex classes, but never closed it. Because an hid_t is not reclaimed when
the native method returns, every read or write of such data leaked at least
one datatype ID for the lifetime of the process, and nested types leaked
one per level.
This PR updates the helpers to close the derived type in their done: blocks,
which covers both the success and the error paths, and to reset the id in the
compound loops to avoid the potential for double closes.
It also has the helpers release the class references that the per-element
helpers look up on entry. These are local references, so they were reclaimed
when the enclosing native method returned, but a compound read calls the helper
once per member per element and held one set per call until then. Releasing
them at the single exit bounds the count of references to the recursion depth.
* Add CHANGELOG entry for the JNI datatype ID leak fix
* Restrict the derived datatype close guards to strictly positive IDs
hid_t 0 is not a valid datatype ID, so H5Tclose(0) would fail.
* Revert "Restrict the derived datatype close guards to strictly positive IDs"
An hid_t of 0 not being a valid ID is a property of the current H5I
encoding rather than a documented guarantee, so the JNI helpers should
not depend on it.
* Reject chunked datasets with mismatched chunk/dspace rank
H5D__chunk_construct() validates that the chunk layout dimensionality
matches the dataspace rank, but that runs only at dataset creation time.
When an existing dataset is opened, H5D__chunk_init() didn't repeat the
check, so a file whose stored chunk rank disagreed with its dataspace rank
was accepted. During chunk I/O the memory-selection rank (from the
dataspace) and the file-selection rank (chunk ndims - 1) then differ, which
produces a zero stride that causes a divide-by-zero in
H5S__hyper_iter_get_seq_list().
H5D__chunk_init() now performs the same dimensionality check on open (the
stored chunk rank includes the extra element-size dimension, so it must be
exactly one greater than the dataspace rank) and rejects a mismatch with an
error.
Added test_chunk_dims_mismatch() as a regression test in test/dsets.c
Fixes#6491
* Fix typo
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Clarify element-vs-byte wording
* Validate chunk/dataspace rank at layout decode time
Move the stored-chunk-rank vs dataspace-rank consistency check out of
H5D__chunk_init() and into H5O__layout_decode(), so a malformed chunked
layout is rejected as the message is decoded (mirroring the fill/datatype
size check in the fill message decode).
* Update release_docs/CHANGELOG.md
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
* Update CHANGELOG
* Pin format version bounds in bad chunk layout generator
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
* Build the examples as C++11 to match the HDF5 C++ library
The standalone examples build forces CMAKE_CXX_STANDARD 98, but
H5public.h includes <cinttypes>, which requires C++11. Any C++
translation unit that includes hdf5.h is therefore affected, not just
users of the C++ API, and the HDF5 C++ library itself is built as C++11
(config/flags/HDFCompilerCXXFlags.cmake). The C++ examples do not
compile as a result, against static or shared HDF5 alike.
Only the standalone build is affected, which is why this is not visible
in ordinary use. The C++98 setting lives in BASIC_SETTINGS, and
HDF5Examples/CMakeLists.txt skips that whole block when
EXAMPLES_EXTERNALLY_CONFIGURED is set -- which HDF5 does for its own
in-tree example build (config/cmake/HDF5ExampleCache.cmake). Built in
tree, the examples inherit HDF5's C++11 and compile normally, and that
is the path the CI workflows exercise. The standalone path, where the
C++98 setting does apply, is driven by the release scripts rather than
by the workflows, and has the C++ examples off by default.
* Select the examples' HL, Fortran and C++ libraries on the right variable
When the examples are built standalone against an installed HDF5, the
HL, Fortran and C++ branches choose between the shared and static
libraries using BUILD_SHARED_LIBS, while the C branch just above them
uses H5EXAMPLE_USE_SHARED_LIBS.
H5EXAMPLE_USE_SHARED_LIBS is what decides whether the "shared" or the
"static" component is requested from find_package, so only the matching
HDF5_<linkage>_<lang>_FOUND variables are ever set. BUILD_SHARED_LIBS
cannot select a linkage on its own; it can only agree or fail to match.
Of its four combinations with H5EXAMPLE_USE_SHARED_LIBS, three produce
no observable difference. In the fourth, H5EXAMPLE_USE_SHARED_LIBS=ON
with BUILD_SHARED_LIBS unset, the shared branch is not taken and the
static branch cannot be, so the HL, Fortran and C++ examples are
disabled with "libs not found" even though the libraries are installed
and were found.
Use H5EXAMPLE_USE_SHARED_LIBS, which is the declared option and is
already what the C branch uses.
A build driven through config/examples/CTestScript.cmake does not reach
the broken combination, because it configures with
HDF5Examples/config/cmake/cacheinit.cmake, which forces
BUILD_SHARED_LIBS=ON. A direct cmake invocation without that cache file
does. In either case the HL, Fortran and C++ examples are off by
default, so this is only visible once they are enabled.
BUILD_SHARED_LIBS remains documented as a user option in
config/examples/HDF5_Examples_options.cmake but no longer influences
library selection; that comment should be revisited separately.
* Abort on infinite loop even when error output is disabled
* Remove abort() on infinite close loop
* Update CHANGELOG.md
* Reference the fixed issue in the CHANGELOG entry
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
The base type lookups in the object-tree helpers tested the
returned hid_t for truth rather than for a negative value. A failed
lookup returns H5I_INVALID_HID (-1), resolving true when checked, so any
failure wouldn't be caught until later.
* Fix JNI datatype ID leak in h5str_detect_vlen_str()
The JNI H5Dread/H5Dwrite/H5Aread/H5Awrite wrappers call h5str_detect_vlen()
on the memory type. For an H5T_ARRAY/H5T_VLEN of a fixed (non-vlen-string)
base type, h5str_detect_vlen_str() acquired the base type via H5Tget_super()
but only closed it when the recursive check returned 1 or a negative error.
When the recursive call returned 0 because no vlen string was found, the base type ID
was leaked.
This PR changes h5str_detect_vlen_str() to close the id unconditionally after the recursive check,
in the same style as the compound-member case in the same function.
A JNI regression test exists at TestH5D.testH5DArray_super_no_id_leak, which reads
an H5T_ARRAY-of-int dataset in a loop and asserts via H5Fget_obj_count() that
no datatype IDs leak.
* Assert non-negative H5Fget_obj_count in array datatype ID leak test
Guard the before/after open-datatype counts against a negative
(failed) H5Fget_obj_count return, which would otherwise let the
equality check pass spuriously. Keep the count scoped to
H5F_OBJ_ALL: the leaked IDs are transient datatypes not attached to
any file, so a per-file count would not see them.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Modify CHANGELOG entry
* Modify CHANGELOG again
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: H. Joe Lee <hyoklee@hdfgroup.org>
* Fix H5DreadVL failing for pre-allocate cmpd-of-seq dsets
* Fix bad vlen of cmpd with null slot read
* Fix bad cmpd of cmpd read in java
`translate_rbuf`'s H5T_VLEN case had a similar bug where when `found_jList` was set to false due to an entyr in `ret_buf` being null, `ret_buf.add()` would be invoked on an array of objects without the list .add() method. This would occur whenever a read was invoked of a vlen sequence with a null (non-preallocated) entry. The pre-existing tests only tested the pre-allocated cases.
I removed the use of the `found_jList` flag, since it conflated the passing of an unallocated slot with `ret_buf` not being an array. Instead use `ret_buflen == 0` as the check to match the pattern in H5T_INTEGER and other branches.
The test for this fix is testH5Dread_vlen_of_compound_nullslot.
---
`translate_atomic_rebuf` had two issues related to handling of nested compounds. First, it discarded recursive returns, resulting in the construction of empty lists. Secondly, its member offset (`char_buf + i * typeSize + memb_offset`) was incorrect. In this case, `i` was the member index and `memberSize` was the entire cmpd size, so the offset would be erroneously large. It seems like this came from copying of the offset computation from `translate_rbuf`, which had to advance over entire elements of compound data. This error was duplicated on the write side in `translate_atomic_wbuf`'s H5T_COMPOUND case (h5util.c:4611).
I changed `translate_atomic_rbuf` to capture the resultant object, and dropped the `i * typeSize` term in both routines.
The new test verifying the fix works is `testH5Dread_vlen_of_nested_compound`.
* Add exception checks
* Update NULL checks in translate_wbuf
* Correct potentially bad array length check
* Clang format
* Fix readVL/writeVL crash on malformed buffer
* Committing clang-format changes
* Add bufSize checks to wbuf/rbuf translation
* Remove vlen pre-allocation support
* Harden JNI buffer interface
* Handle opaque types as byte[] and document JNI buffer data model
Opaque elements were grouped with H5T_INTEGER in the nested-type
translation path, which boxed them as Integer/Long and rejected
arbitrary-sized opaque blobs. Treat H5T_OPAQUE like H5T_REFERENCE
(a byte[] per element) in translate_atomic_rbuf, translate_atomic_wbuf,
and h5validate_atomic_wbuf so nested opaque round-trips correctly.
Also add "Buffer data model" header comments on translate_rbuf() and
translate_wbuf() and note the reference/opaque byte[] leaves in the
H5.java javadocv.
* Initialize typeSize to fix -Werror=maybe-uninitialized
typeSize was assigned only inside the vl_data_class branch but read in
a second, separate vl_data_class branch, which gcc -O2 flags as
maybe-uninitialized under -Werror. Initialize it to 0 at declaration in
H5Aread/H5Awrite/H5Dread/H5Dwrite, matching the existing vl_array_len
pattern.
* Port nested cmpd/vlen tests to java/test and sync reference
The legacy java/test tree's JUnit-TestH5D.txt reference listed the new
nested compound/vlen tests, but the corresponding @Test methods existed
only in java/src-jni/test/TestH5D.java. Port the 10 tests and the
writeCompoundOfVlenDataset helper into java/test/TestH5D.java, remove
debug prints, and
regenerate the reference to match the actual JUnit output.
* Support nested vlen/compound datatypes in Java FFM compat layer
The FFM compatibility layer (java/hdf) lacked the vlen/compound read and
write support that the JNI interface gained, so the nested cmpd/vlen tests
ported into java/test (TestH5D) failed and leaked an id.
VLDataConverter now has recursive encodeValue/decodeValue helpers that pack
and unpack any member class (integer, float, fixed/vl string, nested
compound, and VLEN) in the native HDF5 in-memory layout. These are wired
into convertCompoundDatatype, readCompoundDatatype and convertRawDataToArrayList,
and a type-aware convertToHVLAuto handles top-level VLEN-of-compound writes.
Compound reads now reclaim VL memory, and type/count mismatches raise
IllegalArgumentException instead of silently corrupting data.
H5DwriteVL rejects an undersized buffer up front and routes VLEN writes
through convertToHVLAuto. The JUnit-TestH5D reference regains its trailing
blank line to match the actual JUnit output.
* Committing clang-format changes
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Fix zero-element read failure in H5D__virtual_read() and H5D__virtual_write() in H5Dvirtual.c by skipping H5S_SELECT_BOUNDS if nelmts is 0.
Add test_vds_empty_slice() in rtree.c to verify zero-element read behavior on virtual datasets with r-trees.
Implement the special filename handling that makes the rtree test compatible with the family VFD (and any other VFD that changes how filenames are treated).
Make rtree.c test VFD-compatible by implementing dynamic filename handling and updating function signatures for file access property lists.
Ensure proper closure of resources in gent_array1_big, gent_nested_compound_dt, and gent_intscalars in h5dumpgentest.c.
Add error handling and cleanup logic to prevent resource leaks.
Malformed files can have a zero name-length, which when subtracted lead to an overflow and an out-of-bounds read.
Check that name length is not too small in addition to checking for an overflow directly.
Optimize VDS operations using R-tree spatial index, adding new API functions and tests for improved performance.
Behavior:
Introduces R-tree spatial index for optimizing VDS operations in H5Dvirtual.c.
Adds H5Pset_dset_use_spatial_tree() and H5Pget_dset_use_spatial_tree() to control R-tree usage.
Default behavior uses R-tree for VDS with more than 1000 mappings.
Implementation:
Adds H5RT.c, H5RTprivate.h, and H5RTpkg.h for R-tree implementation.
Updates H5Pdapl.c and H5Pdcpl.c to include R-tree properties.
Modifies H5Dvirtual.c to integrate R-tree in VDS I/O operations.
Testing:
Adds rtree.c for testing R-tree creation, search, and copy operations.
Tests R-tree integration with VDS in test/dsets.c.
Verifies R-tree behavior with different dataset access property list settings.
Implement an r-tree data structure in a new module. It has three exposed methods: H5RT_create(), H5RT_search(), and H5RT_free().
The STR algorithm used during creation is based on the one described here.
Updates CMakeLists.txt to include H5RT.c and related headers.
Adds rtree.c test file to validate R-tree creation and search functionalities.
Add VOL tests for h5dump, update test generation logic, and modify test configurations.
Tests:
Add VOL tests for h5dump in CMakeTests.cmake and CMakeTestsXML.cmake.
Modify ADD_H5_TEST macro to support VOL-specific tests and filtering.
Add gent_tvms() function in h5dumpgentest.c and h5dumpgentest.h for generating test files.
The trefer_extR test depends upon this pair of files, but always uses the pre-generated ones checked into the repo. This adds a routine to generate them, which will soon be used by the h5dump VOL tests.
Refactor ADD_H5_TEST macro in CMakeTests.cmake to consolidate test definitions and improve consistency.
Macros:
Consolidated ADD_H5_TEST and ADD_H5_ERR_TEST into a single ADD_H5_TEST macro in CMakeTests.cmake.
ADD_H5_TEST now accepts RESULT_CODE and optional RESULT_ERRCHECK as keyword arguments.
Test Definitions:
Updated all test invocations to use the new ADD_H5_TEST macro format with RESULT_CODE and RESULT_ERRCHECK where applicable.
Removed redundant ADD_H5_ERR_TEST macro usage.
Behavior:
Ensures consistent test setup and error handling across all h5ls tests.
Move the portion of test_reference_external() in charge of test file creation to its own dedicated function. This will eventually be helpful if/when we change the test file creation here to used a shared routine.
Improve test file cleanup in h5repacktst.c by using h5_delete_test_file and avoiding deletion of default-driver exclusive files with non-default drivers.
Behavior:
Use h5_delete_test_file in h5repacktst.c for file deletions, ensuring compatibility with Family VFD filename requirements.
Avoid deleting default-driver exclusive files with non-default drivers in h5repacktst.c.
File Management:
Reorganize file arrays in h5repackgentest.h to separate default-driver files from others.
* Add header files for tool gentest scripts
* Correct library type for core targets
* Use EXIT_SUCCESS/EXIT_FAILURE on script exit
* Remove h5clear from merged script
* Include gentest header
* Add udfilter files to h5copy gen func
* Add nerror returns to main generator functions
* Move defines to tool headers
* Add copyright header
* Fix bad VOL env var handling in CMake tests
* Preserve VOL connector env during filter plugin test
* Do not load env VOL connector during filter plugin test
Matt L
