The base type lookups in the object-tree helpers tested the
returned hid_t for truth rather than for a negative value. A failed
lookup returns H5I_INVALID_HID (-1), resolving true when checked, so any
failure wouldn't be caught until later.
* Fix JNI datatype ID leak in h5str_detect_vlen_str()
The JNI H5Dread/H5Dwrite/H5Aread/H5Awrite wrappers call h5str_detect_vlen()
on the memory type. For an H5T_ARRAY/H5T_VLEN of a fixed (non-vlen-string)
base type, h5str_detect_vlen_str() acquired the base type via H5Tget_super()
but only closed it when the recursive check returned 1 or a negative error.
When the recursive call returned 0 because no vlen string was found, the base type ID
was leaked.
This PR changes h5str_detect_vlen_str() to close the id unconditionally after the recursive check,
in the same style as the compound-member case in the same function.
A JNI regression test exists at TestH5D.testH5DArray_super_no_id_leak, which reads
an H5T_ARRAY-of-int dataset in a loop and asserts via H5Fget_obj_count() that
no datatype IDs leak.
* Assert non-negative H5Fget_obj_count in array datatype ID leak test
Guard the before/after open-datatype counts against a negative
(failed) H5Fget_obj_count return, which would otherwise let the
equality check pass spuriously. Keep the count scoped to
H5F_OBJ_ALL: the leaked IDs are transient datatypes not attached to
any file, so a per-file count would not see them.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Modify CHANGELOG entry
* Modify CHANGELOG again
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: H. Joe Lee <hyoklee@hdfgroup.org>
* Fix H5DreadVL failing for pre-allocate cmpd-of-seq dsets
* Fix bad vlen of cmpd with null slot read
* Fix bad cmpd of cmpd read in java
`translate_rbuf`'s H5T_VLEN case had a similar bug where when `found_jList` was set to false due to an entyr in `ret_buf` being null, `ret_buf.add()` would be invoked on an array of objects without the list .add() method. This would occur whenever a read was invoked of a vlen sequence with a null (non-preallocated) entry. The pre-existing tests only tested the pre-allocated cases.
I removed the use of the `found_jList` flag, since it conflated the passing of an unallocated slot with `ret_buf` not being an array. Instead use `ret_buflen == 0` as the check to match the pattern in H5T_INTEGER and other branches.
The test for this fix is testH5Dread_vlen_of_compound_nullslot.
---
`translate_atomic_rebuf` had two issues related to handling of nested compounds. First, it discarded recursive returns, resulting in the construction of empty lists. Secondly, its member offset (`char_buf + i * typeSize + memb_offset`) was incorrect. In this case, `i` was the member index and `memberSize` was the entire cmpd size, so the offset would be erroneously large. It seems like this came from copying of the offset computation from `translate_rbuf`, which had to advance over entire elements of compound data. This error was duplicated on the write side in `translate_atomic_wbuf`'s H5T_COMPOUND case (h5util.c:4611).
I changed `translate_atomic_rbuf` to capture the resultant object, and dropped the `i * typeSize` term in both routines.
The new test verifying the fix works is `testH5Dread_vlen_of_nested_compound`.
* Add exception checks
* Update NULL checks in translate_wbuf
* Correct potentially bad array length check
* Clang format
* Fix readVL/writeVL crash on malformed buffer
* Committing clang-format changes
* Add bufSize checks to wbuf/rbuf translation
* Remove vlen pre-allocation support
* Harden JNI buffer interface
* Handle opaque types as byte[] and document JNI buffer data model
Opaque elements were grouped with H5T_INTEGER in the nested-type
translation path, which boxed them as Integer/Long and rejected
arbitrary-sized opaque blobs. Treat H5T_OPAQUE like H5T_REFERENCE
(a byte[] per element) in translate_atomic_rbuf, translate_atomic_wbuf,
and h5validate_atomic_wbuf so nested opaque round-trips correctly.
Also add "Buffer data model" header comments on translate_rbuf() and
translate_wbuf() and note the reference/opaque byte[] leaves in the
H5.java javadocv.
* Initialize typeSize to fix -Werror=maybe-uninitialized
typeSize was assigned only inside the vl_data_class branch but read in
a second, separate vl_data_class branch, which gcc -O2 flags as
maybe-uninitialized under -Werror. Initialize it to 0 at declaration in
H5Aread/H5Awrite/H5Dread/H5Dwrite, matching the existing vl_array_len
pattern.
* Port nested cmpd/vlen tests to java/test and sync reference
The legacy java/test tree's JUnit-TestH5D.txt reference listed the new
nested compound/vlen tests, but the corresponding @Test methods existed
only in java/src-jni/test/TestH5D.java. Port the 10 tests and the
writeCompoundOfVlenDataset helper into java/test/TestH5D.java, remove
debug prints, and
regenerate the reference to match the actual JUnit output.
* Support nested vlen/compound datatypes in Java FFM compat layer
The FFM compatibility layer (java/hdf) lacked the vlen/compound read and
write support that the JNI interface gained, so the nested cmpd/vlen tests
ported into java/test (TestH5D) failed and leaked an id.
VLDataConverter now has recursive encodeValue/decodeValue helpers that pack
and unpack any member class (integer, float, fixed/vl string, nested
compound, and VLEN) in the native HDF5 in-memory layout. These are wired
into convertCompoundDatatype, readCompoundDatatype and convertRawDataToArrayList,
and a type-aware convertToHVLAuto handles top-level VLEN-of-compound writes.
Compound reads now reclaim VL memory, and type/count mismatches raise
IllegalArgumentException instead of silently corrupting data.
H5DwriteVL rejects an undersized buffer up front and routes VLEN writes
through convertToHVLAuto. The JUnit-TestH5D reference regains its trailing
blank line to match the actual JUnit output.
* Committing clang-format changes
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Fix zero-element read failure in H5D__virtual_read() and H5D__virtual_write() in H5Dvirtual.c by skipping H5S_SELECT_BOUNDS if nelmts is 0.
Add test_vds_empty_slice() in rtree.c to verify zero-element read behavior on virtual datasets with r-trees.
Implement the special filename handling that makes the rtree test compatible with the family VFD (and any other VFD that changes how filenames are treated).
Make rtree.c test VFD-compatible by implementing dynamic filename handling and updating function signatures for file access property lists.
Ensure proper closure of resources in gent_array1_big, gent_nested_compound_dt, and gent_intscalars in h5dumpgentest.c.
Add error handling and cleanup logic to prevent resource leaks.
Malformed files can have a zero name-length, which when subtracted lead to an overflow and an out-of-bounds read.
Check that name length is not too small in addition to checking for an overflow directly.
Optimize VDS operations using R-tree spatial index, adding new API functions and tests for improved performance.
Behavior:
Introduces R-tree spatial index for optimizing VDS operations in H5Dvirtual.c.
Adds H5Pset_dset_use_spatial_tree() and H5Pget_dset_use_spatial_tree() to control R-tree usage.
Default behavior uses R-tree for VDS with more than 1000 mappings.
Implementation:
Adds H5RT.c, H5RTprivate.h, and H5RTpkg.h for R-tree implementation.
Updates H5Pdapl.c and H5Pdcpl.c to include R-tree properties.
Modifies H5Dvirtual.c to integrate R-tree in VDS I/O operations.
Testing:
Adds rtree.c for testing R-tree creation, search, and copy operations.
Tests R-tree integration with VDS in test/dsets.c.
Verifies R-tree behavior with different dataset access property list settings.
Implement an r-tree data structure in a new module. It has three exposed methods: H5RT_create(), H5RT_search(), and H5RT_free().
The STR algorithm used during creation is based on the one described here.
Updates CMakeLists.txt to include H5RT.c and related headers.
Adds rtree.c test file to validate R-tree creation and search functionalities.
Add VOL tests for h5dump, update test generation logic, and modify test configurations.
Tests:
Add VOL tests for h5dump in CMakeTests.cmake and CMakeTestsXML.cmake.
Modify ADD_H5_TEST macro to support VOL-specific tests and filtering.
Add gent_tvms() function in h5dumpgentest.c and h5dumpgentest.h for generating test files.
The trefer_extR test depends upon this pair of files, but always uses the pre-generated ones checked into the repo. This adds a routine to generate them, which will soon be used by the h5dump VOL tests.
Refactor ADD_H5_TEST macro in CMakeTests.cmake to consolidate test definitions and improve consistency.
Macros:
Consolidated ADD_H5_TEST and ADD_H5_ERR_TEST into a single ADD_H5_TEST macro in CMakeTests.cmake.
ADD_H5_TEST now accepts RESULT_CODE and optional RESULT_ERRCHECK as keyword arguments.
Test Definitions:
Updated all test invocations to use the new ADD_H5_TEST macro format with RESULT_CODE and RESULT_ERRCHECK where applicable.
Removed redundant ADD_H5_ERR_TEST macro usage.
Behavior:
Ensures consistent test setup and error handling across all h5ls tests.
Move the portion of test_reference_external() in charge of test file creation to its own dedicated function. This will eventually be helpful if/when we change the test file creation here to used a shared routine.
Improve test file cleanup in h5repacktst.c by using h5_delete_test_file and avoiding deletion of default-driver exclusive files with non-default drivers.
Behavior:
Use h5_delete_test_file in h5repacktst.c for file deletions, ensuring compatibility with Family VFD filename requirements.
Avoid deleting default-driver exclusive files with non-default drivers in h5repacktst.c.
File Management:
Reorganize file arrays in h5repackgentest.h to separate default-driver files from others.
* Add header files for tool gentest scripts
* Correct library type for core targets
* Use EXIT_SUCCESS/EXIT_FAILURE on script exit
* Remove h5clear from merged script
* Include gentest header
* Add udfilter files to h5copy gen func
* Add nerror returns to main generator functions
* Move defines to tool headers
* Add copyright header
* Fix bad VOL env var handling in CMake tests
* Preserve VOL connector env during filter plugin test
* Do not load env VOL connector during filter plugin test
Matt L
