Added 3 new functions to support this: H5TSset_internal_threads(), H5Pset_io_threads(), and H5Pget_io_threads().
This feature internally parallelizes read operations on chunked datasets. H5TSset_internal_threads() is used to enable the feature globally, while H5Pset_io_threads() can be used to disable the feature on a per-operation basis. These functions are only available when the library is configured with HDF5_ENABLE_CONCURRENCY=ON. When performing an internally threaded read, the library will concurrently read from disk, unfilter, and scatter to memory all chunks in a read operation on a chunked dataset. Currently each of these sub-operations is serialized (protected by a mutex), so there is not yet likely to be any performance improvement.
* Reject chunked datasets with mismatched chunk/dspace rank
H5D__chunk_construct() validates that the chunk layout dimensionality
matches the dataspace rank, but that runs only at dataset creation time.
When an existing dataset is opened, H5D__chunk_init() didn't repeat the
check, so a file whose stored chunk rank disagreed with its dataspace rank
was accepted. During chunk I/O the memory-selection rank (from the
dataspace) and the file-selection rank (chunk ndims - 1) then differ, which
produces a zero stride that causes a divide-by-zero in
H5S__hyper_iter_get_seq_list().
H5D__chunk_init() now performs the same dimensionality check on open (the
stored chunk rank includes the extra element-size dimension, so it must be
exactly one greater than the dataspace rank) and rejects a mismatch with an
error.
Added test_chunk_dims_mismatch() as a regression test in test/dsets.c
Fixes#6491
* Fix typo
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Clarify element-vs-byte wording
* Validate chunk/dataspace rank at layout decode time
Move the stored-chunk-rank vs dataspace-rank consistency check out of
H5D__chunk_init() and into H5O__layout_decode(), so a malformed chunked
layout is rejected as the message is decoded (mirroring the fill/datatype
size check in the fill message decode).
* Update release_docs/CHANGELOG.md
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
* Update CHANGELOG
* Pin format version bounds in bad chunk layout generator
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
* Avoid converting fill value from memory to file type when reading vlen
data from a nonexistent chunk.
* Committing clang-format changes
* Add Changelog note
* Spelling
* Update changelog note
* Improve test description
* Hopefully make formatter happy
* Hyphenate read-only
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Verify uncompressed chunks are the right size after being uncompressed (reverse filtered)
Verify that the buffer returned from the filter callback is large enough to hold the returned data size
Fix bug in deflate filter that caused it to report the wrong buffer size
Fix bug in chunk copy code that could cause a background buffer overflow
Fix bug in chunk copy code that could cause a double free if the filter realloced the data buffer
Other general cleanup
Closes can_apply and set_local filter callbacks skipped for H5T_VARIABLE dtypes #5942. A more detailed explanation of the rationale for this PR is in the linked issue.
Mandatory filters can be now applied to variable-length data. This notably includes h5py object strings and NpyStrings.
Optional filters applied to variable-length data no longer skip can_apply and set_local, only to jump directly to filter. This fixes hdf5_blosc and HDF5_Blosc2, which rely on set_local. Note that all filters which have neither can_apply nor set_local have always worked fine with vlen data, as long as they were tagged as optional. Notable examples are deflate, lzf, bzip2 (from hdf5plugin), and lz4 (also from hdf5plugin).
Lossy filters, such as scale-offset, are and remain untested and most likely broken, as I expect them to treat the vlen metadata as numbers and corrupt it. The right place to ensure proper behaviour is their can_apply function, which however is broken (filters: can_apply does nothing for optional filters #6161).
Various related changes, including refactoring part of dataset creation, and reworking how layout versions are calculated.
Needs more testing of filters that create very large chunks, but that will need code to turn it off in cases where we can't allocate >4GiB buffers. I tested this manually by increasing the expansion ration in the expand2 test in dsets, and it passed everything up until it tried to expand the datasets with H5Dset_extent() and my laptop ran out of disk space.
We should add code and testing to handle the case where the "size of size" is set less than 8 bytes. This is not a new issue, since it can be set to 2.
I will file issues for these, but I don't think they are necessary for the release.
* Change new chunk indexing methods to always encode chunk size as a 64
bit (size of lengths) integer, when using the 2.0 file format.
* Add CHANGELOG.md note
* Spelling
* Fix errors in parallel build
* Committing clang-format changes
* More parallel fixes.
* Committing clang-format changes
* Another parallel fix
* Fix parallel for real this time I hope
* Update function descriptions in dsets.c
* Fix spelling
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Optimize VDS operations using R-tree spatial index, adding new API functions and tests for improved performance.
Behavior:
Introduces R-tree spatial index for optimizing VDS operations in H5Dvirtual.c.
Adds H5Pset_dset_use_spatial_tree() and H5Pget_dset_use_spatial_tree() to control R-tree usage.
Default behavior uses R-tree for VDS with more than 1000 mappings.
Implementation:
Adds H5RT.c, H5RTprivate.h, and H5RTpkg.h for R-tree implementation.
Updates H5Pdapl.c and H5Pdcpl.c to include R-tree properties.
Modifies H5Dvirtual.c to integrate R-tree in VDS I/O operations.
Testing:
Adds rtree.c for testing R-tree creation, search, and copy operations.
Tests R-tree integration with VDS in test/dsets.c.
Verifies R-tree behavior with different dataset access property list settings.
Add hash tables for detecting repeated source file and dataset names.
Share these repeated strings between mappings in memory. Add new
encoding format for VDS for shared names. Add tests and documentation
for these changes.
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Matthew Larson <matthewjlar@gmail.com>
* Implement support for complex number datatypes
Adds the new datatype class H5T_COMPLEX
Adds the new API function H5Tcomplex_create which creates a complex
number datatype from an ID of a base floating-point datatype
Adds the new feature check macros H5_HAVE_COMPLEX_NUMBERS and
H5_HAVE_C99_COMPLEX_NUMBERS
Adds the new datatype size macros H5_SIZEOF_FLOAT_COMPLEX,
H5_SIZEOF_DOUBLE_COMPLEX and H5_SIZEOF_LONG_DOUBLE_COMPLEX
Adds the new datatype ID macros H5T_NATIVE_FLOAT_COMPLEX,
H5T_NATIVE_DOUBLE_COMPLEX, H5T_NATIVE_LDOUBLE_COMPLEX,
H5T_CPLX_IEEE_F16LE, H5T_CPLX_IEEE_F16BE,
H5T_CPLX_IEEE_F32LE, H5T_CPLX_IEEE_F32BE,
H5T_CPLX_IEEE_F64LE and H5T_CPLX_IEEE_F64BE
Adds hard and soft datatype conversion paths between complex number
datatypes and all the integer and floating-point datatypes, as well as
between other complex number datatypes
Adds a special conversion path between complex number datatypes and
array or compound datatypes where the in-memory layout of data is the
same between the datatypes and data can be converted directly
Adds support for complex number datatypes to the h5dump, h5ls and
h5diff/ph5diff tools. Allows h5dump '-m' option to change floating-point
printing format for float complex and double complex datatypes, as well
as long double complex if it has the same size as double complex
Adds minimal support to the h5watch and h5import tools
Adds support for the predefined complex number datatypes and
H5Tcomplex_create function to the Java wrappers. Also adds initial,
untested support to the JNI for future use with HDFView
Adds support for just the H5T_COMPLEX datatype class to the Fortran
wrappers
Adds support for the predefined complex number datatypes and
H5Tcomplex_create function to the high level library H5LT interface
for use with the H5LTtext_to_dtype and H5LTdtype_to_text functions
Changes some usages of "complex" in the library since it conflicts with
the "complex" keyword from the complex.h header. Also changes various
usages of the word "complex" throughout the library to distinguish
compound datatypes from complex datatypes.
Since each API context is local to a thread, use the stack to
store the context instead of allocating & releasing it each time.
This improves performance (slightly), reduces alloc/free calls,
and eliminates the H5FL package from the push & pop operations,
which helps simplify threadsafe operation.
One effect of this change is that the H5VLstart_lib_state /
H5VLfinish_lib_state API routines for pass through connector
authors now require a parameter that can be used to store
the library's context. It was probably a mistake to assume
that these two routines would not do this previously, so this
is essentially a bug fix for them.
Some other minor things:
* Added API context push+pop operations to cache tests
(I'm not actually certain why this was working before) and
a few other places
* Cleaned up a bunch of warnings in test code (calloc args, mainly)
* Made header file inclusions more standard in some source files
This is where most people will expect to find license information. The
COPYING_LBNL_HDF5 file has also been renamed to LICENSE_LBNL_HDF5.
The licenses are unchanged.
Added new testframe.h header to document testing framework functions and
split them away from h5test.h and from test programs that don't
integrate with the testframe.c testing framework
Added new test setup callback to testframe.c testing framework
Added parameters to AddTest() to specify size of test parameters so they
can be copied for later use
Enabled HDF5 error stacks in testframe.c framework by default and added
some error stack suppressions to some testhdf5 tests
Added new maxthreads option to testframe.c framework to allow specifying
the maximum number of threads a multi-threaded test can use
Moved TestExpress functionality out of testframe.c and into more general
h5test.c for wider use by tests through getter and setter
Updated some tests to not mix and match functionality between h5test.c/h
and testframe.c/h
Moved some functionality from testphdf5.h into testpar.h for parallel
tests that aren't part of testphdf5
Added new parallel test library that contains common shared
functionality for parallel tests (similar to h5test library)
* off_t --> HDoff_t in test
* off_t --> HDoff_t in h5ls, h5dump, & h5repack
* Minor off_t tweak to configure.ac
* off_t --> HDoff_t fixes in src
This changes the type of the offset parameter in H5Pget_external()
to HDoff_t to match H5Pset_external(), along with other minor tweaks.
* off_t --> HDoff_t in C++ wrappers
* Fix off_t usage in Java wrapper
* Clean up random number generator code
Depending on the platform, we use a mix of random, rand, and rand_r
to generate pseudo-random numbers, along with a messy set of ifdefs
in H5private.h. We are not a cryptographic library, only use random
numbers in our test code, and have no need for anything more than the
C standard's (s)rand(). There's no point dithering about using rand()
vs random() when we're also doing bad things like using mod to
restrict the range, which introduces bias.
Also removes CMake/configure checks for rand_r and random
* Remove random/rand_r checks from build system
* Fix missed HDrandom after GitHub merge
Changes Autotools testing to use HDF5_TEST_DRIVER environment
variable to avoid running tests that don't work well with several
VFDs
Restores old h5_get_vfd_fapl() testing function to setup a FAPL
with a particular VFD
Adds a macro for the default VFD name
lseek on Windows uses __int64 for both the offset and return type
instead of off_t like most POSIX systems. This changes ensures we
use HDoff_t (which is typdef'd correctly on Windows) w/ lseek.
* Strip HD prefix from string/char C API calls
* HD(f)(put|get)(s|c)
* HDstr*
* HDv*printf
* HD(s)(print|scan)f
* HDperror
But NOT:
* HDstrcase*
* HDvasprintf
* HDstrtok_r
* HDstrndup
As those are not C99 and have portability work-around
implementations. They will be handled later.
* Fix th5_system.c screwup
* Fixed various -Wmissing-variable-declarations by adding static keyword
* In a few cases, renamed the variable suffix from _g to _s.
* Fixed some -Wmissing-variable-declarations by using different declaration macros
* Fixed various -Wconditional-uninitialized warnings by just initializing variable to zero
* Fixed various -Wcomma warnings
* Fixed clang -Wstrict-prototypes warnings
* Fixed various -Wunused-variable warnings
* Updated some casts to fix the only 3 -Wcast-qual warnings
* Fixed the only -Wsometimes-uninitialized warning
* Adds semicolons to function-like macros
* Adds a do..while(0) loop to some macros
* Removes semicolons when inappropriate, especially H5E_TRY_BEGIN/END
szip (or libaec) is currently not tested in CI. This adds szip to the
the Autotools GitHub CI actions on Linux when building with the
Autotools.
This PR also cleans up a few warnings that remained in the szip-
related code so the -Werror check will pass.
Neil Fortner
jhendersonHDF
Guido Imperiale
Neil Fortner
Allen Byrne
Quincey Koziol
Dana Robinson
Scot Breitenfeld
Sean McBride