Added 3 new functions to support this: H5TSset_internal_threads(), H5Pset_io_threads(), and H5Pget_io_threads().
This feature internally parallelizes read operations on chunked datasets. H5TSset_internal_threads() is used to enable the feature globally, while H5Pset_io_threads() can be used to disable the feature on a per-operation basis. These functions are only available when the library is configured with HDF5_ENABLE_CONCURRENCY=ON. When performing an internally threaded read, the library will concurrently read from disk, unfilter, and scatter to memory all chunks in a read operation on a chunked dataset. Currently each of these sub-operations is serialized (protected by a mutex), so there is not yet likely to be any performance improvement.
* Avoid converting fill value from memory to file type when reading vlen
data from a nonexistent chunk.
* Committing clang-format changes
* Add Changelog note
* Spelling
* Update changelog note
* Improve test description
* Hopefully make formatter happy
* Hyphenate read-only
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Verify uncompressed chunks are the right size after being uncompressed (reverse filtered)
Verify that the buffer returned from the filter callback is large enough to hold the returned data size
Fix bug in deflate filter that caused it to report the wrong buffer size
Fix bug in chunk copy code that could cause a background buffer overflow
Fix bug in chunk copy code that could cause a double free if the filter realloced the data buffer
Other general cleanup
* Implement big I/O support for independent reads. Add test for this.
* Committing clang-format changes
* Update big I/O code in H5FD__mpio_write() to match that in
H5FD__mpio_read().
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* Implement H5ESget_requests() to retrieve request pointers and
corresponding connector IDs from an event set. Add tests for this
function.
* Add "order" parameter to H5ESget_requests, to allow the user to specify
the order in which requests are returned.
* Fix bugs with H5ESinsert_request()
* Change H5ESget_requests() API to have separate input and output
parameters for the allocated array length(s) (in) and the number of events in
the event set (out).
* Fix issue with H5_now_usec() with 32 bit systems. Also improve event
set test.
* Committing clang-format changes
* Add comments to H5_now_usec() explaining rationale for uint64_t casts.
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* Fixed problems with vlens and refs inside compound using H5VLget_file_type()
* Fix date in RELEASE.txt
* Add assertions
* Move some manipulation of H5VL_object_t struct fields into the H5VL
package.
* Cache the pointer to the next point to process after the last call to
H5S__get_select_elem_pointlist. This allows the normal process of
iterating over the points in batches to be much more efficient, as the
library does not need to traverse the entirety of the preceding points
every time the funciton is re-entered.
* Update RELEASE.txt for point selection iteration performance fix.
* commit 'ce653ff82b7dd52b8e36a5eff7b1728c4b75d9f7':
Modify H5VL initialization routines to initialize all VOL-managed object types. Modify H5VLwrap_register() to reject non-VOL-managed object types. Also fix overisights in h5trace.c from previous changes.
* commit '40b2658b4b266b4dc65ec3db2897e0aaf54f8e42':
Fix missing free in H5T__ref_mem_read()
Fix bugs in H5VL file comparison code. Add short circuit success to H5VL_cmp_connector_cls().
Implement file comparison VOL callback. Other changes to allow references to work with non-native connectors. There is a bug somewhere.
* commit '4e12984b77cdd7615843d94f8de8d54db27476ac':
Move checking for zero offset in selection adjust calls to the selection callbacks. This makes the procedure for checking it consistent across selection types and between _s and _u, ensures it is always is performed even when called within the H5S package, and removes the redundant check that would occur when callins H5S_select_adjust_s() from outside the H5S package.
Replace H5Sselect_adjust_u() and H5Shyper_adjust_s() with H5Sselect_adjust. Implement "adjust_s" callback for all selection types. Add range checking to H5Sselect_adjust().
* commit 'c104f4cb87a1558ab6a772967887738dc9800f0d':
Modify H5Ssel_iter_get_seq_list() to accept iterators that have reached the end of iteration (returning zero sequences). Previously it could cause an assertion failure.
procedure for checking it consistent across selection types and between
_s and _u, ensures it is always is performed even when called within the H5S package, and removes the redundant check that would occur when callins H5S_select_adjust_s() from outside the H5S package.
* commit 'db6eab893e5f027f0b535316e0e7a9abe9ff8619':
Implement public H5Sselect_project_intersection(). Updated internal algorithm to (optionally) avoid sharing selection data structures. Tested internal code (including with valgrind) by setting VDS code to avoid sharing selection, has since been changed to share selection for performance, so this code is not yet tested in regression tests. API has not been tested.
algorithm to (optionally) avoid sharing selection data structures.
Tested internal code (including with valgrind) by setting VDS code to
avoid sharing selection, has since been changed to share selection for
performance, so this code is not yet tested in regression tests. API
has not been tested.
* commit '1dbec40d465072540f8507c524ae7b909a6b1cf0':
Implement H5VLget_file_type() to return a copy of a datatype with the location set to be in a file. Only meant to be used by VOL connectors. Implement H5VLpeek_connector_id() to support connectors querying their own IDs. Fix app_ref with connector IDs in a couple places (external VOLs registered as default through ENV should be visible to the application). Modify vlen and reference interfaces to work with arbitrary VOL connectors. Implement file "post open" specific callback, to enable connectors to update their file structs after a wrap context has been set.
location set to be in a file. Only meant to be used by VOL connectors.
Implement H5VLpeek_connector_id() to support connectors querying their
own IDs. Fix app_ref with connector IDs in a couple places (external
VOLs registered as default through ENV should be visible to the
application). Modify vlen and reference interfaces to work with
arbitrary VOL connectors. Implement file "post open" specific
callback, to enable connectors to update their file structs after a
wrap context has been set.
Complete rework of H5S__hyper_project_intersection. Addresses VDS performance issues raised in HDFFV-10693.
* commit 'd3e242eba1ba98726ed64b14e75f6adee724bea7':
Minor fixes/improvements for VDS performance improvement (HDFFV-10693).
Refactor H5S__hyper_project_intersection to operate directly on span trees, improving performance and removing the conditionn that the extents be equal.
Refactor H5S__hyper_project_intersection to operate directly on span trees, improving performance and removing the conditionn that the extents be equal.
Fix issues with direct chunk operations and the single chunk index.
* commit 'c812a6c9ec5ff440dc405167c54b86fdb18dcfa4':
Fix issue with direct chunk write not updating the "last chunk" index cache. Fix issues involving datasets being "no allocated" when they contain cached raw data.
Avoid potential invalid read when decoding unknown object header message.
* commit '685288beabc4f4383d69ec07be05c067fab78544':
Delay checking if decoded message's "shareable" flag is appropriate for the message type until we've verified we understand the message type. Reduce size of H5O_msg_class_g to *not* include space for H5O_BOGUS_INVALID. Make bogus messages shareable. Add new bogus message test with shareable messages to cover the formerly problematic code. Re-run gen_bogus.c to add this test case and also to fix the bogus_invalid messages that were no longer H5O_BOGUS_INVLAID due to a new message class being added in a previous commit. Added comment to remind developers to run gen_bogus.c when adding a new message class.
the message type until we've verified we understand the message type.
Reduce size of H5O_msg_class_g to *not* include space for
H5O_BOGUS_INVALID. Make bogus messages shareable. Add new bogus
message test with shareable messages to cover the formerly problematic
code. Re-run gen_bogus.c to add this test case and also to fix the
bogus_invalid messages that were no longer H5O_BOGUS_INVLAID due to a
new message class being added in a previous commit. Added comment to
remind developers to run gen_bogus.c when adding a new message class.
* commit '837dc4bb51e3d3e823235228a33f6ffeae166b52':
Add RELEASE.txt note for HDFFV-10274. (cherry picked from commit f7a7d0a006)
Initialize hid_t variables in test_Attr_bug9.
Fix HDFFV-10274. When deleting all (or almost all) of the messages in an object header chunk, where the total amount deleted was greater than 64K, an error would occur due to an off by one error in the code that handled that case. Fixed this and added a test case.
an object header chunk, where the total amount deleted was greater than
64K, an error would occur due to an off by one error in the code that
handled that case. Fixed this and added a test case.
* commit 'bca20b40b4966552a844051c3d912d6aec50592f':
Initialize hid_t variables in test_Attr_bug9.
Fix HDFFV-10274. When deleting all (or almost all) of the messages in an object header chunk, where the total amount deleted was greater than 64K, an error would occur due to an off by one error in the code that handled that case. Fixed this and added a test case.
an object header chunk, where the total amount deleted was greater than
64K, an error would occur due to an off by one error in the code that
handled that case. Fixed this and added a test case.
* commit '748d6f318c5ebe4fa7ee5d8e4ca2681d69792b70':
Change check for number of dimensions for old-style arrays in datatype decoding routine from an assertion to an if/HGOTO_ERROR check, since it is inappropriate to assert the contents of a file will be what we expect.
* commit '866d17f57f47414b6e778d63fcdcc2a82b208683':
Fix bug in "nooptype" decode in fix for TALOS-0177.
Fix issues in H5Znbit.c where the decompression algorithm would not check the compressed data for validity, potentially causing a buffer overflow.
* commit '62d7562fbd6c6f9f2892cc9a3894f5a68a2b6fdc':
Replace assertion in H5O_dtype_decode_helper for number of array dimensions with a check and error. The assertion was inappropriate because it is operating on data read from the file, which the library does not always have direct control of.
with an alignment threshold set to be larger than the chunk size but smaller
than the size of the small data aggregator.
Tested: koala, ostrich (h5committest); jelly, ummon
compatible between 32 and 64 bit systems.
Expand cross platform property list encode/decode testing.
Fix warning in H5P__fill_value_enc.
Tested: jam, koala, ostrich (h5committest), ummon
Fix handling of layout version number.
Other minor fixes/cleanup.
NOTE: This commit changes the file format for VDS, all files must be
re-generated.
Tested: ummon
Neil Fortner
